It's a simple AND mask being applied to filter out the lower bits of the counter, before matching the end registers. That's 5 bits fior
Also, the horizontal total is actually 5 more than what's specified(done by hardware). So the value written is wanted total minus 5 (so x-5 for a setting of x, 0x00 being 5 clocks and with 0xFF being 260 horizontal character clocks).
The retrace/blanking end stop their retrace/blanking whenever they are matching all buts but the lower 5, which can be on the current scanline, but also on the next scanline. It can be also be forced out of range for even longer periods(of multiple scanlines) if you're willing to have a very small screen, at least theoretically. Like setting it to 0x1F and horizontal total to lower than (0x1F-5)=0x1B to prevent it from matching.
Also, only the end of the retrace/blanking match the lower 5/6 bits only. The start registers match fully(a range of 0-255 character clock number, which also may be out of range). It ends the first time the horizontal character clock counter's bits match(5-bit for horizontal retrace end and 6-bits for horizontal blanking end).
Look at UniPCemu's CRTC timing for the exact way this masks(hardware/vga/vga_crtcontroller.c, function get_display_x).

Thank you for your explanation. Since I don't have an actual CRT monitor (using an LCD one), I'm not necessarily seeing the actual effects some of these timings have. Your explanation gave me a better idea of how the timing values work.