VOGONS


First post, by clb

User metadata
Rank Member
Rank
Member

Hey all,

There was an earlier thread about Commander Keen 4-6 having choppy hardware scrolling on EGA cards: Keen 4-5-6 choppy scrolling on EGA card

There @K1n9_Duk3 was able to explain that the root cause for that was that Keen assumed vertical retrace would be 70Hz, whereas on EGA cards it is 60Hz. (see also this PCKF thread about the same topic) As result, the code timed out waiting for vblank checks, and updated the scrolling registers during visible scan line area: Re: Keen 4-5-6 choppy scrolling on EGA card

However, in addition to those EGA cards and the 60Hz/70Hz programming bug that some Keen versions had (apparently by Keen Dreams the code was fixed), it is well known that several ATI PCI VGA cards suffer from a similar choppy scrolling issue:

1. there is a PCKF forum thread about this observation: PCKF: Jerky motion on ATI cards: possible community fix?, but no root cause was found
2. and @Gona's test chart widely reports the glitchy scrolling problem on ATI cards as well: Gona's PCI and AGP video card DOS compatibility chart

For these ATI cards, the 320x200 mode vertical retrace is 70 Hz, so the earlier EGA 60Hz/70Hz mismatch would not explain the root cause of the issue for these ATI cards - something else must be amiss.

I have an ATI Mach64 VT PCI card, which showcases the glitch.

ati_mach64_vt_264vt2.jpg
Filename
ati_mach64_vt_264vt2.jpg
File size
1.67 MiB
Views
2535 views
File license
Public domain

What happens is that scrolling the map vertically is butter smooth; but scrolling the map horizontally shows these types of tearing artifacts:

ati_mach64_keen5_hscroll_glitch.png
Filename
ati_mach64_keen5_hscroll_glitch.png
File size
304.53 KiB
Views
2535 views
File license
Public domain

I've uploaded 20% slowed down video to YouTube using that card, that illustrates the issue: https://youtu.be/VhZfStu5GYg

At first impression, one might think "hey, that looks like a typical lack of vsynced screen framebuffer update" artifact that happens in all games if one disables vsync.

However that cannot be the root cause, because a) Keen does not repaint the visible map each frame, but instead it updates the screen start address registers to do hardware scrolling, and b) the artifact never occurs when walking straight up and down in the game map, but that always updates smoothly.

While developing some VGA test code for CRT Terminator Digital VGA Feature Card ISA DV1000 in the SNOOP.EXE: (S)VGA adapter info tool and CRT Terminator config utility utility program, I think I may have found the root cause of this ATI quirk.

It turns out that the ATI cards latch on to changes to the EGA/VGA Attribute Controller register's Horizontal Pixel Shift Count field on each hblank, whereas other cards that I have tested so far (Headland, Cirrus), the adapter latches on to changes to that register only at each vblank. This hshift count register lives in I/O port 3C0h, register index 13h, bits [3:0], see Horizontal Pixel Panning Register at http://www.osdever.net/FreeVGA/vga/attrreg.htm.

I've created a test program that tests whether the VGA adapter reads Horizontal Pixel Shift Count register at each hblank or only during vblank, by feeding random noise to the register during visible picture area, and resetting it back to zero during vblank.

hshift.png
Filename
hshift.png
File size
256.28 KiB
Views
2536 views
File license
Public domain

If the adapter only updates Horizontal Pixel Shift Count register at each vblank, the noise should be ignored, and the program will print a solid one pixel wide green line on the screen.

On my Headland VGA adapter, which does not have scrolling issues in Keen, that is what happens:

headland_video7_ht208_hshift_good.jpg
Filename
headland_video7_ht208_hshift_good.jpg
File size
29.45 KiB
Views
2489 views
File license
Public domain

On the ATI Mach64 VT card, the program instead displays a wide wiggly green line as the ATI adapter latches on to a random value in the Horizontal Pixel Shift Count register on each scanline:

ati_mach_64_hshift_bad.png
Filename
ati_mach_64_hshift_bad.png
File size
30.7 KiB
Views
2535 views
File license
Public domain

I do not have a 100% connection to the ATI Keen scrolling issue, but it does look plausible that this might be the cause. If Keen ends up adjusting the Horizontal Pixel Shift Count register outside vblank, then these types of artifacts would occur specifically on ATI cards only - whereas on other VGA cards changing the register outside vblank is safe and glitch-free, and the change will only apply after the next vblank (just like changing Display Start Address register also does change only at each vblank on all tested cards)

If you'd like to test your own ATI or non-ATI card, you can download the code here:
- HSHIFT.ZIP containing HSHIFT.EXE
- HSHIFT.CPP

What I would be curious to find out: does someone have a graphics card that exhibits horizontal scrolling tearline glitch issues in Keen 4-6 games, but at the same time HSHIFT.EXE displays a straight green line? That is, is there a counterexample that would show that HSHIFT.EXE is not capturing the root cause?

If the above root cause is accurate, then it is plausible that Keen games should be possible to patch for ATI cards by ensuring that the game engine only adjusts the Horizontal Pixel Shift Count register when inside vblank.

Last edited by clb on 2023-09-03, 20:50. Edited 11 times in total.

Reply 1 of 48, by clb

User metadata
Rank Member
Rank
Member

The glitchy ATI Mach64 card in question:

ati_mach64_vt_264vt2_chip.jpg
Filename
ati_mach64_vt_264vt2_chip.jpg
File size
1.72 MiB
Views
2532 views
File license
Public domain
ati_mach64_vt_264vt2_snoop.png
Filename
ati_mach64_vt_264vt2_snoop.png
File size
178.87 KiB
Views
2532 views
File license
Public domain

Reply 3 of 48, by clb

User metadata
Rank Member
Rank
Member

Oh, just realized that my Keen 5 version was 1.0. There is the 1.4 update available for Keen 5, which adds the "Fix Jerky Motion" option.

keen5_1.4_fix_jerky_motion.png
Filename
keen5_1.4_fix_jerky_motion.png
File size
54.08 KiB
Views
2477 views
File license
Public domain

That option does fix up ATI support for Keen 5. Maybe what that fix does is to exactly restrict scroll updates to occur only during vblank.

Reply 5 of 48, by clb

User metadata
Rank Member
Rank
Member
keenmaster486 wrote on 2023-08-31, 17:48:

Keen also has issues on Neomagic laptop chips as well... would be interesting to see if it's the same problem

Looks like that chip is only available as an integrated motherboard/laptop adapter(?) - couldn't find any discrete cards. Would certainly be cool if someone has a setup to test.

Reply 6 of 48, by clb

User metadata
Rank Member
Rank
Member

Browsing through http://gona.mactar.hu/DOS_TESTS/ for cards that have been reported to not work well with Keen 5, I find I have this Matrox MGA2064W card that is marked there to scroll bad:

matrox_mga2064w.jpg
Filename
matrox_mga2064w.jpg
File size
1.85 MiB
Views
2393 views
File license
Public domain

Sure enough, the card does glitch with horizontal scrolling in Keen 5 1.0:

matrox_keen5_hscroll_glitch.png
Filename
matrox_keen5_hscroll_glitch.png
File size
1.59 MiB
Views
2393 views
File license
Public domain

and the "Fix Jerky Motion" option in Keen 5 1.4 fixes that issue up.

In HSHIFT.EXE test the Matrox card behaves just like the ATI Mach64 VT card above, latching onto the Horizontal Pixel Shift Count at each hblank:

matrox_mga2064w_hshift.png
Filename
matrox_mga2064w_hshift.png
File size
31.34 KiB
Views
2393 views
File license
Public domain

SNOOP giving a nice diagnostic about it.

matrox_mga2064w_snoop.png
Filename
matrox_mga2064w_snoop.png
File size
149.58 KiB
Views
2393 views
File license
Public domain

Reply 8 of 48, by maxtherabbit

User metadata
Rank l33t
Rank
l33t

I have a matrox millennium (IS-STORM) and even with the "fix jerky motion" option on in keen4 it still doesn't scroll smoothly. It requires king duke's patch.

Reply 9 of 48, by badmojo

User metadata
Rank l33t
Rank
l33t
maxtherabbit wrote on 2023-09-01, 01:28:

I have a matrox millennium (IS-STORM) and even with the "fix jerky motion" option on in keen4 it still doesn't scroll smoothly. It requires king duke's patch.

Yep same here.

Life? Don't talk to me about life.

Reply 10 of 48, by clb

User metadata
Rank Member
Rank
Member

> I have a matrox millennium (IS-STORM) and even with the "fix jerky motion" option on in keen4 it still doesn't scroll smoothly.

Thanks for the data point. To be more precise, the logical conclusion that I am inferring here is "has_hshift_bug" -> "scrolling_in_unpatched_keen5_1.0_ will_periodically_tear", or equivalently, "scrolling_in_keen5_1.0_is_smooth" -> "no_hshift_bug".

What kind of scrolling artifacts does the IS-STORM card exhibit? Or are there no artifacts, but timing is incorrect and as result the visuals are stuttering?

Would be interesting to see a video capture of it in action in Keen5 1.0 and/or Keen5 1.4, as well as the results of SNOOP.EXE and HSHIFT.EXE on that card.

Reply 11 of 48, by clb

User metadata
Rank Member
Rank
Member

Pierre, CRT Terminator hardware wise is currently in a wait mode. We are waiting for the FPGA manufacturer to finish baking us a batch of chips. The estimated shipping time is marked for early November atm. Meanwhile, I am working on the software side, and writing a user manual for the card.

Reply 12 of 48, by Pierre32

User metadata
Rank Oldbie
Rank
Oldbie
clb wrote on 2023-09-01, 06:48:

Pierre, CRT Terminator hardware wise is currently in a wait mode. We are waiting for the FPGA manufacturer to finish baking us a batch of chips. The estimated shipping time is marked for early November atm. Meanwhile, I am working on the software side, and writing a user manual for the card.

Appreciate the update!

Reply 13 of 48, by maxtherabbit

User metadata
Rank l33t
Rank
l33t
clb wrote on 2023-09-01, 06:44:
> I have a matrox millennium (IS-STORM) and even with the "fix jerky motion" option on in keen4 it still doesn't scroll smoothly […]
Show full quote

> I have a matrox millennium (IS-STORM) and even with the "fix jerky motion" option on in keen4 it still doesn't scroll smoothly.

Thanks for the data point. To be more precise, the logical conclusion that I am inferring here is "has_hshift_bug" -> "scrolling_in_unpatched_keen5_1.0_ will_periodically_tear", or equivalently, "scrolling_in_keen5_1.0_is_smooth" -> "no_hshift_bug".

What kind of scrolling artifacts does the IS-STORM card exhibit? Or are there no artifacts, but timing is incorrect and as result the visuals are stuttering?

Would be interesting to see a video capture of it in action in Keen5 1.0 and/or Keen5 1.4, as well as the results of SNOOP.EXE and HSHIFT.EXE on that card.

It's just stuttering as far as I can see

Reply 14 of 48, by clb

User metadata
Rank Member
Rank
Member

In case we really didn't have enough VGA hardware scrolling test cases, I wrote a VGA hardware scrolling bitmap demo at
- https://github.com/juj/crt_terminator/blob/ma … /bin/SCROLL.zip
- https://github.com/juj/crt_terminator/tree/main/DOS/SCROLL

It pans a very large 2MB 1920x1097 pixel 256-color bitmap by one pixel per vertical refresh in 320x240 Mode X.

SCROLL.png
Filename
SCROLL.png
File size
1.27 MiB
Views
2183 views
File license
Fair use/fair dealing exception

The program can be run in two modes. Just running

SCROLL

implements scrolling that is synchronized to occur inside vblank, whereas

SCROLL hblank

can be used to test how the VGA adapter behaves if hardware scrolling registers (Display Start Address, Horizontal Pixel Shift Count) are modified during active display area.

The whole 256KB VGA memory is used, so this test exposes the Tseng ET4000 and S3 Vision buggy 256KB wraparound.

EDIT: My bad, I completely misremembered the S3 Vision for another card, and it was instead a Alliance Semiconductor ProMotion 3210 that had the wraparound bug. See Which >256KB SVGA adapters forgot to implement the VGA address space wrapping after 256KB mark? for details.

On my Matrox MGA2064W and ATI cards, default SCROLL is smooth, whereas SCROLL hblank is glitchy.

Last edited by clb on 2023-09-09, 11:00. Edited 1 time in total.

Reply 15 of 48, by clb

User metadata
Rank Member
Rank
Member

Oh my.. This weekend I've dug deeper into the behavior of different cards, and there are certainly many quirks to be surfaced in this area. This has given an appreciation to why it is so difficult to write a single hardware scroll implementation that would work on all cards. (it is possible, but the best method I was able to come up with is still somewhat awkward)

Before this weekend, I thought that ATI behavior was particularly "bad", but actually, I am now of the opinion of all the cards, ATI might not have done what was IBM compatible, but they did what made more sense than what the other cards do.

Although thinking about this more, to me it seems that no manufacturer has implemented hardware scrolling according to what would be most ideal/"best practices".

I have this Paradise PVGA1A card:

paradise_pvga1a.jpg
Filename
paradise_pvga1a.jpg
File size
1.14 MiB
Views
2081 views
File license
Public domain

which stuttered horribly when I ran the earlier SCROLL demo on it. That prompted to investigate even more carefully how the latching behavior of the Display Start Address (DS) and Horizontal Pixel Shift Count (HS) registers is implemented on different cards.

I ended up discovering the following data:

Card            | DS Latched  | HS Latched  | SCROLL demo compatible sync methods
----------------+-------------+-------------+-------------------------------------
Oak | Vsync start | Vsync end | 2,3,5,6
Trident | Vsync start | Vsync end | 2,3,5,6
Cirrus | Vsync start | Vsync end | 2,3,5,6
Headland | Vsync start | Vsync end | 2,3,5,6
AHEAD V5000B | Vsync start | Vsync end | 2,3,5,6
ATI Mach64 | Vsync end | Every Hsync | 1,3,6
Matrox MGA2064W | Vblank end | Every Hsync | 1,2,3,4,6
Paradise | Vsync start | Every Hsync | 6
Tseng ET6000 | Vsync start | Vsync start | 1,2,3,4,5,6
S3 Trio64 | Vsync end | Vsync start | 2,3,5,6
SIS 6326 | Vsync end | Vsync end | 1,2,5,6
(ideal) | Vsync end | Vsync end | 1,2,5,6

Combining all the above constraints together, the only timing where scrolling registers could be updated so they will be guaranteed to be properly synchronized on all cards would be to reprogram the registers after vblank start (but before vsync start).

However, there does not unfortunately exist a mechanism on the VGA adapter to synchronize to vblank start. One can cleanly only synchronize to operate after vsync start or after vsync end.

In my opinion the ideal ("best practices") hardware design mechanism would have been to have the VGA adapter latch DS and HS registers both at vsync end. That way one could have either a) cleanly synchronized hardware scrolling at vsync start (using simply this code), or b) the scroll registers could have even been programmed without any synchronization whatsoever! (with a very rare once in a blue moon possibility for a glitchy race condition)

I presume the original IBM VGA behavior was to have DS latch at Vsync start since so many cards above have that behavior, but I would call that an outright hardware flaw. Maybe ATI also observed this behavior as undesirable, and consciously changed their DS implementation to sync at Vsync end. (or maybe it was just a random fluke, 🤣 who knows) Although it would have been better if they would have also had HS register latch at Vsync end, and not every hsync, as that would have enabled programmers to modify the DS and HS registers completely unsynchronized.

Anyhow, I created an interactive set of different synchronization methods into the SCROLL program that one can play around with to check what the latching behavior of a particular card is.

scroll_info.png
Filename
scroll_info.png
File size
30.53 KiB
Views
2081 views
File license
Public domain

SNOOP will also diagnose these combinations. Maybe it will help e.g. emulation developers one day.

If someone has a card that does not follow any of the conventions in the chart above, I'd love to know.

UPDATE: Tested and added Tseng ET6000 (G-Force Jazz 128), which was developed to latch DS and HS on Vsync Start. That is a super compatible method, the best tested so far! (although Tseng has the other 256KB wraparound issue still, which needs custom handling)

UPDATE #2: Tested and added SIS 6326, which implements the most preferable DS,HS both latched at Vsync end form.

UPDATE #3: Tested and added Matrix Millennium MGA2064W.

Last edited by clb on 2023-09-12, 20:32. Edited 4 times in total.

Reply 16 of 48, by clb

User metadata
Rank Member
Rank
Member

Looks like this SIS 6326 AGP (PCI bus) graphics card

sis_6326.jpg
Filename
sis_6326.jpg
File size
804.38 KiB
Views
2034 views
File license
Public domain

does implement the ideal format: both Display Start Address and Horizontal Pixel Shift Count registers are latched at Vsync end. This means that it is very easy to program code that updates the scrolling registers without glitching or without awkward polling performance losses. Fancy!

Reply 17 of 48, by clb

User metadata
Rank Member
Rank
Member

Matrix Millennium MGA2064W (from this earlier comment: Re: Solved(?): Why does Commander Keen 4-6 hardware scrolling glitch on ATI (Mach) PCI video cards ) takes the crown as the most uniquely peculiar card yet: it latches DS register at the end of vertical *blank*, not at the end of vertical retrace. No other card has synchronized to blank period so far. Well, that is quite compatible fortunately at least, but it does have the quirk that it latches HS at every hsync, giving Keen the creeps.

Reply 18 of 48, by clb

User metadata
Rank Member
Rank
Member

I went through and disassembled and decompiled Keen 4 1.0, 1.2 and 1.4 versions to figure out exactly how they are synchronizing, and what that Fix Jerky Motion option does.

(sidenote, this is not "new" information, as many Keen modders have already discovered this information and delved through this area of the code; although I am not sure if a writeup of this area existed anywhere)

Original Keen 4 v1.0 had this code:

keen4_v1.0_screen_scrolling.png
Filename
keen4_v1.0_screen_scrolling.png
File size
104.8 KiB
Views
1865 views
File license
Public domain

That is, it looks like they try to find the start edge of vertical blank by observing a period of five consecutive "display in blank area" status register reads. "Horizontal blanks are short, shorter than it takes time to do 5 ISA bus I/O port reads, right?"

That kind of code is likely brittle to ISA bus overclocking, and display card I/O read speed. And once you have a graphics card on the PCI bus, I/O port reads become mighty more faster, so the above code would then think hblanks were vblanks, and lead to incorrect synchronization.

Another problem are interrupts: they play against this code to mess the timings. In Keen 4, the PIT0 timer ticks at 8*70 = 560 Hz, which is about ~25% more often than there are scanlines in the output image (449), so it could make the code occassionally miss finding the short vertical front porch window.

Keen 4 v1.2 scrolling code is very much the same, but they added a new /NOPAN command line option:

keen4_v1.2_screen_scrolling.png
Filename
keen4_v1.2_screen_scrolling.png
File size
106.92 KiB
Views
1865 views
File license
Public domain

There are two possible reasons for this:
a) there were some graphics cards where the Horizontal Pixel Shift Count register (also called the Pixel Panning or PEL Panning register) was buggy (I haven't found one yet), or, to me more likely is that
b) ID got reports that there were some graphics cards where the synchronization interaction with Display Start Address and Horizontal Pixel Shift Count registers was working unexpectedly. From the ASCII table in the earlier comment, we know that such cards were in abundance; there was no standardization whatsoever on the latching timings between manufacturers. Disabling panning effectively removes one source of synchronization headache, and better to have an option to revert to blocky scrolling than to have outright tearing/glitchy scrolling.

The /NOPAN option would have been an easy way for tech support to help people mitigate/diagnose their setups.

And then, we get to the Keen 4 v1.4 scrolling implementation:

keen4_v1.4_screen_scrolling.png
Filename
keen4_v1.4_screen_scrolling.png
File size
97.41 KiB
Views
1865 views
File license
Public domain

Oh boy. If 5 ISA I/Os isn't enough to distinguish a hblank from a vblank, "let's just repeat for a five more, that oughta do it!". They could have automated the mechanism by actually benchmarking the durations of hblanks and vblanks, and no "Fix Jerky Motion" option would have been necessary. This fix looks like something that software near the end of its lifecycle would receive, when late bug reports are piling in and only one sorry maintainer guy is left to work on the codebase. We can only wonder 😁

Anyhow, even after I applied the Fix Jerky Motion option to the ATI card, I found that the game was still stuttering. But now it was stuttering like "I remember it used to". Back in my day, I had played Keen on a 16MHz 386 SX, so it would be obvious not to run smoothly. But now in this instance I have a 80MHz 486 with a PCI graphics card, so surely Keen should be able to run perfect smooth without stutters?

With help from PCKF community at https://pckf.com/viewtopic.php?t=11505&start=45 (more conversation at https://pckf.com/viewtopic.php?t=16773 ), the experts there were able to point out that the game logic is ticked at a timer that is decoupled from the vertical refresh rate - likely due to the lack of reliable vertical refresh interrupt.

With help, I was able to rebuild Keen 4 from reverse engineered/recreated sources, and I added some hardware assistance via CRT Terminator to acquire such a reliable frame timing source (a frame counter and a scanlines-until-vblank counter), and now I have a very smoothly running Commander Keen 4. Well, not only is it running at smooth locked at vsync 35 Hz, I adjusted its refresh rate to run at 70Hz! Oh boy is this the smoothest Keen I've ever seen.

The relevant synchronization code can be found at https://github.com/juj/crt_terminator/blob/ma … L.CPP#L255-L333 . I really really hope to make these features into the production version of CRT Terminator so that developers can leverage them, although I have no clue yet as to how useful they would actually be, and the tricky part is that the board is so full already that I am unsure where to make it all fit.

Last edited by clb on 2023-09-07, 17:09. Edited 2 times in total.

Reply 19 of 48, by rasz_pl

User metadata
Rank l33t
Rank
l33t

On cards that do not generate Vsync interrupts would an AND gate on VGA V H and going straight to one of ISA interrupt lines accomplish same thing?

Open Source AT&T Globalyst/NCR/FIC 486-GAC-2 proprietary Cache Module reproduction