I think VGA Mode Q is infact 256x256, and uses the whole 64kb window http://swag.outpostbbs.net/EGAVGA/0175.PAS.html

Also you can try Tweak, this is a tool that maybe can help you creating the resolution you want:

Thanks, I will play around with the tool. Yes, Mode Q does do that, but if I change it to only show a 160x144 window while in that resolution (using the same number of scanlines and horizontal pixel counts), it will only use a small part of the screen. The strange character mode idea was a way to:

1) mostly fill the screen (understanding that the gameboy aspect ratio is 10:9, not 4:3, so you'll never fill all of the screen without a huge stretch),
2) be able to window the 160x144 pixel output within a larger 256x256 pixel buffer, and
3) limit the number of bytes I have to write to vram. Writing 1 byte populates the colors for two of my "gameboy pixels", each of which fills 12 (4x3) of the "onscreen 720x480 pixels" (if I can use that term loosely), which is very efficient in terms of bytes I have to copy around.

So you want a 256x256 pixel buffer, using a character mode which represents two gameboy pixels using one (8x3) character? That would mean a 128x256 text mode. Isn't there only a 32KB memory area available for text modes? At 0xB800 segment?

Since the gameboy has only 4 gray levels, I bet you could also pack 4 gameboy pixels into (the color attribute byte of) one character.

BTW, I'm not sure how having the oversized 256x256 buffer would help with emulator performance, since that suggests that you are going to render things that are outside the visible area. Doesn't seem worth the time.

bakemono wrote on 2023-09-15, 20:41:

...Isn't there only a 32KB memory area available for text modes? At 0xB800 segment?

Yep, only 32kb is available for text modes. The maximum tweaked resolution for VGA cards on text mode is 80x200 (using half-width characters, 160x200)

https://www.youtube.com/watch?v=XfxqMl9hmYc

Thanks for the replies.

The buffer is only the memory buffer. Only a portion of it is displayed so I don't need a 128x256 text mode. As I mentioned, I would use the offset and start address registers to tell the hardware what parts of the memory segment to display. "Back in the day " (~1994), I'm pretty sure I remember doing similar tricks in text mode. You can pan both directions within a continuous memory buffer. This can't be new ground... (several searches later) ...have a look at this link, which describes what Commander Keen did, complete with diagrams. https://fabiensanglard.net/ega/ I'm talking about a similar technique, just using a character mode and the performance reasons for doing it are the same as for Commander Keen and match very well with the gameboy's design. The reason for the character mode is to be able to write two large "pixels" with 1 byte. If we can switch out of odd/even addressing mode when in text mode (?), we might be able to just write the colors and ignore the character codes (after they've been properly initialized).

As far as packing 4 pixels per byte--that would be great, if we were in a 4 color mode, but we're in 16 color mode...which brings up another way I thought about tackling this. That would be trying to create a tweaked graphics mode, starting with 320x200x4color cga compatible, keeping the 4 color mode, but switching to 360x240 timings (480 line mode vs 400) which then gets us down to the desirable 60hz, and then trying the line tripling trick I mentioned above to get down to 160 lines, and then (which was my central question about tweaking overscans) making the overscan larger to get to 320x144 addressable pixels. Each 2-bit gameboy pixel would have to be written twice so we don't end up with a (very narrow output) to get to 160x144, but we still end up with being able to do 2 pixels per byte. The end result should look identical to my proposed character example. I just don't know if either of these is feasible. I can test and find out, but thought I would first consult with y'all to speed up the process in case I'm missing something obvious.

That's a good point that text modes only have a 32kb segment. I'm not sure if that can be overcome (another experiment to run with the Memory Map Select register), but maybe (?)...

>The maximum tweaked resolution for VGA cards on text mode is 80x200 (using half-width characters, 160x200)
I have a guess that that isn't perfectly accurate. If you started with timings for an 80x60 mode and decreased the character height to 2 pixels high, you'd be able to put out 204 lines before running out of memory, no? Absent being able to increase vertical overscan, I don't know if the hardware would wrap back around and repeat what was at the top of the screen or what, but that'll be another fun experiment!

For the 4-color issue, all I can think of are custom 8x8 textmode chars for 33%/66% dithered/filled in all possible 2x2 combinations per char for every 2x2 pixels of a GB buffer all on an extended text mode

bakemono wrote on 2023-09-15, 20:41:

Isn't there only a 32KB memory area available for text modes? At 0xB800 segment?

The selection of text/graphics mode is independent of the selection of the memory window. You can configure an EGA/VGA card to have a 128K address window which can be completely used for character/attribute bytes, using the complete plane 0 (64KB) for character codes and the complete plane 1 (another 64K) for attribute codes. The processor sees a linear 128K area if interleaved character/attribute codes as usual.

leileilol wrote on 2023-09-15, 23:40:

For the 4-color issue, all I can think of are custom 8x8 textmode chars for 33%/66% dithered/filled in all possible 2x2 combinations per char for every 2x2 pixels of a GB buffer all on an extended text mode

There is another hack that should work on a lot of classic VGA cards, maybe less well on newer SVGA cards: You can switch the attribute into 256-color mode, but keep the remaining part of the graphics card set to text mode. It will then take 2 text mode pixels (each having one of 16 color values) and combine them into one 8-bit value to index the DAC, which creates a way to generate 4 pixels of 4 colors instead of 8 pixels of two colors from a single character. The most prominent issue with this approach is that it depends on the VGA chipset, whether the low nibble or the high nibble of the 256-color value is determined from the left pixel.

ViTi95 wrote on 2023-09-15, 16:28:

Also you can try Tweak, this is a tool that maybe can help you creating the resolution you want:
vgatweak.zip

Attached are some Tweak16b files for 160xYYY graphics modes. It includes a 160x144 mode.

I didn't create these files and I can't remember where I downloaded these files from, so use at your own risk.

mkarcher wrote on 2023-09-16, 07:30:

The selection of text/graphics mode is independent of the selection of the memory window. You can configure an EGA/VGA card to have a 128K address window which can be completely used for character/attribute bytes, using the complete plane 0 (64KB) for character codes and the complete plane 1 (another 64K) for attribute codes. The processor sees a linear 128K area if interleaved character/attribute codes as usual.

Good to get confirmation on that. Any chance that you can further confirm if there is a suitable addressing mode resulting in a continuous memory block of just the character attributes? (not odd-even addressing). The thought is that for my application, I just need to have the entire screen filled with one character (the one that is split into left/right halves of foreground-only/background-only). After I've filled the screen with that character, I never need to touch that memory and the only memory I need to be working with are the colors, so there are obvious advantages if they're available in a continuous block.

mkarcher wrote on 2023-09-16, 07:30:

There is another hack that should work on a lot of classic VGA cards, maybe less well on newer SVGA cards: You can switch the attribute into 256-color mode, but keep the remaining part of the graphics card set to text mode. It will then take 2 text mode pixels (each having one of 16 color values) and combine them into one 8-bit value to index the DAC, which creates a way to generate 4 pixels of 4 colors instead of 8 pixels of two colors from a single character. The most prominent issue with this approach is that it depends on the VGA chipset, whether the low nibble or the high nibble of the 256-color value is determined from the left pixel.

Controlling 4 pixels of 4 colors would be ideal for the 4-color (grayscale) gameboy! Can you point me to more information on this? I'm not following exactly how that could work. I obviously understand how you can pack 4 sets of 4 colors into the 8 bits, but I don't know how, within a text mode, those could get spread out into 4 different regions. The character just has foreground and background, unless the behavior of how the character pattern is somehow affected or those colors somehow are used on two different character patterns. (?)

Calvero wrote on 2023-09-17, 14:14:

ViTi95 wrote on 2023-09-15, 16:28:

Also you can try Tweak, this is a tool that maybe can help you creating the resolution you want: vgatweak.zip

Attached are some Tweak16b files for 160xYYY graphics modes. It includes a 160x144 mode. I didn't create these files and I can't remember where I downloaded these files from, so use at your own risk.

Great! This week, I will pull out some early PS/2s and try experimenting on real hardware to confirm operation. And then I will see if I can tweak that even further (since ideally the 160 pixels would be the programmed logically width but wouldn't fill the entire available physical width since the gameboy aspect ratio is much closer to square than a regular monitor).

leileilol wrote on 2023-09-15, 23:40:

For the 4-color issue, all I can think of are custom 8x8 textmode chars for 33%/66% dithered/filled in all possible 2x2 combinations per char for every 2x2 pixels of a GB buffer all on an extended text mode

That might work, but I am hoping to avoid dithering to have a more authentic look. It would be cool to try out sometime just to see how such a thing might look. I definitely have an appreciation for some of those old mac 1-bit dithered images. 😀

xorlof wrote on 2023-09-17, 15:11:

mkarcher wrote on 2023-09-16, 07:30:

The selection of text/graphics mode is independent of the selection of the memory window. You can configure an EGA/VGA card to have a 128K address window which can be completely used for character/attribute bytes, using the complete plane 0 (64KB) for character codes and the complete plane 1 (another 64K) for attribute codes. The processor sees a linear 128K area if interleaved character/attribute codes as usual.

Good to get confirmation on that. Any chance that you can further confirm if there is a suitable addressing mode resulting in a continuous memory block of just the character attributes? (not odd-even addressing).

That's perfectly possible. The text mode hardware of the EGA/VGA card works by having all character codes in plane 0 and all attribute codes in plane 1.

• Odd/Even mode (the default addressing mode for text mode) uses the lowest CPU address bit to toggle between planes 0 and 1, which is something you do not want. So you need to disable odd/even mode. If I remember correctly, there are two or three bits in different sections of the EGA/VGA cards dealing with odd/even mode, so set them all to "odd/even disabled".
• After disabling odd/even mode, ensure that you only interact with plane 1 by setting the read plane select register in the graphics controller to "plane 1", and the write mask register in the timing sequencer to "plane 1 only".
• As Odd/Even mode keeps the higher address bits as they are, in odd/even mode, the first character (CPU address zero) gets written to plane 0, address 0 and the first attribute (CPU address 1) byte gets written to plane 1, address 0. The second second character (CPU address 2) gets written to plane 0, address 2. Plane 0, address 1 is not used until you go 128K. After you disabled odd/even mode, you get to see all bytes in plane 0 or plane 1, so you need to disable the mode of the CRTC that causes every other byte to be skipped. This mode is called "word mode", and for your purpose, you need to reconfigure the CRTC to "byte mode".
• To get easy access to all 64K attributes at the same time, configure the card to use the A000-AFFF memory window.

xorlof wrote on 2023-09-17, 15:11:

mkarcher wrote on 2023-09-16, 07:30:

There is another hack that should work on a lot of classic VGA cards, maybe less well on newer SVGA cards: You can switch the attribute into 256-color mode, but keep the remaining part of the graphics card set to text mode. It will then take 2 text mode pixels (each having one of 16 color values) and combine them into one 8-bit value to index the DAC, which creates a way to generate 4 pixels of 4 colors instead of 8 pixels of two colors from a single character. The most prominent issue with this approach is that it depends on the VGA chipset, whether the low nibble or the high nibble of the 256-color value is determined from the left pixel.

Controlling 4 pixels of 4 colors would be ideal for the 4-color (grayscale) gameboy! Can you point me to more information on this?

I don't know any good write-up about this. The idea is to "hack" the data processing pipeline of the EGA/VGA card. The pipeline for text modes works like this:

1. (this is text mode specific) Planes 0&1 are accessed at the current screen location to retrieve the character and attribute code to display. Then Plane 2 is accessed at the location combined from the currently selected character set, the character code retrieved in the previous step and the current scan line counter, to get the required 8 bits to chose between foreground and background for 8 pixels.
2. (again text mode specific) The attribute bits retrieved from plane 1 and the character data retrieved from plane 2 are used by the graphics controller to generate a series of 8 (or 9) 4-bit values which are sent to the attribute controller. Each of these 4-bit values is either the foreground color or the background color.
3. (now text and graphics mode work alike, but this step is different in 256-color mode) Each 4-bit value received by the attribute controller is used as an index into the EGA palette registers and converted into a 6-bit color code, which is extended by another 2 fixed programmable bits (0 by default) to generate an 8-bit value, which is sent to the RAMDAC.
4. The RAMDAC uses this 8-bit value is used to look up one out of 262144 colors, and that color is sent to the monitor as analog signal.

In 256-color graphics mode, the pipeline works like this:

1. (common to all graphics modes) all four planes are fetched at the current address to retrieve 32 bits of pixel data (4 pixels of 256 colors)
2. (order of bits specific to 256-color graphics mode) the 32 bits of pixel data is split by the graphics controller into eight pieces of 4 bits each, containing the upper nibble and the lower nibble of the color number of one pixel one after the other to the attribute controller.
3. Each two 4-bit values received by the attribute controller are combined into one 8-bit value, which is sent to the RAMDAC.
4. The RAMDAC uses this 8-bit value is used to look up one out of 262144 colors, and that color is sent to the monitor as analog signal.

As you see, step 4 is the same in both text mode and 256-color mode. The idea to get "4-color text mode" is to splice step 3 of the 256-color mode as replacement step 3 into the text-mode pipeline. You should first ensure that you don't run in a 9-pixel text mode. You select which step 3 is used in this pipeline by setting or clearing bit 6 of attribute controller register 0x10. This means that every two pixels generated from the character display step 2 are combined into one DAC index in step 3. For example, if you have a character with blue background (color 1) and black foreground (color 0), this will make colors 0/0 = 0, 0/1 = 1, 1/0 = 16 and 1/1 = 17 accessible. The downside of this approach is that the 256-color step 3 is only meant to be used with the 256-color graphics step 2, which is selected by setting bit 6 in the graphics controller register 5. For normal hardware operation of the VGA hardware, it doesn't matter whether the graphics controller sends the low 4 bits of the 256-color pixel value first, or it sends the high four bits first, as long as the attribute controller combines them in the correct order. The last time I mentioned the idea of cross-combining a serialization mode (step 2) aimed at 16 colors and the attribute controller mode designed for 256 colors (or the other way around), some VGA expert here on VOGONs pointed out that both orders are implemented in VGA-compatible hardware by different vendors.

More modern SVGA chipsets may not have this "4-bit bottleneck" anymore, and pass 256-color pixels in 256-color modes using one 8-bit channel instead the the classic EGA-type 4-bit channel. This hack will not work on this kind of card.

mkarcher wrote on 2023-09-17, 16:56:

That's perfectly possible. The text mode hardware of the EGA/VGA card works by having all character codes in plane 0 and all attribute codes in plane 1.

(detailed response snipped)

mkarcher wrote on 2023-09-17, 16:56:

xorlof wrote on 2023-09-17, 15:11:

Controlling 4 pixels of 4 colors would be ideal for the 4-color (grayscale) gameboy! Can you point me to more information on this?

I don't know any good write-up about this. The idea is to "hack" the data processing pipeline of the EGA/VGA card. The pipeline for text modes works like this:

(another amazing response snipped)

Perfect, thank you for your expertise. If I end up using this 2-bit-in-character-mode technique and doing a writeup, do I credit it as an mkarcher-original? 😀

xorlof wrote on 2023-09-18, 14:31:

Perfect, thank you for your expertise. If I end up using this 2-bit-in-character-mode technique and doing a writeup, do I credit it as an mkarcher-original? 😀

You may credit this as "based on an a idea suggested by Michael Karcher", but I am definitely not the only one who thought of it. Nevertheless, I am the one who explained the idea to you, so crediting it that way seems fair.