megatron-uk wrote on 2021-01-18, 17:12:

I'm using a double buffer, so I have a 256KB buffer in memory, write all of the updates to that, then flush it to VRAM in one operation (which will be in 64KB chunks compared to the X68000 and PC-98). There's no direct setting of pixels, and since there is no animation involved (only a scrollbar moving from one highlighted game to the next, or each games boxart or screenshot loading as it is selected) the latency of doing that once, say, every 500ms is probably going to be good enough. The rest of the application should easily fit in the other 300KB or thereabouts.

Hi,
by using VESA VBE functions 06 and 07 you could avoid using your buffer in system memory and skip the usually not so efficient system memory to video memory copy operation. By using function 06 you can define a larger video memory area than displayed (e.g. in your case 1280x400 that practically means 2 pages) and by using function 07 you can set a different display start (e.g. in your case alternating between start at 0 and start at 640 pixel) without any copy operation. This way you can achieve double buffering using only video memory.

Falcosoft wrote on 2021-01-31, 18:34:

by using VESA VBE functions 06 and 07 you could avoid using your buffer in system memory and skip the usually not so efficient system memory to video memory copy operation. By using function 06 you can define a larger video memory area than displayed (e.g. in your case 1280x400 that practically means 2 pages) and by using function 07 you can set a different display start (e.g. in your case alternating between start at 0 and start at 640 pixel) without any copy operation. This way you can achieve double buffering using only video memory.

Drawbacks:
- it would exclude 256KB SVGA cards, need at least 512KB
- operations on a banked frame buffer are more difficult than on a single array in main memory

That 16-bit Watcom allows for >64KB arrays, doesn't it?

I think using the remainder of video memory for page flipping or to hold off-screen bitmaps is very sensible - however I really do want to try and aim for as low a common denominator as possible in terms of hardware; all the way down to 8086/286 with basic 256KB VGA.

I could do a couple of code paths and do run-time detection of the available hardware, but that's extra complexity and I'm already juggling 3 distinct hardware classes. The fact that I can't use the same development tools for the three different platforms (a custom version of gcc/DJGPP for PC-98, gcc-human68k for Sharp and now Open Watcom for PC/DOS) is enough of a headache that I'm having to keep the code in seperate trees when ideally I'd have just the one and abstract out the hardware specific stuff.

Grzyb wrote on 2021-01-31, 19:58:

That 16-bit Watcom allows for >64KB arrays, doesn't it?

It certainly complains about the vram_buffer[256000] declaration if you don't use the large (-ml) or huge (-mh) memory model at compile time:

1Open Watcom Make Version 2.0 beta Jan 13 2021 00:22:00 (32-bit)
2Copyright (c) 2002-2021 The Open Watcom Contributors. All Rights Reserved.
3Portions Copyright (c) 1988-2002 Sybase, Inc. All Rights Reserved.
4Source code is available under the Sybase Open Watcom Public License.
5See http://www.openwatcom.org/ for details.
6	/opt/toolchains/x86/watcom/binl64/wcc -0 -bc -d0 -ox -ms -zq -i=/opt/toolchains/x86/watcom/h src/gfx.c -fo=obj/gfx.o
7src/gfx.c(578): Warning! W124: Comparison result always 0
8src/gfx.c(35): Error! E1157: Variable must be 'huge'

I'm using large (-ml) and there isn't any warning from the compiler in that mode.

The different models are described in the Open Watcome C language reference: https://open-watcom.github.io/open-watcom-v2- … kidocs/clr.html

The definitions are:

small
small code (code < 64K), small data (data < 64K)
compact
small code (code < 64K), big data (total data > 64K, all objects < 64K)
medium
big code (code > 64K), small data (data < 64K)
large
big code (code > 64K), big data (total data > 64K, all objects < 64K)
huge
big code (code > 64K), huge data (total data > 64K, objects > 64K)

The data definition being:

"With either small or big data, each object is restricted in size to a maximum of 64K bytes. However, an object may be declared as huge, allowing the object to be bigger than 64K bytes. Pointers to huge objects are the least efficient because of extra code required to handle them, especially when doing pointer arithmetic."

It appears that in large or huge memory model mode the compiler automatically tags the 256k array with the __huge prefix, as seen by the warning when trying to compile it in the other modes.

Memory model doesn't refer to the "size" of memory. If refers to how the memory can/is accessed.

So, in 16bit real mode you can only allocate up to 64Kb of memory (limit size of a segment because the "offset" part of the address is 16bit). In a small memory model (usually a com file) all code and data live in the same segment. This means that the size of the code (instructions) and variables can not exceed 64Kb.

Other memory models allow for multiple data segments (space where you store variables to be allocated) or multiple code segments (where instructions can live) to be allocated. Each of these memory models require different addressing techniques. Hence, the memory model options on your compiler.

In your case, you probably want a single code segment and multiple data segments which is in my experience the most common model for 16bit applications.

I understand that, and I understand that normally you would not be able to have any single object larger than 64KB as it wouldn't fit in a single segment. However the explanation from the Open Watcom documentation describes how single objects larger than 64KB can used, regardless of the memory model by internally referencing them by larger pointer types, rather than the usual 16bit pointers which would be the limit for 64KB segments (and the data objects therein). Of course with pointers larger than the systems word size, will come performance issues, so it does warn against their liberal use.

In the documentation it describes the use of data structures >64KB as:

Even using the big data model, each object is restricted in size to 64K. Some applications will need to get beyond this limitation. The Open Watcom C/16 compiler provides the keyword __huge to describe those objects that exceed 64K in size. The code generated for these objects is less efficient than for __far objects.

The declaration of such objects follows the same pattern as above, with the keyword __huge preceding the name of the object if the object itself is bigger than 64K, or preceding the * if the pointer is to an object that is bigger than 64K.

The keyword __huge can only be applied to arrays. Huge objects may be used in both the small and big data models.

I haven't done any detailed digging, but purely adding __huge to the front of the declaration of the display buffer array adds 112 bytes to the final compiled executable. There is no difference in behaviour (that I can see, yet) in the access or use of that 256k array whether I use the __huge keyword or not.

Okay, sure. I didn't see that last sentence on your previous post.

If you're going for the __huge approach than you probably need to figure out how you can memcpy in 64kb chunks to video memory since the addressing of the __huge buffer has some special addressing techniques, unless Watcom has a variation of memcpy specifically for this purpose.

Edit: The other option would be to copy the buffer to video memory pixel by pixel I guess.

It does indeed:

void __far *_fmemcpy( void __far *dst, const void __far *src, size_t length );

The _fmemcpy function is a data model independent form of the memcpy function. It accepts far
pointer arguments and returns a far pointer. It is most useful in mixed memory model applications.

It seems to be working, in that it is copying the the entire contents of the buffer into the video window (in 64k chunks). I've clearly got something wrong somewhere, but I'm not unconvinced that I still have some integer conversion issues as per my other thread, which are causing the duplicated/weird offsets in the source buffer.

megatron-uk wrote on 2021-01-31, 21:55:

It seems to be working, in that it is copying the the entire contents of the buffer into the video window (in 64k chunks). I've clearly got something wrong somewhere, but I'm not unconvinced that I still have some integer conversion issues as per my other thread, which are causing the duplicated/weird offsets in the source buffer.

So, it's working but it's not working?

The copy from the local buffer to the (four) VESA video windows is working... but I've got a bug somewhere which is causing the offset/duplication errors.

Aha! Another integer casting error was causing the x/y coordinate error - anything over a y coordinate of ~100 was causing the images to loop back around and get copied in to an earlier offset in the video buffer.

I think I've fixed that (as well as a similar offset calculation overflow bug in the bitmap font printing routine) and it looks to be placing elements in the correct location now, and it's clear that the RAM to Video memory copy routine is working correctly:

Still some palette bugs to squash (the bitmap font should be grey outline, white fill), and some of the half-toning/antialiasing of the splash logo is not showing correctly. Definitely some palette entries not getting set correctly.

I also think I have a memory issue loading up all of the UI bitmap assets (menu bars, backgrounds, scrollfields etc). So I'm going to have to load them in and use them one-by-one, rather than keep them in-memory all of the time. 640KB should be enough for anyone.... 😀

I'm fairly pleased that other than the integer overflow and casting issues going from a 32bit world, and writing the VESA window flip code, almost everything else is working without modification. So once these few glitches are sorted it should pick up the pace quite rapidly.

Also, that splash logo is rubbish.. I need to do something about that!

Currently working on moving this version of the launcher code to preload bitmap headers and palette data from the disk only - instead reading actual pixel data a line at a time, so the memory in use by a single open bitmap should be a couple of KBytes (headers + file object + palette table) at the most, regardless of actual bitmap size.

I'll keep the various small buttons, icons etc fully loaded in memory as they change the most frequently, but backgrounds and the like will just exist in memory as their headers. It will be slower (especially when repainting the main screen when you exit out of an option screen, for example), but it should help to keep the memory requirements of this Dos version under the 640KB limit.

I can always look at an enhancement to use EMS/XMS to have them preloaded in the future.

Sorted the palette mapping problem, it looks much nicer now (as well as a fancy SVG logo I converted to a 32 colour greyscale bitmap):

So I figured out after setting and reading back the palette entries that they weren't getting set to the correct values... almost as if only part of the rgb triplet was getting stored... oh, wait....

normal VGA DAC is 6bit by default, isn't it?

(sound of hand slapping forehead)

So my palette setting went from:

1outp(VGA_PALLETE_MASK_ADDR, 0xFF);
2outp(VGA_PALLETE_SEL_ADDR, idx);
3outp(VGA_PALLETE_SET_ADDR, r);
4outp(VGA_PALLETE_SET_ADDR, g);
5outp(VGA_PALLETE_SET_ADDR, b);

to

1outp(VGA_PALLETE_MASK_ADDR, 0xFF);
2outp(VGA_PALLETE_SEL_ADDR, idx);
3outp(VGA_PALLETE_SET_ADDR, r >> 2);
4outp(VGA_PALLETE_SET_ADDR, g >> 2);
5outp(VGA_PALLETE_SET_ADDR, b >> 2);

The line-reading bitmap display routine is also working nicely, so I'll transplant that to the rest of the ui display routines and see how we go in terms of reducing memory use. Not too far off getting the ui up and running - once we're there everything should fall in to place from the rest of the support libraries which are doing the game data scraping.

megatron-uk wrote on 2021-02-01, 17:17:

normal VGA DAC is 6bit by default, isn't it?

Normal VGA DAC - IMS G171 - is 6-bit, period.
Some later RAMDACs are indeed 6-bit by default, with the 8-bit option.

I vaguely recall seeing a vesa function to test/set DAC width, I guess I could try detecting the type and alter my palette setting code to suit based on that result - 8bit DACs would get a slightly nicer colour mapping for more graduated images like box art and screenshots.

The "capabilities" field of the VBE info seem to indicate the supported width of the DAC.

1The Capabilities field describes what general features are supported in the
2video environment.  The bits are defined as follows:
3
4        D0      = DAC is switchable
5                  0 = DAC is fixed width, with 6-bits per primary color
6                  1 = DAC width is switchable
7        D1-31   = Reserved

https://web.archive.org/web/20090114055246/ht … nl/vesasp12.txt

Ah, is the VGA DAC fixed as 6bit in Dosbox?

1void vesa_PrintVBEInfo(vbeinfo_t *vbeinfo){
2	// Print the current contents of the vbeinfo structure
3	
4	printf("%s.%d\t VESA BIOS information follows\n", __FILE__, __LINE__);
5	printf("VBE Signature:\t %s\n", vbeinfo->vbe_signature);
6	printf("VBE Vendor:\t %s\n", vbeinfo->oem_string_ptr);
7	printf("VBE Version:\t %d\n", vbeinfo->vbe_version);
8	printf("SW Version:\t %d\n", vbeinfo->oem_software_rev);
9	printf("Vendor:	\t %d\n", vbeinfo->oem_vendor_name_ptr);
10	printf("Product:\t %d\n", vbeinfo->oem_product_name_ptr);
11	printf("Version:\t %d\n", vbeinfo->oem_product_rev_ptr);
12	printf("Capabilities:\t %d\n", vbeinfo->capabilities);
13	if (vbeinfo->capabilities & 0x01){
14		printf("DAC Type:\t Switchable, 6bit + other\n");
15	} else {
16		printf("DAC Type:\t Fixed, 6bit\n");
17	}
18	printf("Total RAM:\t %dKB\n", (vbeinfo->total_memory * 64));
19	printf("%s.%d\t End of VESA BIOS information\n", __FILE__, __LINE__);
20	printf("----------\n");
21}

...

src/vesa.c.197 VESA BIOS information follows
VBE Signature: VESA
VBE Vendor: S3 Incorporated. Trio64
VBE Version: 512
SW Version: 0
Vendor: 0
Product: 0
Version: 0
Capabilities: 0
DAC Type: Fixed, 6bit
Total RAM: 2048KB
src/vesa.c.212 End of VESA BIOS information

I guess I'm going to have to do some real hardware testing to make sure that the 6/8bit palette mode changes work correctly.

megatron-uk wrote on 2021-02-02, 09:29:

Ah, is the VGA DAC fixed as 6bit in Dosbox?

I would imagine that it depends on the VESA BIOS used, which I assume would match the hardware it is trying to emulate. Yes, it would probably be good to test with real hardware and knowing what the specs of it are.

Your project is interesting. 😀

pan069 wrote on 2021-02-02, 09:43:

Your project is interesting. 😀

Thank you!

Hopefully when it gets to a point of feature-parity with the X68000 and PC-98 versions, it will become useful for other Dos gamers as another games launcher and game browser option.