I wouldn't really rely on DOSBox's tick counting since it just treats every instruction as taking 1 cycle (and iirc there is/was a bug that treats REP MOVS as one cycle regardless of the repeat count).

jmarsh wrote on 2021-02-03, 21:38:

I wouldn't really rely on DOSBox's tick counting since it just treats every instruction as taking 1 cycle (and iirc there is/was a bug that treats REP MOVS as one cycle regardless of the repeat count).

No problem - I certainly am not going to rely on it to state that "this bit is fast enough" or "function is now performant enough", but it's definitely helpful in at least pointing me in the right direction.

Next up is to try the same performance test (400 games scraped and loaded, as well as the new fill function) on my 286 (I can imagine it's CF card is going to struggle against a pair of mirrored SSD's!!!) It's difficult to judge from Dosbox just how it compares to a real machine... It can't be beaten in terms of speeding up the development cycle though!

Updated version:

https://youtu.be/LJ1fFGvar1k

Faster artwork display and menu navigation. I optimised a few of the other block fill routines and moved all of the artworking to stream-from-disk method to reduce base memory use.

Getting quite pleased with the responsiveness of it. Still need to track down a couple of bugs:

- What is exactly causing the corruption between (what appears to be) 64kb video segments
- What is causing the the interface to crash when my directory of RPG or Adventure titles is included in the scraping

Game browsing, including artwork (if you put any in!)

Selecting a filter to apply (F):

Genre filters are automatically generated from whatever genres are applied to your games in the metadata file:

Game series filters are again generated from any series' you define in your game metadata. In this case I only set "Arkanoid" as a series name for both Arkanoid 1 and 2:

And, of course, selecting a given filter then filters the games that you see. You can see that a filter is active by the icon next to the 'Filter' text:

megatron-uk wrote on 2021-02-03, 14:56:

[…]
Show full quote

Copy code to clipboard

1	char name[MAX_STRING_SIZE];			// Just the directory name; e.g. FinalFight

Copy code to clipboard

1typedef struct launchdat {
2	char realname[MAX_STRING_SIZE];		// A 'friendly' name to display the game as, instead of just the directory name
3	char genre[MAX_STRING_SIZE];			// A string to represent the genre, in case we want to filter by genre
4	char series[MAX_STRING_SIZE];		// Series name; e.g. Gradius, Streetfighter, etc.
5	char publisher[MAX_STRING_SIZE];		// The name of the publisher
6	char developer[MAX_STRING_SIZE];		// The name of the developer
7	char start[MAX_FILENAME_SIZE];		// Name of the main start file
8	char alt_start[MAX_FILENAME_SIZE];	// Name of an alternative start file (e.g a config utility)
9	char images[IMAGE_BUFFER_SIZE];		// String containing all the image filenames
10} launchdat_t;

this is whats killing you. they should all just be uint8_t *something;

and just strdup them when you read them in, or better use a hash/arry for things that would get repeated (publisher, developer, series, genre).

this is whats killing you. they should all just be uint8_t *something;

and just strdup them when you read them in, or better use a hash/arry for things that would get repeated (publisher, developer, series, genre).

In what way is it killing me?

megatron-uk wrote on 2021-02-05, 16:10:

I optimised a few of the other block fill routines and moved all of the artworking to stream-from-disk method to reduce base memory use.

Very good decision, that should help a lot, and theoritically should be able to handle over a thousand entries without issues.

You're doing great work here.

carlostex wrote on 2021-02-05, 21:21:

megatron-uk wrote on 2021-02-05, 16:10:

I optimised a few of the other block fill routines and moved all of the artworking to stream-from-disk method to reduce base memory use.

Very good decision, that should help a lot, and theoritically should be able to handle over a thousand entries without issues.

You're doing great work here.

Thanks, it's appreciated. Although the source is up on https://github.com/megatron-uk/x86Launcher, once I've got it to a state where I'm reasonably happy with its stability I'll start putting binaries up on my website and posting in the releases sub-forum. I'll probably also do some pre-made screenshot and metadata file packs to go with it.

One idea I've just had for a further improvement to responsiveness is rather than immediately loading the box art or first screenshot in the list upon scrolling to a game, I should instead start a timer the last time the cursor keys were touched... after a certain period (250ms? 500ms?) the artwork will then load. It means scrolling through the list will limited only by the speed at which the miniscule metadata files (~200 bytes each) for each game can be loaded from disk, rather than the metadata file and the first artwork resource.
That should increase the scroll/navigation speed even more, without having to further optimise any of the graphics functions.

megatron-uk wrote on 2021-02-05, 19:43:

this is whats killing you. they should all just be uint8_t *something;

and just strdup them when you read them in, or better use a hash/arry for things that would get repeated (publisher, developer, series, genre).

In what way is it killing me?

because those fixed arrays are 416+ bytes each when they probably use like 40-50.. switching to strdup/malloc's thats enough space for several more data structures.
your wasting heaps of memory this way.

I only have one of those structures in memory at any given time. It is parsed from the metadata file loaded when a game is selected.

The linked-list representing all games found is far more simplistic and smaller:

1typedef struct gamedata {
2	int gameid;			
3	char drive;				
4	char path[65];			
5	char name[32];
6	int has_dat;				
7	struct gamedata *next;
8} gamedata_t;

That's what, 102 bytes total, plus the pointer to the next instance. Even a thousand games would be barely 100KB in memory.

my bad, I thought you held the entire ini in memory and parsed the whole thing into structs.

No, just the basic game+location information which is scraped at launch time. I even make the parsing of the 'real' game name from the metadata file an optional at-run-time parameter (so that the game list can use the real name and be sorted on them), or, alternatively just use the found 8+3 Dos name..... purely because there may be some systems that are so IO starved that even reading a few hundred bytes for each scraped game may slow it to a crawl.

The X68000 for example, is pretty slow and benefits from doing this (it's a powerhouse in terms of graphics and sound, but IO less so; it's just a 68000 at the end of the day). The PC-98 (well, my 9821An, anyway) less so. I've yet to test any large number of games on my 286 - but that's what I'm using as a yardstick for whether the application will be usable by the majority of hardware configurations; hence keeping it real mode and to 640x400.

Add the metadata file / artwork bitmap delay code to the keyboard handling routines. Now the launch.dat metadata file for a game and the first artwork bitmap for a game only fire if a timer threshold is exceeded (used 500 clock ticks = ~0.5s as a starting default). It makes scrolling up and down and paging up and down far, far quicker. Quite pleased with the outcome 😀

Also thinking about extending the metadata file format with [sound] and [video] attribute sections so that we can add additional filters to the search function to restrict games by audio device type and video adapter type.

Finally, since we're no longer dealing with the X68000 or PC-98, i'm tempted to remove the RS-MIDI icon from the interface, since I don't know of any standard serial port midi interface for Dos. I'll keep the main 'MIDI' indicator, but move audio/video device compatability to a pop-up window, or overlay it on the artwork window instead.

Well the code is totally broken at the moment! 😁

I made the decision to try and remove as much code that relies on dynamic memory allocation (malloc/_fmalloc etc) as possible and to statically define as much as possible, in order to cut down on some of the non-deterministic behaviour of the code (amount of objects of different types which are in memory, size of data, etc).

So it's currently quite broken, as there are lots of dependencies (as you would imagine) on some functions being able to dynamically create certain data structures as needed. I'm gradually moving through the code sorting this out.

The upshot of this is that although the data section of the application may grow somewhat (there will likely be a small increase in data due to some larger structures which are declared in the main routine and accessed extern in other functions), it should drastically reduce the possibility of memory fragmentation and the overhead of creating/destroying objects in many of the loops within the code. My hope is that it will make the application more robust.

Haven't done a great deal more over the last few days (going through a restructuring exercise at work, so my mind is elsewhere right now), but did get a few bits of optimisation done:

- Artwork loading is further improved by approx 25%

- Reduced the amount of malloc()/free() calls in the main supporting functions which deal with loading the metadata files from disk for each game, as well as processing the list of artwork images into an array (instead of a linked list).

Bearing in mind that the Dosbox timings are purely synthetic, the current time to stream-in a 320x200 8bpp image line-by-line from disk and render it in the artwork window has now dropped from 220 clock ticks to 165 clock ticks.

I'd like to get to a situation where I'm only allocating memory for an object once, and passing references to the low level functions as a means to try and reduce the possible sources of memory leaks and further bugs. The main ones left are the initial metadata scraper, which mallocs() a new data object for each game directory on the disk... but that's only done the once, at startup, so it can probably be left as-is (I don't want to get into statically declaring the game list to be a fixed size of 200, 500, 1000 entries, or whatever). Everything else is a one time malloc() call, except the filtering code.

In the filtering code I need to parse the metadata file for each game in the inventory to examine the fields to see if it matches the publisher/developer, genre, series, selected sound device etc. I currently malloc/free() each file in turn, so I will probably change that to just the one object and pass a reference instead.

Once that's done, there shouldn't be any further malloc calls once the application is running and at the main UI screen and we should have a good idea of what the steady-state memory use actually is.

Well, I can say that I am now able to load over 500 games into the interface without it locking up any longer; including those 'odd' RPG and Adventure sub-folders which were crashing it previously.

There are now no functions which dynamically allocate or free memory (other than on the stack) once the interface is running - so I should have a pretty steady-state of memory use once that dynamic list of game objects has been scraped from the disk.

Next job is to make the 'filter' selections scrollable - by that I mean if you select to filter by genre or company, and there are more choices than can currently fit in the window (~30 checkboxes will fit), then allow the user to page up/down through the available filter options. Clearly with many hundreds of games the chance of having more than 30 genres is likely, and the chance of having more than 30 companies (developers or publishers) is almost a certainty!

Once I've got the latter implemented I'll try to put up some binaries for people to have a go with.

Added the ability to search by technical specifications:

You can then select one or more of the hardware/technical attributes of a game (such as audio or video device support):

The list of games is then searched and the on-disk metadata interrogated for an AND-ed set of the selection criteria (in the case of the above, all games listed as supporting Soundblaster and General MIDI):

As a result, the metadata definitions for a game (Duke Nukem 3D data filled in as an example) have been expanded to the following:

[default]
name=Duke Nukem 3D
developer=3D Realms
publisher=GT Interactive
midi_mpu=1
midi_serial=0
year=1996
genre=FPS
images=cover.bmp,screen1.bmp,screen2.bmp,box.bmp
series=Duke Nukem
start=DUKE3D.EXE
alt_start=SETUP.EXE

[sound]
beeper=
adlib=1
soundblaster=1
mt32=1
gm=1
tandy=
ultrasound=1
disney=
covox=

[video]
text=
cga=
ega=
vga=1
tandy=
hercules=
svga=1

[cpu]
xms=
ems=
dpmi=1
8086=
286=
386=
486=1
586=1

All of the metadata fields are optional, obviously the more that's there, the better use the filter is in finding the game you're after.

Genre, Series and Company (a set which includes the Publisher and Developer fields) filtering has also been extended to allow multiple pages of selection criteria - if you have 200 genres then you can page through the list until you find the one you want. There's a hard limit of 255 unique genre/series/company entries though as a sane default to try and limit memory somewhat.

I think the application is pretty much near a release quality - there's a few minor bugs (the graphical corruption) and user interface improvements (a better display in the event that no results come back from a filter operation) to make, but then I'll start publishing some pre-compiled versions and make a post in the announcements area.