Wow! This is a good one!
I could not get it to work properly, though. Not sure if my version is different from the expected or I am missing some step -- my copy comes from Sci-Fi Ultimate Archives collection. The game was "factory pre-cracked" for that budget release so I have starcon2.com and starcon2.exb. I've deleted .com and renamed .exb to .exe but the game still exits after attempting to enter star name. Just for reference, the file starcon2.exe in this release is 342819 bytes long and has file date 12/03/92.

vasyl wrote:

Just for reference, the file starcon2.exe in this release is 342819 bytes long and has file date 12/03/92.

342819 is the right length. The CRC32 should be 126d0616.
Did you put both the dosbox.conf and the sc2.dof (or movie.dof, I forget which one was the desired filename in that version) file in the game's directory?

why not use ZMBV for video ?

You are removing way too much in the keyboard port 60 write handler. (the mem_a20 stuff is important!)

I'm planning on rewriting the callback/run_realint stuff a bit so there is less recursiveness in dosbox. I think that this essential for more complex games. (and that there is only one machine running at one time)

Your way is quite "messy" compared to how dosbox is organized now, but maybe I'm thinking too much in the defined borders. One of my ideas is to use the config class (that startups and shutdown all devices) to make savestates.

I'm not sure what you mean with branching ?

I doubt that your savestates are endian safe. but that will be tricky because of all the pointers we use internally as well. I'm not sure if that can be easely corrected in your patch.

It's a nice initative though, certain things you handled quite nice. We will develop savestate support eventually (no suprise here).

Yep, looks nice 😀
Though as you noted the main problems remain (callback recursion,
pic/callback function pointers). They're better worked around/ignored
for the moment, will get really nasty otherwise.

The CRC is the same. movie.dof is there and is functional -- I can see dosbox trying to input star name. There is also input going on after the game exits.
From what I can see, the game asks for orange dwarf at 581.2:120.8 which is Sculptoris Delta but the input shows Horologii (a constellation around 680:740). Tomorrow I will try to find some other copy of the game (with all luck I may be able to find the original release) and see if any other files were modified in Sci-Fi Archives.

Qbix wrote:

why not use ZMBV for video ?

What is ZMBV? Is it open source? Does it work in mencoder?

Qbix wrote:

You are removing way too much in the keyboard port 60 write handler. (the mem_a20 stuff is important!)

Oh, I kind of forgot about the A20 stuff. Wonder how Descent still managed to get started, though.

But why I rewrote the keyboard handler is because I needed a way to introduce a keyboard queue that is not really a queue at all; instead, a list of key states. So that for example, if you hit A, but then, you undo the hitting of A with a savestate, no A press/release IRQ will be triggered at all; and a neat way to save the keyboard state in movie so that it can be replayed.

The thing with savestates is that not only need they restore the previous situation, they also need to act as if no savestate was done at all.

For example, if your movie is created like this:
. Frame 0000 -- input A, advance
. Frame 0001 -- input B, advance
. Frame 0002 -- input C, advance
. Frame 0003 -- savestate -- input D, advance
. Frame 0004 -- input E, advance
. Frame 0005 -- loadstate
. Frame 0003 -- input F, advance
. Frame 0004 -- input G, advance
And later, you replay the movie, the emulated system must progress exactly as if the original progress was like this:
. Frame 0000 -- Input A, advance
. Frame 0001 -- Input B, advance
. Frame 0002 -- Input C, advance
. Frame 0003 -- Input F, advance
. Frame 0004 -- Input G, advance
The game must react exactly the same way, and every internal aspect of the emulated system (including things like current rendering scanline, CPU cycle counter, etc) should be identical to the situation that was when the movie was under making. Loading the snapshot cannot involve resetting any device to its default values (such as zeroing the CPU cycle counters), etc.
I do not mean to imply that you meant that, but just to be safe, I clarified it.

By branching, I meant that while the official version undergoes development, the TAS community maintains their own emulator version which also undergoes development, but the differences between them will increase. This is a bad thing.

Qbix wrote:

I doubt that your savestates are endian safe. but that will be tricky because of all the pointers we use internally as well. I'm not sure if that can be easely corrected in your patch.

No, they are not endian-safe. In fact, these savestates are only valid for the binary they were saved with. When you do even trivial changes to dosbox and recompile it, the old savestates no longer work, because they contain pointers which were saved raw, without any semantics.

One of the greatest features, but also the most troublesome, in this patch, is the "fsemu" part. It is a replacement / middle-end to act between the INT 21h API and the DOS_* function calls. Basically, it implements a virtual copy-on-write filesystem that uses the real mounted filesystem as a template (ignoring actual mountpoints though), but any writes into the filesystem will be simulated, without actual writes happening, and the simulated state is snapshottable so that any revision of the filesystem can be restored ... without a single write to actual files happening. In TAS making, the existing of this middle-end is necessary, because the savestate must save and restore the complete system state -- including the list of open files&handles, their seek positions and their contents, and the movie must be replayable from beginning with always the same beginning situation and always the same end result.

vasyl: Sounds like some timing difference. With this information alone I cannot guess what causes it.

What is ZMBV? Is it open source?

Maybe look at the dosbox source tree 😉 (hint: src/libs)

it might be easier to put your changes in the sdl_mapper file (to save keyboard input)

My idea was to have an alternate constructor for each module of dosbox that gets the values for all variables involved from a file. So a restoring of the savestate would rebuild the machine with values from a file.

wd wrote:

What is ZMBV? Is it open source?

Maybe look at the dosbox source tree 😉 (hint: src/libs)

Oh, but I would really suppose that H.264 maintains a better quality/size ratio than ZMBV...

Re: sdl_mapper. You might be right. However, I had already created the SimKeyboard system once (before I started working on dosbox, I tried to make my own emulator), and it was easiest to plug it into where I did.
Also, the sdl_mapper mechanism is quite complex and hard to understand in my opinion... I eventually figured my way around it, and I implemented some hotkeys by putting them in CKeyEvent::Active(), but I'm sure this is also the wrong way.

zmbv is lossless
and fast.
(and can be recoded later by ffmpeg)

Regarding all the pointers, my idea was to issue a "declare a callback" call in each object's constructor for each callback function they use, in the same way as it does the RegisterSaveStateField calls.
The "declare a callback" would assign an unique name for each callback function, and when savestates are saved, the callback pointers in structures are replaced with those names.

When the savestate is loaded, each callback function name loaded from the savestate, is replaced with the actual pointer, using the pointer declared by the currently running binary.

For example, in sblaster.cpp, SBLASTER_Init(), there would be a call "RegisterCallback(DMA_Silent_Event)", and it would insert the pointer and the name into a static map in state.cpp.
In pic.cpp, PIC_Init(), pic.entries[].event would be introduced as a callback pointer to the savestate mechanism.

However, this seems like large work (cataloging each callback function and changing the savestate mechanism to make an exception for the function pointers), so I haven't done it yet.

Dynamically allocated memory would still need their own save&load functions, as is done in fsemu.hh for example.

These changes would resolve the "recompile dosbox, and all your savestates become invalid" issue, but it wouldn't make them portable.

I was thinking something comparable.
having a name match for the callbacks and linking those together.

I've located the original version of Star Control II and while some files are different it still produces exactly the same result. I don't think it's timing issue, I rather suspect host CPU dependency. I don't remember if SC2 uses FPU or not but if it does and if it is used as part of randomizer then it may behave differently on Intel and AMD. The way DOSBox works those subtle FPU differences may leak from host to the emulated environment -- the differences are irrelevant in almost all cases anyway. My system is Athlon64 3500+. Let's see if we get more reports.

I have tested it on Athlon XP, Pentium M... and Athlon XP, all successfully.
I thought Dosbox does not depend on host CPU?

Tested versions are alpha3 and alpha5.

(Five minutes prior to this posting I tested it on Pentium M.)

Contents of directory where I run it (taken in Cygwin):

1bisqwit@Gon:~/sc2$ ls -l
2total 16496
3-rwxrwxrwx    1 bisqwit Ei mitään 2241967 Mar 24 23:57 SDL.dll
4-rw-r--r--    1 bisqwit Ei mitään   53103 Jun 27  1996 androsyn.shp
5-rw-r--r--    1 bisqwit Ei mitään   46739 Jun 27  1996 arilou.shp
6-rw-r--r--    1 bisqwit Ei mitään   87325 Jun 27  1996 blackurq.shp
7-rw-r--r--    1 bisqwit Ei mitään   97521 Jun 27  1996 chenjesu.shp
8-rw-r--r--    1 bisqwit Ei mitään   82821 Jun 27  1996 chmmr.shp
9-rw-r--r--    1 bisqwit Ei mitään 2053204 Jun 27  1996 con1.pkg
10-rw-r--r--    1 bisqwit Ei mitään 1443224 Jun 27  1996 con2.pkg
11-rw-r--r--    1 bisqwit Ei mitään    6688 Mar 21 20:31 dosbox.conf
12-rwxrwxrwx    1 bisqwit Ei mitään 2409472 Mar 31 23:26 dosbox.exe
13-rw-r--r--    1 bisqwit Ei mitään   56261 Jun 27  1996 druuge.shp
14-rw-r--r--    1 bisqwit Ei mitään   81403 Jun 27  1996 human.shp
15-rw-r--r--    1 bisqwit Ei mitään   66461 Jun 27  1996 ilwrath.shp
16-rw-r--r--    1 bisqwit Ei mitään  947953 Jun 27  1996 ip.pkg
17-rwxr-xr-x    1 bisqwit Ei mitään   30784 Jun 27  1996 keys.exe
18-rwxr-xr-x    1 bisqwit Ei mitään  170177 Jun 27  1996 melee.exe
19-rw-r--r--    1 bisqwit Ei mitään   67742 Jun 27  1996 melee.pkg
20-rw-r--r--    1 bisqwit Ei mitään   87127 Jun 27  1996 melnorme.shp
21-rw-r--r--    1 bisqwit Ei mitään   74011 Jun 27  1996 mmrnmhrm.shp
22-rw-r--r--    1 bisqwit Ei mitään 3080928 Mar 25 00:55 movie.dof
23-rw-r--r--    1 bisqwit Ei mitään   83953 Jun 27  1996 mycon.shp
24-rw-r--r--    1 bisqwit Ei mitään   89887 Jun 27  1996 orz.shp
25-rw-r--r--    1 bisqwit Ei mitään   89261 Jun 27  1996 pkunk.shp
26-rw-r--r--    1 bisqwit Ei mitään 2370872 Jun 27  1996 setup.pkg
27-rw-r--r--    1 bisqwit Ei mitään   67817 Jun 27  1996 shofixti.shp
28-rw-r--r--    1 bisqwit Ei mitään   62665 Jun 27  1996 slylandr.shp
29-rw-r--r--    1 bisqwit Ei mitään   48297 Jun 27  1996 spathi.shp
30-rw-r--r--    1 bisqwit Ei mitään    1434 Jun 27  1996 starcon.pkg
31-rwxrwxr-x    1 bisqwit Ei mitään  342819 Jun 27  1996 starcon2.exe
32-rw-r--r--    1 bisqwit Ei mitään   48093 Jun 27  1996 supox.shp
33-rw-r--r--    1 bisqwit Ei mitään   56583 Jun 27  1996 syreen.shp
34-rw-r--r--    1 bisqwit Ei mitään   48333 Jun 27  1996 thradd.shp
35-rw-r--r--    1 bisqwit Ei mitään   49083 Jun 27  1996 umgah.shp
36-rw-r--r--    1 bisqwit Ei mitään   93513 Jun 27  1996 urquan.shp
37-rw-r--r--    1 bisqwit Ei mitään   65059 Jun 27  1996 utwig.shp
38-rw-r--r--    1 bisqwit Ei mitään   36793 Jun 27  1996 vux.shp
39-rw-r--r--    1 bisqwit Ei mitään   44527 Jun 27  1996 yehat.shp
40-rw-r--r--    1 bisqwit Ei mitään   43445 Jun 27  1996 zoqfot.shp

Here's a screenshot of it running…
http://bisqwit.iki.fi/kala/snap/sc2dosboxsnap.png

subtle FPU differences

Which? Either c lib routines are used (consistent over fpus just has
not full precision) or the host fpu is used (dynamic core, but then
send a mail to amd/intel if they produce non-consistent fpu units).

Which core was used for recording?

wd wrote:

Which core was used for recording?

"simple".

Which? Either c lib routines are used (consistent over fpus just has
not full precision) or the host fpu is used (dynamic core, but then
send a mail to amd/intel if they produce non-consistent fpu units).

Unfortunately, even C RTL routines can be affected. A few years ago I had first-hand experience working on certain network game that made a lot of physics calculations. We had a lot of fun dealing with all bugs caused by FPU discrepancies. Granted, it was P2 vs K6, and I think the discrepancies were mostly in trig routines but things like that happen.
Now I don't think this is the case here. I've got the exact same file set as Bisqwit -- btw, that appears to be the same version as in SciFi Archives, slightly newer than the original release. I tried to run it on Centrino-based notebook and got the same result. The must be some other variable. So far I can only think about machine locale (all of my machines are set to US and Bisqwit's one is obviously non-US) but I don't see how that could affect random number generator.

vasyl, ensure yet again that you actually have that dosbox.conf file (which was included in the zip) in the directory where you launch dosbox (i.e. the game's directory). If the dosbox titlebar says "3000" instead of "20000", it is wrong.

For pmode/paging games you should try running them using the
dynamic core, as it avoids recursion for (most) pagefaults.