Alright, I removed the CLI/STI, it was just a precaution anyway. The (...) is there because I was compiling is as CPP and it otherwise complained, wanting the ... to be there. Changed it to C and now, as it is plain C, and it takes just () giving warnings about an unknown funcion prototype, but no error. Compiled it again and its giving out the same sound and freezes.

WhiteFalcon wrote on 2024-05-07, 10:04:

Alright, I removed the CLI/STI, it was just a precaution anyway.

It's not a precaution if it's not needed and damaging things when used 😀
Share the updates.

WhiteFalcon wrote on 2024-05-07, 10:04:

The (...) is there because I was compiling is as CPP and it otherwise complained, wanting the ... to be there.

Ok, I see in the doc getvect/setvect use function types that has ... in arguments when used with C++.

vstrakh wrote on 2024-05-07, 10:48:

It's not a precaution if it's not needed and damaging things when used 😀
Share the updates.

It was during the phase when I added tons of pushes and pops everywhere 😀 I just removed the CLI/STI and replaced (...) with (), no more updates yet. Going to start testing it by commenting parts out.

Ok, I see in the doc getvect/setvect use function types that has ... in arguments when used with C++.

Yes, those two functions complained. I switched to C now so its just warnings about undefined prototypes.

I have narrowed it down to one line, surprisingly its nothing to do with the store/restore or interrupts per se, its this line:

while (inportb(0x60) != 1) oldKeyHandler();

When I comment it out, it works. Apparently DOS, unlike DOSBox, doesnt like the keyboard port to be hammered, even by a read. So I think the way to do keyboard reading withing mainTSR (the body of the whole program) may be to pass the scancode from the interrupt handler to mainTSR as an argument. At the moment its a (void). Reasonable?

EDIT: Not reasonable. The new key handler works as the launchpad for mainTSR, if I was to change that, any keypress would start mainTSR all over again, including the store and restore. I would have to install a different key handler inside mainTSR. This is getting clumsy. Actually maybe I could just restore the old handler while in mainTSR and restore the new one after leaving it, hmm.
And no, that doesnt work either for somer reason.

WhiteFalcon wrote on 2024-05-07, 11:24:

I have narrowed it down to one line, surprisingly its nothing to do with the store/restore or interrupts per se, its this line:

while (inportb(0x60) != 1) oldKeyHandler();

I didn't look into this before. This feels very wrong.
First - you can't (or rather shouldn't) spin any long task in the interrupt handler. Everything locks up, not being able to execute anything or trigger hardware events.
Second - you can't treat old handler as a function to call freely at any time. Probably Borland wraps up things for you, so IRET in the original handler is treated as far return with popf (I see pushf before doing far call into old handler). But the old handler also talks to PIC, repeatedly acknowledging interrupts no matter what those are.

Instead you should flag the new state in some variable, and restore the screen the next time you enter the interrupt handler and you see ESC scancode.
If you want to take controls away from the programs, you should not call the old handler at all, but then you must acknowledge the IRQ with PIC yourself.

The buzz you hear is likely the BIOS screaming the keyboard buffer is full, as you call the interrupt (i.e. event) handler repeatedly when there are no actual keypress events.
Normally you get handler called when the keyboard IRQ signaled there is a _new_ scancode. But now you spin in the loop, calling the original handler, which will see the same last scancode, regardless if the key was pressed or not, spamming the keyboard buffer.

WhiteFalcon wrote on 2024-05-07, 11:24:

Apparently DOS, unlike DOSBox, doesnt like the keyboard port to be hammered, even by a read.

It's not DOS, it's hardware. You abuse the PIC by repeatedly calling original handler, and you do not do so on actual keypress events.

After reading this, I am clearly in way over my head. But I am not giving up just yet. I take it by acknowledging the PIC you mean outputting 0x20 to port 020, right?
I rewrote the program to use the typical simple game key handling. It not longer freezes unless I try to wait for ESC to be pressed. I cannot figure why its not changing the keys[] array to correspond to the key presses. It works fine in my normal tiny key handler used for games.
The code it even more messy now as I am trying to figure out every possible scenario happening withing the TSR 😜

https://paste.ofcode.org/33eShMD7LiRkBAvyGEZsbrE

I ignored the extended scan code here, but ESC should still work as "1" I hope.

WhiteFalcon wrote on 2024-05-07, 14:25:

I take it by acknowledging the PIC you mean outputting 0x20 to port 020, right?

Yes, correct.

WhiteFalcon wrote on 2024-05-07, 14:25:

I rewrote the program to use the typical simple game key handling. It not longer freezes unless I try to wait for ESC to be pressed. I cannot figure why its not changing the keys[] array to correspond to the key presses. It works fine in my normal tiny key handler used for games.

It doesn't look right.
The 'running' is set to one on mainTSR() entry and cleared to zero on exit. So the check for 'running' in newKeyHandler never sees that 'running' as 1.
Feels like you're treating the mainTSR as some kind of independently running application thread, but in essence it's all a single function call - a single call to newKeyHandler when the key is pressed/released.
So the mainTSR needs to be rewritten as to move through the TSR states each time the newKeyHandler is called. The state of the TSR must not live in the mainTSR, so no thousands of loops with delays and printfs within the mainTSR. You only update the state machine on the mainTSR entry, and you immediately exit after the update. If the update caused the required change - save screen, or restore screen - you do only that operation.

Also, we've established the DS gets loaded with the program's data segment by borland, so you can drop "mov ax, SEG screen_state; mov ds,ax" as superfluous in StoreScren/RestoreScreen.

I see, you are right, I did somehow considered mainTSR to be the main program and expected that when the execution was in there and a key was pressed, the handler would get called anew. Which is why I have the complicated switching within the handler itself. But as you describe it, its probably not how that works and the handler does not get called again until it finished first? Which completely changes the way the whole thing is made, oh my. Ok, I will get to making big changes to it. Still I learned some things I would not otherwise 😀

The delays were there just for testing to have time to press another key and see if the keys[] array got changed, but it could not if the handler does not get called recursively.

So I have rewritten most of the code, its much less of a mess now and it almost works 😀 I can now process the keys within the TSR and exit it too, but when back to the state before calling the TSR, keyboard no longer works. Its not frozen though as I can all the TSR again and it still works, the old hanlder just wont process anything anymore despite being called when the TSR is not active. I assume I am not doing some "cleaning" after quitting the TSR and cannot find any info on what to do.

https://paste.ofcode.org/6DBCJEDj5i3nqu2Q6XAWWL

EDIT: Forgot to remoce the redundant code, removing now 😀 Also doing all this in DOSBox again to prevent constant reboots so it will again be a surprise when tried on the 486.
EDIT2: It actually works, the only problem is that the ALT key is somehow not "cleared" so when I press it after quitting the TRS, all is well again. Just have to find why it gets "stuck".
EDIT3: And I solved it! 😀 Not sure its the correct way, but it seems to work fine now. I just manually clear the ALT bit in the kbd flag location before starting the main part of the TSR.
EDIT4: Moved the ALT clearing part to the section after restoring the screen instead, it makes more sense and takes care of any ALT presses during the main TSR. Promised a friend to post the working version here so here goes:

https://paste.ofcode.org/Bm4SQbWScwKGNyFSHy2rJs

WhiteFalcon wrote on 2024-05-08, 08:10:

And I solved it! 😀

Congrats 😀

Thank you for all your help with this 😀