Looking at the code, it could be that CFifo class is bugged? I'm not seeing anything that could corrupt the buffers themselves and issues occur with both command input and received data.

If the BBSes are telnet-based you need to issue the ATNET1 command before dialing.

jmarsh wrote on 2020-02-27, 23:52:

If the BBSes are telnet-based you need to issue the ATNET1 command before dialing.

Yes, I'm aware. Issues #5 and #6 describe behavior with telnet mode enabled with ATNET1 command.

On sourceforge, Marco Maccafferri posted a fix addressing at least some of these issues: https://sourceforge.net/p/dosbox/patches/287

To test it, you need the patch applied to both your DOSBox as well as the BBS's server-side DOSBox (assuming the BBS itself is running under DOSBox).

If you don't have a build environment setup to try patches, then you can try one of the binaries from dosbox-staging (https://github.com/dreamer/dosbox-staging), which accepted Marco's patch a couple weeks ago: https://github.com/dreamer/dosbox-staging/pull/172

Also of note are the com port settings, mentioned in section 9, Serial Modem, under the "BBS Gaming" subsection:
https://github.com/dreamer/dosbox-staging/blob/master/README

Curious if any of this helps.

Thanks, I'll give it a shot!

Well, I've compiled the patch and it was of no use whatsoever, it didn't resolve any of my issues. It looks like it's only intended to fix file uploading. But at least I have the build environment set up now so I can try and do my own investigation.

EDIT: Ok, found a bug with backspace handling at line 686, if you type backspace while cmd buffer is empty, it will write backspace symbol to the buffer and screw up the command.
EDIT 2: Seems like backspace characters are what is screwing up FTP server commands as well. This is not a DosBox bug, however, this seems to be a problem with how FTP server handles backspace.
EDIT 3: Duke Nukem 3D sends three ESC (0x1B) chars to the modem before init string which are not handled by virtual modem. Not sure what this is, kind of looks like escape sequence?.. Making the command parser skip those seems harmless although it doesn't fix invalid packet crash.
EDIT 4: Doom issue appears to be gone, maybe I was doing something wrong.
EDIT 5: This still leaves 3 issues out of 6, I don't think I can figure those out.
EDIT 6: Duke Nukem 3D crash only appears to happen on version 1.3D (original retail release) and it's the only version to send 3 ESC characters. There might be a connection. I'm thinking it's a bug because I've checked several pages detailing Hayes command set and there's no mention of command mode handling ESC character.

I saw the same behaviour with Duke3D (1.3D) but every few tries it would actually work correctly; the data being transferred to/from the game was exactly the same in all cases so it looks like it's just some internal state bug in the game.

jmarsh wrote on 2020-03-01, 07:44:

I saw the same behaviour with Duke3D (1.3D) but every few tries it would actually work correctly; the data being transferred to/from the game was exactly the same in all cases so it looks like it's just some internal state bug in the game.

Has it been tested on the real hardware? I might be able to do that once I get that Retro WiFi SI modem.

EDIT: I've re-tested NFSSE using two instances on the same machine (instead of one of them running on an old WinXP rig) and it doesn't drop to single digit framerate. However, it still runs worse than in singleplayer. This suggests that modem play in NFSSE is extremely taxing under DOSBox for some reason to the point where the game becomes unplayable on Pentium 4.

Modem emulation uses blocking TCP transfers, null-modem uses non-blocking. Like I said in the other thread, it's preferable to use null-modem whenever possible.

A wifi "modem" won't accurately emulate a serial port+hardware modem combination any more than DOSBox does already.

jmarsh wrote on 2020-03-01, 12:51:

Modem emulation uses blocking TCP transfers, null-modem uses non-blocking. Like I said in the other thread, it's preferable to use null-modem whenever possible.

A wifi "modem" won't accurately emulate a serial port+hardware modem combination any more than DOSBox does already.

Considering the data sent is the same, it seems more like a CPU emulation issue and not modem issue.

No. Interrupts are not predictable, even with fixed clock speeds. The fact that the game wanders off and tries to execute unmapped memory is a dead-giveaway that it's bugged.

jmarsh wrote on 2020-03-01, 13:04:

No. Interrupts are not predictable, even with fixed clock speeds. The fact that the game wanders off and tries to execute unmapped memory is a dead-giveaway that it's bugged.

It seems highly unlikely to me that the game shipped with a completely broken modem support.

You mean like all the games with adlib detection routines that fail on any PC made after ~2000?

jmarsh wrote on 2020-03-01, 13:31:

You mean like all the games with adlib detection routines that fail on any PC made after ~2000?

But they weren't broken at launch, were they? I don't understand your point.

Just because something worked (sometimes) at launch, doesn't mean it isn't bugged.

jmarsh wrote on 2020-03-01, 13:40:

Just because something worked (sometimes) at launch, doesn't mean it isn't bugged.

By that logic the majority of DOS games are bugged since most of them have compatibility issues on newer hardware. But the whole point of DosBox is to emulate old hardware to address that. I think it's reasonable to assume that DN3D wasn't broken at launch and, as such, this is DosBox compatibility issue.

You think they would have patched the code if there was nothing wrong with it?

jmarsh wrote on 2020-03-01, 14:54:

You think they would have patched the code if there was nothing wrong with it?

What? I'm not saying the game wasn't bugged, period; I'm saying that the current issue where the game crashes (almost) every time when you start a modem play was not present on the real hardware at launch.

Nicknine, even though ATNET1 works for small downloads, I have also confirmed that large downloads (ie: DOOM shareware, ~1.2 MB), fails after 5+ minutes downloading; like you've suggested.
This was confirmed when downloading DOOM1_2A.ZIP from bbs.dmine.net:24, using DOSBox at 6000 cycles running Terminate 5.00 with the serial port set to 19,200 baud.