Moved as per request.

Edit: Actually, I think this belongs in the multiplayer sub-forum. Moved (again).

MOVED again to the DOSBox-does-not-run-windows-executables-forum......erm.....
...oh we don't have that. Hm leaving it here maybe.

I thought you'd run dooms setup and start multiplayer from there.

I think both is possible, though the "Returned from doom.exe" sounds strange.

Yeah, sounds really strange. Doom and Doom II were not "Windows-Aware", so there should be no Windows-apps in the game folders. The error message is pretty clear - one of the programs you're using for launching Doom is trying to run a Windows executable.

Here's some stuff to try:
- If you're feeling really anal, try to identify which program gives the "can't run in DOS mode" message. List all executables in the game directory, and execute them one by one, until you have the one that's causing the error message. Oh, don't forget the .BAT files, too.
- Reinstall Doom and Doom 2, try again
- Use one of the many source ports (this is what i'm doing when i feel the need for some multiplayer Doom)

The one that's causing the trouble is IPXSETUP.EXE, because DM.EXE switches to that EXE when everything is set up.

BTW, if all else fails, I'll just use Chocolate DOOM, because I know that port's multiplayer works fine. I just wanted to get the original DOS versions' multiplayer working because I'm really obsessed with playing the original version of DOOM.

EDIT: I tried doing multiplayer with a Serial/NullModem connection, and it did the exact same thing! And those DOOM installations were fresh, BTW. The program is SERSETUP.EXE

Doom games use the following executables for network or modem/serial play.

======================
Ipxsetup.exe and sersetup.exe
======================

Those were never windows-only.

I used to play multiplayer Doom II on dos a long time ago. Currently, I have switched to Zdaemon for online play, but my memory is still clear. The error that you are receiving indicates only one thing. I think your doom files are corrupted. Ipxsetup.exe is a dos executable. Dm.exe is also a dos executable. There aren't any windows variants; Doom 95 comes with another launcher and does not include ipxsetup.exe or sersetup.exe.

I have Doom Collector's edition which I have transformed into a dos version, using the shareware files, found on ftp.idsoftware.com. I've just run another test with ipxsetup.exe (run independently, and also through dm.exe), and it executes normally under Dosbox 0.72, trying to find players and establish a connection. The Ultimate Doom v1.9 and Doom II: Hell on Earth v1.9 are the games I have tested just now. I started two dosbox instances, started an ipx server on the first, and connected through 127.0.0.1 on the other. Then back to the first, I started dm.exe, configured for two player ipx play, deathmatch, executed and it went to ipxsetup.exe, looking for nodes. Then I switched to the second dosbox, started dm.exe, configured, executed, it found a node, and the game booted and I was inside. Hehe, I even moved the far away character to player two, switched back to green, and I made one frag, that was fun actually.Then I quit both dosbox-es. Test complete, works great.

The bottom line is, I think you need to check your ipxsetup.exe file size, and most likely reinstaill Doom and Doom II from your original cd or floppy disks. That should solve your problems.

I GOT IT WORKING!!

There were two things I did wrong:

"This program cannot be run in DOS mode"

I set DOSBox to automatically mount to a strange type of drive (B cls something). That's what was causing the DOS Compatibility problem. I switched my configuration to "mount c \doom" and it was solved!

"Redirected from DOOM.EXE"

I renamed DOOM.EXE to DOOMX.EXE because I wanted to name dosbox.exe to the former. I had to change it back.

Now both DOOM and DOOM II work perfect on multiplayer mode! 😉

Thanks for trying to help though. I appreciate it.

Yeah, it was a bad idea to mount it to a B drive. And yes, altering doom.exe to doomx.exe was even worse, as ipxsetup.exe tries to launch doom.exe or doom2.exe after it finds a node.

Remember that it is always good practice to check dosbox's readme.txt out, plenty of useful information there. Or if you don't feel like it, you can at least type intro when you launch dosbox itself, that's a very nicely done tutorial as well.

But I am glad to hear you got it working. 😀