Doom games use the following executables for network or modem/serial play.
======================
Ipxsetup.exe and sersetup.exe
======================
Those were never windows-only.
I used to play multiplayer Doom II on dos a long time ago. Currently, I have switched to Zdaemon for online play, but my memory is still clear. The error that you are receiving indicates only one thing. I think your doom files are corrupted. Ipxsetup.exe is a dos executable. Dm.exe is also a dos executable. There aren't any windows variants; Doom 95 comes with another launcher and does not include ipxsetup.exe or sersetup.exe.
I have Doom Collector's edition which I have transformed into a dos version, using the shareware files, found on ftp.idsoftware.com. I've just run another test with ipxsetup.exe (run independently, and also through dm.exe), and it executes normally under Dosbox 0.72, trying to find players and establish a connection. The Ultimate Doom v1.9 and Doom II: Hell on Earth v1.9 are the games I have tested just now. I started two dosbox instances, started an ipx server on the first, and connected through 127.0.0.1 on the other. Then back to the first, I started dm.exe, configured for two player ipx play, deathmatch, executed and it went to ipxsetup.exe, looking for nodes. Then I switched to the second dosbox, started dm.exe, configured, executed, it found a node, and the game booted and I was inside. Hehe, I even moved the far away character to player two, switched back to green, and I made one frag, that was fun actually.Then I quit both dosbox-es. Test complete, works great.
The bottom line is, I think you need to check your ipxsetup.exe file size, and most likely reinstaill Doom and Doom II from your original cd or floppy disks. That should solve your problems.