Almost any game that allows IPX connections (most commonly associated with small LAN's), can be played with things like Hamachi. Direct play normally means that the game has built in modem commands for direct connect modems. This allows one computer to launch the game, and use it to directly dial another computer that is also running the game. From there, the two computers establish a handshake and one acts as a host for the game. This was a very common way to play with others in the mid-late 90's.
IPX on the other hand was meant to be used on a private network and back then, not to many people had networks in their homes. A simple but well built switch could cost $100 alone and even if you made your own Ethernet cables, you didn't save that much and then there was the cost of the computers themselves and the fact that you had to add a network card that could cost another $50-$150 per machine. They were commonly used at the birth of LAN parties and a few companies that didn't bother to check what the IS/IT departments were doing (god knows how many hours I and the rest of my team played games over the company network back then.
Things like Hamachi or the old Battlenet (3rd party program that was named after the Blizzard network) convert and run the IPX packets into TCP/IP so they can work over a WAN connection (in short, over the internet). Games like the original ROTT, Blood, Duke 3D, Heretic, and Hexen (and countless others) from that time period almost always had an IPX option along with direct connect. You should be able to connect to anybody with a service like Hamachi using the IPX option, even in DOSBox. Trying to go old school with direct connect and modems will be much more complex if you have never dealt with it and that would include whomever you are trying to connect too.
If you decide to try modem play, I would recommend a real hardware modem and not the junky Winmodems that flourished in the late 90's. Winmodems have no microcontrollers and are entirely run by the computer (ie the CPU, RAM, and overall speed of the system have huge impacts on performance) and chew up a nice chunk of the systems resources. In all honesty, the direct connect modem option simply isn't worth the time, and effort. You would be far better off with using Kali on a vintage machine (its still out there and still in use), and putting in a NIC instead.
EDIT
By the way, the serial option was for connecting two computers directly to each other via a null modem serial cable and creating a sort of mini lan. it was often the worst option as most games didn't not allow the serial UART to run any faster than 9600bps and that was often a minimum of speed needed to play. Most games at the time were optimized for 33.6k sending. The few games that did allow greater than 9600 at the serial port tended to run like crap because most consumer grade rigs of the time also used a software based serial UART like the Winmodem to save a buck and yet again, it caused the systems resources to be used up on running the port and directing data transfer instead of a full blown (but more costly) hardware based microcontroller. I would HIGHLY recommend avoiding that option at all costs unless you know that your computer and the second one have quality serial UART's.
Oh and a final note, IPX could allow more than two people to play if the game allowed it. The direct connection and serial connection limit the connection to only two computers.
