On-hook means the modem is disconnected and the phone line is not in use (when you hang up the phone, it is sitting "on the hook"). Off-hook is when the modem is talking on the phone line, either before or after a connection is made.

Once a connection is made, the data that is passed to and from the modem is no different to a serial port connected directly to another computer. It is entirely application defined. You can use SLIP to take TCP/IP traffic and send it down a serial line, or the more modern (relatively speaking) replacement for SLIP, called PPP. SLIP is much simpler (I have implemented SLIP on an Arduino to get TCP/IP working without any shields) however PPP is more versatile.

If you are running an old operating system inside your emulator, then you will need to configure it to speak SLIP or PPP when you set up a dial-up Internet connection. Your emulator can then listen to that data coming from the serial port and decode it using the SLIP or PPP protocols, and send the appropriate response data to funnel Internet traffic through the connection.

Most old DOS games will not support TCP/IP for networking anyway. You're likely better off working on null modem connections and/or IPX.

Is it possible to automatically detect whether the application is using SLIP or PPP?

Probably - it's been a while but you could just parse the first packet with both protocols and if one comes up with rubbish values and the other doesn't, then you'll know which it is.

I seem to remember PPP having heaps of "~" (tilde) characters, so it was very obvious when you got a heap of PPP data spewing up on the screen.

But how will I know an entire packet is buffered in one of those fomats? If a packet end signal is given in one packet with the other protocol used? Like SLIP packet end in a PPP packet?

For DOS games, they use direct serial connections so it's just null modem cable through phone line.
If something uses PPP or SLIP, it means the game itself (running in Windows perhaps) uses TCP/IP or UDP/IP, and those IP packets are then sent out of the machine by the networking layer, and depending on the LAN/internet connection, it could be the Ethernet card or modem. If it is the modem, it means the IP connection is through SLIP or PPP instead of Ethernet and the game does not know that.

And when a connection is established, the endpoints negotiate what protocol is used, so they know it's either SLIP or PPP and it won't change during this connection.

Exactly. You're better off with the emulator being able to establish generic serial connections without having to do fake dialing.

The only reason you'd want to do this is if you want to use dial-up networking in Windows to establish a (slow) Internet connection. It might be easier than emulating an Ethernet card. As others have said, games rarely if ever use IP, it's always IPX or a direct serial/modem connection.

There is no such thing as PPP inside SLIP. SLIP only carries IP traffic, and PPP only carries IP traffic. Think of it like SLIP is MP3, and PPP is Ogg. They both carry data from .wav files, but it doesn't make sense to consider an Ogg file being inside an MP3 file. If you ever had to do that, you'd decode the MP3 back to .wav, then re-encode it as Ogg instead. Likewise you can turn SLIP back into IP, then put it inside PPP instead, but both formats will only contain IP traffic, just encoded with a different algorithm.

But I want to know how to know if the data sent by the software(one byte at a time) is to be interpreted as a SLIP or PPP protocol, based on the data sent up to that point.

It would be bad if my application interprets a SLIP end of packet byte, while the software is actually in the middle of a PPP packet. The same, but backwards also applies. How will my software know if it's processing a full SLIP packet to be sent and not an partial PPP packet(or a partial SLIP packet that looks like a PPP packet)?

Back in the day, the user had to choose SLIP or PPP depending on what the ISP provided. You could just make it configurable for each emulated modem and leverage existing PPP or SLIP stacks. There should be some BSD licensed versions around.