As many users have problems setting up directserial/modem connections, here's a short guide to getting Dosbox talk to the outside world. Since directserial/modem support is still being updated regularly, you need a recent CVS, links can be found in the Dosbox Development Forum (For the most recent Windows binaries see this page).

Configuration of directserial/modem functionality is quite easy and is done (surprise!) via dosbox.conf.

Suppose you want an emulated modem on COM1 listening on port 23, then the following configuration can be used.

[serial]
serial1=modem listenport:23
serial2=dummy
serial3=disabled
serial4=disabled

phonelistsize=3
phone1=localhost:1023
phone2=xxxxx.dialin.net
phone3=192.168.0.1

Only the settings of the first paragraph are really needed. The second paragraph specifies phonenumbers to be used with programs that don't support dots in between numbers. Which brings me to the next point: setting up a connection. On a second computer (or with a second instance of Dosbox, for testing) the same configuration is used. Start dosbox and the game you want to use the modem with (one side has to be configured for answering, the other dials). The number dialed should be the full IP address of the computer to which you want to connect (with dots). If it is not possible to dial such an address, you have to specify the address in the phone list (second paragraph of [serial] config) and simply dial x (where x is the number in the phonelist, eg 1 for phone1).

Ok, so much about modems. Directserial is a lot easier, as in most cases there's not much to be set up. The following configuration can be used to set up a directserial connection to the real COM1 of your computer (realport) which shows up in Dosbox as COM1 (ie serial1).

[serial]
serial1=directserial realport:com1
serial2=dummy
serial3=disabled
serial4=disabled

Well, if you're lucky, it already works with this simple setup. If you're not you may try to configure the more advanced options as there are: irq, startbps, bytesize,stopbits, parity (all optional).
These are all give by adding the parameter (in the same line) in the form parameter:value. Example:

serial1=directserial realport:com1 startbps:1200 parity:e bytesize:8 stopbits:1 irq:4

I can't give any further advice on this, though, since I know almost nothing about this. Perhaps Hal (who by the way, greatly enhanced or added to dosbox all this functionality) can add some information here. If you have an app requiring additional parameters you should know what you're doing anyway...

Troubleshooting:

1)Check every line of dosbox.conf in the [serial] section. Small errors can be disastrous...
If the modem is correctly set up, the dosbox console should have a line "MODEM: Port listener installed at port 23" in it on startup. For directserial, the corresponding line is "Serialport at xxx: Opening COMx"

Also, if the modem is initialized or dialing starts, there should be corresponding output on the console. If this is missing, either the modem is not correctly configured or the game/app is not yet supported.
2)Check your firewall setup. Is Dosbox blocked? Are the modem ports blocked?
3)Set core=normal, some games/apps have still problems with the dynamic core
4)Make sure the com port isn't used my something else.
5)Increase or decrease cycles.

I'll gladly add information I forgot or correct mistakes. Feel free to add info or to submit the names of successfully tested games. If you have questions concerning directserial/modem these should go here.