Managed to improve the connection some and fix some bugs with it.
When connecting to a TCP server(making a TCP connection) as a client(dial-up modem dialling) it now won't count the connection as established instead of failing to make a connection(this was due to an incorrect result code when failing to establish a TCP connection). So dialing an invalid address will now no longer make the serial modem(or pseudo-client for a passthrough connection) think it's connected(while in reality it isn't).
I also added two new options to the passthrough connection. The first is a simple sending of the DTR/RTS signal to the other side(using Dosbox's protocol 0xFF escape). Said signal is the ANDed with UniPCemu's connected status in much the same way on a receiving UniPCemu connected. The signals mentioned before are simply ANDed with said bits for their effective results on the DSR/CTS inputs to the UART.
RTS is completely ignored for being able to receive in these new modes.
The second mode is like the first one, but it dials out to the first entry in the phone book(without command parsing and ignoring (stuff like ; etc.), although ip:port is still interpreted in the same way(using 12 digits and a variable sized port number)) whenever DTR is raised(hanging up and redialing when raised again will lower the other side(in the second of those mode) and will disconnect and reconnect(like the ATHDT0 command) whenever becoming high again.
I also added a settings menu option to change the passthrough mode(between the mentioned four modes(which includes the normal modem mode as the first mode)). This setting requires a emulator restart to apply(due to switching modem emulation modes).
When the settings menu sees that the modem is put in a passthrough mode, it will show a new option. This option is the connect/disconnect option. It will allow effectively dialing out to an inputted entry like the phonebook(but using normal text input, using the keyboard(Direct Input mode) or OSK in UniPCemu, instead of a setting of the loaded settings(not stored)). Once confirmed, it will make the modem in passthrough mode dial said address. That's a normal IP:port address, or alternatively an address like "google.com:80". The slash-character is allowed as well(besides alphanumeric, colon and dot).
It's function can also be used to hang up the connection when connected (since the DTR signal lowering doesn't do this).
So basically it's emulated what Dosbox does as well, within UniPCemu's constraints(except telnet protocol, special delays and having a forced transparent/non-local setting(both disabled effectively) as far as I can see from the code/manuals of Dosbox). Also a slight difference on the masking instead of plain receiving of the 0xFF parameter on the connected line with the connected status. Although that shouldn't be a problem?
Also, the 0xFF break bit is stored, but currently unused by both the modem and UART emulation. This is mainly because I don't know how this works in actual hardware.