I don't have windows 11 (microsoft won't let me put it on an old computer), but I assume it would be similar to windows 10.
Start, Settings (little cogwheel), Network & Internet, Change Adapter Settings, your adapter will be listed, right-click and choose Properties.
Turn everything off, then turn on File and Print Sharing, Internet Protocol Version 4, and any Link-Layer Topology things.
Double-click on Internet Protocol Version 4, you get the page where you can enter your IP address and subnet (leave everything else off/blank). IP address must be the same first 3 numbers with the last number different between the 2 computers (eg 192.168.1.10 for one computer, and 192.168.1.11 for the other one). The subnet mask must be 255.255.255.0 on both computers.
There's an important thing with ethernet cables. If you plan to directly plug one cable from one computer to the other, you need a crossover cable. The more common way with standard cables is to use a spare hub, switch, or gateway/router, and use 2 cables one for each computer going to the switch or whatever. No need for an internet connection.
One last thing is to make sure that both computers are in the same Workgroup. I use WORKGROUP, but you can use anything you like. This setting is in the same place where you set your computer name, which must be different for each computer.