The story continues on 286 pc.
The newer TD 5.0 does not run on 286, with the issue buried in the dpmi16bi.ovl not recognizing my machine. I've ran the dpmiinst, it did some tests and altered the DPMI server internals, but TD still did not run. So I've tried earlier Turbo debugger v3.2. It has separate versions for real mode, 286 protected mode, and 386 virtual mode debuggers.
The real mode TD.EXE does not use XMS but eats up lots of conventional memory. The TD386 obviously does not run on a 286 pc 😀
Now the TD286 embeds its own protected mode management stuff, and has its own td286inst.exe, very much the same as there is dpmiinst.exe for dpmi16bi.ovl, which is not used by TD286.EXE.
Since it's not the DPMI extender, it doesn't look for environment variables, and the debugger has no switches to reserve XMS for application, like in TD386.
I've looked into td286 with Hiew to find the spots where I could patch in the code that will limit the XMS allocations. The final solution was even easier than I thought.
The TD286 protected mode manager does not really allocate every bit of extended memory, it only grabs the largest contiguous block. So the easiest way to prevent it from allocating the whole xms memory is to fragment it 😀
I wrote a tiny app that allocates half of the extended memory, then allocates smallest block possible (1KB), and then release the previously allocated large block. That tiny block is then simply left dangling, not being used for anything but causing the memory fragmentation, so now TD286 will grab only one of two large blocks available, leaving the rest for the applications.
Note that fragmentation only works for TD286 with its rudimental memory handling.
This does not work for TD that uses DPMI host, apparently it's more advanced and can stitch scattered physical memory blocks into one contiguous virtual address space.