jez wrote on Yesterday, 18:45:

IDA 5 can give you a decent call graph of a real mode DOS program, and you can do analysis of later Windows stuff. But am I right in thinking that extended mode DOS programs are virtually impossible?

Huh, while I don't have the capabilities of all IDA versions and editions at hand, I am very confident that at least commercial IDA in the 5.x range was perfectly capable of loading the protected mode part of DOS4GW software (IIRC that's typicall "LX", a "linear executable") just fine.

mkarcher wrote on Yesterday, 19:01:

jez wrote on Yesterday, 18:45:

IDA 5 can give you a decent call graph of a real mode DOS program, and you can do analysis of later Windows stuff. But am I right in thinking that extended mode DOS programs are virtually impossible?

Huh, while I don't have the capabilities of all IDA versions and editions at hand, I am very confident that at least commercial IDA in the 5.x range was perfectly capable of loading the protected mode part of DOS4GW software (IIRC that's typicall "LX", a "linear executable") just fine.

I just took a look at a DOS/4GW executable (from Wacky Wheels) and it seems more subtle than this. [3ch] points outside the range of the file and IDA's efd (which is freeware and understands a lot of these formats) doesn't identify it as LE. However, DOS32A is open source and seemingly can run the file, and its source suggests it uses LE, possibly compressed. DOS32A is all assembly and in taking a 10 minute look I haven't been able to pinpoint how the code identifies which executable format it's parsing (it supports multiple, not just LE).

ignore or remove the stub. just decode the LE format.

the biggets problem your probably going to run into is that you will have to work out the calling convention that was used (and it could be multiple if using third party libraries), and yes it can still have real mode code that it calls back into (some audio libraries had 16bit trampolines for IRQs)

mkarcher wrote on Yesterday, 19:01:

jez wrote on Yesterday, 18:45:

IDA 5 can give you a decent call graph of a real mode DOS program, and you can do analysis of later Windows stuff. But am I right in thinking that extended mode DOS programs are virtually impossible?

Huh, while I don't have the capabilities of all IDA versions and editions at hand, I am very confident that at least commercial IDA in the 5.x range was perfectly capable of loading the protected mode part of DOS4GW software (IIRC that's typicall "LX", a "linear executable") just fine.

Not sure what's going on with mine then because it just seems to get lost at the bit where the far call (presumably to DOS4GW's loaded code) happens.

In fact, that makes sense. If DOS4GW's code is loaded into memory at runtime, IDA doesn't have it in the file being analyzed so how can it come up with anything meaningful?

jez wrote on Yesterday, 19:23:

mkarcher wrote on Yesterday, 19:01:

jez wrote on Yesterday, 18:45:

IDA 5 can give you a decent call graph of a real mode DOS program, and you can do analysis of later Windows stuff. But am I right in thinking that extended mode DOS programs are virtually impossible?

Huh, while I don't have the capabilities of all IDA versions and editions at hand, I am very confident that at least commercial IDA in the 5.x range was perfectly capable of loading the protected mode part of DOS4GW software (IIRC that's typicall "LX", a "linear executable") just fine.

Not sure what's going on with mine then because it just seems to get lost at the bit where the far call (presumably to DOS4GW's loaded code) happens.

In fact, that makes sense. If DOS4GW's code is loaded into memory at runtime, IDA doesn't have it in the file being analyzed so how can it come up with anything meaningful?

Because the DOS-extended portion is encoded as a Linear Executable (LE) a standard executable format.
It's not as simple as the stub+LE portion concatenated together (as a PE executable is--it's a DOS program saying "This program cannot be run in DOS mode" plus the Windows executable, and at file offset 3C, a DWORD pointing to the start of the PE), but in any case, I was able to use SB /U from sb.exe here (https://github.com/amindlost/dos32a/tree/main/binw) and unbind the stub/extract the LE out of the DOS4GW-extended program I was using for testing.

This produced an LE file, which IDA's extensive file dumper (efd) can understand. Note that IDA free won't be able to understand LE files.

mkarcher wrote on Yesterday, 19:01:

This produced an LE file, which IDA's extensive file dumper (efd) can understand. Note that IDA free won't be able to understand LE files.

You sure about that? IDA 5's open dialog seems to have an option for MZ/LE files.

I see; for retrocomputing stuff I've been using IDA Free 7.0 linux version which has everything but PE support removed. Maybe it will work in 5, but I remember trying it on VxDs (another case of LE files) and couldn't get it to work. Ghidra doesn't do well on those either unfortunately.

Well I diid manage to unbind the program and open it in IDA 5, though it recognized almost no subs and most of the code is just displayed as literal bytes.