I think you should super clean the edge contacts with a pencil eraser and then IPA before doing anything else, looking at the pictures they do not look clean.....

Re-insert all socketed chips a few times before you do anything else. Especially the addon card above the RAM

Slight oxidisation on memory chips/conncetors is the number 1 causes of failure than anything else

The PCI card is a donor if some chips on ISA card are bad.

ISA card works as shown on the photos. Only BIOS chips are socketed. I already cleaned the ISA connector.

How might someone diagnose what to replace? Memory chips, DAC...

Checked for any loose pins on the chips and it seems all the pins have good connection.

About memory size. It's recognized only 1MB of VRAM. M64 diagnostics says 8x 256K VRAM - so it should be 2MB.

I have also put the 2MB memory upgrade card on it and it still recognizes only 1MB of VRAM. Probably bad memory chip(s).

Do you have the manual for it, for jumper settings?

Some cards do have an IRQ (interrupt request) selection in jumper settings. An interrupt error is mentioned in the screen shots.

Any BIOS IRQ settings? Any IRQ conflicts with other cards?

Even if there is a jumper with IRQ setting (I haven't seen it on any ISA Mach64 card tho). The more pressing problems aren't associated with it (memory size, image corruption, green tint).