jal wrote on 2024-07-01, 09:41:

GloriousCow wrote on 2024-06-26, 18:36:

I can't ever concentrate on one thing for very long, so I started working on VGA.

Ouch, this is so relatable 😁. Nevertheless, great work as always!

Just to give you an idea of how bad the problem is I stopped in the middle of my unfinished VGA and did Adlib via nuked-opl3. I really wanted to hear the music in Wolf3D now that it's running!

I made a crate/library out of it so that anyone else making an emulator in Rust someday can use it.
https://crates.io/crates/opl3-rs

Along with Adlib support comes a new, threaded sound system that can receive input from multiple sound devices.

Oh, wow, Nuked OPL3 wrapped in a Rust crate. That will surely be useful for other projects as well. Thanks for sharing this stuff with the world! 🙂

Here's something a few people stated they wanted in a PC emulator - a memory visualizer.
You can view the contents of memory starting at the specified address as 1bpp, 2bpp, 4bpp or 8bpp graphics. You can drag the dimensions to smoothly change the aperture of the bitmap used to interpret the data for any bitmap span from 4-1024. (Would we need any wider??)

Here we found Prince of Persia's back buffer in memory. Kinda cool.

Oh, and it's updated in realtime. You can always poke around with the emulator paused, of course.

I have implemented directory mounting. Only for floppies, initially, but the basic technique should be extendable to DOS partitions on hard disks as well.

This is not quite a live filesystem like DosBox provides, but it's very quick to rebuild the image when you make changes. I have left syncing the directory a manual trigger at the moment, as there are some concerns about filesystem corruption I need to iron out before the image can be automatically remounted when the host filesystem changes. It will probably be possible to crash the emulated system or at least see a bunch of garbage, by changing the host filesystem if you have an open file handle. I still consider this feature useful enough to offer despite the potential caveats.

If a 'bootsector.bin' file is present in the mounted directory, that will be installed as the resulting disk image's boot sector. MartyPC will extend this file to 512 bytes if it is not and add the boot marker - this is pretty handy for testing code, as you can assemble a COM file right into a boot sector and immediately boot from it.

In addition, MartyPC can recognize DOS kernels from MSDOS, PCDOS and FreeDOS and ensure they are installed first.
This enables booting directly from a host directory:

The FAT filesystem is created recursively, so subdirectories are also supported.

This makes getting files in and out of MartyPC much more convenient, and it doesn't require any drivers or TSRs.

Sounds cool, gotta try this when I have some time 😀.

One of the biggest shortcomings of MartyPC to date has been lack of proper low level floppy emulation. I am aiming to rectify that. There is a lot of work to do, but I am moving MartyPC to an internal bitstream representation. I think this is sufficient for most PC floppy protections, as they can't get as crazy as Apple II or Commodore FDC shenanigans.

This means a whole new array of disk image support! So far, I have added support for HFE, MFM, PRI, PSI, IMD, with TC0 in progress.

I couldn't resist making adding a routine to visualize a disk, inspired by HxC:

Fancy.

Hey there, just I came across MartyPC is because I was hoping to re-discover and introduce my kids to a lot of classic PC games I played way, way back in the day. I've never actually created an account on this forum until recently, but I go by "Nemesis" in other circles. I'm behind the (rather neglected) Exodus emulator (https://www.exodusemulator.com) for the Mega Drive. I've just learned about MartyPC, and I wanted to reach out to say you've done an absolutely amazing job on it. My emulator had a very similar philosophy - aiming for cycle-accurate emulation, with a focus on extensive debugging features, and like you I did a LOT of testing on the hardware back in the day. The demands of life (work, emarriage, raising four kids, life stuff) has left me with little time to work on projects for the last... decade or so. Things are getting easier as my kids get older though, and I'm starting to ease back into things.

I'm at a crossroads of sorts. I'm trying to decide whether I resurrect Exodus, make it fully properly cycle accurate like it was always intended to be, but never achieved, and carry it forward to its logical conclusion (MegaCD, 32x, maybe onto the Saturn, Arcade hardware, etc), or whether I leave it for longer and tackle new challenges, possibly smaller ones as I ease back into things.

One thing I'm currently working on is a 128+ channel "logic analyzer"/"data capture" device, based on the Cypress FX3 paired with an FPGA, to do streaming capture of large parallel buses to PC, to serve as "reference traces" from the original hardware. My plan was similar to what I see you've already done - take hardware verification to the next level by comparing emulated bus activity against physical bus traces, to flush out timing errors and prove perfect timing accuracy, and ultimately regression test to keep it there. As you've encountered, you can either get small snippets of large parallel bus activity (I have an old 128 channel logic analyzer on my bench for this purpose), or endless streams of a smaller number of channels through devices like Saleae clones, but nothing today seems to be out there to capture a large number of channels in an endless stream to a multi-GB file sift through on a computer later. I do intend to complete that work, so that might be of interest to you when it's done.

I was wondering though, want to collaborate a bit? I notice you don't seem to have savestate support in MartyPC. I built Exodus to support this, and I'm well aware of the challenges in making it happen in complex systems that aim for perfect timing accuracy. I've had a bit of a poke through the code, and I notice you seem to have reads and writes on your bus as complete indivisible steps. Everyone does that, and I did the same in Exodus originally. I had a major redesign of the bus system for Exodus planned around separating reads and writes into lower level component steps (IE, don't perform a "read", just assert your various lines/strobes, execute the requisite number of wait states based on internal state/external signals, then latch the current contents of the data bus at the appropriate time), so that you can insert unpredictable wait states in there for example, handle perfect timing for faults/exceptions during bus operations, or step through a processor bus operation a cycle at a time, and even savestate the entire system at that point, without having threads being blocked or anything like that. Is this a problem in the 8088 architecture, such that it makes savestates hard right now? This is something Byuu/Near had a lot of problems with back in the day. I vaguely recall we had some big discussions around this kind of stuff over 15 years ago now, probably still around on SpritesMind somewhere. Happy to talk shop about this kind of stuff anytime. Might even tinker and send a pull request your way for an enhancement or two if I find myself wanting a feature, it'll give me an excuse to cut some more code in Rust. I would have written Exodus in Rust if I was starting it today.

Anyway, wanted to say again, well done on this amazing achievement! I hope to see this project continue to advance. A 486DX2 with VGA graphics, a Gravis Ultrasound paired with a Sound Blaster Pro, and a couple of IDE HDDs and an ATAPI CDROM will be supported any day now, right? 😉

DivByZero wrote on 2024-08-14, 04:25:

Hey there, just I came across MartyPC is because I was hoping to re-discover and introduce my kids to a lot of classic PC games I played way, way back in the day. I've never actually created an account on this forum until recently, but I go by "Nemesis" in other circles. I'm behind the (rather neglected) Exodus emulator (https://www.exodusemulator.com) for the Mega Drive. I've just learned about MartyPC, and I wanted to reach out to say you've done an absolutely amazing job on it. My emulator had a very similar philosophy - aiming for cycle-accurate emulation, with a focus on extensive debugging features, and like you I did a LOT of testing on the hardware back in the day. The demands of life (work, emarriage, raising four kids, life stuff) has left me with little time to work on projects for the last... decade or so. Things are getting easier as my kids get older though, and I'm starting to ease back into things.

Thanks for the kind words.

DivByZero wrote on 2024-08-14, 04:25:

One thing I'm currently working on is a 128+ channel "logic analyzer"/"data capture" device, based on the Cypress FX3 paired with an FPGA, to do streaming capture of large parallel buses to PC, to serve as "reference traces" from the original hardware. My plan was similar to what I see you've already done - take hardware verification to the next level by comparing emulated bus activity against physical bus traces, to flush out timing errors and prove perfect timing accuracy, and ultimately regression test to keep it there. As you've encountered, you can either get small snippets of large parallel bus activity (I have an old 128 channel logic analyzer on my bench for this purpose), or endless streams of a smaller number of channels through devices like Saleae clones, but nothing today seems to be out there to capture a large number of channels in an endless stream to a multi-GB file sift through on a computer later. I do intend to complete that work, so that might be of interest to you when it's done.

USB3 based logic analyzers seem quite happy to stream data to you forever, at least mine does. I have made several minute long captures in the GB range, although my analyzer only has 32 channels. The real bottleneck is not the capture but the decoding - I wanted to publish a full trace of the IBM 5150 going through POST, but my Pulseview decoder requires on the order of 5GB of memory *per second* to decode 8088 instructions. Even if you have enough memory, the user interface becomes so sluggish it is barely usable with a capture much bigger than 5 seconds or so. Perhaps this is a limitation of using Python for this sort of thing.

DivByZero wrote on 2024-08-14, 04:25:

I notice you don't seem to have savestate support in MartyPC. I built Exodus to support this, and I'm well aware of the challenges in making it happen in complex systems that aim for perfect timing accuracy.

I just really haven't gotten around to it. Rust actually makes this fairly straightforward with the Serde crate - you just derive Serialize and Deserialize for your structures and you can write them all to disk in some format and read them back in later. There are some lingering questions about how to handle hard drives - they could be quite big, but they are technically part of the "state." There's also just a lot of miscellaneous glue logic needed to tie it all together, handle versioning, compression, figure out from a user perspective where save states get stored, and how to associate them with a particular machine configuration. I am brainstorming moving to a more user-friendly model involving a 'VM' definition that is a specific instance of a machine configuration, this would live in its own directory, store its own VHD there, as well as nvram, etc. That would make storing save states in the VM directory straightforward. Right now media assets are all kind of pooled together.

DivByZero wrote on 2024-08-14, 04:25:

I've had a bit of a poke through the code, and I notice you seem to have reads and writes on your bus as complete indivisible steps. Everyone does that, and I did the same in Exodus originally. I had a major redesign of the bus system for Exodus planned around separating reads and writes into lower level component steps (IE, don't perform a "read", just assert your various lines/strobes, execute the requisite number of wait states based on internal state/external signals, then latch the current contents of the data bus at the appropriate time), so that you can insert unpredictable wait states in there for example, handle perfect timing for faults/exceptions during bus operations, or step through a processor bus operation a cycle at a time, and even savestate the entire system at that point, without having threads being blocked or anything like that.

You might want to have another look at the code, especially the logic in cycle.rs in the CPU core. Bus transfers are not indivisible; they are fully emulated with each individual T-state accounted for. The job of 'biu_bus_begin' is to start a bus transfer, but it does not finish it. A state machine in the cycle.rs advances from one T-state to another. Actual bus reads and writes happen only on a specific CPU cycle, namely either T3, or the last wait state I call TwLast. This corresponds to when the READY line is asserted after T2. If the target of a bus operation is a device that can manipulate the ready line, it is queried on T2 for the appropriate number of wait states. A bus read is simply what sets the data bus, when the data bus would be set in a real system. A bus write simply delivers the value of the data bus to some device. This only occurs after any wait states have been accounted for. The relevant pins of an 8088 in maximum mode are simulated on a per-cycle basis. This is all accurate enough that you can directly log CPU activity in the same format as a logic analyzer capture and load it in PulseView and use the same decoder on it that I use with hardware captures.

You cannot currently step the CPU one cycle at a time because instructions themselves are atomically executed. But you can have a fully cycle-accurate, instruction-stepped emulator. The proper way to do an 8088 that was cycle-stepped would be to directly emulate the microcode. I've been considering it.

DivByZero wrote on 2024-08-14, 04:25:

Might even tinker and send a pull request your way for an enhancement or two if I find myself wanting a feature, it'll give me an excuse to cut some more code in Rust. I would have written Exodus in Rust if I was starting it today.

I wouldn't reject a PR outright, but I would ask that you coordinate your ideas with me. There are things that I am very much looking forward to implementing myself, even if it takes time to get there. This is after all a deeply personal project. I do have a contributor who is working on PC-98 emulation, simply because that's far enough outside my intended scope and area of knowledge that they can do their own thing with that. The best way to reach me is probably Discord - i have the same username there.

DivByZero wrote on 2024-08-14, 04:25:

Anyway, wanted to say again, well done on this amazing achievement! I hope to see this project continue to advance. A 486DX2 with VGA graphics, a Gravis Ultrasound paired with a Sound Blaster Pro, and a couple of IDE HDDs and an ATAPI CDROM will be supported any day now, right? 😉

I know you're joking, but I am not sure really how far it makes sense to take MartyPC and where one should just simply use something like 86box. If I cannot make a cycle-accurate 386, then there's really no point. I think 286 is doable. So MartyPC might stop there. I suppose time will tell. I do want to make an emulator that runs DOOM (and not just one of the various real-mode ports). But that may be a separate project entirely, emulated at a higher level with a fixed hardware configuration.

Proper floppy emulation is a lot of work! I'll have to take a break soon to finalize my slides for VCFMW, but I've added support for Teledisk 2.x images (including advanced compression) and 86f for compatibility with images created by 86Box.

Now, just because I've added support for an image format doesn't mean copy protected images will all start working. That will be a long process of debugging each protection type.

Do copy-protected images even exist? I thought that the whole idea of these copy-protected disks (at least the ones for the PC) was adding some bad sectors that couldn't be read? So trying to extract it as image would always fail (or at least, would fail to copy the bad sector as bad).

jal wrote on 2024-08-27, 08:58:

Do copy-protected images even exist? I thought that the whole idea of these copy-protected disks (at least the ones for the PC) was adding some bad sectors that couldn't be read? So trying to extract it as image would always fail (or at least, would fail to copy the bad sector as bad).

Absolutely - there were more advanced capture formats back in the day, such as TeleDisk and ImageDisk that could store 'bad sectors'. But the modern method of backing up a floppy disk is taking a flux capture directly from the drive, using a device like a Kryoflux https://www.kryoflux.com/ or Greaseweazle https://github.com/keirf/greaseweazle?tab=readme-ov-file. A disk drive reports the magnetic flux transitions read from the disk. These devices control the drive and record those transitions for a few revolutions per track, and the results can be processed into a bit-level representation of the disk usable in an advanced emulator. Differences found in each revolution can be used for error-correction or detection of disk surface aberrations such as intentional damage, or weak/fuzzy bits. Properly encoded, an emulator can use this metadata to reproduce the behavior of a copy-protected disk.

I've got some more information about various disk formats on the MartyPC wiki now:

https://github.com/dbalsom/martypc/wiki/Flopp … ith-Disk-Images

Here's a blog post I wrote about emulating Copy-Lock copy protection:

https://martypc.blogspot.com/2024/08/pc-flopp … -formaster.html

Cool, thanks.

And now a mention on Hackaday 😀.

JAL

GloriousCow wrote on 2024-08-27, 16:24:

Absolutely - there were more advanced capture formats back in the day, such as TeleDisk and ImageDisk that could store 'bad sect […]
Show full quote

jal wrote on 2024-08-27, 08:58:

Do copy-protected images even exist? I thought that the whole idea of these copy-protected disks (at least the ones for the PC) was adding some bad sectors that couldn't be read? So trying to extract it as image would always fail (or at least, would fail to copy the bad sector as bad).

Absolutely - there were more advanced capture formats back in the day, such as TeleDisk and ImageDisk that could store 'bad sectors'. But the modern method of backing up a floppy disk is taking a flux capture directly from the drive, using a device like a Kryoflux https://www.kryoflux.com/ or Greaseweazle https://github.com/keirf/greaseweazle?tab=readme-ov-file. A disk drive reports the magnetic flux transitions read from the disk. These devices control the drive and record those transitions for a few revolutions per track, and the results can be processed into a bit-level representation of the disk usable in an advanced emulator. Differences found in each revolution can be used for error-correction or detection of disk surface aberrations such as intentional damage, or weak/fuzzy bits. Properly encoded, an emulator can use this metadata to reproduce the behavior of a copy-protected disk.

I've got some more information about various disk formats on the MartyPC wiki now:

https://github.com/dbalsom/martypc/wiki/Flopp … ith-Disk-Images

Here's a blog post I wrote about emulating Copy-Lock copy protection:

https://martypc.blogspot.com/2024/08/pc-flopp … -formaster.html

Hi,

Interresting the disk images support 😀

I wonder if it is really needed (If lot of image in other format exist than raw)

FreddyV wrote on 2024-08-28, 17:42:

Interresting the disk images support 😀

I wonder if it is really needed (If lot of image in other format exist than raw)

I'm not really sure what you mean.

A lot of original floppy preservation efforts ended up using Teledisk, ImageDisk, Transcopy and similar advanced tools and formats, so supporting (at least some of) them is a wonderful addition! Great job 😀

What I think is a decent overview over there: https://forum.winworldpc.com/discussion/7877/ … -archival-tools

Boohyaka wrote on 2024-08-28, 18:22:

A lot of original floppy preservation efforts ended up using Teledisk, ImageDisk, Transcopy and similar advanced tools and formats, so supporting (at least some of) them is a wonderful addition! Great job 😀

What I think is a decent overview over there: https://forum.winworldpc.com/discussion/7877/ … -archival-tools

About all i'm missing, besides flux format support, is Transcopy. I can't find any documentation on the format.

EDIT: Of course immediately after I post this I do another google search and see
https://www.robcraig.com/download/transcopy-5-x-file-format/

There's a few other obscure formats, like CP2, PDI, EPL(?) , DCF, etc, but I'm not sure if there are enough unique images in these formats to make support worthwhile.

GloriousCow wrote on 2024-08-28, 17:55:

FreddyV wrote on 2024-08-28, 17:42:

Interresting the disk images support 😀

I wonder if it is really needed (If lot of image in other format exist than raw)

I'm not really sure what you mean.

I mean is there enaugh disk image available in all the format so that it is worth adding support ?

I am also wondering the more important ones to add for the PicoMEM.

FreddyV wrote on 2024-08-28, 19:28:

I mean is there enaugh disk image available in all the format so that it is worth adding support ?

There is a significant body of images from both historical archival efforts as well as modern ones. It might be enough to support just the modern formats, but supporting the older ones may be helpful in the case where such images may be the only existing images of that particular software. It's true you can always convert some format to another, but such efforts can be tricky. I've seen posts on VOGONS with someone explaining a dozen different steps using the PSI tool, that even I barely understand. It would be nice to just be able to throw any format you might encounter at MartyPC and have it work without needing a doctorate in floppy engineering.