If the sector size field is different it will report sector not found since the sector size is part of the address. If the sector size field is correct then it'll keep reading all the requested data even if it passes though the gap into the next sector (some copy protection exploits this) and it'll report success or possibly a crc error.

crazyc wrote on 2024-06-16, 19:47:

If the sector size field is different it will report sector not found since the sector size is part of the address. If the sector size field is correct then it'll keep reading all the requested data even if it passes though the gap into the next sector (some copy protection exploits this) and it'll report success or possibly a crc error.

OK. So that's correct behaviour.
What about read/write sector result phase's sector size result?
For read/write I simply report the byte from the command input. ST1 is set to 01h and ST2 to 05h.
The read ID command otoh tries to approximate from the read sector size (same ST1&ST2 failure if not found) the closest matching size (the biggest fitting match for non-standard sizes, like say 513 byte sector becomes reported as 512 and limiting to closest possible (<128 reports 255(unknown, should this be 0 instead?), >max(128x2^255) reports max), all rounded down to nearest match).
Although if non-standard read/write sector commands, for those don't match exactly, it'll fail with the specified ST1(01h) and ST2(05h) reported.
How are the 1-255 byte ones detected? Also documentation says that with 128 byte selected(128x2^0), the actual size is specified using the final field(specifying 0-255 bytes) for byte #8(data length)? What is sector size=0 and data length=0? UniPCemu really tries a 0 byte transfer in that case (which will obviously time out).

What about read/write sector result phase's sector size result?

Why would this be different? The actual sector size has to be the same as the command size. Read track is different though. It reads the command size no matter what the size field is in the address so it can be used to overread into the gaps (again there's copy protection the exploits this).

How are the 1-255 byte ones detected?

It'll have 0 in the size field and it's up to the software to set dtl properly. That said out of the dozen or so different machines I've worked on that have 765 fdcs I have never seen the dtl field used, even for copy protection.

superfury wrote on 2024-06-16, 23:38:

UniPCemu really tries a 0 byte transfer in that case (which will obviously time out).

That's not obvious to me. As I understand it, a 0-byte transfer should be finished immediately, and not wait for any data byte. A sector with size code 0 and DTL 0 shoud just consist of the CRC16 of the data address mark (which is included in the sector CRC, IIRC).

OK. Just implemented some things:
- Transferring 0 byte sectors actually just uses a timed 'read' from the disk (like READ ID and VERIFY). It's counted as a non-data transfer on the CPU side, like READ ID and VERIFY. Internally it's just a timed like those, executing an empty sector being read (like the sector has been fully read from disk, but within hardware timing itself instead of the CPU reading n bytes using DMA or PIO). If data IS read using DMA/PIO during that, it will perform an abnormal polling error instead.
- Fixed transferring less than a sector's worth of data to complete for reads. If more is requested to read than is available, it will fail the read with a sector ID not found. Less will succeed, with the sector ID as possible (rounded down now) reporting the size requested.
- Read ID now reports a sector size of 00h if it's less than 128 bytes in size (software itself will need to figure out this if that's the case by performing dummy reads until it can read), instead of the previous FFh.
- Writes with non-matching sector size fail always (sector ID not found). The FDC documentation doesn't say anything about writing less than a sector somehow?
- Floppy disk formatting performs more strict checks on sector size now (forcing the sector sizes to match, except when the IMD disk format is used (which allows individual sector sizes to be modified)).

superfury wrote on 2024-06-18, 18:14:

- Transferring 0 byte sectors actually just uses a timed 'read' from the disk (like READ ID and VERIFY). It's counted as a non-data transfer on the CPU side, like READ ID and VERIFY. Internally it's just a timed like those, executing an empty sector being read (like the sector has been fully read from disk, but within hardware timing itself instead of the CPU reading n bytes using DMA or PIO). If data IS read using DMA/PIO during that, it will perform an abnormal polling error instead.

Trying to transfer a zero-byte sector should only work if the specified sector (including a size code byte of zero) is actually present. So trying to read a zero-byte sector is going to fail ("Sector ID not found") on sectors with a different size code (i.e. 256 bytes or a bigger power of two), or if the sector number exceeds the sectors per track. Furthermore (looking at how real floppy works), it is supposed to yield a CRC error unless the first two bytes following the sector header contain a valid CRC. You might want to ignore the CRC issue in your emulator right now, but you should issue a "Sector ID not found" error if your image has sector sizes exceeding 255 bytes.

superfury wrote on 2024-06-18, 18:14:

- Fixed transferring less than a sector's worth of data to complete for reads. If more is requested to read than is available, it will fail the read with a sector ID not found. Less will succeed, with the sector ID as possible (rounded down now) reporting the size requested.

Transferring more than a sector's worth of data is supposed to work (that is: Not asserting TC (DMA terminal count) after one sector has been transferred. The FDC will happily read as many sectors as required to fulfill the read by just incrementing the sector number, leaving the cylinder number and head number alone. This mode is limited by the sectors per track value that is part of the read command. There is a multi-track mode that should allow the controller to continue reading even after the end of one track, by continuing to the next head(?) or even stepping to the next cylinder(?), but they say this mode doesn't work for some technical reasons in IBM-PC-type systems, so no PC software uses this mode.

As the sector size is part of the ID, issuing a read command with a size code of 1 (256 bytes) will return "sector ID not found" on a standard PC formatted floppy disk consisting of 512-byte sectors. The 765 FDC does not support non-power-of-two sector sizes above 255 bytes, so there should be no need to round anything.

superfury wrote on 2024-06-18, 18:14:

- Writes with non-matching sector size fail always (sector ID not found). The FDC documentation doesn't say anything about writing less than a sector somehow?

Failing writes with non-matching size is correct. Reads need to fail, too, as already stated in this reply. If the sector size code is 0 (0..255 bytes), the write command contains the actual size to write, and let's just hope the space on the disk till the next sector header is sufficient so writing the sector data does not overwrite the next sector header. If a disk has been formatted at 128 bytes per sector, writing 255 bytes will be accepted by the FDC (as the size code is still 0), but writing the extra 127 bytes is likely to overwrite the subsequent sector header and mess up the medium format. As sectors have a CRC appended, you can't update the start of a sector and leave the remaining part of the sector alone, so supplying partial sectors (premature TC) to a write will cause an unreadable sector or the FDC is going to pad up to the required sector size, and append the correct CRC.

mkarcher wrote on 2024-06-18, 21:51:

Trying to transfer a zero-byte sector should only work if the specified sector (including a size code byte of zero) is actually […]
Show full quote

superfury wrote on 2024-06-18, 18:14:

- Transferring 0 byte sectors actually just uses a timed 'read' from the disk (like READ ID and VERIFY). It's counted as a non-data transfer on the CPU side, like READ ID and VERIFY. Internally it's just a timed like those, executing an empty sector being read (like the sector has been fully read from disk, but within hardware timing itself instead of the CPU reading n bytes using DMA or PIO). If data IS read using DMA/PIO during that, it will perform an abnormal polling error instead.

Trying to transfer a zero-byte sector should only work if the specified sector (including a size code byte of zero) is actually present. So trying to read a zero-byte sector is going to fail ("Sector ID not found") on sectors with a different size code (i.e. 256 bytes or a bigger power of two), or if the sector number exceeds the sectors per track. Furthermore (looking at how real floppy works), it is supposed to yield a CRC error unless the first two bytes following the sector header contain a valid CRC. You might want to ignore the CRC issue in your emulator right now, but you should issue a "Sector ID not found" error if your image has sector sizes exceeding 255 bytes.

superfury wrote on 2024-06-18, 18:14:

- Fixed transferring less than a sector's worth of data to complete for reads. If more is requested to read than is available, it will fail the read with a sector ID not found. Less will succeed, with the sector ID as possible (rounded down now) reporting the size requested.

Transferring more than a sector's worth of data is supposed to work (that is: Not asserting TC (DMA terminal count) after one sector has been transferred. The FDC will happily read as many sectors as required to fulfill the read by just incrementing the sector number, leaving the cylinder number and head number alone. This mode is limited by the sectors per track value that is part of the read command. There is a multi-track mode that should allow the controller to continue reading even after the end of one track, by continuing to the next head(?) or even stepping to the next cylinder(?), but they say this mode doesn't work for some technical reasons in IBM-PC-type systems, so no PC software uses this mode.

As the sector size is part of the ID, issuing a read command with a size code of 1 (256 bytes) will return "sector ID not found" on a standard PC formatted floppy disk consisting of 512-byte sectors. The 765 FDC does not support non-power-of-two sector sizes above 255 bytes, so there should be no need to round anything.

superfury wrote on 2024-06-18, 18:14:

- Writes with non-matching sector size fail always (sector ID not found). The FDC documentation doesn't say anything about writing less than a sector somehow?

Failing writes with non-matching size is correct. Reads need to fail, too, as already stated in this reply. If the sector size code is 0 (0..255 bytes), the write command contains the actual size to write, and let's just hope the space on the disk till the next sector header is sufficient so writing the sector data does not overwrite the next sector header. If a disk has been formatted at 128 bytes per sector, writing 255 bytes will be accepted by the FDC (as the size code is still 0), but writing the extra 127 bytes is likely to overwrite the subsequent sector header and mess up the medium format. As sectors have a CRC appended, you can't update the start of a sector and leave the remaining part of the sector alone, so supplying partial sectors (premature TC) to a write will cause an unreadable sector or the FDC is going to pad up to the required sector size, and append the correct CRC.

The rounding might still be needed though. The backend file formats (dsk, imd, fdi) might support arbitrary sector sizes (imd supporting anywhere from 0 through 64K in byte granularity).

With the latest changes, all sector size fields (both on disk and from commands(or format track data)) are translated to 16-bit byte granularity length.
During reads a comparison is made to verify if it's valid to read (after CHS passes for a sector). The same during writes.

That comparison is simply converting to 16-bit length and if requested<=present it will:
- For reads read the 'present' size and transfer 'requested' bytes.
- For writes read the 'present' size, modify 'requested' bytes and writeback 'present' size (so basically a RMW of 'present' size data from the start of the sector.
- For formats enforce exactly the same 'present' and 'requested', except for IMD disk images (which support fully reformatting tracks with different sector IDs and sizes), taken the sector ID and size from the data phase only (filler from parameter phase though, size field unused).

The 0-case translation from 8-bit 'request' is done the same for read data/track and write data command right now, before the sector IDs are read and parsed into valid sectors depending on backend (IMA(and FDI as a subfunction of that, due to same type but with an extra header to check special cased) vs DSK vs IMD).
After the sector ID (CHS) is found (ignored for read track command), it will check for the 'request' size vs 'present' size.
If 'request'>'present', fail the sector ID and continue to the next sector (spinning emulation on non-IMA/FDI disk images) or (in the case of spinning until reaching the index hole twice while scanning) abort (error out with sector not found (ST1=05h, ST2=01h, ST0=01xxxxxxb).
If 'request'<='present', it'll transfer 'request' bytes out of 'present' bytes (as NEC upd765 and 82077AA document), then:
- For read commands, next sector or result phase.
- For write commands, perform RMW on the sector, modifying 'request' out of 'present' bytes, then next sector or result phase.

So what is it? Modify(write using RMW) or read part (for reads) or force exact match for reads and/or writes only? Documentation says read part always, but write (on 82077AA documentation, NEC upd765 documents roughly 'same as read track') doesn't say anything other than 'C/H/R/N' match (indicating forcing exact match?)

OK. Just checked the 82077AA manual:
About the sector size, it says:
- N determines the number of bytes per sector.
- If N is set to zero, the sector size is 128. The DTL determines the number of bytes to be transferred.
- If DTL is less than 128, the 82077AA transfers the specified number of bytes to the host.
- For reads, it continues to read the entire 128 byte sector and checks for CRC errors.
- For writes, it completes the 128 byte sector by filling in zeroes.
- If N is not set to 00 Hex, DTL, should be set to FF Hex, and has no impact on the number of bytes transferred.

So basically:
- Always exact match on sector size based on N only(ignoring DTL), with N=0 always using 128 byte reads/writes from medium(even with DTL>128).
- 128 byte sectors use the DTL to limit transfers if transferring only? (What if DTL is above 128?) Write data pads up to 128 bytes from byte #DTL (0-based) onwards with zeroes?
So non-power of 2 sectors aren't detected at all?

I was under the impression that the controller didn't give a crap about sector sizes unless using hard sectored disks, where everything in the PC world since IBM 5150 is soft sectored.

Just adjusted the sector size handling as follows:
- Size #0 (128 bytes) reads/writes 128 byte sectors only, otherwise specified power of 2 only.
- Size #0 with non-128 bytes reads/writes 128 byte sectors, with DTL specifying how many to pad till 128 bytes (so up to sector size is padded with zeroes when written, and not returned when read).
- Mismatch on 128 bytes or 256 bytes and up against on-disk sector size fails the read/write.

superfury wrote on 2024-06-16, 11:52:

What does a FDC report in it's ST registers if the sector size read doesn't match the sector size requested?
Like 1024 bytes or 128 bytes sector disks being read using a 512 byte sector read sector command?

It's been years ... so won't go into details I don't remember, I *think* there was a way to scan what sectors
are available, or maybe I had to try for everything the PC''s 765 FC could read

I wrote ImageDisk to be able to read/write any floppy disk format the the PC's 765 FDC could physically handle.
It can read/write disks with any sector size supported by the FDC - including mixed sector sizes.

I did publish the source code - look on "Daves Old Comptuers", or the "Retirement project" section
of my personal site.

Dave ::: https://dunfield.themindfactory.com ::: "Daves Old Computers"->Personal

DaveDDS wrote on 2024-06-21, 15:11:

It's been years ... so won't go into details I don't remember, I *think* there was a way to scan what sectors are available, or […]
Show full quote

superfury wrote on 2024-06-16, 11:52:

What does a FDC report in it's ST registers if the sector size read doesn't match the sector size requested?
Like 1024 bytes or 128 bytes sector disks being read using a 512 byte sector read sector command?

It's been years ... so won't go into details I don't remember, I *think* there was a way to scan what sectors
are available, or maybe I had to try for everything the PC''s 765 FC could read

I wrote ImageDisk to be able to read/write any floppy disk format the the PC's 765 FDC could physically handle.
It can read/write disks with any sector size supported by the FDC - including mixed sector sizes.

I did publish the source code - look on "Daves Old Comptuers", or the "Retirement project" section
of my personal site.

Dave ::: https://dunfield.themindfactory.com ::: "Daves Old Computers"->Personal

Oh, you wrote the IMD file format? Nice.
Implemented the format for a while in my emulator now.
It's written to support any sector size in the module itself (as long as the on-disk and read/write sector sizes match). Also one thing that my emulator can do is create unformatted IMD images (all tracks unformatted). And of course format them however software likes (using the format track command's 128x2^n field in the 4-byte packets), unlike other software x86 emulators I know of.
Ofc it can only create 40 and 80 track unformatted images right now (other track counts I haven't heard of).
Formatting in this case is buffering all sector IDs and sizes to format first, then once the entire track is finished buffering (track length of the command) take the file contents before(previous tracks) and after (next tracks) and add the track data (using the compressed single byte form until it's uncompressed by writing sectors that aren't compressable anymore) followed by the next track(s) if any.

Do you have any idea what's the difference between the Sector Size field in the command and the Sector Size field in the data phase? Rn I'm only using the data phase one (every 4th byte) for sector size to add, but perhaps it's got something to do with the size=0 case(128 bytes officially)? If different sizes for sectors are supported, it seems useless to specify it in the command phase?

Didn't OS/2 update pack disks use a non standard sector/track number ?
Also the fdformat program comes to mind which allows floppies to be formatted in a non standard ie different from the normal 40/80 9/18 layout way.
https://github.com/christoh/fdformat

sorry for the sidetracking.

DaveDDS wrote on 2024-06-21, 15:11:

It's been years ... so won't go into details I don't remember, I *think* there was a way to scan what sectors
are available

There is. This is the "READ ID" command. The "READ ID" commands waits until a the controller reads something that looks like a valid sector ID, and returns the first ID found to the CPU. You have to repeatedly invoke this command to collect all the IDs on a track. If two ID marks are quite close to each other, it might be difficult to issue a "read ID" command at the correct point in time to catch the second ID header. In practice, this is not an issue, because there will be sector data between ID headers, providing enough separation. On the other hand, if you go crazy making creative copy-protected disks, you could try some setup in which there are fake spurious ID headers on the track.

superfury wrote on 2024-06-21, 15:54:

Do you have any idea what's the difference between the Sector Size field in the command and the Sector Size field in the data phase? Rn I'm only using the data phase one (every 4th byte) for sector size to add, but perhaps it's got something to do with the size=0 case(128 bytes officially)? If different sizes for sectors are supported, it seems useless to specify it in the command phase?

The sector size field in the command phase is used by the floppy controller to determine how many F6 bytes it is going to write after each sector ID. This size, combined with the GAP size that is also specified in the command phase specifies the physical distance between sector headers on the floppy. On the other hand, the sector sizes in the data phase are just what is written into the sector headers. If the sector size field in the command phase mismatches the sector size fields in the data phase, strange things will happen. You need to do stuff like this to format mixed-sector-size tracks (like OS/2 XDF or MS DMF, or 2M). Usually these formats are formatted by sending a format command with a size code of 0 (128 bytes) and a minimal gap, so a lot of sector headers are written. This includes some sector headers that are acutally required for that format, and a lot of "stub" sector headers. After formatting, you won't be able to read any sector, as there won't be any sector header that has its size code set to 0, but all data on the medium is 128 bytes sectors. You will get CRC errors on all sectors you try to read. The situation will change of course as soon as you write data using the write sector command specifying the size codes you sent during the data phase: This will write the intended large sectors (up to 4K on HD floppies, IIRC), and overwrite all the stub sector headers.

superfury wrote on 2024-06-21, 15:54:

Oh, you wrote the IMD file format? Nice.
Implemented the format for a while in my emulator now.
It's written to support any sector size in the module itself (as long as the on-disk and read/write sector sizes match). Also one thing that my emulator can do is create unformatted IMD images (all tracks unformatted). ..

IMD doesn't really have the concept of "unformatted" - you can create a file containing only "comment"<0x1A>
but that only makes a 0-track disk! (probably the closest you can get to unfornatted) - .IMD was designed to preserve as
much of "what is there" as possible - "nothing at all" was never a scenario I considered when designing the .IMD file).

There is no way to specify tracks exist without information about the format of that track.
You can use a sector mode value indicating the sector is "unavailable" (usually because of an unrecoverable read error),
but that could just get you to a formatted disk full of read errors!

Almost every system supporting emulation of diskette drives from files, supports raw binary images (just a series of
sector blocks) - IMDU can convert a .IMD into such a raw binary file, and BIN2IMD can gp th other way.

I've not seen much need to have .IMD itself supported in such environments (and re-writing compressed sectors can
make it "fun").

Dave ::: https://dunfield.themindfactory.com ::: "Daves Old Computers"->Personal

DaveDDS wrote on 2024-06-22, 04:25:

IMD doesn't really have the concept of "unformatted" - you can create a file containing only "comment"<0x1A> but that only makes […]
Show full quote

superfury wrote on 2024-06-21, 15:54:

Oh, you wrote the IMD file format? Nice.
Implemented the format for a while in my emulator now.
It's written to support any sector size in the module itself (as long as the on-disk and read/write sector sizes match). Also one thing that my emulator can do is create unformatted IMD images (all tracks unformatted). ..

IMD doesn't really have the concept of "unformatted" - you can create a file containing only "comment"<0x1A>
but that only makes a 0-track disk! (probably the closest you can get to unfornatted) - .IMD was designed to preserve as
much of "what is there" as possible - "nothing at all" was never a scenario I considered when designing the .IMD file).

There is no way to specify tracks exist without information about the format of that track.
You can use a sector mode value indicating the sector is "unavailable" (usually because of an unrecoverable read error),
but that could just get you to a formatted disk full of read errors!

Almost every system supporting emulation of diskette drives from files, supports raw binary images (just a series of
sector blocks) - IMDU can convert a .IMD into such a raw binary file, and BIN2IMD can gp th other way.

I've not seen much need to have .IMD itself supported in such environments (and re-writing compressed sectors can
make it "fun").

Dave ::: https://dunfield.themindfactory.com ::: "Daves Old Computers"->Personal

Well, as UniPCemu handles that, it's actually quite simple: just create the header and define all tracks (40 or 80) without ANY sectors in them.
Once it's been given a format track command, it will remove the track's sector data and replace it with compressed sectors as the command data specifies (including creating the track header and substructures).
Read/write sector commands return (using the sector's physical 'number' and track number) the nth sector on said track, the FDC then checks if it's usable or not (invalid, deleted filtering etc.). Once those match ((deleted) valid sector with matching CHS and size fields(only powers of 2 from 128 shl 0 and up)) it then checks the transfer size (0-255 or normal 256^(n-1)), reads the sector or writes (padding zeroes up to 128th byte for 0-255 bytes written if less than sector size) the sector according to documentation. For reads it then starts the data phase as requested, for writes data phase for next sector, or termination otherwise. Internally this is handles as a command phase execution (read or write, both scanning the disk as documented (using index hole mechanics, it's untimed though)), then data transfer phase, then the next phase is read or write data execution phase. The read data execution phase moves to the next sector and terminates or restarts a next read sector phase. The write sector execution phase does the same, but actually performs the same logic as the command phase execution, just writing the data to the disk instead of just checking if it's valid.
Edit: My bad, the CHS, invalid/deleted and sector size are checked in a different order:
1. CHS
2. Invalid/deleted (data mark), ignored bc of overwrite during writes.
3. Sector size
Write sector counts INVALID data marks as non-existant (same common logic as unmatching CHS), all others are counted as present and are overwritten as the command implies (using either a normal or deleted data mark, depending on write sectors or write deleted sectors command).
Sector sizes 128-255 bytes are currently ignoring all data bytes written past 128, writes ignoring them completely(forcing 128 byte sectors) after the data phase finishes and reads ...

superfury wrote on 2024-06-22, 09:10:

Write sector counts INVALID data marks as non-existant (same common logic as unmatching CHS)

That sounds wrong, as it seems to be impossible to implement that behaviour on actual hardware. The "write (deleted) data" command reads from the floppy drive until it sees the index mark that indicates the correct CHSS (CHS+size) and then switches over into write mode during the gap between the sector header and the data address mark and writes a new (deleted) data address mark immediately followed by the data. This is necessary because only when you write the data address mark and the data as one unit, you can make sure you don't lose synchronization in the MFM decoder while reading. This is comparable to writing CDs before we had buffer-underrun protection. In the CD case, the whole medium as to be written as "one unit", while on a floppy (and CDs in packet-writing mode), only a single physical block including some synchronization pattern needs to be written as one unit.

So the "correct" behaviour is: "write data" is able to convert deleted data into non-deleted data and the other way around, because the deletion is indicated in the data address mark that doesn't get read, but instead re-written every time a sector is written.

superfury wrote on 2024-06-22, 09:10:

Well, as UniPCemu handles that, it's actually quite simple: just create the header and define all tracks (40 or 80) without ANY sectors in them.

Ok, I interpreted your original as meaning you implemented unformatted in a .IMD
it's perfectly reasonable to implemented it anyway you want in the emulator.

Just curious though - your descriptions sounds like the number of tracks is pre-defined for an unformatted disk.
How does it handle things like 5.25"s with differing number of tracks (originally formatted in 40 track (360Kish)
drive, and later formatted in 80 track (720Kish) drive - IMD handles this by a "double step" option within IMD
itselfm as a .IMD knows only what tracks exist - not the track density? (I specifically avoided encoding anything
about "type of drive" - that can make it more difficult to retrieve the data if you don't have the drive it wamts -
eg: I can recreate working 8" disk images on a 80 track drive)

Dave ::: https://dunfield.themindfactory.com ::: "Daves Old Computers"->Personal