Keep also in mind that in FAT/FAT32 partitions the file clusters begin right after the FAT table(s), whose size is not a nice round cluster number.

konc wrote on 2020-02-03, 08:18:

Keep also in mind that in FAT/FAT32 partitions the file clusters begin right after the FAT table(s), whose size is not a nice round cluster number.

From mkfs.vfat (Linux) manpage:

1...
2OPTIONS
3       -a  Normally,  for any filesystem except very small ones, mkfs.fat will
4           align all the data structures to cluster size, to make sure that as
5           long  as  the  partition  is properly aligned, so will all the data
6           structures in the filesystem.  This option disables alignment; this
7           may  provide a handful of additional clusters of storage at the ex‐
8           pense of a significant performance degradation on RAIDs, flash  me‐
9           dia or large-sector hard disks.
10...

I was going to use mkfs.vfat because it let's you control every single aspect of filesystem creation... even to the extent of creating an unusable disk 😉 I wonder if other tools (i.e.: Rufus, or the standard windows formatting tool) also try to do this data alignment...

In this case, the option to choose cluster size is -s, although you can choose -S and choose sector size (to match it with your device? Has anyone seen a filesystem with 4K sector size?), but I guess that it would break compatibility with older OSs (which ones would support such a thing?).

1...
2       -s SECTORS-PER-CLUSTER
3           Specify the number of disk sectors per cluster.  Must be a power of
4           2, i.e. 1, 2, 4, 8, ... 128.
5
6       -S LOGICAL-SECTOR-SIZE
7           Specify the number of bytes per logical sector.  Must be a power of
8           2  and  greater  than  or equal to 512, i.e. 512, 1024, 2048, 4096,
9           8192, 16384, or 32768.  Values larger than 4096 are not  conforming
10           to the FAT file system specification and may not work everywhere.
11...

How to get data, FAT and partitions aligned is easy, but I'm still trying to find how to find the erase page size for USB pendrives...

I think you're well beyond my experience level...but I failed at doing this. I wanted to format my SSD's and SD cards (have IDE adapters for my older computers for these) with 4K partition Alignment and couldn't get it to work. Everybody says to just use a modern Linux system, but honestly, who has one of those? (ok, wrong question for this group perhaps, but I'm just being realistic for a laymen like myself 😉

So since I don't have access to a Linux system to format with, I tried things like GParted (which 3x different versions or GParted wouldn't boot on my 3x different P3 systems...) and other bootable CD solutions, and even software which is compatible in Windows7/8/10 (like EaseUS PartitionGuru I read from a tutorial), and although that could format to 4k, and even tested as such, Windows 98 would NOT recognize the drive no matter that I did. Had to use fdisk to delete the partition and reformat (which people say kills the 4k alignment).

I would LOVE to know of a tutorial which will work for the masses (i.e. people who don't have a Linux system to boot into...and furthermore lack the knowledge to use high-level command/terminal level commands to format)... everybody on this forum tells me this is a MUST when using SSD's...but when I ask other enthusiasts away from here they have absolutely no clue what I'm talking about. Tutorials are VERY few and far between on the web (especially for FAT32 formatting for Windows 98), I followed all the non-Linux ones I could find and still struck out...

Dochartaigh wrote on 2020-02-04, 19:07:

I wanted to format my SSD's and SD cards (have IDE adapters for my older computers for these) with 4K partition Alignment and couldn't get it to work.

Not sure if this of any help, but on Windows, there's SD Formatter.. It's meant to correctly format SD cards with FAT (FAT32).
The older version 4 had several useful features, also (secure erase, size correction etc).
https://www.retro-programming.de/download/sd-formatter-4-0/

Zup wrote on 2020-02-02, 22:04:

And how can I see the erase page size of a pendrive?

You can try flashbench: https://github.com/bradfa/flashbench
I'm not sure there's much you can do to optimise FAT for solid-state storage given that it was originally designed for a floppy disk. The most you can do is align the partition itself. I'd leave the sector size at 512B, there's a reason newer "Advanced Format" hard drives continued to expose 512B sectors even though they used 4096B internally (many operating systems simply couldn't handle anything other than 512B sectors).

Dochartaigh wrote on 2020-02-04, 19:07:

So since I don't have access to a Linux system to format with, I tried things like GParted (which 3x different versions or GParted wouldn't boot on my 3x different P3 systems...) and other bootable CD solutions, and even software which is compatible in Windows7/8/10 (like EaseUS PartitionGuru I read from a tutorial), and although that could format to 4k, and even tested as such, Windows 98 would NOT recognize the drive no matter that I did. Had to use fdisk to delete the partition and reformat (which people say kills the 4k alignment).

I've got a Toshiba laptop with a P3 600 processor, and it has installed (along with DOS and Windows 98SE) a Puppy Linux 5. So I guess (provided you have enough RAM) that, if you're interested in trying Linux to format that drives, you can use it.

About the 4K alignment, and your troubles... well, there are two ways to do this:
- Using 4K sectors. Almost every modern disk use a sector size of 4096 bytes (instead of 512 bytes), and that's a feature only supported in Windows XP or later. You can format a partition using 4K sectors, but probably Windows 98 won't be able to use it.
- Using 4K clusters. A cluster is the minimal size that an OS can write to a filesystem and it's composed by a number of sectors. Most disks that use 4096 bytes sectors can still be accessed in a "legacy mode" where the OS uses 512 bytes sectors and the disk does the translation (so it can work with non-4K aware OSs like Windows 98). In that case, for maximum performance, you should align the start of your partition on a 4K boundary (usually, modern partition tools align it with 1M), and use a filesystem with 4K cluster (may be referenced as 8 sectors/cluster) or a multiple of that.

If you use 512 bytes/sector, Windows 98 wouldn't have problems dealing with that partition (keep on mind that Windows 98 can have a cluster size limit, but I don't know how much it is).

jtchip wrote on 2020-02-04, 23:33:

You can try flashbench: https://github.com/bradfa/flashbench
I'm not sure there's much you can do to optimise FAT for solid-state storage given that it was originally designed for a floppy disk. The most you can do is align the partition itself. I'd leave the sector size at 512B, there's a reason newer "Advanced Format" hard drives continued to expose 512B sectors even though they used 4096B internally (many operating systems simply couldn't handle anything other than 512B sectors).

Yup, I'll take a look... I've seen some programs that can identify the memory chip on an USB, but fails on Sandisk pendrives...

Partitions don't have "sector sizes". It's a physical aspect of the disk which cannot be changed and there is no "legacy mode" which lets you select which size a drive uses.
A drive has either 4KB sectors or 512 byte sectors, nothing to do with how it is partitioned will change that.

I guess someone has to differ between physical and logical sector sizes.
Ideally, both overlap each others seamlessly.

Some really old DOSes, likely OEM versions, used bigger sector sizes to overcome the ~30MB limit of original FAT16 (before FAT16B existed).
This was in the times before MS-/PC-DOS 4.x, I recall. The ancient fixed disks they ran on were using the usual 512 bytes per sector, though.

Also, I believe some early flash media used other sector sizes, too, like 512B, 1KiB and 2KiB.#
Unfortunatelly, I can't provide examples right now. The internals of flash cards are usually "secret sauce". 🙁

Speaking of 4k (4KiB/4KB) sectors.. On Windows, that's what both NTFS and the Virtual Memory (-> 4K Pages) uses..
On the retro side, even Windows 3.x did translate the 64K segments -that Win16 programs worked with- into 4K chunks when run in Enhanced Mode.

This means that modern AF drives match perfectly into the sheme, also.
With AF-aware (advanced format) OSes, like Win 8.1 and higher, memory performance could be really awesome.

Anyway, what you meant to say is right and this was no offense. 😀

Jo22 wrote on 2020-02-05, 13:52:

I guess someone has to differ between physical and logical sector sizes.
Ideally, both overlap each others seamlessly.

Yup, I missed the "logical" bit.

In any case, you can take two approach to partition alignment: using clusters multiple of 4K (in HDDs that uses 4K sectors, in flash media it should be a multiple of erase page size) or using 4K logical sectors in FAT filesystems. I haven't fiddled too much with filesystem headers, so were not aware that you could define a logical sector size... so maybe I've used a filesystem with other logical sector size but I didn't know that.

Jo22 wrote on 2020-02-05, 13:52:

Speaking of 4k (4KiB/4KB) sectors.. On Windows, that's what both NTFS and the Virtual Memory (-> 4K Pages) uses..
On the retro side, even Windows 3.x did translate the 64K segments -that Win16 programs worked with- into 4K chunks when run in Enhanced Mode.

Not a Windows thing, but a 80386 thing. The MMU is composed of various units, and the paging unit divides memory on 4K pages (this is independent of segmentation unit, that can use segments up to 4Gb). In later processors, you could use bigger memory pages... but every OS that use virtual memory must do it in 4K chunks.

BTW, I've found in some russian site a tool (ChipGenius) that examines USB pendrives and (hopefully) tell which controller and memory are used. Looking into the datasheet of that memory you can find the page erase size... but (being paranoid) I don't know if the controller uses the same size or another size (i.e.: the memory has a page size of 8K, but the controller uses page sizes of 16K).

Having just written a FAT library, I've spent quite a bit of time deeply involved in the layout of that particular file system. 😁

So, about alignment: There are a few data regions on the disk, and there's a strict formula for defining their starting location and length. You CANNOT alter that and still be compatible with FAT drivers. Many of those regions don't even have pointers in the headers anywhere. Their location is known relative to other aspects of the layout.

Here's how it looks on-disk:

Sector 0 is the volume boot record (VBR), or "boot sector" region. This contains the 0x55 0xAA boot signature used by the MBR boot loader to detect a bootable partition, some descriptive fields, and the BIOS Parameter Block (BPB), which defines the characteristics of the underlying disk and the file sytem as originally created upon that disk. (Any space not allocated to the above can be used for stashing additional boot loader code.)

Sector 0 is part of what is known as the "reserved sectors." With FAT12/16, there is usually only the one reserved sector -- #0. With FAT32, there are customarily 32 reserved sectors, including sector 0. However, the BPB includes a field where the number of reserved sectors is defined, so you CAN tweak this. AFAIK, it was originally designed to allow you to carve out additional space for boot loader code, but there's no reason you can't use it to force alignment goals.

FAT32 introduces some new structures -- a backup boot sector in case the original is unreadable, and the "FS Info" structure, which caches the number of available clusters (and the first available cluster number) for faster calculation of free space, and allocation of new data. These use sector 1 (FS Info), sector 6 (backup boot sector), and sector 7 (backup FS Info) traditionally. It isn't advisable to change these, but the backup sector and the FS Info sector offset can be moved (their block numbers are in the BPB). The rest of the reserved sectors are left blank.

After the reserved sectors is the FAT itself. The space consumed by this is entirely dependent on the number of clusters in the volume. FAT12 uses 1.5 bytes per cluster. FAT16 uses 2 bytes per cluster. FAT32 uses 4 bytes per cluster. Num_Clusters (+2 reserved clusters) * Entry_Size = FAT_Size. This is rounded up to an integer number of sectors. There are usually 2 copies of the FAT, but that is configurable in the BPB as well. The 2nd FAT is located at (0 + Reserved_Sectors + FAT_Sectors) sectors.

For FAT12/16, the next region is the Root Directory. It follows on the very next sector after the FATs. The number of entries is chosen when the file system is created, and each entry takes 32 bytes. Ergo, the size of the root directory is ((Num_Entries * 32) / Sector_Size). This is again rounded up to full sectors.

FAT32 does not use a dedicated region for the root directory, it's just allocated in the data region like any other directory. The BPB field that defines the number of root entries will be set to 0.

Finally, the data region. It starts at Cluster #2 and goes all the way up to ((Sectors_Total - (Reserved_Sectors + Root_Sectors + (FAT_Sectors * FATs))) / Sectors_Per_Cluster), rounded down to the nearest integer number of clusters.

FAT12 can contain 1 to 4084 clusters. FAT16 can contain 4085 to 65,524 clusters. FAT32 can contain 65,525 to 268,425,446 clusters. This is actually how you determine which FAT to use -- the number of clusters. It's not set explicitly anywhere in the BPB or any other place. (Although there is a text field in the BPB that says "FAT16" or whatever. It is not meant to be used authoritatively, but some tools -- like apparently sys.com -- will fail to work if it's not set correctly.)

Now, if you're trying to assure that the data region starts on a cluster boundary, you have to make sure that the reserved sectors, FATs, and root directory (if not FAT32) all take up exactly X cluster(s). This can be difficult to do, as the size of the FAT tables depends on the number of clusters. So, as you change the number of sectors available to be used as data clusters, it changes the size of the FATs, which changes the number of sectors available to be used as data clusters, and so on. I wrote a routine that does this iteratively to find the most efficient layout given a certain number of total sectors and the cluster size you want to use, but it does not attempt to align the data region by manipulating the number of reserved sectors. (It could, though.)

SirNickity wrote on 2020-02-17, 20:35:

FAT12 can contain 1 to 4084 clusters. FAT16 can contain 4085 to 65,524 clusters. FAT32 can contain 65,525 to 268,425,446 clusters. This is actually how you determine which FAT to use -- the number of clusters. It's not set explicitly anywhere in the BPB or any other place.

The 16-bit sector count (offset 0x13) will always be 0 for FAT32 volumes and non-zero for FAT12/16.

I did a manual edit on 4K/512e disks to get an aligned partition scheme for my DOS compatible systems.

The 2 M alignment in partition utility since Windows Vista ignores CHS alignment.
DOS fdisk ignores 4 K alignment.

In DOS, partition track starts at cylinder 0, head 0, sector 1 and has 63 sectors.
The first partition starts at cylinder 0, head 1, sector 1 - that is block 63 and not 4 K aligned.
So, you have to manually set the start of the partition to cylinder 0, head 8, sector 1 - that is block 504 and both CHS and 4 K aligned.

In the FAT itself you can adjust the root directory size to get a matching alignment.

Here is a new DOS tool YSDDT able to align FAT16 and FAT32 partitions and disks:
http://ysbits.net/ysddt/