NAND FAT file system management

The usual CompactFlash format is the MS-DOS style FAT. If your flash is supposed to be read and written by other devices, you'll need to stick with the standard file systems, even if they weren't designed with a concern for wear-levelling properties. If your flash is read only by your own software, then you have more freedom to choose other systems.

I've had to deal with three commercial flash file systems. They were all marketed for the same reason -- reliable persistent storage, with the emphasis on reliable, as for telecomm devices. These products are supposed to be robust against random loss of power, resets, and the like.

Not one of them really worked. It would be fairly routine for the code to announce the file system was corrupted after we pulled a card out of a shelf and plugged it back in. It's nice that the file system detects the fact that it's corrupted and reformats itself, but it still loses all your data. The whole point of paying for such a thing is for someone to have done the hard work of getting it right. Anybody can write file system code that corrupts data on loss of power.

If you try to write such a thing yourself, be warned that it's more difficult than it seems. You essentially have to consider every word in your data structur and every instruction in your module as to what will happen if hardware suddenly stops working at any point.

I looked briefly at the JFS code for Linux, but it was way too big and bulky for my needs.

NAND flash adds an extra layer of complexity, in that sectors of NAND have a much higher failure rate. NAND devices are usually treated much like hard disk drives, where there is more space than really needed in NAND chip, bad sectors are detected and remembered, and extra good sectors used in their place. A NAND flash more or less requires a full-blown file system to handle these bit failures during operation.

NOR devices are simpler to use and more reliable on a bit-by-bit basis, but less dense and thus more expensive. The largest sizes, like those 1GB CF cards, are generally not available in NOR technology.

Typical flash lifetimes (even for NOR flash) are 100,000 erase cycles, and it's not hard to find parts rated for 1,000,000 cycles. 100K erase cycles is 30 erases a day for 10 years.

Questions to ask yourself:

What are the needs of your application for update frequency? Are you really changing data that often?

Do you really need a file system, which is to say file names, a directory structure, variable sized files generally composed of lists of fixed size blocks, random access to small amounts of data at arbitrary locations inside the file, and all that? Or do you just need to store some data in flash?

Do you need redundant data storage in case of bit failure? Loss of power during an erase operation?

The usual CompactFlash format is the MS-DOS style FAT. If your flash is supposed to be read and written by other devices, you'll need to stick with the standard file systems, even if they weren't designed with a concern for wear-levelling properties. If your flash is read only by your own software, then you have more freedom to choose other systems.

I've had to deal with three commercial flash file systems. They were all marketed for the same reason -- reliable persistent storage, with the emphasis on reliable, as for telecomm devices. These products are supposed to be robust against random loss of power, resets, and the like.

Not one of them really worked. It would be fairly routine for the code to announce the file system was corrupted after we pulled a card out of a shelf and plugged it back in. It's nice that the file system detects the fact that it's corrupted and reformats itself, but it still loses all your data. The whole point of paying for such a thing is for someone to have done the hard work of getting it right. Anybody can write file system code that corrupts data on loss of power.

If you try to write such a thing yourself, be warned that it's more difficult than it seems. You essentially have to consider every word in your data structur and every instruction in your module as to what will happen if hardware suddenly stops working at any point.

I looked briefly at the JFS code for Linux, but it was way too big and bulky for my needs.

NAND flash adds an extra layer of complexity, in that sectors of NAND have a much higher failure rate. NAND devices are usually treated much like hard disk drives, where there is more space than really needed in NAND chip, bad sectors are detected and remembered, and extra good sectors used in their place. A NAND flash more or less requires a full-blown file system to handle these bit failures during operation.

NOR devices are simpler to use and more reliable on a bit-by-bit basis, but less dense and thus more expensive. The largest sizes, like those 1GB CF cards, are generally not available in NOR technology.

Typical flash lifetimes (even for NOR flash) are 100,000 erase cycles, and it's not hard to find parts rated for 1,000,000 cycles. 100K erase cycles is 30 erases a day for 10 years.

Questions to ask yourself:

What are the needs of your application for update frequency? Are you really changing data that often?

Do you really need a file system, which is to say file names, a directory structure, variable sized files generally composed of lists of fixed size blocks, random access to small amounts of data at arbitrary locations inside the file, and all that? Or do you just need to store some data in flash?

Do you need redundant data storage in case of bit failure? Loss of power during an erase operation?

Hi Drew,

Thanks for your reply, some very good points.

My device is a data logger updating at 4sec intervals, hence the wear concerns for flash. If I used MMC cards would the wear leveling be done by the MMC protocal?

All these card based digital camcorders must have the same problem repeatedly writing to the memory card so does that mean they have a tranlation layer?

Best regards,

Malcom

The "flash translation layer" is more or less the bottom half of the file system. They usually handle bad block replacement. They may or may not cope with wear levelling; some of them do. Unfortunately, I'm not intimately familiar with the MMC spec.

a data logger updating at 4sec intervals

One key point is that flash lifetime is limited by the number of erase cycles, not programming cycles. You can write to flash all you want; you just can't erase a sector more than 100K or 1M times. Erasing sets all bits to 1; programming changes 1s to 0s. You need not erase flash every time you issue the program command. And you can even program the same word of flash multiple times without erasing -- just so long as you never have to set a bit back to 1. (You might, for example, have some sort of used/unused bitmap that gets checked off every time you log a set of data.)

So, if you log, say, 16 bytes of data per sample, and your flash sector is 64K, you can store 4096 samples in one sector before having to erase it. If you have, say, 8 64k sectors of flash, you could just round-robin your way through it all, 32000 samples or roughly 36 hours of operation per erase cycle.

That observation might help your particular application, depending on the size of your samples.

As far as I know SecureDigital cards do wear leveling on their own internally. SD cards can be interfaced using SPI using the MMC compatibility layer.