BigFAT — Backward compatible FAT extension for unlimited file size
BigFAT is a free and open specification allowing storage of files larger than 4 GB on FAT-formatted drives, maintaining full FAT compatibility.
No patents, no fees.
Overview
FAT (short for File Allocation Table) is probably the most widely used file system. Introduced in 1977 by Microsoft for floppy disks, it has been enhanced multiple times, moving from FAT12 to FAT16, then FAT32. It has also been extended in other respects, such as adding support for long file names, while generally maintaining backward compatibility.
FAT is probably the most popular choice of format for removable media, such as SD cards, USB memory sticks as it is supported by all operating systems, and embedded devices. Unfortunately, FAT has one significant limitation: A storage medium formatted as FAT can only hold files smaller than 4 GB, which can be a limitation for some types of data, such as movies, databases or virtual HD containers.
SEGGER’s BigFAT is a specification defining how to store large files well beyond 1 TB on any FAT-formatted drive. Regular FAT systems see big files as multiple files, which they can read, write, and copy.
Key features
- Maintains full compatibility with FAT
- Supports all storage types, such as NAND, NOR, SD/MMC/CompactFlash cards
- Free utility to read and write big files from a PC
- Specifications fully public and open
- Not encumbered by any patents
- emFile component available to handle BigFAT on embedded systems
Why BigFAT?
FAT is a great file system for removable storage, such as USB memory sticks. It can be read and written by pretty much anything in this world today. There is only one problem: File size is limited to less than 4 GB.
BigFAT addresses this by defining how to split a file of 4 GB or more into chunks and how to name the chunks. Files smaller than 4 GB are stored the same way, so only big files are stored as multiples files.
The specification is fully open the to public. Anybody is welcome to add support for BigFAT to their software. No license fees or permissions of any kind are needed. Whoever wants to implement it or make something compatible is free to do so. SEGGER's BigFAT converter is a free command line program, available on Windows, macOS or Linux to anyone.
We hope to see widespread adoption as a new, open standard!
Why not exFAT?
exFAT was introduced by Microsoft in 2006. While it is claimed that it is "optimized for flash memory such as USB flash drives and SD cards" [1], one of the major reasons for introducing it was probably commercial: All patents on FAT have expired, meaning Microsoft no longer controls the technology and can no longer charge license fees for FAT. With exFAT however, the situation is different: Microsoft owns several patents, and anyone who implements or uses exFAT technology needs Microsoft's permission, which typically also includes paying fees to Microsoft.
Unfortunately, exFAT has been adopted by the SD Association as the default file system for SDXC cards larger than 32 GB. In our view, this should never have happened, as it forces anyone who wants to access SDXC cards to get a license from Microsoft, basically making this a field owned by Microsoft.
Fortunately, there is a solution: The SD or SDXC card can simply be reformatted with the good old, proven and free FAT file system. And with BigFAT, the 4 GB per file restriction is also lifted.
__________
[1] exFAT (2022-07-22) in Wikipedia, [online] https://en.wikipedia.org/wiki/ExFAT.
Specification
BigFAT is built on top of the FAT. It splits the data of a file of 4 GB or larger into smaller files (chunks) that can be handled by the underlying FAT file system. BigFAT directs file operations to the appropriate chunks, depending on the file position that is accessed.
Files smaller than 4 GB - 128 Kbytes are treated like normal files and the file access works like in a standard file system implementation. As soon as the file grows above the limit, the file gets extended. An extended file consists of one base file and one or more continuation files. The continuation files are stored in the same directory as the base file, with the name derived from the name of the base file plus an additional file extension. The name of the base file remains unchanged. The file extension of a continuation file has the format .<Index>.BigFAT where the Index is a three digit decimal number. The index of the first continuation file is set to one and increases by one with each additional continuation file. BigFAT requires support for long file names on the underlying file system layer in order to be able to apply the additional file extension of continuation files.
When the base file grows larger than 4 GB - 128 Kbytes, the first continuation file is created. This procedure is required in order to guarantee that the first byte in the continuation file is aligned to a logical sector boundary. This helps increase the performance of the file system accessing the file data. The same procedure is applied when the first continuation file exceeds the 4 GB - 128 Kbytes limit, with a second continuation file being created and so on.
BigFAT Converter
SEGGER's BigFAT Converter is a simple command line program, available for Windows, Linux and macOS. It can convert large files (≥ 4 GB) into chunks which are then recognized as a single big file by any BigFAT compatible system and vice versa, convert the small chunks into a single large file.
The command line utility is able to merge the fragment files generated on a target device into a single file larger than 4 GB. In addition, the command line utility is also able to split a file larger than 4 GB into fragment files that can later be copied to a target device in order to be accessed by a target application.
emFile BigFAT component
The emFile BigFAT component is an optional add-on of emFile that allows a target application to access files that are compliant with the SEGGER BigFAT specification. The access to the files is done via a dedicated API that follows the same design as the regular emFile API. The target application can use the BigFAT API to open a file, perform read and write operations on it and then to close it. In addition, API functions are available for performing operations on large files such as copying, moving and deleting.
The integration of the emFile BigFAT component in a target application is straightforward. It only requires the addition of a single source file and the activation of the support for long file names. No other special configuration is required.
Use cases
Transfer of large files such as movies to TVs
Movie files are in many cases larger than 4GB. Transferring these to a TV can be quite tricky or require splitting up the movie into different parts or reducing quality. With BigFAT, these transfers can be easily performed.
Creation of large files on embedded devices
Nowadays, more and more data is collected, e.g. in production environments where sensors in machines record data on temperature, pressure, fill levels, etc. over long periods of time. Other examples are battery management systems (BMS), which collect data on charge, temperature, current, voltage for each cell over very long periods of time. This data must be stored, ideally as a single file, which can now be done even on standard SD card using BigFAT and the standard underlying FAT format.
Large databases on embedded systems
BigFAT allows random read and write access to any file, even if larger than 4GB, as required by databases. With BigFAT, the only limit to the size of the database on an Embedded System is the size of the storage available.
FAQ
Q: What is the advantage of BigFAT over exFAT?
A: BigFAT is an open specification while exFAT is a closed specification protected by patents owned by Microsoft. exFAT cannot be legally used without permission from Microsoft.
Q: Can I implement BigFAT myself?
A: Absolutely. BigFAT is a specification made available by SEGGER. Anybody is free to write a piece of software implementing it. No fees, no royalties, no headaches. You do not even have to let SEGGER or anybody else know.
Q: What is in it for SEGGER? Why do you make it available free of charge to anyone?
A: We see emFile customers asking for solutions for bigger files. Implementing exFAT is not an option for us, as it is patent encumbered. SEGGER would need Microsoft's permission to implement and offer it, and our customers need to deal with Microsoft again to be able to use it in their products. This can be time-consuming and also expensive. We feel there should be a free alternative. The more popular BigFAT becomes, the better.