FileSystem

A file system (filesystem) is means to organize data expected to be retained after a program terminates by providing procedures to store, retrieve and update data, as well as manage the available space on the device(s) which contain it. A file system organizes data in an efficient manner and is tuned to the specific characteristics of the device.

There is usually a tight coupling between the operating system and the file system. Some file-systems provide mechanisms to control access to the data and metadata. Insuring reliability is a major responsibility of a file-system. Some file-systems provide a means for multiple programs to update data in the same file nearly at the same time.

Without a filesystem programs would not be able to access data by file name or directory and would need to be able to directly access data regions on a storage device.

File systems are used on data storage devices such as magnetic storage disks or optical discs to maintain the physical location of the computer files. They may provide access to data on a file server by acting as clients for a network protocol (e.g., NFS, SMB, or 9P clients), or they may be virtual and exist only as an access method for virtual data (e.g., procfs). This is distinguished from a directory service and registry.

Aspects of file systems

1. Space Management

File systems allocate space in a granular manner, usually multiple physical units on the device. The file system is responsible for organizing files and directories, and keeping track of which areas of the media belong to which file and which are not being used. For example, in Apple DOS of the early 1980s, 256-byte sectors on 140 kilobyte floppy disk used a track/sector map.

This results in unused space when a file is not an exact multiple of the allocation unit, sometimes referred to as slack space. For a 512-byte allocation, the average unused space is 255 bytes. For a 64KB clusters, the average unused space is 32KB. The size of the allocation unit is chosen when the file system is created. Choosing the allocation size based on the average size of the files expected to be in the filesystem can minimize the amount of unusable space. Frequently the default allocation may provide reasonable usage. If it can be anticipated that a file system will contain mostly small files a small cluster size should be chosen. Choosing an allocation size that is too small results in excessive overhead if the file system will contain mostly very large files.

File system fragmentation occurs when unused space or single files are not contiguous. As a filesystem is used, files are created, modified and deleted. When a file is created the filesystem allocates space for the data. Some filesystems permit or require specifying an initial space allocation and subsequent incremental allocations as the file grows. As files are deleted the space they were allocated eventually is considered available for use by other files. This creates alternating used and unused areas of various sizes. This is free space fragmentation. When a file is created and there is not an area of contiguous space available for its initial allocation the space must be assigned in fragments. When a file is modified such that it becomes larger it may exceed the space initially allocated to it, another allocation must be assigned elsewhere and the file becomes fragmented.

A file system may not make use of a storage device but can be used to organize and represent access to any data, whether it is stored or dynamically generated (e.g., procfs).

2. Restricting and permitting access

There are several mechanisms used by file systems to control access to data. Usually the intent is to prevent reading or modifying files by a user or group of users. Another reason is to insure data is modified in a controlled way so access may be restricted to a specific to program. Examples include passwords stored in the metadata of the file or elsewhere and file permissions in the form of permission bits, access control lists, or capabilities. The need for filesystem utilities to be able to access the data at the media level to reorganize the structures and provide efficient backup usually means that these are only effective for polite users but are not effective against intruders.

See also password cracking.

Methods for encrypting file data are sometimes included in the filesystem. This is very effective since there is no need for filesystem utilities to know the encryption seed to effectively manage the data. The risks of relying on encryption include the fact that an attacker can copy the data and use brute force to decrypt the data. Losing the seed means losing the data.

3. Maintaining integrity

One of the filesystems significant responsibilities is to insure that, regardless of the actions by programs accessing the data, the structure remains consistent. This includes actions taken if a program modifying data terminates abnormally or neglects to inform the filesystem that is has completed its activities. This may include updating the metadata, the directory entry and handling any data that was buffered but not yet updated on the physical storage media.

Other failures which the filesystem must deal with include media failures or loss of connection to remote systems.

In the event of an operating system failure or "soft" power failure, special routines in the filesystem must be invoked similar to when an individual program fails.

The filesystem must also be able to correct damaged structures. These may occur as a result of an operating system failure for which the OS was unable to notify the file system, power failure or reset.

The filesystem must also record events to allow analysis of systemic issues as well as problems with specific files or directories.

4. Multiple Filesystems within a single system

Frequently retail systems are configured with a single filesystem occupying the entire hard disk.

Another approach is to partition the disk so that several filesystems with different attributes can be used. One filesystem, for use as browser cache, might be configured with a small allocation size. This has the additional advantage of keeping the frantic activity of creating and deleting files typical of browser activity in a narrow area of the disk and not interfering with allocations of other files. A similar partition might be created for email. Another partition, and filesystem might be created for the storage of audio or video files with a relatively large allocation. One of the filesystems may normally be set read only and only periodically be set writable.

Multiple filesystems on a single system has the additional benefit that in the event of a corruption of a single partition, the remaining filesystems will frequently be still intact. This includes virus destruction of the system partition or even a system that will not boot. Filesystem utilities which require dedicated access can effectively be completed piece meal, in addition defragmentation may be more effective. Several system maintenance utilities such as virus scans and backups can also be processed in segments. For example it is not necessary to backup the filesystem containing videos along with all the other files if none have been added since the last backup.

5. Design limitations

All file systems have some functional limit that defines the maximum storable data capacity within that system. These functional limits are a best-guess effort by the designer to determine how large the storage systems will be right now, and how large storage systems are likely to become in the future. Disk storage has continued to increase at near exponential rates (see Moore's law), so after a few years file systems have kept reaching design limitations that require computer users to repeatedly move to a newer system with ever greater capacity.

File system complexity typically varies proportionally with the available storage capacity. The file systems of early 1980s home computers with 50KB to 512KB of storage would not be a reasonable choice for modern storage systems with hundreds of gigabytes of capacity. Likewise, modern file systems would not be a reasonable choice for these early systems, since the complexity of modern file system structures would consume most or all of the very limited capacity of the early storage systems.

Types of file systems