File System

 FileSystem

A filesystem is a method used by operating systems to organize and store data on storage devices such as hard drives, solid-state drives (SSDs), USB drives, and other storage media. It provides a structured way to store, retrieve, and manage files and directories within a storage device.

Key characteristics and functionalities of a filesystem include:

Hierarchical Structure: Filesystems organize data in a hierarchical structure consisting of directories (also known as folders) and files. Directories can contain other directories (subdirectories) and files, forming a tree-like structure.

File Naming: Each file within a filesystem is identified by a unique name, which allows users and applications to access and manipulate individual files.

File Attributes: Filesystems typically store additional information, known as metadata, about each file, including attributes such as file size, creation date, modification date, and permissions.

Data Storage: Filesystems manage the allocation of storage space on the underlying storage device. They divide the storage space into smaller units called blocks or clusters and allocate these units to store file data efficiently.

File Access and Retrieval: Filesystems provide mechanisms for users and applications to access and retrieve data from files stored on the storage device. This includes read, write, and execute permissions that control how files can be accessed and modified.

Filesystem Operations: Filesystems support various operations for managing files and directories, including creating, deleting, renaming, moving, copying, and searching for files.

Filesystem Types: There are different types of filesystems designed for specific purposes and compatible with different operating systems. Common filesystem types include FAT32, NTFS, exFAT, ext2, ext3, ext4, HFS+, APFS, and XFS, among others.

One host only have one filesystem?

Typical host may have only one filesystem. However, there are scenarios where hosts may have multiple filesystems, especially on servers or systems with complex storage configurations. For instance:

Servers with Multiple Disks or Partitions: Servers may have multiple physical disks or partitions, each representing a separate filesystem. This is common in environments where administrators partition storage devices to separate data, applications, or operating system files.

Network Attached Storage (NAS) or Storage Area Network (SAN): Servers connected to NAS or SAN may have multiple filesystems mounted from remote storage devices. Each mounted filesystem represents a distinct storage volume or share.

Virtual Machines (VMs): Virtualized environments often involve multiple virtual machines running on a single physical host. Each VM may have its own filesystem, esspecially if it's configured with dedicated storage resources.

Containerized Environments: Containers deployed on a host may have their own filesystems. In containerized architectures like Docker, each container typically has its own filesystem, separate from the host's filesystem.

Specialized Configurations: In certain specialized configurations, such as clustered storage systems or distributed file systems, a single host may have access to multiple filesystems for redundancy, scalability, or performance reasons.

In these scenarios, Zabbix's filesystem discovery feature becomes particularly useful as it automatically detects and monitors all filesystems present on a host. Even if a host has only one filesystem, the filesystem discovery process in Zabbix still provides a standardized way to manage filesystem monitoring across all hosts, ensuring consistency and scalability in monitoring configurations.

Relationship between Database and Disk?

How is the data being stored??

Let's break down the relationship between databases and data storage on disk with a simple example:

Data Storage on Disk:

• Imagine you're running an e-commerce website. When a user makes a purchase, information about the order (such as product details, customer information, and order timestamp) needs to be stored somewhere for future reference.
• This data is initially stored on the disk of the server where your website is hosted. Each piece of information is written to files on the disk in a structured or unstructured format.

Database Management System (DBMS):

• To organize and manage this data effectively, you employ a Database Management System (DBMS) such as MySQL, PostgreSQL, or MongoDB.
• The DBMS provides software tools and services to store, retrieve, update, and manage the data efficiently. It acts as an intermediary layer between your application and the physical storage on disk.

Database Structure and Schema:

• Before storing data, you define a database schema, which specifies the structure of the data, including tables, columns, data types, and relationships.
• For example, in your e-commerce application, you might have tables for "Products," "Customers," and "Orders," each with specific columns to store relevant information.

Data Interaction through SQL:

• To interact with the data stored in the database, you use a query language such as SQL (Structured Query Language).
• SQL allows you to perform operations such as inserting new data, querying existing data, updating records, and deleting data from the database.

Example Cycle:

• Let's say a customer places an order on your e-commerce website. When the order is submitted, your application collects the necessary information (e.g., product details, customer information) and sends it to the database.
• Using SQL commands, your application inserts this data into the appropriate tables in the database, following the defined schema.
• The DBMS then takes care of writing this data to the disk, organizing it efficiently according to the database schema.

Data Retrieval and Processing:

• Later, when you need to retrieve information about orders or customers, your application sends SQL queries to the database.
• The DBMS processes these queries, retrieves the requested data from the disk, and returns it to your application for further processing or display on the website.

In summary, the database acts as an organized repository for storing and managing data, providing a structured and efficient way to interact with data stored on disk. It abstracts away the complexities of low-level disk operations and provides high-level functionality for data management, retrieval, and processing.

What Is the Master Boot Record (MBR)?

• MBR: "Master Boot Record" is stored on hard drives or other storage devices (less than 2 TB) with data needed for the boot process. Data of partition table that includes information of the number of partitions, their size, and the one actively working to boot the system.
• The Master Boot Record (MBR) is the first sector on a hard drive that contains special data about the operating system that was used to start the computer. “Sector zero,” “master boot block,” or “master partition boot sector” are other terms for the Master Boot Sector.
• When the computer starts by pressing the power on button, BIOS checks the boot loader present in the MBR which is present in the first sector of the hard disc. The bootloader then loads the operating system into the main memory. That process is known as Booting.
• MBR only supports the overall storage of a maximum of 2 TB. If your disk storage is more than 2 TB, then the remaining space will remain unallocated.
• MBR is created when a drive is partitioned, yet it's not located within a partition.
• The master boot record is located on the first sector of a disk. And the specific address is Cylinder: 0, Head: 0, Sector: 1.

MBR is consists of three significant pieces necessary for the boot process:

· The master boot code

· The master partition table

· The disk signature

MBR and GPT are two different methods of storing partition information. When you're partitioning a hard drive or using a disk partitioning tool, you can choose one of them based on your knowledge, or you can:

· Convert MBR to GPT(GUID Partition Table)

· Convert GPT to MBR

• MBR only supports the overall storage of a maximum of 2 TB. If your disk storage is more than 2 TB, then the remaining space will remain unallocated. Thus, if you wish to utilize the entire storage on your disk, then you should go with GPT.
• The maximum capacity of MBR partition tables is only about 2 terabytes. You can use a drive that's larger than 2 terabytes with MBR, but only the first 2 terabytes of the drive will be used. The rest of the storage on the drive will be wasted.
• In contrast, GPT partition tables offer a maximum capacity of 9.7 zetabytes. 1 zetabyte is about 1 billion terabytes, so you're unlikely to run out of space anytime soon.
• Next, MBR partition tables can have a maximum of 4 separate partitions. However, one of those partitions can be configured to be an extended partition, which is a partition that can be split up into an 23 additional partitions. So the absolute maximum number of partitions an MBR partition table can have is 26 partitions.
• GPT partition tables allow for up to 128 separate partitions, which is more than enough for most real world applications.

Disk Partitioning in Linux:

A physical hard disk (say HDD) can be divided into different storage units and those storage units are known as partitions. Thus, dividing a hard disk drive into different parts is known as partitions for that particular hard disk.

For example, if you have a 1TB hard disk drive, and you divide it into 4 parts (say 250GB each), those parts are called partitions. You can create four primary partitions or three primary partitions and one extended partition. Here is what that means:

• Primary partitions are those where you can install Windows and put other data. For example, if you have installed Windows 11/10 on a C drive, then the C drive is a primary partition
• Extended partition works as a container that you can use to create multiple logical partitions
• Logical partitions are those where you can create folders, add your media files, and other personal data you want.

·

Note: Disk Partitioning is the process of dividing a disk into one or more logical areas, often known as partitions, on which the user can work separately. It is one step of disk formatting.

If a partition is created, the disk will store the information about the location and size of partitions in the partition table. With the partition table, each partition can appear to the operating system as a logical disk, and users can read and write data on those disks. The main advantage of disk partitioning is that each partition can be managed separately.

A logical disk, logical volume or virtual disk (VD[1] or vdisk[2] for short) is a virtual device that provides an area of usable storage capacity on one or more physical disk drive(s) in a computer system. The disk is described as logical or virtual because it does not actually exist as a single physical entity in its own right.

In order to successfully partition a disk and to make it useful, we need to follow the four steps:

1) Attach the disk to the proper port.

2) Create partitions in the disk.

3) Create a file system on the partition.

4) Mounting the file systems

Once the disk is attach to the proper port, will proceed to create the partitions in the disk.

Create Partitions in the Disk

To view the available Hard Disks in our system, use the command lsblk or cat /proc/partitions.

Command- lsblk

· We can find that the system has two disks -> sda and sdb.

· sda is our older Hard Disk. We can say that disk is already partitioned as sda1 and sda2.

Command: fdisk -l

fdisk -l

We found that the Hard Disk that we are going to partition can be found at /dev/sdb.

fdisk /dev/sdb

fdisk is command line utility used for disk partioning, it allow the users to create , delete and manipulate the diskpartitions. This command instructs the OS to run the “fdisk” utility.

It will take us to a different console where we can use the partitioning specific commands. We will be concentrating more on the following commands (or flags).

m -> help

p -> print partition table

n -> create new partition

d -> delete partition

q -> quit without writing

w -> write to disk

While partitioning, we should be aware of certain factors.

· On a disk, we can have a maximum of four partitions.

The partitions are of two types.

1. Primary

• Primary partition is the major partition where Windows OS and other data can be stored, and it’s the only partition that can be set to active. It cannot be divided into other types of partitions. This type of partition stores the boot files and only one
• primary partition on one hard disk can be set to active at a time. If not, Windows will be unbootable.

2. Extended

• Extended partitions can have logical partitions inside it.

Note: On MBR hard disks, users can create 4 primary partitions or 3 primary partitions plus 1 extended partition at most. However, a GPT disk can be divided into 128 primary partitions at most. Besides, the GPT disk only contains primary partition.

Among the four possible partitions, the possible combinations are.

• All 4 primary partitions
• 3 primary partitions and 1 extended partition

fdisk /dev/sdb

Command: n

This will create a new partition. Specify the type of partition using the p for primary and e for extended.

Command: p

• This will create a primary partition. The console will prompt for the number to be given to the partition. We can give any number from 1 to 4. Let me give 1. Then choose the starting position (cylinder) of partition 1. Press enters to start partitioning from the beginning of the disk.
• You can specify the size of the partition in two ways, either as the last cylinder number or by specifying the size directly. If we need partition1 to be of 4 GB size (as a whole number), use.

Command: +4G

• Pressing enter will create our 1st partition successfully. Follow the same steps, until we create 4 partitions on our newly attached disk.
• create our 1st partition successfully
• We have created 3 primary partitions and an extended partition in the disk. The extended partition, as the name suggests, can be further divided into multiple logical partitions.
• Once 4 partitions are created, no more partitions can be created on the same disk. We can check whether we have done the partitioning in the right way by printing the partition table using the p command. If everything is as per the expectation, then write the changes to the disk using the w command, else use the q command to quit without writing.

Note: The term "formatting" a partition refers to the process of preparing a storage device, such as a hard drive or a partition, for use by an operating system. When you format a partition, you typically specify a file system to be used on that partition.

Note: When you format a partition, you are essentially organizing the disk space into a structure that the operating system can understand and utilize efficiently. This involves setting up data structures and metadata necessary for storing and accessing files and directories.

Create a File System on the Partition

• File system provide the structure to data
• The commonly used Linux file systems are ext2, ext3, ext4, JFS, ReiserFS, XFS, FAT (usually in Windows OS) and B-treeFS.
• A file system is used to control how data is stored and retrieved. Without a file system, data placed in a storage medium would be one large body of data with no way to tell where one piece of data stops and the next begins.
• File system can separate the data into pieces and give each piece a name. In this way, the data is easily isolated and identified. Each group of data is called a “file”, while the structure and logic rules used to manage the groups of data and their names are called “file system.”

Example: If we have a video file, so file system divide the data into block(each piece of data called file) and store it in the different sector of disk. when the user send the request it retrieve the info and send back to the user. file system API, it is an application programming interface through which a utility or user program requests services of a file system. A file system is a data management scheme

Note: When you create partitions on a storage device, the information about these partitions is stored in Partition Table that is located on the first sector of disk (MBR sector). Then, you need to add file systems to these partitions by formatting.

Note: In a file system partition, a part of the storage space is used to store the logical structure information of the file system. This part is called different names in different file systems, for example, it is called FAT table in FAT file system, called MFT table in NTFS file system, etc. Then, the remaining space is used to store data in the form of files and directories.

Command: mkfs.ext4 -j /dev/sdb1

What this command does is that it will make the first partition’s file system to be ext4 (format the partition to ext4). -j flag is used to allow/support journaling. It helps in throwing errors into the journal, in case of system failure.

Command: mkfs.fat /dev/sdb2

This command is used to format the 2nd partition available in /dev/sdb2 to the FAT file system. Even though we formatted the disks, it is of no use to us, unless we mount it on a directory.

Root file system: 

It is the file system that holds the root directory “/” and is mounted as the root directory during system boot

About Linux Directory Structure

• Linux uses directory tree to manage directories and files. The directory tree information should be also stored on storage device and this part is called root file system (the “/” directory).
• In Linux system, the root directory (/) is the most important directory, because:
• All other directories are derived from the root directory;
• The root directory is closely related to the boot, repair and restore of the Linux system.

The root directory contains the following subdirectories (first-level directories):

1. /bin: This directory includes user executable files or It contains common system commands that can be executed by any users.

2. /boot: It saves files related to system startup, such as kernel files and bootloader (grub) files, etc.

3. /dev: It contains the device files for every hardware device attached to the system. These are not device drivers, rather they are files that represent each device on the computer and facilitate access to those devices.

4. /etc: It contains the local system configuration files for the host computer, such as user information, service startup scripts, configuration files for common services, etc. It’s similar to Windows Registry.

5. /home: It’s the default location to log in and save user data. Each user has a subdirectory in /home.

6. /lib: It contains shared library files that are required to boot the system.

7. /media: It’s the place used to mount external removable media devices such as floppy disks, CDs, and USB thumb drives that may be connected to the host.

8. /mnt: It’s the traditional mountpoint for regular hard drive partitions (file systems). But most people like to mount hard drives under /home

9. /misc: It’s the place used to mount the shared directory of the NFS service.

10. /opt: It’s the place used to place and install other software.

11. /usr: Its full name is Unix Software Resource. It is the default installation location of the software and similar to the complex of “C:Windows + C:Program files” in the Windows system.

12. /root: It is the home directory for the root user. It’s the default location to log in and save root user data.

13. /sbin: It saves commands related to system environment settings. Only root can use these commands, but there are also commands that allow ordinary users to view.

14. /srv: It contains data for services. After some system services are started, they can call out or save necessary data in this directory.

15. /tmp: It is a place where the system stores temporary files, under which all users can access and write.

16. /lost+found: When the system crashes unexpectedly or shuts down unexpectedly, some file fragments will be stored here. During system startup, the fsck tool will check this directory and repair the damaged file system. Please note this directory only appears in each partition (file system).

17. /proc: The data in this directory is not saved on the hard disk, but in the memory. It mainly saves the system’s kernel, process, external device status and network status, etc.

18. /sys: It’s similar to the /proc directory and the data in this directory is stored in memory, but it mainly stores information related to the kernel.

19. /var: It is used to store dynamic data, such as caches, log files, and files generated during software operation.

Mounting the file systems:

Let us make three directories, for mounting the first, second, and third partition using the mkdir (make directory) command.

mkdir /mount1

mkdir /mount2

mkdir /mount3

Mounting the formatted partitions can be achieved using the mount command.

mount /dev/sdb1 /mount1

mount /dev/sdb2 /mount2

mount /dev/sdb3 /mount3

The first part of the command is the keyword mount, followed by the partition and the directory to which the mounting is to be done.

To view the disk details, we can use the df command. h flag helps to display the output in a human-readable format.

Command: df –h

Note: Unmounting a disk can be achieved by unmount command.

Command: unmount /dev/sdb2

All these mountains are temporary in nature. Once we reboot the system, mounting will be reverted. To make it permanent, we must edit the File System Table of the Operating System. We should be highly careful whirling with this file. A small error in this file can cause the system to be unbootable and can make the entire system to be useless. Let us open the file in any editor (I will be using the VI editor).

vi /etc/fstab

fstab(File System Table) file

Add our mounted file systems’ details in the file, in the order.

mounted partition [ space ] directory [ space ] file system type [ space ] defaults [ space ] 0 [ space ] 0

Save the file and come back to the terminal. Mount the partitions permanently using the mount -a command

Command: mount –a

We can verify whether our partitions are available for our purposes by visiting the root folder of the Operating System (since we have mounted the partitions on the mount1, mount2, mount3 directories in the root folder). We can also verify the existence of our newly created partitions using the lsblk command.

Command: lsblk

We can find our partitions in the table, each of 4 GB size.

Note: Disk-->Partitioning(divide the disk)-->Create a file system for each partition(manage the data on disk: Define how the data will store in disk)----> Mouning the each filesystem(mount the each partition with directory)

What is mounting

• In Linux, "mounting" refers to the process of attaching a file system to a specific directory in the Linux directory tree.
• This allows the files and directories on that file system to be accessed through the specified directory i.e we can access the data of the logical disk through the specified directory. This is particularly useful for accessing external drives, network shares, or other storage devices.

Example: How to mount a USB drive in Linux

Plug in the USB Drive: Insert your USB drive into an available USB port on your computer.

Identify the USB Drive: Open a terminal and use the lsblk command to list all block devices, including your USB drive. Look for your USB drive in the list. It will typically be something like /dev/sdX (where "X" is a letter representing the specific drive, e.g., /dev/sdb).

Create a Mount Point: Choose or create a directory where you want to mount the USB drive. For example, you can create a directory named "usb" in your home directory:

mkdir ~/usb

Mount the USB Drive: Use the mount command to mount the USB drive to the mount point you created:

sudo mount /dev/sdX ~/usb

Note: Replace /dev/sdX with the actual device name of your USB drive.

Access the Files: You can now access the files on your USB drive through the directory you mounted it to (~/usb in this example). Navigate to that directory using the cd command and use standard file operations to read, write, or modify files.

Unmounting the Drive: When you're finished accessing the files on the USB drive, you should unmount it before removing it physically. Use the umount command followed by the mount point:

sudo umount ~/usb

Remember to replace ~/usb with the actual mount point if you've chosen a different one. Failure to properly unmount the drive before removing it can lead to data corruption or loss

What /dev/sdx?

In Linux, /dev/sdx is a special device file that represents a specific block storage device, such as a hard drive, solid-state drive (SSD), USB drive, or any other storage device recognized by the system. Let's break down each component:

/dev/: This is a directory in the Linux filesystem hierarchy that contains device files. These device files represent various hardware devices connected to the system. The /dev directory stands for "device."

sdx: This part of the path represents the specific block storage device. The sd prefix is typically used for devices connected via the SCSI (Small Computer System Interface) subsystem, which includes most modern storage devices like hard drives and SSDs, whether they are SATA, SCSI, or USB. The letter x represents a specific device identifier.

Examples of x: The letter x can be any lowercase letter from a to z, indicating different storage devices recognized by the system. For instance, /dev/sda might represent the first hard drive, /dev/sdb might represent the second hard drive, and so on. If you have multiple partitions on a single drive, they would be represented as /dev/sda1, /dev/sda2, etc.

Overall, /dev/sdx represents a block storage device (such as a hard drive or USB drive) connected to the system, with x representing a specific identifier for that device. This identifier can vary depending on the order in which devices are detected by the system and the configuration of your hardware.

What is the swap space?

Traditionally Linux was developed for server system. Server systems are built to process several processes simultaneously. Sometime some process may use more than expected memory. For example a university website which is hosted in Linux server system may consume more memory on the day when exam result is announced. This kind of memory spike is very common in server system. Any service may use more memory in its peak time. Linux uses swap space to deal with the shortage of physical memory. The swap space is the space of hard