Linux for DevOps, Part 5: Storage & Disk I/O

Part 1 covered the VFS as the abstraction over filesystems. This post goes underneath it — from raw block device to the bytes your application reads — and where Kubernetes storage plugs into all of it.

The Storage Stack

graph TB
    APP["Application
read()/write()"]
    VFS["VFS
(Part 1)"]
    PC["Page Cache
(RAM)"]
    FS["Filesystem driver
(ext4 / xfs / btrfs)"]
    BIO["Block I/O layer
+ I/O scheduler"]
    LVM["LVM (optional)
logical volumes"]
    PART["Partition table"]
    DEV["Block device
(/dev/sda, /dev/nvme0n1)"]

    APP --> VFS --> PC --> FS --> BIO --> LVM --> PART --> DEV

    style PC fill:#1e3a1e,color:#7ec87e,stroke:#2d6a2d
    style BIO fill:#1e3a5f,color:#7ec8e3,stroke:#2d6a9f

Every layer here is a place things can go wrong or need tuning — worth knowing which one you’re looking at when disk performance is the symptom.

Block Devices and Partitions

lsblk                            # tree view: disks -> partitions -> LVM -> mount points
lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT
blkid                            # UUIDs and filesystem types per device
fdisk -l /dev/sda                # partition table detail

A block device is addressed in fixed-size blocks (traditionally 512 bytes, often 4096 now) rather than a byte stream. /dev/sda is the whole disk; /dev/sda1 is a partition — a contiguous slice described by a partition table (GPT on anything modern, MBR on legacy/small disks).

LVM: A Flexible Layer Between Partitions and Filesystems

Partitions are fixed at creation time. LVM (Logical Volume Manager) adds a layer that can be resized, snapshotted, and spread across multiple disks without unmounting.

graph LR
    D1["/dev/sda1"] --> PV1["Physical Volume"]
    D2["/dev/sdb1"] --> PV2["Physical Volume"]
    PV1 --> VG["Volume Group
(pooled capacity)"]
    PV2 --> VG
    VG --> LV1["Logical Volume
/var"]
    VG --> LV2["Logical Volume
/home"]
    VG --> LV3["Logical Volume
(unallocated)"]

pvcreate /dev/sda1 /dev/sdb1         # mark raw partitions as physical volumes
vgcreate data-vg /dev/sda1 /dev/sdb1  # pool them into a volume group
lvcreate -L 50G -n var-lv data-vg     # carve out a logical volume
mkfs.xfs /dev/data-vg/var-lv
lvextend -L +20G /dev/data-vg/var-lv  # grow it later, online
xfs_growfs /var                        # tell the filesystem to use the new space

This is exactly what most cloud-provisioned root disks use under the hood, and it’s why you can grow a cloud volume without downtime: extend the underlying block device, extend the LV, grow the filesystem — three separate operations, none of which require unmounting.

Choosing a Filesystem

Filesystem	Strengths	Watch out for
ext4	Mature, predictable, low overhead, default on most distros	Online shrink not supported; max practical volume ~50TB
XFS	Excellent large-file and parallel I/O throughput, online grow	No online shrink at all; less ideal for tiny files at scale
Btrfs	Native snapshots, checksums, built-in RAID-like features	Higher CPU overhead; RAID5/6 historically had stability issues
overlayfs	Not for real data — layered, used for container images	Copy-up on first write to a lower-layer file; see Part 6
tmpfs	RAM-backed, extremely fast	Not persistent; competes with application memory for RAM

XFS is the default on RHEL/CentOS and a common choice for databases needing high parallel throughput; ext4 remains the safe general-purpose default. The choice rarely matters until it does — snapshot requirements or extreme file counts are the usual deciding factors.

The Page Cache and Writeback

Linux never lets a write go straight to disk if it can help it. Writes land in the page cache (RAM) first and are marked “dirty”; a kernel thread flushes dirty pages to disk asynchronously.

free -h                           # "buff/cache" column — this is largely page cache, reclaimable
cat /proc/meminfo | grep -i dirty
sync                               # force all dirty pages to disk right now
echo 3 > /proc/sys/vm/drop_caches # (rarely needed) drop clean caches, for benchmarking only

This is why free -h showing low “available” memory is usually not a problem — page cache is reclaimed instantly under memory pressure. It’s also why a hard power loss can lose recent writes that an application believed were “written”: write() returning success only means the data reached the page cache, not the disk. Applications that need durability call fsync() explicitly (databases do this on every commit).

cat /proc/sys/vm/dirty_ratio          # % of RAM that can hold dirty pages before writes block
cat /proc/sys/vm/dirty_background_ratio  # % at which background flushing kicks in

In containers, emptyDir volumes without medium: Memory are backed by the node’s normal filesystem — same page cache behavior. medium: Memory skips the block layer entirely and is tmpfs, trading persistence for guaranteed RAM speed.

I/O Schedulers

The block layer queues and reorders I/O requests before they hit the device, because reordering can dramatically cut seek time on spinning disks (less relevant, but not irrelevant, on SSD/NVMe).

cat /sys/block/sda/queue/scheduler
# [mq-deadline] kyber none

Scheduler	Best for
`none`	NVMe — device is fast enough that reordering overhead isn’t worth it
`mq-deadline`	General purpose — bounds max latency per request
`kyber`	Latency-sensitive workloads on fast SSDs, tunable target latency
`bfq`	Desktop-style fairness between processes sharing one disk

Measuring I/O

iostat -xz 1                 # per-device utilization, await, throughput — the first command to run
iotop                        # per-process I/O, like top but for disk
blktrace -d /dev/sda -o -    | blkparse -i -   # trace every I/O request through the block layer
df -hT                       # space usage per mounted filesystem
df -i                        # inode usage — can hit 100% while df -h still shows free space

iostat -xz 1’s %util near 100 with high await means the device is saturated — the fix is fewer/ smaller I/Os, a faster device, or spreading load, not more application threads (which will just queue harder against the same disk).

Kubernetes Storage, Mapped to This Stack

graph LR
    PVC["PersistentVolumeClaim"] --> PV["PersistentVolume"]
    PV --> SC["StorageClass
(provisioner)"]
    SC --> CSI["CSI driver"]
    CSI --> BLOCK["Cloud block device
(EBS/PD/Azure Disk)
or local disk"]
    BLOCK --> FS2["Filesystem
(ext4/xfs, formatted by CSI)"]
    FS2 --> MOUNT["Bind-mounted
into the pod's mount namespace"]

A PersistentVolumeClaim is a request; the CSI driver provisions an actual block device (an EBS volume, a GCE PD, or a local disk partition), formats it with a filesystem, and bind-mounts it into the pod’s mount namespace at the path you specified. securityContext.fsGroup (from Part 2) recursively chowns that mount on attach. All the concepts above — page cache, I/O scheduler, filesystem choice — apply exactly the same way to that volume as to any other mounted disk; Kubernetes doesn’t change the storage stack, only who provisions it.

Next: Part 6 — Namespaces & cgroups Deep Dive, returning to containers with the storage (OverlayFS) and resource-limiting mechanics covered in full depth.