Linux for DevOps, Part 1: Ecosystem, Kernel, and What You Must Know

Linux is not just an operating system. It is a kernel — the software that arbitrates access to hardware — surrounded by a rich ecosystem of tools, libraries, and distributions that together form what most people call “Linux”. Understanding the distinction, and where each layer lives, is the foundation of everything else in systems work.

The Linux Ecosystem

When you log into a Linux server you are interacting with many layers stacked on top of each other. From bottom to top:

graph TB
    HW["Hardware
(CPU · RAM · Disk · NIC)"]
    KS["Kernel Space
(Linux Kernel)"]
    SC["System Call Interface"]
    LIBC["C Standard Library
(glibc / musl)"]
    US["User Space
(Daemons · Shells · Applications)"]
    DIST["Distribution Layer
(Package manager · init system · config)"]

    HW --> KS
    KS --> SC
    SC --> LIBC
    LIBC --> US
    US --> DIST

    style HW fill:#1a1a1a,color:#aaa,stroke:#333
    style KS fill:#1e3a5f,color:#7ec8e3,stroke:#2d6a9f
    style SC fill:#2d2d2d,color:#ccc,stroke:#444
    style LIBC fill:#2d2d2d,color:#ccc,stroke:#444
    style US fill:#1e3a1e,color:#7ec87e,stroke:#2d6a2d
    style DIST fill:#2d2d2d,color:#ccc,stroke:#444

Hardware is physical — CPU cores, RAM chips, disk controllers, network cards.

Kernel Space is privileged memory where the Linux kernel runs. It has unrestricted access to hardware. Code here can corrupt the entire system if it misbehaves — which is why only the kernel and kernel modules run there.

User Space is unprivileged memory where every process you run lives — your shell, nginx, Python scripts, container runtimes. User space processes cannot touch hardware directly. They must ask the kernel.

The System Call Interface is the gated bridge between the two. A syscall is how user space requests a kernel service: open a file, allocate memory, send a packet, create a process.

Kernel Space: What Lives There

The kernel is not a monolith you can observe directly. It is a collection of subsystems, each managing a class of resources.

Process Scheduler

Decides which process runs on which CPU core and for how long. Linux uses the Completely Fair Scheduler (CFS) — it tracks CPU time owed to each process and always runs the one with the most deficit. Real-time processes (SCHED_FIFO, SCHED_RR) bypass CFS and are relevant when running latency-sensitive workloads like kernel bypass networking.

Memory Manager

Manages physical RAM and the virtual address space each process sees. Key concepts:

Virtual memory — every process thinks it owns a flat address space. The kernel + MMU translate virtual → physical addresses via page tables.
Pages — memory is divided into 4 KiB pages (or 2 MiB huge pages). The kernel maps, unmaps, and swaps pages.
OOM Killer — when RAM is exhausted the kernel picks a process to kill. In container environments this kills your app, not the noisy neighbour.

Virtual File System (VFS)

An abstraction layer that presents a unified open()/read()/write() interface regardless of the underlying filesystem type — ext4, XFS, tmpfs, procfs, sysfs, NFS. Everything in Linux is a file because everything goes through the VFS.

Network Stack

Full TCP/IP implementation inside the kernel. Packets travel:

sequenceDiagram
    participant App as Application (user space)
    participant Sock as Socket API
    participant TCP as TCP Layer
    participant IP as IP Layer
    participant NIC as NIC Driver

    App->>Sock: send(data)
    Sock->>TCP: segment + seq numbers
    TCP->>IP: add IP header
    IP->>NIC: hand off frame
    NIC-->>App: (async) tx complete

eBPF (extended Berkeley Packet Filter) lets you attach programs to kernel hook points — including network ingress/egress — without modifying the kernel. Cilium and Calico use it for container networking and policy enforcement.

Namespaces and cgroups

These two primitives are what containers actually are:

Primitive	What it isolates
`pid` namespace	Process ID space — PID 1 inside a container is not global PID 1
`net` namespace	Network stack — each container gets its own interfaces and routing table
`mnt` namespace	Mount points — container filesystem is isolated from the host
`uts` namespace	Hostname
`user` namespace	UID/GID mapping — container root can be unprivileged on the host
cgroups	Resource limits — CPU, RAM, I/O bandwidth, PIDs per group

Namespaces provide isolation. cgroups provide accounting and limits. Docker and Kubernetes do nothing magic — they call clone() with namespace flags and write cgroup files.

The System Call Path

Every interaction between user space and hardware goes through a syscall. Here is what happens when your program calls read() on a file:

sequenceDiagram
    participant Prog as Your Program
    participant LIBC as glibc wrapper
    participant CPU as CPU (trap)
    participant Kern as Kernel syscall handler
    participant VFS as VFS layer
    participant FS as ext4 driver

    Prog->>LIBC: read(fd, buf, len)
    LIBC->>CPU: syscall instruction (nr=0)
    CPU->>Kern: privilege escalation → ring 0
    Kern->>VFS: vfs_read()
    VFS->>FS: ext4_file_read_iter()
    FS-->>VFS: data from page cache or disk
    VFS-->>Kern: bytes read
    Kern-->>CPU: return to ring 3
    CPU-->>LIBC: return value
    LIBC-->>Prog: bytes or -1 + errno

strace intercepts this at the boundary — it shows every syscall your process makes, with arguments and return values. It is the most useful debugging tool you are not using enough.

User Space: Distributions and the Init System

A distribution packages the kernel with:

A C library (glibc on most, musl on Alpine)
An init system — systemd dominates; it is PID 1, starts services, manages mounts, handles logging (journald)
A package manager (apt, dnf, pacman, apk)
A base set of userland tools (coreutils, util-linux, procps)

The init system is critical for platform engineers. systemd units define how services start, restart, log, and depend on each other. Containers replace systemd with a single entrypoint process — which is why PID 1 behaviour matters (signal handling, zombie reaping).

What a DevOps / Platform Engineer Must Know

1. Process Management

ps aux                      # snapshot of all processes
pstree -p                   # parent/child hierarchy
kill -SIGTERM <pid>         # graceful shutdown request
kill -SIGKILL <pid>         # unconditional termination

Key signals:

Signal	Number	Meaning
SIGTERM	15	Polite shutdown — process can clean up
SIGKILL	9	Immediate kill — cannot be caught
SIGHUP	1	Traditionally reload config
SIGINT	2	Keyboard Ctrl-C

Kubernetes sends SIGTERM, waits terminationGracePeriodSeconds, then SIGKILL. If your app ignores SIGTERM, Kubernetes hard-kills it — connections drop, in-flight requests die.

2. systemd

systemctl status nginx
systemctl restart nginx
journalctl -u nginx -f          # follow logs
systemctl list-units --failed
systemd-analyze blame           # boot time by unit

A unit file anatomy:

[Unit]
Description=My Service
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/myapp
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

3. Filesystem and Storage

df -hT                          # disk usage per filesystem + type
lsblk                           # block devices and mount points
mount | grep "^/dev"            # active mounts
findmnt                         # tree view of mounts
du -sh /var/log/*               # directory sizes
inode usage: df -i              # inodes, not bytes — can exhaust independently

tmpfs is RAM-backed. Kubernetes emptyDir with medium: Memory is tmpfs — writes go to RAM, not disk. Pod OOM is possible if you write too much.

Bind mounts (--mount type=bind) expose a host path into a container. The kernel implements this as an additional VFS reference — no data copy.

4. Networking

ip addr                         # interfaces and IPs
ip route                        # routing table
ss -tlnp                        # listening TCP sockets with PIDs
ip netns list                   # network namespaces (container networks)
tc qdisc show dev eth0          # traffic control / queueing disciplines

iptables / nftables is how the kernel filters and routes packets. Every kubectl expose or Service of type ClusterIP writes iptables rules (or ipvs rules in kube-proxy IPVS mode). When networking breaks, iptables -L -n -v --line-numbers and conntrack -L are your starting points.

DNS resolution order is controlled by /etc/nsswitch.conf and /etc/resolv.conf. In Kubernetes, the kubelet writes a custom resolv.conf into every pod pointing at CoreDNS.

5. Observability: Performance Tools

graph TD
    Perf["Performance problem"]
    CPU["CPU bound?"]
    MEM["Memory bound?"]
    IO["I/O bound?"]
    NET["Network bound?"]

    Perf --> CPU & MEM & IO & NET

    CPU --> T1["top / htop
perf stat
mpstat -P ALL 1"]
    MEM --> T2["free -h
vmstat 1
/proc/meminfo
valgrind / heaptrack"]
    IO --> T3["iostat -xz 1
iotop
blktrace"]
    NET --> T4["ss -s
nethogs
iperf3
tcpdump"]

/proc and /sys are virtual filesystems that expose kernel internals as files — no syscall overhead beyond read(). /proc/<pid>/ holds everything about a running process: open files (fd/), memory maps (maps), syscall counts (status), cgroup membership (cgroup).

6. Containers: Connecting the Dots

Containers are not VMs. There is no hypervisor. A container process runs directly on the host kernel — isolated via namespaces, limited via cgroups, with a layered filesystem (OverlayFS).

When kubectl schedules a pod:

kubelet calls the CRI (containerd) to pull the image
containerd unpacks OCI layers into OverlayFS
runc/crun calls clone() with namespace flags → new namespaces created
cgroup hierarchy created under /sys/fs/cgroup/kubepods/
Container process starts as PID 1 in its own PID namespace

7. Security Primitives

Capabilities — the kernel divides root privilege into ~40 capabilities. CAP_NET_ADMIN lets you modify routing tables. CAP_SYS_PTRACE lets you trace other processes. Drop capabilities you don’t need. capsh --print shows the current set.

seccomp — a syscall allowlist enforced in the kernel. Docker’s default profile blocks ~44 syscalls. Kubernetes securityContext.seccompProfile applies one per container.

AppArmor / SELinux — Mandatory Access Control. Defines what files, sockets, and capabilities a process is allowed to access beyond DAC (owner/group bits). Openshift enforces SELinux by default.

Mental Model to Keep

mindmap
  root((Linux))
    Kernel Space
      Process Scheduler
      Memory Manager
      VFS
      Network Stack
      Device Drivers
      Namespaces + cgroups
      LSM
    User Space
      Shells + coreutils
      Init system (systemd)
      Libraries (glibc)
      Daemons + services
    DevOps Layer
      Processes + signals
      Storage + mounts
      Networking + iptables
      Observability tools
      Container primitives
      Security (caps + seccomp)

The mental model that pays the most dividends: every abstraction in the DevOps stack is kernel primitives with a friendly API on top. Kubernetes pods are namespaced cgroup-limited processes. Service meshes are eBPF programs or iptables rules. S3-backed volumes are VFS mounts. When things break, you trace back down through the layers until you find the kernel resource that is exhausted, misconfigured, or missing a permission.