Linux for DevOps, Part 6: Namespaces & cgroups Deep Dive

Part 1 established that containers are namespaced, cgroup-limited processes — no hypervisor, no magic. This post goes hands-on with both primitives and traces exactly what happens when a container runtime starts a container.

Namespaces, One at a Time

A namespace scopes a global kernel resource so a set of processes sees its own private instance of it. unshare creates new namespaces and runs a command inside them, directly from your shell — no container runtime needed.

# PID namespace: the new shell sees itself as PID 1
unshare --pid --fork --mount-proc bash
ps aux    # only shows processes inside this namespace

# mount namespace: new mounts here don't leak to the host
unshare --mount bash
mount -t tmpfs tmpfs /mnt    # invisible outside this shell

# UTS namespace: independent hostname
unshare --uts bash
hostname sandbox

A container runtime combines nearly all of these via one clone() call with multiple flags OR’d together — the process is born already inside all of them, rather than joining them one at a time.

User namespaces deserve special attention: they let UID 0 inside the namespace map to, say, UID 100000 outside. A process can be “root” for filesystem/capability checks inside its own namespace while being a fully unprivileged, ordinary user from the host’s point of view. This is the basis of rootless containers (Podman’s default mode, rootless mode in containerd) — a compromised container process has zero extra privilege on the host, because outside the namespace it was never root to begin with.

podman run --rm -it alpine id       # uid=0(root) inside
# meanwhile on the host:
podman unshare cat /proc/self/uid_map
#          0     100000      65536
# "namespace UID 0 maps to host UID 100000, for a range of 65536 UIDs"

Inspecting and Joining Namespaces

ls -la /proc/<pid>/ns/                # one symlink per namespace, e.g. net:[4026532xxx]
lsns                                   # list all namespaces on the system, with owning processes
nsenter -t <pid> -n -m bash            # join another process's net + mount namespaces

Two processes sharing the same inode number under /proc/<pid>/ns/net are in the same network namespace — this is how you can confirm two containers share a pod’s network (as all containers in a Kubernetes pod do).

cgroups: Accounting and Limits

Where namespaces control what a process can see, cgroups control how much it can use — CPU, memory, I/O bandwidth, PID count.

cgroups v2: The Unified Hierarchy

graph TD
    ROOT["/sys/fs/cgroup
(root)"]
    ROOT --> KUBEPODS["kubepods.slice"]
    KUBEPODS --> BE["besteffort.slice"]
    KUBEPODS --> BURST["burstable.slice"]
    KUBEPODS --> GUAR["(guaranteed pods,
direct children)"]
    BURST --> POD1["pod-abc123.slice"]
    POD1 --> C1["container-app.scope"]
    POD1 --> C2["container-sidecar.scope"]

    style ROOT fill:#1e3a5f,color:#7ec8e3,stroke:#2d6a9f

cgroups v2 (default since kernel 5.x, universal on modern distros) uses a single tree — every process belongs to exactly one cgroup, and controllers (cpu, memory, io, pids) are enabled per-subtree. This is a deliberate simplification over v1, which had a separate, independently-mountable hierarchy per controller and let a process be in different groups for CPU vs memory — powerful but a frequent source of bugs.

mount | grep cgroup2                          # confirm unified v2 mode
cat /sys/fs/cgroup/cgroup.controllers          # which controllers are available
cat /sys/fs/cgroup/kubepods.slice/*/cpu.max    # CPU quota for a slice: "200000 100000" = 2 cores
cat /sys/fs/cgroup/kubepods.slice/*/memory.max # hard memory ceiling in bytes
systemd-cgls                                   # tree view of the whole cgroup hierarchy
systemd-cgtop                                  # live resource usage per cgroup, like top but per-group

Kubernetes resources.limits.cpu: "2" becomes cpu.max = "200000 100000" (200ms of CPU time per 100ms period = 2 cores). resources.limits.memory becomes memory.max in bytes. When a container hits memory.max, the kernel’s OOM killer targets a process in that exact cgroup — not the node, not a neighbor — which is why one pod exceeding its memory limit doesn’t (in the normal case) take down others on the same node.

cat /sys/fs/cgroup/kubepods.slice/*/memory.events   # oom_kill counter — did this cgroup get OOM-killed?
dmesg | grep -i "killed process"                     # kernel's OOM killer log, system-wide

OverlayFS: The Container Filesystem

A container image is a stack of read-only layers plus one writable layer on top, unified into a single mount point by OverlayFS.

graph TB
    MERGED["Merged view
(what the container sees at /)"]
    UPPER["Upper dir
(writable — this container's changes)"]
    LOWER3["Lower: app layer"]
    LOWER2["Lower: dependencies layer"]
    LOWER1["Lower: base OS layer"]
    WORK["Work dir
(OverlayFS internal bookkeeping)"]

    MERGED -.reads through.-> UPPER
    UPPER -.falls through to.-> LOWER3
    LOWER3 -.falls through to.-> LOWER2
    LOWER2 -.falls through to.-> LOWER1
    UPPER --- WORK

mount -t overlay overlay \
  -o lowerdir=/layers/base:/layers/deps:/layers/app,upperdir=/container/upper,workdir=/container/work \
  /container/merged

Reading a file walks the layers top-down and returns the first match — this is why deleting a base layer file inside a container doesn’t touch the image; OverlayFS writes a whiteout marker in the upper layer that hides the lower file instead. Writing to an existing lower-layer file triggers copy-up: the whole file is copied into the upper layer first, then modified — which is why containers with very large files that get modified even slightly (databases, log files) perform badly on OverlayFS and should use a volume mount instead.

Multiple containers from the same image share identical lower layers on disk — this is the mechanism behind fast image pulls and low disk usage for common base images, and it’s the same hard-linking concept from Part 2 applied at the layer level.

The Full runc Lifecycle

When containerd (via the CRI) needs to actually start a container, it hands off to runc (or crun), a low-level OCI-compliant runtime that does the kernel plumbing.

pivot_root() is worth calling out: it’s how the container’s mount namespace stops seeing the host’s filesystem entirely and switches to the OverlayFS-merged rootfs as / — stronger than chroot, which only changes the apparent root but leaves the old root reachable via tricks like .. from an open file descriptor.

Practical Checklist

# is this "container" actually isolated the way I think?
cat /proc/<pid>/ns/pid /proc/<pid>/ns/net    # compare inode numbers across processes

# what's this pod's actual resource ceiling, from the kernel's point of view?
crictl inspect <container-id> | grep -A5 linux
cat /sys/fs/cgroup/kubepods.slice/.../memory.max

# container was OOM-killed — confirm and see by how much it exceeded
kubectl describe pod <pod> | grep -A3 "Last State"
cat /sys/fs/cgroup/.../memory.events

# rootless podman: what's this container's UID really mapped to on the host?
podman unshare cat /proc/self/uid_map

Next: Part 7 — Security Hardening, covering the layers runc applies last: capabilities, seccomp, and mandatory access control.