Linux for DevOps, Part 7: Security Hardening

Part 6 ended with runc applying capabilities, seccomp, and MAC policy right before execve(). This post covers each in depth — these three layers are what turn “isolated by namespaces” into “actually hard to break out of.”

Layered Defense, in Order

graph TD
    P["Container process
about to call a syscall"]
    P --> NS["Namespaces
(Part 6)
what it can see"]
    NS --> CG["cgroups
(Part 6)
how much it can use"]
    CG --> CAP["Capabilities
which root powers it has"]
    CAP --> SC["seccomp
which syscalls are allowed at all"]
    SC --> LSM["SELinux / AppArmor
what files/sockets it can touch"]
    LSM --> OK["Syscall executes"]

    style CAP fill:#1e3a5f,color:#7ec8e3,stroke:#2d6a9f
    style SC fill:#1e3a5f,color:#7ec8e3,stroke:#2d6a9f
    style LSM fill:#1e3a5f,color:#7ec8e3,stroke:#2d6a9f

Each layer is independent and enforced by the kernel — bypassing one (say, a capability grant) still leaves seccomp and MAC in place. This is defense in depth, not one big on/off switch.

Capabilities: Splitting Up Root

Traditionally, a process either had UID 0 (root — bypasses essentially all permission checks) or it didn’t. Capabilities split root’s power into roughly 40 independent bits, so a process can get exactly the privileged operations it needs.

The Four Capability Sets

A process doesn’t just “have” a capability — it tracks four separate sets, which is where most confusion comes from:

capsh --print                              # full breakdown of the current process's sets
getpcaps <pid>                             # capabilities of a running process
getcap /usr/bin/ping                       # file-attached capabilities
setcap cap_net_raw+ep /usr/local/bin/mytool  # grant a capability to a binary directly, no setuid needed

setcap on a binary is the modern, narrower replacement for setuid root (Part 2) — ping needs CAP_NET_RAW to open a raw socket; giving it that one capability via file attributes is strictly safer than making it setuid-root.

Dropping Capabilities in Containers

Docker drops a default set of dangerous capabilities and grants a minimal safe set (CAP_CHOWN, CAP_NET_BIND_SERVICE, CAP_SETUID, etc. — about 14 by default, far short of the full ~40). Kubernetes lets you go further:

securityContext:
  capabilities:
    drop: ["ALL"]
    add: ["NET_BIND_SERVICE"]   # only if this specific container needs it
  allowPrivilegeEscalation: false
  runAsNonRoot: true

drop: ["ALL"] then adding back only what’s proven necessary is the correct default posture — start from zero, not from Docker’s default set.

seccomp: Filtering Syscalls Themselves

Capabilities gate what a syscall is allowed to do. seccomp gates whether the syscall can be called at all, independent of privilege. It’s a BPF program the kernel runs on every syscall entry, deciding allow, deny, or kill.

docker run --rm --security-opt seccomp=unconfined alpine sh   # disable filtering (don't do this in prod)
docker run --rm --security-opt seccomp=/path/to/profile.json alpine sh

Docker’s default seccomp profile blocks roughly 44 syscalls out of ~300+ that exist — things almost no application legitimately needs: mount(), reboot(), swapon(), kernel module loading, and several obscure/legacy syscalls with a history of kernel vulnerabilities (clone with certain flag combinations, ptrace variants). A minimal custom profile that allows only the syscalls your specific binary actually makes (traceable with strace -c under representative load) is far stronger than the generic default, at the cost of maintenance burden.

securityContext:
  seccompProfile:
    type: RuntimeDefault    # use the container runtime's default profile
    # or: type: Localhost, localhostProfile: profiles/my-app.json

Mandatory Access Control: SELinux vs AppArmor

Discretionary Access Control (DAC) is the owner/group/other model from Part 2 — the file’s owner decides who gets access. Mandatory Access Control adds a system-wide policy that even root cannot override without explicitly changing the policy itself.

# SELinux
getenforce                            # Enforcing / Permissive / Disabled
ls -Z /var/www/html                   # show SELinux context on files
ps -eZ | grep nginx                   # context a running process has
semanage fcontext -a -t httpd_sys_content_t "/srv/web(/.*)?"
restorecon -Rv /srv/web
audit2allow -a                        # turn denials in the audit log into a candidate policy

# AppArmor
aa-status                             # loaded profiles and their mode (enforce/complain)
aa-complain /etc/apparmor.d/usr.sbin.nginx   # switch a profile to log-only, for testing
journalctl -k | grep -i apparmor       # denial log

A container labeled with the wrong SELinux type is the classic “works when I docker run it locally (SELinux disabled or permissive), fails on the RHEL/OpenShift node (enforcing)” bug — the fix is almost always restorecon on a bind-mounted volume, or :z/:Z on the volume flag to have the runtime relabel it automatically.

docker run -v /host/data:/data:Z myimage   # :Z relabels the volume privately for this container

Mapping to Kubernetes Pod Security Standards

graph LR
    PRIV["Privileged
(no restrictions)"] --> BASE["Baseline
(blocks known privilege escalations)"]
    BASE --> REST["Restricted
(current best practice)"]

    style REST fill:#1e3a1e,color:#7ec87e,stroke:#2d6a2d

The restricted Pod Security Standard is essentially a checklist assembled from everything above:

securityContext:
  runAsNonRoot: true
  runAsUser: 10000
  allowPrivilegeEscalation: false
  capabilities:
    drop: ["ALL"]
  seccompProfile:
    type: RuntimeDefault
  readOnlyRootFilesystem: true

allowPrivilegeEscalation: false sets the kernel’s no_new_privs bit — it stops a process from gaining capabilities it didn’t start with, even via a setuid binary it executes. Combined with runAsNonRoot, dropped capabilities, and a read-only root filesystem, this closes off nearly every practical container escape path that doesn’t rely on an actual kernel vulnerability.

Practical Checklist

# what's actually blocking this container from doing X?
kubectl logs <pod>                                   # app-level error first
dmesg | tail -30                                      # seccomp kills show up here (SIGSYS)
journalctl -k | grep -i -E "avc|apparmor"             # SELinux/AppArmor denials

# audit a running container's effective privilege
docker inspect --format '{{.HostConfig.CapAdd}} {{.HostConfig.Privileged}}' <container>
crictl inspect <id> | grep -A20 securityContext

# find over-privileged pods across a cluster
kubectl get pods -A -o json | jq '.items[] | select(.spec.containers[].securityContext.privileged==true) | .metadata.name'

Next: Part 8 — Performance & eBPF Observability, the final part: finding why a system is slow, using the same kernel hook points that seccomp and cgroups rely on.