Linux for DevOps, Part 3: Processes, Signals & systemd

Part 1 introduced the process scheduler at a high level. This post goes one level down: how a process is actually born, what state it can be in, how signals interrupt it, and how systemd — PID 1 on nearly every modern distro — keeps a fleet of them alive.

fork, exec, wait: How a Process Is Born

Linux has no “create process from scratch” syscall. Every process is created by copying an existing one.

sequenceDiagram
    participant Shell as bash (PID 100)
    participant Kernel
    participant Child as new process

    Shell->>Kernel: fork()
    Kernel->>Child: duplicate address space (copy-on-write)
    Kernel-->>Shell: return child PID
    Kernel-->>Child: return 0
    Child->>Kernel: execve("/usr/bin/curl", argv, envp)
    Kernel->>Child: replace memory image with curl binary
    Note over Child: now running curl, same PID
    Child-->>Shell: exit(0) on completion
    Shell->>Kernel: wait() / waitpid()
    Kernel-->>Shell: exit status, reap zombie

• fork() duplicates the calling process. Both copies continue running from the instruction right after fork(). The kernel uses copy-on-write — no memory is physically copied until one side writes to a page, so fork() is cheap even for large processes.
• execve() replaces the calling process’s memory image with a new program. Same PID, entirely different code. This is why ps shows bash becoming curl without a new PID appearing.
• wait()/waitpid() lets a parent collect a child’s exit status. Skip this and the child becomes a zombie — dead but still occupying a process table entry until the parent reaps it or the parent itself dies (at which point init/PID 1 adopts and reaps it).

Shells fork+exec for every external command. bash built-ins (cd, export) skip this entirely — they run in the shell’s own process, which is why cd in a subshell doesn’t change your shell’s working directory.

Process States

stateDiagram-v2
    [*] --> Runnable: fork()
    Runnable --> Running: scheduled on CPU
    Running --> Runnable: time slice ends (preempted)
    Running --> Sleeping: blocks on I/O, lock, or syscall
    Sleeping --> Runnable: event / data ready
    Running --> Zombie: exit()
    Zombie --> [*]: reaped by parent (wait())
    Running --> Stopped: SIGSTOP
    Stopped --> Runnable: SIGCONT

ps aux shows these as a single-letter STAT column: R running/runnable, S interruptible sleep, D uninterruptible sleep (usually stuck on disk I/O — cannot even be killed with SIGKILL until the I/O completes, a classic symptom of a dying NFS mount or failing disk), Z zombie, T stopped.

ps aux | awk '{print $8}' | sort | uniq -c    # count processes by state
ps aux | grep ' D '                            # find processes stuck in uninterruptible sleep

A host with many D-state processes has an I/O problem, not a CPU problem — no amount of kill -9 will fix it.

Signals

A signal is an asynchronous notification delivered to a process — by the kernel, by another process (kill), or by hardware (illegal instruction → SIGILL).

kill -TERM 1234           # graceful — process can catch and clean up
kill -9 1234               # SIGKILL — kernel removes it immediately, no cleanup runs
kill -l                    # list all signal names
trap 'echo caught SIGTERM; cleanup; exit 0' TERM   # bash: catch a signal

Why this matters in containers: the container runtime sends SIGTERM to PID 1 inside the container, waits terminationGracePeriodSeconds (Kubernetes) or --stop-timeout (Docker), then escalates to SIGKILL. If your entrypoint is a shell script wrapping the real binary, the shell — not your app — receives the signal, and shells do not forward signals to children by default. This is the single most common cause of pods that take the full grace period to terminate, or that drop connections mid-request on every rollout.

# Bad: shell is PID 1, SIGTERM goes to bash, curl never sees it
ENTRYPOINT ["sh", "-c", "myapp --config /etc/app.conf"]

# Good: exec form — myapp becomes PID 1 directly
ENTRYPOINT ["myapp", "--config", "/etc/app.conf"]

# Or if you must use a shell, forward the signal explicitly
ENTRYPOINT ["sh", "-c", "trap 'kill -TERM $PID' TERM; myapp & PID=$!; wait $PID"]

tini or dumb-init solve this generically — they run as PID 1, forward signals to the real child, and reap zombies, which matters if your app itself forks subprocesses.

systemd: Managing Processes at Scale

Running one binary is fork+exec. Running a fleet of services reliably — restart on crash, start in dependency order, capture logs, apply resource limits — is what systemd does as PID 1 on Debian, RHEL, Ubuntu, Fedora, and most cloud images.

Unit Types

Anatomy of a Service Unit

# /etc/systemd/system/myapp.service
[Unit]
Description=My Application
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
ExecStart=/usr/bin/myapp --config /etc/myapp/config.yaml
Restart=on-failure
RestartSec=5
User=myapp
Group=myapp
LimitNOFILE=65536
MemoryMax=512M
CPUQuota=200%

[Install]
WantedBy=multi-user.target

• Type= tells systemd how to know the service is “ready”: simple (default, ready as soon as ExecStart runs), forking (parent exits, daemon child continues), notify (service calls sd_notify(READY=1) — the most reliable for dependency ordering), oneshot (runs to completion, used for setup tasks).
• Restart=on-failure + RestartSec=5 is the crash-loop policy — systemd is your first line of self-healing, before Kubernetes ever gets involved.
• MemoryMax= / CPUQuota= are implemented via cgroups — every systemd service already gets its own cgroup, which is the same primitive containers use (see Part 1 and the deeper dive in Part 6 of this series).

Boot Sequence

Targets don’t “do” anything themselves — they’re synchronization points. multi-user.target is reached only once everything it Wants=/Requires= has started, and your service reaching WantedBy=multi-user.target is what makes it start at boot.

Operating systemd

systemctl status myapp                  # current state, recent log lines, cgroup
systemctl restart myapp
systemctl enable --now myapp             # enable at boot AND start now
systemctl list-units --failed            # anything that failed to start
systemctl list-dependencies myapp        # what it waits on
systemd-analyze blame                    # slowest units at boot
systemd-analyze critical-chain           # critical path of the boot
journalctl -u myapp -f                   # follow this service's logs
journalctl -u myapp --since "10 min ago" -p err   # recent errors only
journalctl -k                            # kernel ring buffer messages

journald captures stdout/stderr from every service unit automatically — no need for the application to open a log file. This is the same mechanism that lets docker logs and kubectl logs work: the container runtime captures the entrypoint’s stdout/stderr the same way.

Practical Checklist

# service won't start — where's it stuck?
systemctl status myapp
journalctl -u myapp -n 50 --no-pager

# is a "hung" process actually stuck on I/O?
ps aux | awk '$8 ~ /D/'
cat /proc/<pid>/stack        # kernel stack — shows what syscall it's blocked in

# container pod stuck "Terminating"
kubectl describe pod <pod>   # check terminationGracePeriodSeconds
# then check: does PID 1 in the container actually handle SIGTERM?

# find what's holding a file open before you can unmount
lsof +D /mnt/data
fuser -vm /mnt/data

Next: Part 4 — Networking Deep Dive, where namespaces and cgroups from this post and Part 1 come together to explain how packets actually move between containers.