Linux for DevOps, Part 2: Users, Permissions & the Filesystem

Every file operation in Linux passes through one question: does this process have permission? This post covers how Linux answers that — the user/group model, permission bits, special bits, ACLs, and the standard directory layout every distro follows. This builds on Part 1, which covered kernel space vs user space.

The Multi-User Model

Linux was built for many users sharing one machine. Every process runs as a user, and every file is owned by a user and a group.

id
# uid=1000(abin) gid=1000(abin) groups=1000(abin),27(sudo),999(docker)

Identity is numeric at the kernel level — UID and GID are just integers. /etc/passwd maps a UID to a username; /etc/group maps a GID to a group name. The kernel never looks at names, only numbers.

cat /etc/passwd | head -3
# root:x:0:0:root:/root:/bin/bash
# daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
# abin:x:1000:1000:Abin M:/home/abin:/bin/bash
#     ^field2 is a placeholder — real hash lives in /etc/shadow

UID 0 is root — the kernel grants it a bypass on most permission checks (this is what capabilities in Part 1 later split into fine-grained pieces). Every other UID is subject to full permission checking.

graph LR
    P["Process
(runs as UID:GID)"] -->|open/write/exec| F["File
(owned by UID:GID, mode bits)"]
    F --> K{Kernel
permission check}
    K -->|allow| OK["Operation proceeds"]
    K -->|deny| ERR["EACCES / EPERM"]

    style K fill:#1e3a5f,color:#7ec8e3,stroke:#2d6a9f

Permission Bits

Every file has three permission triads — owner, group, others — each with read/write/execute.

ls -l /usr/bin/podman
# -rwxr-xr-x 1 root root 12345678 Jun  1 10:00 /usr/bin/podman
#  ^^^ ^^^ ^^^
#  own grp oth

A directory with r but not x is nearly useless — you can list names but not stat() or open anything in it. This trips people up constantly.

chmod 755 script.sh        # rwxr-xr-x — owner full, others read+execute
chmod u+x script.sh        # add execute for owner only
chmod g-w,o-rwx secret.txt # tighten group and others
umask 022                  # default mask: new files get 644, dirs get 755

umask subtracts from the maximum (666 for files, 777 for dirs) at creation time. It’s set per-shell or per-process — services started by systemd get their umask from the unit’s UMask= directive, not your shell.

Special Bits: setuid, setgid, sticky

Three more bits sit above the standard rwx triads.

ls -l /usr/bin/passwd
# -rwsr-xr-x 1 root root ... /usr/bin/passwd

passwd needs to write /etc/shadow, which only root can touch. setuid lets any user run it as root for that program only — the classic controlled-privilege-escalation pattern that predates sudo. /tmp has the sticky bit (drwxrwxrwt) so any user can create files there but not delete someone else’s.

chmod u+s binary      # setuid
chmod g+s /shared/dir # setgid — new files inherit the dir's group
chmod +t /tmp/shared  # sticky

setuid binaries are a common attack surface — find / -perm -4000 -type f 2>/dev/null lists every setuid binary on a box, worth auditing on any host you harden.

Access Control Lists (ACLs)

Owner/group/other gives you exactly one owner and one group per file. Real-world sharing needs more granularity — ACLs extend permissions to arbitrary additional users or groups.

setfacl -m u:deploy:rw /var/log/app.log   # grant user 'deploy' read+write
getfacl /var/log/app.log
# user::rw-
# user:deploy:rw-
# group::r--
# mask::rw-
# other::r--

ls -l shows a + after the mode bits when ACLs are present. Kubernetes and container tooling rarely touch ACLs directly, but you’ll meet them on shared NFS mounts and multi-tenant build servers.

Everything Is a File — Inodes and Links

Linux stores file metadata (owner, permissions, timestamps, block pointers) in an inode, separate from the filename. A directory is just a table mapping names to inode numbers.

graph LR
    subgraph Directory Entry
    N1["/etc/hosts"]
    N2["/etc/hostname.bak"]
    end
    subgraph Inode Table
    I["inode 884201
mode, owner, size,
block pointers"]
    end
    N1 --> I
    N2 -.->|hard link, same inode| I

ls -i /etc/hosts        # show inode number
stat /etc/hosts          # full inode metadata

Hard links (ln a b) point two names at the same inode — indistinguishable from the original, sharing the same data blocks. Deleting one name just decrements the inode’s link count; data survives until the count hits zero.

Symbolic links (ln -s a b) are a separate inode whose data is a path string. Break the target and the symlink still exists but resolves to nothing (ls -l shows it in a different color when broken).

ln /etc/hosts /etc/hosts.hardlink       # same inode, same data
ln -s /etc/hosts /etc/hosts.symlink     # new inode containing the path "/etc/hosts"

Container images use hard links heavily (OverlayFS layer deduplication); Kubernetes ConfigMap/Secret volume mounts are implemented with symlinks under ..data for atomic updates — kubectl exec into any pod with a mounted ConfigMap and ls -la the mount to see it.

The Filesystem Hierarchy Standard (FHS)

Every mainstream distro agrees on a rough directory layout so tooling can rely on fixed paths.

graph TD
    ROOT["/"]
    ROOT --> BIN["/usr/bin
user commands"]
    ROOT --> ETC["/etc
system-wide config"]
    ROOT --> VAR["/var
variable data: logs, spool, cache"]
    ROOT --> HOME["/home
user home directories"]
    ROOT --> PROC["/proc
virtual — kernel/process state"]
    ROOT --> SYS["/sys
virtual — kernel/device objects"]
    ROOT --> TMP["/tmp
ephemeral, sticky bit"]
    ROOT --> OPT["/opt
third-party software"]
    VAR --> LOG["/var/log
application + system logs"]
    VAR --> LIB["/var/lib
persistent app state (databases, etc.)"]

/proc and /sys deserve a second look: they don’t hold real files. Reading /proc/loadavg asks the kernel to format live scheduler state on demand — nothing is stored on disk. This is why df never shows meaningful usage for proc or sysfs filesystem types.

Practical Checklist

# who owns what, and can this process actually read it?
ls -la /path/to/file
stat /path/to/file
getfacl /path/to/file          # any ACLs hiding beyond the mode bits?

# find dangerous setuid binaries
find / -xdev -perm -4000 -type f 2>/dev/null

# check what a container's entrypoint runs as
docker inspect --format '{{.Config.User}}' <image>

# fix "permission denied" on a mounted volume
# 1. check the UID the container process runs as
# 2. check the UID that owns the files on the host/volume
# 3. they must match, or use fsGroup / an initContainer chown

The single most common Kubernetes permission bug: a container image runs as a non-root UID (good security practice) but the mounted PersistentVolume was formatted and owned by root. Kubernetes’ securityContext.fsGroup exists specifically to fix this by recursively chowning the volume’s group ownership on mount.

Next: Part 3 — Processes, Signals & systemd, where the same UID/GID model determines what a running process is allowed to touch.