Linux for DevOps, Part 4: Networking Deep Dive

Part 1 mentioned network namespaces and iptables in passing. This post builds the full picture: how a namespace becomes an isolated network, how packets actually traverse the kernel’s netfilter hooks, and how that machinery becomes a Kubernetes Service.

Network Namespaces: An Isolated Stack

A network namespace is a completely separate instance of the network stack — its own interfaces, routing table, iptables rules, and /proc/net. Two processes in different net namespaces can both bind to port 80 without conflict.

ip netns add test-ns
ip netns exec test-ns ip addr        # empty — only loopback, and it's down
ip netns exec test-ns ip link set lo up

A brand-new namespace has no connectivity at all. To give it any, you need a veth pair — a virtual Ethernet cable with one end in each namespace.

graph LR
    subgraph "Host (root) namespace"
    br["bridge: cni0 / docker0"]
    veth1["veth-host-a"]
    veth2["veth-host-b"]
    end
    subgraph "Container A namespace"
    eth1["eth0"]
    end
    subgraph "Container B namespace"
    eth2["eth0"]
    end
    br --- veth1
    br --- veth2
    veth1 <-.->|virtual wire| eth1
    veth2 <-.->|virtual wire| eth2

ip link add veth-host-a type veth peer name eth0 netns test-ns
ip link set veth-host-a master cni0    # attach host end to the bridge
ip netns exec test-ns ip addr add 10.244.1.5/24 dev eth0
ip netns exec test-ns ip link set eth0 up

This — a veth pair per pod, all attached to a bridge — is exactly what every CNI plugin (Calico, Flannel, Cilium in non-eBPF mode) does when a pod is scheduled. kubectl exec <pod> -- ip addr shows you the container-side eth0; ip link on the node shows you the host-side veth peer.

The netfilter Hook Chain

Every packet transiting the kernel’s network stack passes through five fixed points where netfilter can intercept it. This is the actual substrate that iptables, nftables, and most CNI network policies are built on.

flowchart LR
    IN["Packet arrives
on NIC"] --> PRE[PREROUTING]
    PRE --> ROUTE{routing
decision}
    ROUTE -->|destined for
local process| IN2[INPUT] --> LOCAL["Local process
(socket)"]
    ROUTE -->|destined
elsewhere| FWD[FORWARD] --> POST[POSTROUTING] --> OUT["Packet leaves
via NIC"]
    LOCAL --> OUT2[OUTPUT] --> POST

    style PRE fill:#1e3a5f,color:#7ec8e3,stroke:#2d6a9f
    style IN2 fill:#1e3a5f,color:#7ec8e3,stroke:#2d6a9f
    style FWD fill:#1e3a5f,color:#7ec8e3,stroke:#2d6a9f
    style OUT2 fill:#1e3a5f,color:#7ec8e3,stroke:#2d6a9f
    style POST fill:#1e3a5f,color:#7ec8e3,stroke:#2d6a9f

iptables -t nat -L PREROUTING -n -v --line-numbers
iptables -t filter -L FORWARD -n -v
conntrack -L | head           # connection tracking table — NAT needs this to reverse-translate replies

DNAT vs SNAT, concretely: when you curl a NodePort service, PREROUTING DNATs the destination from <node-ip>:30080 to <pod-ip>:8080 before routing. When a pod calls out to the internet, POSTROUTING SNATs (masquerades) the pod’s source IP to the node’s IP, because the pod’s private IP isn’t routable outside the cluster. conntrack remembers both translations so return traffic gets un-translated correctly — this table filling up (conntrack -L | wc -l approaching sysctl net.netfilter.nf_conntrack_max) is a classic cause of mysterious connection drops under load.

iptables and nftables

iptables organizes rules into tables (filter, nat, mangle, raw) each containing chains that map to the hooks above. Rules are evaluated top to bottom; the first match wins.

iptables -t filter -A INPUT -p tcp --dport 22 -j ACCEPT
iptables -t filter -A INPUT -j DROP
iptables -t nat -A PREROUTING -p tcp --dport 8080 -j DNAT --to-destination 10.244.1.5:80

nftables is the modern replacement — one syntax for all address families, atomic rule replacement (no window where rules are half-applied), and better performance at scale. Most distros now translate iptables commands to nftables rules under the hood via iptables-nft.

nft list ruleset                    # dump everything, all tables and chains
nft add rule inet filter input tcp dport 22 accept

How this maps to Kubernetes: kube-proxy in iptables mode (the default) watches the API server for Service and Endpoints changes and rewrites DNAT rules accordingly — every ClusterIP is just a DNAT target rotated across pod IPs via probabilistic matching (statistic mode random). In IPVS mode it uses the kernel’s IP Virtual Server instead, which scales better with rule count but the concept is identical: intercept traffic to a virtual IP, rewrite to a real pod IP.

DNS Resolution

cat /etc/resolv.conf         # nameserver, search domains, options
cat /etc/nsswitch.conf       # resolution order: files, dns, mdns...
getent hosts myservice       # resolve using the full nsswitch chain, not just DNS

In Kubernetes, the kubelet writes a pod-specific /etc/resolv.conf pointing at CoreDNS’s ClusterIP, with search domains for the pod’s namespace so curl myservice resolves without a fully-qualified name. dnsPolicy: ClusterFirst (the default) is what triggers this; dnsPolicy: Default inherits the node’s own resolv.conf instead — a common cause of “works on the node, fails in the pod” DNS bugs.

Troubleshooting Toolkit

ip addr                        # interfaces and assigned IPs
ip route get 10.244.2.9        # what route/interface would this destination use?
ss -tlnp                       # listening TCP sockets with owning PID
ss -tnp state established      # active connections
tcpdump -ni eth0 port 443      # capture on the wire
dig +short myservice.default.svc.cluster.local
nsenter -t <pid> -n ip addr    # inspect another process's network namespace without exec-ing into it

nsenter -t <pid> -n is the debugging trick worth remembering: given any PID, you can jump into its network namespace from the host without needing a shell inside the container — useful when the container image has no shell at all (distroless, scratch-based images).

Next: Part 5 — Storage & Disk I/O, covering the layer below networking: how bytes actually get to and from disk, and what backs a Kubernetes PersistentVolume.