Linux Process Diagnostics: top, /proc, Network Listeners, CPU Affinity, and Memory

A collection of Linux system diagnostics techniques you’ll reach for repeatedly when troubleshooting production systems.

top — Reading and Navigating

Launch with top. The summary header shows system-wide stats; the task list below shows per-process info.

Key fields in the task list

Column	Meaning
PID	Process ID
USER	Owner
PR	Kernel priority (lower = higher priority)
NI	Nice value (-20 to +19)
VIRT	Total virtual memory claimed
RES	Resident (physical) memory in use
SHR	Shared memory
S	State: R=running, S=sleeping, D=uninterruptible sleep, Z=zombie
%CPU	CPU usage since last update
%MEM	Physical memory as % of total RAM
TIME+	Total CPU time consumed
COMMAND	Process name (truncated by default)

Interactive commands inside top

Key	Action
M	Sort by memory
P	Sort by CPU (default)
T	Sort by CPU time
k	Kill a process (prompts for PID)
r	Renice a process
1	Toggle per-CPU breakdown
H	Toggle showing threads
u	Filter by user
q	Quit

Show full command line in top

By default top truncates the COMMAND column to just the binary name. Press c to toggle the full command line (with arguments) for all processes. This is invaluable when multiple instances of the same binary are running with different flags.

Alternatively, launch with the flag:

top -c

To make it permanent, save your preferences with W (writes to ~/.config/procps/toprc).

---

Diagnosing High-CPU Processes

1. Identify the offending PID

top -b -n 1 | head -20          # one-shot batch output
# or
ps aux --sort=-%cpu | head -10   # sorted snapshot

2. Drill into threads

A single PID at 400% CPU means it’s using 4 cores. See which threads are responsible:

top -H -p <PID>     # show threads for one process
# or
ps -T -p <PID>      # list threads with their TIDs

3. Sample what the process is actually doing

# Stack trace via /proc (no external tools needed)
cat /proc/<PID>/wchan          # kernel function the process is waiting in
cat /proc/<PID>/stack          # kernel stack (requires root)

# perf — if installed
perf top -p <PID>              # live symbol-level CPU breakdown
perf record -g -p <PID> sleep 10 && perf report

# strace — what syscalls is it making?
strace -p <PID> -c             # count syscalls for 10s then Ctrl-C

---

/proc — Finding Where Code Is Running

/proc/<PID>/ is a virtual filesystem exposing every detail of a running process.

Maps: what code is loaded where

cat /proc/<PID>/maps

Each line is a memory region:

address           perms offset  dev   inode   pathname
7f3a2c000000-7f3a2c021000 r-xp 00000000 fd:01 1234567 /usr/lib/libc.so.6

• r-xp — readable, executable, private (typical for code segments)
• rw-p — readable, writable, private (heap/stack)
• Anonymous regions (no path) are heap allocations or JIT-compiled code.

smaps: per-region memory stats

cat /proc/<PID>/smaps           # detailed breakdown per region
cat /proc/<PID>/smaps_rollup    # totals (much shorter)

Status: quick process summary

cat /proc/<PID>/status

Shows name, state, PID, parent PID, thread count, memory stats (VmRSS, VmSwap), and capability sets.

fd: open file descriptors

ls -la /proc/<PID>/fd           # what files/sockets are open
ls -la /proc/<PID>/fd | wc -l   # total fd count

Symlinks point to the actual file or socket:[inode]. Cross-reference socket inodes with ss -tlnp or /proc/net/tcp.

cmdline: full command as launched

cat /proc/<PID>/cmdline | tr '\0' ' '   # args are null-separated

environ: environment variables

cat /proc/<PID>/environ | tr '\0' '\n'  # null-separated

cwd and exe: where is it running from?

ls -la /proc/<PID>/cwd    # current working directory
ls -la /proc/<PID>/exe    # absolute path to the binary

exe is especially useful when a binary has been deleted from disk but is still running — you’ll see (deleted) appended.

schedstat and stat: scheduling info

cat /proc/<PID>/schedstat   # time spent running, waiting, # of switches
cat /proc/<PID>/stat        # raw stats including CPU time per thread

---

Where Is a Process Stuck? Kernel Space vs User Space

When a process is blocked or behaving unexpectedly, the first question is: is it stuck in your code (user space), waiting on the kernel (kernel space), or genuinely just busy on CPU?

The process state is your first clue

From top or ps, the S column (state) tells you a lot:

State	Meaning
R	Running or runnable — actively on CPU or ready to be scheduled
S	Interruptible sleep — waiting on an event (I/O, timer, signal). Normal for idle processes.
D	Uninterruptible sleep — blocked inside the kernel, usually on disk I/O. Cannot be killed.
Z	Zombie — process exited but parent hasn’t called wait()
T	Stopped (SIGSTOP or being traced by a debugger)

D state is the most actionable: a process stuck in D for more than a few seconds means something below it (disk, NFS mount, kernel driver) is not responding.

wchan — what kernel function is it waiting in?

cat /proc/<PID>/wchan

This outputs a single kernel function name — the function the process is currently sleeping inside. Examples:

wchan output	Likely cause
pipe_wait	Blocked reading from a pipe
futex_wait	Waiting on a mutex/lock (very common in multithreaded apps)
do_sys_poll / ep_poll	Waiting in poll()/epoll() — normal for event-driven servers
wait_woken	Sleeping in a generic wait queue
io_schedule	Blocked on disk I/O
nfs_file_write	Stuck on NFS
0 / (null)	Running in user space — not currently in the kernel

When wchan shows 0 or nothing, the process is executing user-space code, not sleeping in a syscall.

/proc//stack — full kernel stack trace

cat /proc/<PID>/stack    # requires root

Shows the full kernel call stack, not just the leaf function. Useful when wchan alone doesn’t give enough context:

[<0>] futex_wait_queue_me+0xd4/0x130
[<0>] futex_wait+0x17e/0x2c0
[<0>] do_futex+0x1a8/0x1d0
[<0>] __x64_sys_futex+0x13b/0x1e0
[<0>] do_syscall_64+0x5b/0x90

This tells you the process called futex() and is waiting on a lock, not just that it’s somewhere in futex.

strace — which syscall is it blocked in right now?

strace -p <PID>           # attach and show syscalls in real time
strace -p <PID> -e trace=read,write,futex   # filter to specific calls

If the process is truly stuck, strace will show one line and hang there — that’s the blocking syscall. If lines are flying by rapidly, it’s spinning (likely a busy-loop bug or legitimate CPU work).

The distinction:

• One syscall, no movement → blocked in kernel space waiting on something external
• Rapid syscall stream → active work, possibly inefficient (e.g., calling read() in a tight loop)
• No output at all from strace → running in user space (no syscalls happening)

perf — where is CPU time actually spent?

When a process is consuming CPU (not blocked), perf shows you where in the code:

perf top -p <PID>                          # live symbol breakdown
perf record -g -p <PID> sleep 10          # sample for 10s with call graphs
perf report                               # interactive viewer

perf top output has a [k] prefix for kernel symbols and [.] for user-space symbols:

Overhead  Shared Object       Symbol
  42.3%   [kernel]            [k] copy_user_generic_string
  31.1%   myapp               [.] process_records
   8.7%   libc.so.6           [.] malloc

Heavy [k] overhead means your process is spending most time in kernel code (system calls, page faults, etc.). Heavy [.] overhead is user-space — look at your own code or libraries.

Putting it together: a diagnostic flow

Process behavior            → What to check
────────────────────────────────────────────────────────
High %CPU, R state          → perf top -p <PID>  (where in code?)
Low %CPU, S state, healthy  → Normal. Nothing to do.
Low %CPU, D state, stuck    → cat /proc/<PID>/wchan  (what kernel fn?)
                              cat /proc/<PID>/stack   (full trace, root)
                              dmesg | tail            (kernel errors?)
Low %CPU, S state, slow     → strace -p <PID>  (which syscall is slow?)
Threads fighting each other → cat /proc/<PID>/wchan showing futex_wait
                              perf record + report to find lock contention

---

Kernel vs User Space Memory

A process’s memory is split between memory it directly manages (user space) and memory the kernel uses on its behalf (kernel space).

User space memory in /proc//status

cat /proc/<PID>/status | grep -i vm

Key fields:

Field	Meaning
VmPeak	Peak virtual memory size ever used
VmSize	Current virtual memory size (total address space claimed)
VmRSS	Resident Set Size — physical RAM currently in use
VmAnon	Anonymous RSS: heap, stack, mmap’d anonymous regions
VmFile	File-backed RSS: code, shared libs, mmap’d files
VmShr	Shared memory (shared with other processes)
VmData	Size of data + stack segments
VmStk	Stack size
VmExe	Text (code) segment size
VmLib	Shared library code size
VmSwap	Swapped-out memory

VmRSS = VmAnon + VmFile roughly. A process with high VmSize but low VmRSS has reserved virtual address space it hasn’t touched yet — not a problem. High VmRSS or VmSwap is where to look for actual pressure.

smaps_rollup — the clearest single-process summary

cat /proc/<PID>/smaps_rollup

Example output:

Rss:              142336 kB    ← total physical RAM in use
Pss:               98412 kB    ← proportional share (shared pages divided among users)
Shared_Clean:      12288 kB    ← shared file-backed pages, unmodified (e.g. shared libs)
Shared_Dirty:          0 kB    ← shared pages written to
Private_Clean:     18432 kB    ← private file-backed pages (mmap'd files, read-only)
Private_Dirty:    111616 kB    ← private anonymous pages (heap, stack, written data)
Swap:                  0 kB

PSS (Proportional Set Size) is the most accurate memory cost per process. If a shared library is loaded by 10 processes, each one counts 1/10th of that library’s pages. RSS counts the full shared library for each, so the sum of all RSS values exceeds actual RAM used.

System-wide kernel vs user space memory

cat /proc/meminfo

Key fields to understand:

MemTotal:       32768000 kB   ← total physical RAM
MemFree:         1024000 kB   ← completely unused RAM
MemAvailable:   12000000 kB   ← estimated RAM available without swapping (more useful than MemFree)
Buffers:          512000 kB   ← kernel block I/O buffers (disk metadata cache)
Cached:          8192000 kB   ← page cache (file contents cached by kernel)
SwapTotal:       4096000 kB
SwapFree:        4096000 kB
Slab:             640000 kB   ← kernel slab allocator total
  SReclaimable:   512000 kB   ← slab memory that can be freed under pressure
  SUnreclaim:     128000 kB   ← slab memory the kernel will NOT give back
KernelStack:       32000 kB   ← memory used for kernel thread stacks
PageTables:        16000 kB   ← memory used for page table entries
AnonPages:       9000000 kB   ← user-space anonymous pages (heap, stack)
Mapped:          1200000 kB   ← file-backed pages currently mapped into processes

Approximate breakdown:

• User space RSS ≈ AnonPages + Mapped
• Kernel overhead ≈ Slab + KernelStack + PageTables
• Reclaimable kernel cache ≈ Buffers + Cached + SReclaimable — the kernel will evict these under memory pressure, so they’re not “lost”

Watching kernel slab usage

The kernel allocates its own internal objects (dentries, inodes, socket buffers, etc.) from the slab allocator:

cat /proc/slabinfo | sort -k3 -rn | head -20    # sort by number of objects
slabtop                                          # live top-like view (if installed)

High dentry counts can indicate a directory traversal leak. High kmalloc-* entries can indicate a driver or subsystem accumulating memory.

Identifying kernel memory growth over time

# Watch slab total
watch -n2 "grep Slab /proc/meminfo"

# Watch the biggest slab consumers
watch -n2 "cat /proc/slabinfo | awk 'NR>2{print \$3, \$1}' | sort -rn | head -10"

If SUnreclaim in /proc/meminfo grows over time and never drops, you likely have a kernel memory leak (driver bug, BPF program, module issue).

Mapping regions: user vs kernel in a process

cat /proc/<PID>/maps | awk '{print $6}' | sort | uniq -c | sort -rn

The kernel’s VDSO and vvar pages appear in every process’s map but are kernel-owned:

7fff12345000-7fff12346000 r-xp 00000000 00:00 0   [vdso]
7fff12344000-7fff12345000 r--p 00000000 00:00 0   [vvar]

• [vdso] — Virtual Dynamic Shared Object: a kernel-provided page mapped into every process so certain syscalls (gettimeofday, clock_gettime) can run without a full context switch into the kernel.
• [vvar] — kernel variables the vdso reads (current time, etc.).
• [heap], [stack] — user-space anonymous regions.
• [stack:<tid>] — per-thread stacks in multithreaded processes.

---

Viewing TCP/UDP Listeners

ss (preferred, replaces netstat)

ss -tlnp    # TCP listeners
ss -ulnp    # UDP listeners
ss -tlnp4   # IPv4 only
ss -tlnp6   # IPv6 only

Flag breakdown:

• -t / -u — TCP / UDP
• -l — listening sockets only
• -n — numeric (don’t resolve hostnames/ports)
• -p — show process name and PID

Example output:

Netid  State   Recv-Q Send-Q  Local Address:Port   Peer Address:Port  Process
tcp    LISTEN  0      128     0.0.0.0:22            0.0.0.0:*          users:(("sshd",pid=1234,fd=3))
tcp    LISTEN  0      4096    127.0.0.1:5432        0.0.0.0:*          users:(("postgres",pid=5678,fd=7))

Show all connections (not just listeners)

ss -tnp     # established TCP connections with process info
ss -s       # summary statistics

netstat (legacy, still widely available)

netstat -tlnp   # TCP listeners with PID
netstat -ulnp   # UDP listeners with PID
netstat -an | grep LISTEN

/proc/net directly

cat /proc/net/tcp     # hex-encoded TCP sockets (local_address, rem_address, state, inode)
cat /proc/net/tcp6    # IPv6
cat /proc/net/udp
cat /proc/net/udp6

The local address is in hex_ip:hex_port format (little-endian). Useful in containers where ss may not be available. The inode can be matched against /proc/<PID>/fd to identify the owning process.

---

taskset — CPU Affinity

taskset pins a process to specific CPU cores. This is about CPU affinity (which cores can run this process), not scheduling priority.

Check current affinity

taskset -p <PID>                    # shows affinity mask in hex
taskset -cp <PID>                   # shows affinity as a core list

Pin a running process to cores

taskset -cp 0,1 <PID>               # restrict to cores 0 and 1
taskset -cp 2-5 <PID>               # restrict to cores 2 through 5
taskset -p 0x3 <PID>                # hex mask: cores 0 and 1 (bits 0 and 1 set)

Launch a new process with affinity

taskset -c 0 ./my_program           # run on core 0 only
taskset -c 3,4,5 python3 heavy.py   # run on cores 3-5

When to use taskset

• Isolate a latency-sensitive process to dedicated cores
• Prevent a noisy neighbor from competing with a critical service
• Benchmark reproducibly by fixing to a single core
• Work around NUMA effects on multi-socket systems

Scheduling priority (separate from affinity)

taskset controls where a process runs, not how urgently it’s scheduled. For priority:

nice -n 10 ./program          # launch with lower priority (nice 10)
renice -n -5 -p <PID>         # raise priority of running process (needs root for negative values)
chrt -r -p 50 <PID>           # set real-time FIFO scheduling policy

---

Other Useful /proc Entries

System-wide

Path	Contents
/proc/cpuinfo	Per-core info: model, MHz, cache size, flags
/proc/meminfo	Detailed memory breakdown: MemFree, Buffers, Cached, SwapUsed
/proc/loadavg	1/5/15-min load averages, running/total threads, last PID
/proc/uptime	Seconds since boot, idle time
/proc/version	Kernel version string
/proc/cmdline	Kernel boot parameters
/proc/mounts	Currently mounted filesystems
/proc/filesystems	Filesystems supported by the kernel
/proc/net/dev	Per-interface RX/TX byte and packet counters
/proc/net/arp	ARP table
/proc/diskstats	Per-disk I/O stats (reads, writes, time spent)
/proc/sys/	Live kernel tunable parameters (sysctl)

Useful sysctl paths under /proc/sys

cat /proc/sys/net/ipv4/ip_forward          # IP forwarding enabled?
cat /proc/sys/vm/swappiness                # swap aggressiveness (0-100)
cat /proc/sys/fs/file-max                  # system-wide fd limit
cat /proc/sys/kernel/pid_max              # maximum PID value
echo 1 > /proc/sys/net/ipv4/ip_forward    # enable IP forwarding (root)

/proc/interrupts — IRQ distribution

cat /proc/interrupts
watch -n1 cat /proc/interrupts    # live view

Shows which CPUs are handling which hardware interrupts. High interrupt counts on a single core can indicate IRQ affinity problems.

/proc/buddyinfo and /proc/slabinfo

cat /proc/buddyinfo    # buddy allocator free pages per order (memory fragmentation)
cat /proc/slabinfo     # kernel slab allocator stats (cache names, object counts)

Kernel threads

ps -ef | grep '\[.*\]'     # kernel threads have names in brackets
cat /proc/2/status         # PID 2 is kthreadd, parent of all kernel threads

---

Quick Reference Cheatsheet

# High CPU PID
ps aux --sort=-%cpu | head -5

# Full command line for PID
cat /proc/<PID>/cmdline | tr '\0' ' '
# or in top: press c

# What binary is running
ls -la /proc/<PID>/exe

# Open files/sockets
ls -la /proc/<PID>/fd

# TCP/UDP listeners with process names
ss -tlnp && ss -ulnp

# Pin process to CPU cores
taskset -cp 0,1 <PID>

# Thread-level CPU usage
top -H -p <PID>

# Memory regions (what code is loaded)
cat /proc/<PID>/maps | grep 'r-xp'

# System memory summary
cat /proc/meminfo | grep -E 'MemTotal|MemFree|MemAvailable|Buffers|Cached|Slab|AnonPages'

# Network interface stats
cat /proc/net/dev

# Where is a process stuck?
cat /proc/<PID>/wchan              # kernel function it's waiting in
cat /proc/<PID>/stack              # full kernel stack (root)
strace -p <PID>                    # which syscall is blocking it

# User-space memory breakdown for a process
cat /proc/<PID>/smaps_rollup       # RSS, PSS, private/shared split
cat /proc/<PID>/status | grep Vm   # VmRSS, VmSwap, VmAnon

# Kernel memory consumers
grep -E 'Slab|SReclaimable|SUnreclaim|KernelStack|PageTables' /proc/meminfo