Linux Freeze Recovery, Magic-SysRq, and OOM Kill Guide

# Linux Mint: Freeze Recovery, Magic SysRq, and OOM Kill Guide

This guide documents a practical playbook for recovering from “barely responding” system states on Linux, using Magic SysRq, quick non-interactive process inspection, and optional early userspace OOM helpers.

One very common source of your computer’s “freezes” or is “very sluggish”, is an “out of memory” (OOM) situation.

– Audience: Heavy users constantly running many applications, and Developers running local environments. People who sometimes hit CPU or memory spikes; or who have applications that have memory leaks that freeze the system.

– Scope: Kernel Magic-SysRq setup/usage, fast identification/kill of memory hogs, dmesg interpretation, and optional earlyoom. Disk swapfile and RAM compression, to reduce OOM.

– This setup: Linux Mint 21.2 Cinnamon, should work with many Linux types and versions.

## TL;DR

• Enable Magic SysRq fully or at least include OOM-kill.
• On Windows keyboards the Magic key is Alt; on OS/X keyboards the Magic key is Option but there is no SysRq/PrintScreen key (you can use a “for Windows” keyboard on Apple hardware running Linux, or map a key)
• During a freeze:
  • Try Alt+SysRq+f (invoke kernel OOM killer).
  • Or Alt+SysRq+k/e/i as last resort.
• If you can type in a TTY, pick and kill the top RSS process via `ps` (non-interactive, fast).
• Set up “systemd user service” or “wrapper script” to start your important programs to not be OOM-killed, or as “most likely to OOM-kill”.

## Enable Magic SysRq

The setting controls which SysRq actions are permitted.

– Current config file: `/etc/sysctl.d/10-magic-sysrq.conf`

– Bits of interest: 16 sync, 32 remount RO, 64 signals/oom-kill, 128 reboot/poweroff, 4 keyboard controls (SAK, unraw)

Less-used Bits: bit 2 (console loglevel) and bit 8 (debug dumps)

Recommended persistent values:

– Minimal with OOM-kill: `kernel.sysrq = 240` (128+64+32+16)

– Add keyboard controls (k/r): `kernel.sysrq = 244` (128+64+32+16+4)

– Enable everything: `kernel.sysrq = 1`

Apply and verify without reboot:

sudo sysctl --system
cat /proc/sys/kernel/sysrq

Quick temporary enable (until reboot):

echo 1 | sudo tee /proc/sys/kernel/sysrq

## Which keys make Magic-SysRq?

• PC/Windows‑layout keyboards on Linux: SysRq shares the Print Screen key. Hold Alt+PrintScreen, then press the action letter.
• Laptops: Often Alt+Fn+Print Screen+letter (model‑dependent).
• Apple keyboards used on Linux: Option acts as Alt, but there is no Print Screen/SysRq key. Use an external keyboard, map a key to SysRq, or trigger via `/proc`:
• If the screen blanks briefly, that’s normal; the kernel prints to a text virtual terminal/console. Return to your display VT (e.g., Ctrl+Alt+F1/F7 depending on your Linux distro and display manager).

# As root, equivalent to Alt+SysRq+f
echo f > /proc/sysrq-trigger

Confirm key handling by testing safe actions:

• – Alt+SysRq+h → prints help into the kernel log.
• – Alt+SysRq+m → memory info.
• – Alt+SysRq+l → NMI backtraces (may blank the screen briefly).

Note on SysRq mask values (244 vs 252/254):

• – With `kernel.sysrq = 244` (4+16+32+64+128):
  • – Enabled: k, r, s, u, e, i, f, b/o.
  • – Disabled: m, t, w, l, p, q, z, etc. (debug-dump group), and loglevel 0–9.
• – Result: `Alt+SysRq+h` works (help), but `Alt+SysRq+m` (Show Memory) is blocked → “This sysrq operation is disabled.”
• – If you want debug dumps (m/t/w/l), add bit 8:
  • – Set `kernel.sysrq = 252` (244 + 8) to enable dumps while keeping loglevel control off.
• – If you also want SysRq loglevel control (0–9), add bit 2 as well:
  • – Set `kernel.sysrq = 254` (244 + 8 + 2). This enables almost everything except the rarely used RT nicing bit.

Quick runtime test (until reboot): `echo 254 | sudo tee /proc/sys/kernel/sysrq`

Set so Persists across reboot: edit `/etc/sysctl.d/10-magic-sysrq.conf` and find the line with kernel.sysrq, edit it to → `kernel.sysrq = 252` or `254`, and save it. Then `sudo sysctl –system`.

## Magic-SysRq actions quick reference

Give each time to complete before additional ones e.g. s → u → b)

Hold Magic+SysRq (often Alt+PrintScreen; notebooks may need Fn), and press one of these:

• f — Invoke OOM killer (selects a victim process)
• r — “unRaw” keyboard; switch to XLATE mode to regain control from X/Wayland/logind.
• s — Sync disks (flush dirty buffers)
• u — Remount filesystems read-only (reduce fs corruption risk)
• e — Send SIGTERM to all processes except PID 1
• i — Send SIGKILL to all processes except PID 1
• k — Secure Attention Key: kill all processes on the current virtual console (not system-wide; spares PID 1)
• b — Reboot immediately (no sync; use s → u → b instead when possible)
• o — Power off immediately (if supported by firmware/ACPI)
• m — Dump memory info to kernel log
• t — Dump all tasks’ states and backtraces to kernel log
• w — Dump blocked (D-state) tasks to kernel log
• – l — NMI backtrace of all CPUs (heavy; availability varies)
• – 0–9 — Set console loglevel (0=quietest, 7=most verbose; higher may be ignored)

Note: Available letters can vary by kernel/architecture/config. Use Alt+SysRq+h to see the authoritative list on your system, in the logs.

Check logs after pressing:

journalctl -k -b | tail -n 200
# or
dmesg | tail -n 200

Example log snippets you might see:

• `sysrq: HELP : loglevel(0-9) … memory-full-oom-kill(f) …`
• `Sending NMI from CPU …` and `NMI backtrace …` when using `l`.

## Using Magic SysRq during a freeze

• – Alt+SysRq+f — Ask the kernel OOM killer to select and kill a memory hog.
• – Will kill only if the kernel considers memory truly tight (global) or a cgroup is OOM.
• – Expect logs like: `SysRq : invoked oom-killer` and victim selection.
• – Alt+SysRq+k — Kill all processes on the current virtual console (requires bit 4).
• – Alt+SysRq+e — Send TERM to all processes except PID 1 (graceful, broad).
• – Alt+SysRq+i — Send KILL to all processes except PID 1 (forceful, broad).
• – REISUB sequence for a safer reboot: Alt+SysRq then press R, E, I, S, U, B slowly.

Caution: `e` and `i` are heavy-handed; prefer `f` first.

## When OOM doesn’t kill

With plenty of free/available memory and swap, the kernel’s automatic OOM killer will not fire. `Alt+SysRq+f` may log but do nothing if there’s no actual OOM, unless a specific memory cgroup is under pressure.

Example snapshot indicating no OOM:

• MemAvailable: ~7.3 GB
• Swap free: ~2.0 GB

In this state, pick a victim manually or consider userspace helpers (next section).

## Fast, non-interactive ways to find a memory hog

Use these at the first sign the system is sluggish (but not frozen); these run once and exit quickly.

– Biggest resident memory users (RSS):

ps -eo pid,ppid,cmd,%mem,%cpu,rss --sort=-rss | head -n 25

– Highest kernel OOM scores (what the kernel would likely kill):

sudo sh -c 'for p in /proc/[0-9]*; do \
pid=${p#/proc/}; [ -r "$p/oom_score" ] || continue; \
score=$(cat "$p/oom_score"); \
cmd=$(tr -d "\0" < "$p/cmdline" | sed "s/\x0/ /g"); \
printf "%6d %6d %s\n" "$pid" "$score" "${cmd:-[kernel]}"; \
done | sort -k2,2nr | head -n 25'

Or less verbose:

#!/usr/bin/env bash
# Top 25 by OOM score: pid, score, short command
for p in /proc/[0-9]*; do
pid=${p#/proc/}
[ -r "$p/oom_score" ] || continue
score=$(<"$p/oom_score")
comm=$(<"$p/comm")
printf "%6d %6d %s\n" "$pid" "$score" "$comm"
done | sort -k2,2nr | head -n 25

Save to ~/sites/hi-oom.sh and add to ~/.bash_aliases this line:

# List highest scores for Out-of-Memory, most likely to be killed by Alt-PrtScrn-f
alias hi-oom='bash ~/sites/hi-oom.sh'

This outputs simply:

$ hi-oom
19871 879 brave
20156 876 brave
24374 876 windsurf
26488 876 vlc.bin
19798 872 brave
22720 872 brave
53006 872 vlc.bin
20217 871 brave
19846 870 brave
19863 870 brave
19880 870 brave
19913 870 brave
24345 870 brave
20001 869 brave
50746 869 brave
19845 868 brave
56961 868 brave
60644 868 brave
60645 868 brave
60650 868 brave
26479 866 bwrap
26484 866 bwrap
26485 866 xdg-dbus-proxy
26487 866 bwrap
52997 866 bwrap

That shows many Brave browser windows, Windsurf AI coding, VLC Media Player, plus bwrap/xdg-dbus-proxy are used by Flatpak sandboxes.

You could use killall to quickly kill your preference of those, e.g. `killall vlc.bin` or `killall brave` (if you have no unsaved work in Brave). Remember though, Alt-PrtScrn-f is quicker (often 40 minutes sooner) if the system is frozen.

Batch-mode top memory (quits immediately):

top -b -n1 -o +%MEM | head -n 40

Kill the offender:

sudo kill -TERM <pid> || sudo kill -KILL <pid>

If you suspect I/O stalls (high `wa%` in `top`), identify IO hogs:

sudo iotop -oPa -n 3 -d 1

## Optional: Userspace helpers before true OOM

### Choosing between earlyoom and systemd-oomd

– neither, if you get infrequent freezes and Magic-SysRq-f unfreezes well enough

– earlyoom: simpler, works everywhere, easy to reason about. Good default for Mint 21.x where systemd-oomd isn’t present/enabled.

– systemd-oomd: smarter under varied workloads, better with containers/cgroups, but requires PSI and some tuning knowledge.

You can run both, but generally pick one to avoid overlapping policies unless you know how you want them to interact.

Mint 21.x often does not run `systemd-oomd` by default (`systemctl status systemd-oomd` may say not found).

### earlyoom

If you want proactive killing under sustained memory pressure, consider earlyoom (recommended: simple, lightweight). Earlyoom has simple thresholds based on free RAM/swap.

sudo apt install earlyoom
sudo systemctl enable --now earlyoom
systemctl status earlyoom
# Verify is running
systemctl status earlyoom
journalctl -u earlyoom -b

Tune thresholds in `/etc/default/earlyoom`; -m 5 -M 50 is a reasonable start if you want earlier kills to avoid system freeze.

Key flags:

-m N → minimum free RAM percent. Kill when free RAM < N%.

-M N → minimum free swap percent. Kill when free swap < N%.

Helpful extras:

-r SECONDS → check interval (default 1s).

-n → prefer killing newer processes.

-i → ignore positive oom_score_adj (be more aggressive).

-p → print chosen victim and reason to syslog.

# /etc/default/earlyoom (in Debian/Ubuntu/Mint packages)
EARLYOOM_ARGS="-m 5 -M 50 -r 1 -p"

systemd-oom

Another useful tools is systemd-oomd (if available in your repos).

systemd-oomd is smarter than earlyoom, it uses Linux PSI (pressure stall information) to detect sustained memory pressure:

apt-cache policy systemd-oomd
sudo apt install systemd-oomd
sudo systemctl enable --now systemd-oomd
journalctl -u systemd-oomd -b

systemd-oomd relies on PSI to detect sustained pressure rather than just “low free” memory.

PSI = Pressure Stall Information, exposed in /proc/pressure/*.

systemd-oomd triggers when memory pressure stays above a threshold for some duration, optionally also considering high swap usage.

It can act system-wide or per-cgroup (e.g., user.slice, session.slice, service units), killing the “worst offenders” within a pressured cgroup.

Check PSI:

cat /proc/pressure/memory

Typical output:

some avg10=0.00 avg60=0.00 avg300=0.00 total=12345
full avg10=0.00 avg60=0.00 avg300=0.00 total=6789

some = proportion of time at least one task is stalled due to memory contention.

full = proportion of time all non-idle tasks are stalled (system-wide stall).

avg10/60/300 = moving averages over the last 10/60/300 seconds.

Values are percentages (0.00–100.00). Higher = more time stalled.

systemd-oomd typically looks at sustained “some/full” averages exceeding thresholds.

Examples (adjust to match your systemd version; confirm exact option names via man oomd.conf):

System-wide defaults:

# /etc/systemd/oomd.conf
[OOM]
# Trigger if memory pressure stays high for a while (example values)
DefaultMemoryPressureLimitPercent=60
DefaultMemoryPressureDurationSec=30s
DefaultSwapUsedLimitPercent=90

## Set OOM priorities for common apps (systemd-run and wrappers)

`oom_score_adj` tunes who the kernel kills first under OOM. Range -1000 to 1000.

• – -1000 = protect (kill last)
• – 0 = normal
• – positive (e.g., 300) = prefer to kill

Child processes inherit `oom_score_adj` if set before they fork. Use `systemd-run` or small wrapper scripts.

### Quick launch via systemd-run

systemd-run creates a transient unit (service or scope). It does not persist across reboots.

OOMScoreAdjust is a per-process runtime attribute. It must be applied each time you launch an app.

Tip: You can wrap desktop launchers by prefixing their Exec= with `systemd-run –user -p OOMScoreAdjust=400 –`.

Protect Kate editor (kill last, even with autosave set to 5 minutes I don’t want to lose work):

systemd-run --user -p OOMScoreAdjust=-1000 -- /usr/bin/kate

Protect Windsurf (adjust path if needed):

systemd-run --user -p OOMScoreAdjust=-600 -- /usr/share/windsurf/windsurf

Prefer to kill VLC first (Flatpak):

systemd-run --user -p OOMScoreAdjust=300 -- flatpak run org.videolan.VLC

## Always-works wrappers (inherit for the whole app tree)

Create tiny launchers and point your menu/shortcuts at them:

# ~/bin/kate-protected
#!/usr/bin/env bash
echo -1000 > /proc/$$/oom_score_adj
exec /usr/bin/kate "$@"

# ~/bin/windsurf-protected
#!/usr/bin/env bash
echo -1000 > /proc/$$/oom_score_adj
exec /usr/share/windsurf/windsurf "$@"

# ~/bin/vlc-prefer (Flatpak)
#!/usr/bin/env bash
echo 300 > /proc/$$/oom_score_adj
exec flatpak run org.videolan.VLC "$@"

Make them executable: `chmod +x ~/bin/*` (ensure `~/bin` is in your PATH).

### BOINC daemon (service) — prefer to kill

The compute daemon runs as a system service. Set a service override so it’s favored as a victim:

sudo systemctl edit boinc-client.service

Add:

[Service]
OOMScoreAdjust=300

Then apply:

sudo systemctl daemon-reload
sudo systemctl restart boinc-client.service

### Verify current OOM scores

# VLC Flatpak payload
pgrep -f '/app/bin/vlc.bin' | while read p; do echo vlc.bin $p: adj=$(cat /proc/$p/oom_score_adj) score=$(cat /proc/$p/oom_score); done

# Kate
pgrep -x kate | while read p; do echo kate $p: adj=$(cat /proc/$p/oom_score_adj) score=$(cat /proc/$p/oom_score); done

## Interpreting dmesg/journal entries

– `sysrq_handle_keypress` stack shows the kernel processed your SysRq keystrokes.

– `sysrq: HELP : …` confirms that `Alt+SysRq+h` worked.

– `Sending NMI … / NMI backtrace …` indicates `Alt+SysRq+l` (CPU backtraces) ran; screen blanking to a VT is expected.

– Absence of `invoked oom-killer` after `Alt+SysRq+f` usually means no actual OOM condition at that moment.

## Practical playbook

1. Try `Alt+SysRq+f` first to free RAM quickly.

2. If still sluggish and you can reach a TTY (Ctrl+Alt+F3):

– Run the `ps` RSS sort or `oom_score` loop, kill the top offender.

3. If unresponsive, use `Alt+SysRq+k` on the active VT (virtual terminal) or escalate to `Alt+SysRq+e`/`i`.

4. For recurrent issues, install and tune `earlyoom`.

5. After recovery, review logs for GPU/driver resets or IO issues:

journalctl -b -k | grep -Ei 'gpu|amdgpu|i915|nvrm|nouveau|drm'

## Appendix: Example memory snapshot

KiB Mem : 15934548 total, 414260 free, 6821648 used, 8698640 buff/cache
KiB Swap: 2097148 total, 2096636 free, 512 used, 7407444 avail Mem

With substantial `avail Mem` and free swap, the kernel’s OOM killer won’t trigger automatically; manual selection or userspace helpers are appropriate.

## Example journalctl Output

$ journalctl -b -1 -k -o short-full | egrep -i 'out of memory|oom-killer|invoked oom-killer|killed process'

Mon 2025-08-04 22:23:00 MST GL-Lat3410 kernel: HangWatcher invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=300

Mon 2025-08-04 22:23:01 MST GL-Lat3410 kernel: Out of memory: Killed process 3628 (slack) total-vm:1476408588kB, anon-rss:212268kB, file-rss:384kB, shmem-rss:2636kB, UID:1000 pgtables:4164kB oom_score_adj:300

Mon 2025-08-04 22:42:03 MST GL-Lat3410 kernel: Out of memory: Killed process 4482 (brave) total-vm:1466675100kB, anon-rss:227340kB, file-rss:0kB, shmem-rss:2176kB, UID:1000 pgtables:2060kB oom_score_adj:300

Tue 2025-08-05 02:10:11 MST GL-Lat3410 kernel: dockerd invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=-500

Tue 2025-08-05 02:10:04 MST GL-Lat3410 kernel: Out of memory: Killed process 4191 (brave) total-vm:1459828748kB, anon-rss:166836kB, file-rss:384kB, shmem-rss:1460kB, UID:1000 pgtables:1588kB oom_score_adj:300

Tue 2025-08-05 02:10:42 MST GL-Lat3410 kernel: ib_log_writer invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0

Tue 2025-08-05 02:10:42 MST GL-Lat3410 kernel: Out of memory: Killed process 4443 (brave) total-vm:1461547800kB, anon-rss:194792kB, file-rss:512kB, shmem-rss:992kB, UID:1000 pgtables:1928kB oom_score_adj:300

Fri 2025-08-08 23:31:44 MST GL-Lat3410 kernel: thermald invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0

Note that the ‘-o short-full’ changes the time to show in your timezone.

## Post‑mortem checklist and mitigations

Use these steps after a freeze and reboot to find root causes and prevent repeats.

### Rapid post‑mortem (previous boot)

# Kernel log from previous boot
journalctl -b -1 -k

# OOM events
journalctl -b -1 -k | egrep -i 'oom-killer|out of memory|killed process'

# GPU/graphics hangs or resets
journalctl -b -1 -k | egrep -i 'gpu|amdgpu|i915|nvrm|nouveau|drm|radeon|Xid'

# Lockups / watchdog / RCU stalls
journalctl -b -1 -k | egrep -i 'soft lockup|hard lockup|watchdog|rcu[^:]*stall'

# Blocked tasks / I/O errors
journalctl -b -1 -k | egrep -i 'blocked for more than|hung task|D state'

journalctl -b -1 -k | egrep -i 'i/o error|EXT4-fs error|btrfs: error|xfs .*error|nvme .*timeout|ata[0-9].*error'

# Display manager (adjust for your DM)
journalctl -b -1 -u lightdm || journalctl -b -1 -u gdm

# User coredumps
coredumpctl list --boot=-1

Tip during a stall: leave breadcrumbs with Magic SysRq so you have artifacts to read later: Alt+SysRq+m (memory), w (blocked tasks), l (NMI backtraces).

### Mitigations that improve resilience

– Increase swap capacity and prefer compressed zram to smooth spikes.

– Enable SysRq with OOM‑kill and diagnostics.

– Nudge swappiness up so zram is used proactively.

#### Swapfile (example: 8G for 8 Gigabyte swap file)

“`bash

sudo swapoff /swapfile

sudo fallocate -l 8G /swapfile # or: sudo dd if=/dev/zero of=/swapfile bs=1M count=8192

sudo chmod 600 /swapfile

sudo mkswap /swapfile

echo ‘/swapfile none swap sw 0 0’ | sudo tee -a /etc/fstab

sudo swapon /swapfile

swapon –show

“`

#### zram settings (recommended on laptops)

Edit `/etc/default/zramswap`:

“`ini

ALGO=zstd # lz4 is fastest; zstd compresses better → more effective swap

PERCENT=50 # ~50% of RAM → ≈8G on a 16G system

PRIORITY=100 # ensure zram is preferred over disk swap

“`

Then restart and verify:

“`bash

sudo systemctl restart zramswap

zramctl

swapon –show

“`

Example expected state for swapon –show:

“`text

/dev/zram0 ~7–8G ALGORITHM=zstd PRIO=100

/swapfile 8G PRIO=-2

“`

#### Swappiness and SysRq

Set swappiness so the kernel leverages zram earlier (reduces OOMs/stalls):

“`bash

echo ‘vm.swappiness=60’ | sudo tee /etc/sysctl.d/99-swappiness.conf

sudo sysctl –system

cat /proc/sys/vm/swappiness # expect 60

“`

Ensure Magic SysRq includes OOM‑kill and diagnostics:

“`bash

# /etc/sysctl.d/10-magic-sysrq.conf

kernel.sysrq = 252 # or 254

sudo sysctl –system

cat /proc/sys/kernel/sysrq # expect 252 or 254 (or 1)

“`

#### Hardware note: RAM upgrades matter

Modern browsers/IDEs/containers can exceed 16 GB under heavy workloads. If your system supports it, upgrading RAM (e.g., from 8GB to 16GB or 32 GB) is one of the most impactful responsiveness improvements. zram+swap helps, but more physical RAM reduces pressure and the likelihood of OOMs.

Adding or replacing RAM so your computer has the maximum it can use, is an inexpensive speed-enhancement.