Initial commit: swap-watchdog + Force Quit Monitor

Polls /proc/meminfo swap and /proc/pressure/memory PSI every 3s; on a
rising edge into critical, reveals an always-running, normally-minimized
GTK3 process-killer window via kdotool rather than spawning anything new
at the exact moment the system is least able to launch a process.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

.gitignore

@@ -0,0 +1 @@
+/bin/

README.md

@@ -0,0 +1,98 @@
+# swap-watchdog
+
+A background watchdog + always-ready "Force Quit" GUI for Linux desktops,
+for when swap fills up and the whole desktop grinds to a halt. Built after
+a real Firefox flatpak process ran away to 3.3GB RSS + 5.8GB swap, swap hit
+100% full, and the desktop stayed unresponsive for a long stretch before
+`systemd-oomd` eventually killed it on its own.
+
+## What it does
+
+- **`cmd/swap-watchdog`** polls every 3s for real memory distress:
+  - `SwapFree/SwapTotal` from `/proc/meminfo` — critical if under 1%.
+  - `full avg10` from `/proc/pressure/memory` (kernel PSI — the same signal
+    `systemd-oomd` itself uses) — critical if over 10.0.
+  - On the rising edge into "critical" (not on every poll while it stays
+    critical, so it doesn't keep yanking the window back if you minimize it
+    again while still investigating), it reveals the Force Quit Monitor
+    window.
+  - Both thresholds are overridable via `SWAP_PCT_THRESHOLD` /
+    `PSI_THRESHOLD` env vars, specifically so this can be tested without
+    rebuilding.
+  - Logs every trigger to `~/.local/state/swap-watchdog/watchdog.log`.
+
+- **`cmd/force-quit-gui`** is a plain GTK3 window (via `gotk3`) titled
+  "Force Quit Monitor", listing processes sorted by RSS+Swap descending,
+  refreshed every 2s, with a **Force Quit Selected** button
+  (`SIGKILL`s the selected PID directly — no `pkexec`, since it only needs
+  to kill the current user's own processes).
+
+## The cold-start problem, and how this avoids it
+
+The whole point is to still work when the system is least able to do
+anything — including launch a new process. So neither binary is spawned
+fresh at crisis time:
+
+- The GUI starts once (at login, via its systemd unit) and stays running,
+  minimized the rest of the time. The watchdog's job at trigger time is
+  just to *reveal* it — `kdotool search -t "Force Quit Monitor"` →
+  `kdotool windowstate --remove MINIMIZED --add ABOVE` →
+  `kdotool windowactivate`. No process spawn happens at the moment it's
+  needed.
+- Closing the GUI's window (the X button) just re-minimizes it rather than
+  exiting — it has to keep running to be revealed again later.
+- Both services get `OOMScoreAdjust=-1000` in their systemd units, so
+  neither is itself a candidate for the kernel OOM killer or
+  `systemd-oomd` exactly when memory is critical.
+- Polling two small `/proc` pseudo-files every few seconds is negligible
+  cost regardless of system load.
+
+Scope note: this only keeps things workable within the existing KWin/Wayland
+session (always-on-top + activate). It does not fall back to a virtual
+terminal if the compositor itself is too starved to render — that's a
+deliberately separate, bigger problem, not handled here.
+
+## Requirements
+
+- `kdotool` (KWin window control on Wayland — used by the watchdog to
+  reveal the GUI window).
+- `gtk3-devel` to build `force-quit-gui` (cgo bindings via `gotk3`).
+- Built and tested against `gotk3 v0.6.3` — v0.6.4 fails to compile its
+  `gdk` package against this system's GTK3 (3.24.52) with an `undefined:
+  callback` error in `gdk_since_3_22.go`'s `Seat.Grab`, an unrelated
+  function this project never calls but which still blocks the build since
+  Go compiles the whole package.
+
+## Install
+
+```
+go build -o bin/swap-watchdog ./cmd/swap-watchdog
+go build -o bin/force-quit-gui ./cmd/force-quit-gui
+mkdir -p ~/.config/systemd/user
+cp systemd/*.service ~/.config/systemd/user/
+systemctl --user daemon-reload
+systemctl --user enable --now force-quit-gui.service swap-watchdog.service
+```
+
+The unit files' `ExecStart=` point at `%h/Code/swap-watchdog/bin/...` — if
+you clone this somewhere other than `~/Code/swap-watchdog`, edit the unit
+files to match before copying them.
+
+## Uninstall
+
+```
+./uninstall.sh
+```
+
+Stops and disables both services and removes the unit files. Leaves the
+built binaries and the log in place.
+
+## Testing without a real crisis
+
+```
+SWAP_PCT_THRESHOLD=100 PSI_THRESHOLD=-1 ./bin/swap-watchdog
+```
+
+Forces an immediate trigger regardless of actual system state (thresholds
+set so the very first poll counts as critical) — confirms the reveal path
+end-to-end against the real running GUI instance.

cmd/force-quit-gui/main.go

@@ -0,0 +1,194 @@
+// force-quit-gui is a minimal, always-running process killer. It starts
+// minimized at login and is revealed by swap-watchdog when memory pressure
+// gets critical. Deliberately plain GTK3 (via gotk3) rather than anything
+// heavier, so it has the best chance of already being alive and responsive
+// exactly when the system is least able to start something new.
+package main
+
+import (
+	"bufio"
+	"fmt"
+	"log"
+	"os"
+	"path/filepath"
+	"sort"
+	"strconv"
+	"strings"
+	"syscall"
+
+	"github.com/gotk3/gotk3/glib"
+	"github.com/gotk3/gotk3/gtk"
+)
+
+const refreshIntervalMs = 2000
+
+type procInfo struct {
+	pid     int
+	name    string
+	rssKB   int64
+	swapKB  int64
+	totalKB int64
+}
+
+func readProcesses() []procInfo {
+	entries, err := os.ReadDir("/proc")
+	if err != nil {
+		return nil
+	}
+	var procs []procInfo
+	for _, e := range entries {
+		pid, err := strconv.Atoi(e.Name())
+		if err != nil {
+			continue
+		}
+		status, err := os.Open(filepath.Join("/proc", e.Name(), "status"))
+		if err != nil {
+			continue
+		}
+		var rss, swap int64
+		scanner := bufio.NewScanner(status)
+		for scanner.Scan() {
+			fields := strings.Fields(scanner.Text())
+			if len(fields) < 2 {
+				continue
+			}
+			switch fields[0] {
+			case "VmRSS:":
+				rss, _ = strconv.ParseInt(fields[1], 10, 64)
+			case "VmSwap:":
+				swap, _ = strconv.ParseInt(fields[1], 10, 64)
+			}
+		}
+		status.Close()
+
+		if rss == 0 && swap == 0 {
+			continue
+		}
+
+		name := e.Name()
+		if commBytes, err := os.ReadFile(filepath.Join("/proc", e.Name(), "comm")); err == nil {
+			name = strings.TrimSpace(string(commBytes))
+		}
+
+		procs = append(procs, procInfo{pid: pid, name: name, rssKB: rss, swapKB: swap, totalKB: rss + swap})
+	}
+	sort.Slice(procs, func(i, j int) bool { return procs[i].totalKB > procs[j].totalKB })
+	return procs
+}
+
+func main() {
+	gtk.Init(nil)
+
+	win, err := gtk.WindowNew(gtk.WINDOW_TOPLEVEL)
+	if err != nil {
+		log.Fatalf("could not create window: %v", err)
+	}
+	win.SetTitle("Force Quit Monitor")
+	win.SetDefaultSize(700, 450)
+
+	// Closing the window just hides it — this process must keep running so
+	// the watchdog can reveal it again later without spawning anything new.
+	win.Connect("delete-event", func() bool {
+		win.Iconify()
+		return true
+	})
+
+	box, _ := gtk.BoxNew(gtk.ORIENTATION_VERTICAL, 8)
+	box.SetBorderWidth(8)
+	win.Add(box)
+
+	label, _ := gtk.LabelNew("Sorted by memory + swap used, highest first. Select a process and force quit it.")
+	label.SetHAlign(gtk.ALIGN_START)
+	box.PackStart(label, false, false, 0)
+
+	// Columns: PID, Name, RSS (MB), Swap (MB), Total (MB)
+	store, _ := gtk.ListStoreNew(glib.TYPE_INT, glib.TYPE_STRING, glib.TYPE_STRING, glib.TYPE_STRING, glib.TYPE_STRING)
+
+	treeView, _ := gtk.TreeViewNewWithModel(store)
+	addColumn := func(title string, idx int) {
+		renderer, _ := gtk.CellRendererTextNew()
+		col, _ := gtk.TreeViewColumnNewWithAttribute(title, renderer, "text", idx)
+		col.SetSortColumnID(idx)
+		treeView.AppendColumn(col)
+	}
+	addColumn("PID", 0)
+	addColumn("Name", 1)
+	addColumn("RSS (MB)", 2)
+	addColumn("Swap (MB)", 3)
+	addColumn("Total (MB)", 4)
+
+	scrolled, _ := gtk.ScrolledWindowNew(nil, nil)
+	scrolled.SetPolicy(gtk.POLICY_AUTOMATIC, gtk.POLICY_AUTOMATIC)
+	scrolled.Add(treeView)
+	box.PackStart(scrolled, true, true, 0)
+
+	statusLabel, _ := gtk.LabelNew("")
+	statusLabel.SetHAlign(gtk.ALIGN_START)
+	box.PackStart(statusLabel, false, false, 0)
+
+	killButton, _ := gtk.ButtonNewWithLabel("Force Quit Selected")
+	box.PackStart(killButton, false, false, 0)
+
+	refresh := func() {
+		store.Clear()
+		for _, p := range readProcesses() {
+			iter := store.Append()
+			store.Set(iter,
+				[]int{0, 1, 2, 3, 4},
+				[]interface{}{
+					p.pid,
+					p.name,
+					fmt.Sprintf("%.1f", float64(p.rssKB)/1024),
+					fmt.Sprintf("%.1f", float64(p.swapKB)/1024),
+					fmt.Sprintf("%.1f", float64(p.totalKB)/1024),
+				},
+			)
+		}
+	}
+	refresh()
+
+	glib.TimeoutAdd(refreshIntervalMs, func() bool {
+		refresh()
+		return true // keep repeating
+	})
+
+	killButton.Connect("clicked", func() {
+		sel, err := treeView.GetSelection()
+		if err != nil {
+			return
+		}
+		model, iter, ok := sel.GetSelected()
+		if !ok {
+			statusLabel.SetText("No process selected.")
+			return
+		}
+		val, err := model.(*gtk.TreeModel).GetValue(iter, 0)
+		if err != nil {
+			return
+		}
+		pidVal, err := val.GoValue()
+		if err != nil {
+			return
+		}
+		pid, ok := pidVal.(int)
+		if !ok {
+			return
+		}
+		if err := syscall.Kill(pid, syscall.SIGKILL); err != nil {
+			statusLabel.SetText(fmt.Sprintf("Failed to kill PID %d: %v", pid, err))
+		} else {
+			statusLabel.SetText(fmt.Sprintf("Killed PID %d.", pid))
+		}
+		refresh()
+	})
+
+	win.ShowAll()
+	// Deferred so the window has actually been mapped before we ask the
+	// compositor to minimize it, rather than racing the initial show.
+	glib.TimeoutAdd(200, func() bool {
+		win.Iconify()
+		return false // run once
+	})
+
+	gtk.Main()
+}

cmd/swap-watchdog/main.go

@@ -0,0 +1,147 @@
+// swap-watchdog polls system memory pressure and, on a rising edge into
+// "critical", reveals an already-running Force Quit Monitor window via
+// kdotool rather than spawning anything new at the exact moment the system
+// is least able to launch a process.
+package main
+
+import (
+	"bufio"
+	"fmt"
+	"log"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"time"
+)
+
+const (
+	pollInterval      = 3 * time.Second
+	defaultSwapPct    = 1.0  // percent SwapFree/SwapTotal below which swap counts as exhausted
+	defaultPSI        = 10.0 // percent "full avg10" above which memory pressure counts as critical
+	forceQuitWinTitle = "Force Quit Monitor"
+)
+
+func envFloat(name string, def float64) float64 {
+	v := os.Getenv(name)
+	if v == "" {
+		return def
+	}
+	f, err := strconv.ParseFloat(v, 64)
+	if err != nil {
+		return def
+	}
+	return f
+}
+
+// swapStatus reports whether swap is critically exhausted. ok is false if
+// there's no swap configured at all (nothing to measure).
+func swapStatus(pctThreshold float64) (critical bool, pct float64, ok bool) {
+	f, err := os.Open("/proc/meminfo")
+	if err != nil {
+		return false, 0, false
+	}
+	defer f.Close()
+
+	var total, free float64
+	scanner := bufio.NewScanner(f)
+	for scanner.Scan() {
+		line := scanner.Text()
+		fields := strings.Fields(line)
+		if len(fields) < 2 {
+			continue
+		}
+		switch fields[0] {
+		case "SwapTotal:":
+			total, _ = strconv.ParseFloat(fields[1], 64)
+		case "SwapFree:":
+			free, _ = strconv.ParseFloat(fields[1], 64)
+		}
+	}
+	if total <= 0 {
+		return false, 0, false
+	}
+	pct = (free / total) * 100
+	return pct < pctThreshold, pct, true
+}
+
+// memoryPressure reads the kernel PSI "full avg10" figure — the same signal
+// systemd-oomd uses — for /proc/pressure/memory.
+func memoryPressure(psiThreshold float64) (critical bool, avg10 float64, ok bool) {
+	f, err := os.Open("/proc/pressure/memory")
+	if err != nil {
+		return false, 0, false
+	}
+	defer f.Close()
+
+	scanner := bufio.NewScanner(f)
+	for scanner.Scan() {
+		line := scanner.Text()
+		if !strings.HasPrefix(line, "full ") {
+			continue
+		}
+		for _, field := range strings.Fields(line) {
+			if val, found := strings.CutPrefix(field, "avg10="); found {
+				avg10, _ = strconv.ParseFloat(val, 64)
+				return avg10 > psiThreshold, avg10, true
+			}
+		}
+	}
+	return false, 0, false
+}
+
+func revealForceQuitWindow() error {
+	out, err := exec.Command("kdotool", "search", "-t", forceQuitWinTitle).Output()
+	if err != nil {
+		return fmt.Errorf("kdotool search failed: %w", err)
+	}
+	id := strings.TrimSpace(strings.SplitN(string(out), "\n", 2)[0])
+	if id == "" {
+		return fmt.Errorf("no window titled %q found", forceQuitWinTitle)
+	}
+	if err := exec.Command("kdotool", "windowstate", "--remove", "MINIMIZED", "--add", "ABOVE", id).Run(); err != nil {
+		return fmt.Errorf("kdotool windowstate failed: %w", err)
+	}
+	if err := exec.Command("kdotool", "windowactivate", id).Run(); err != nil {
+		return fmt.Errorf("kdotool windowactivate failed: %w", err)
+	}
+	return nil
+}
+
+func main() {
+	stateDir := filepath.Join(os.Getenv("HOME"), ".local", "state", "swap-watchdog")
+	if err := os.MkdirAll(stateDir, 0o755); err != nil {
+		log.Fatalf("could not create state dir: %v", err)
+	}
+	logFile, err := os.OpenFile(filepath.Join(stateDir, "watchdog.log"), os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644)
+	if err != nil {
+		log.Fatalf("could not open log file: %v", err)
+	}
+	defer logFile.Close()
+	logger := log.New(logFile, "", log.LstdFlags)
+
+	swapPctThreshold := envFloat("SWAP_PCT_THRESHOLD", defaultSwapPct)
+	psiThreshold := envFloat("PSI_THRESHOLD", defaultPSI)
+
+	logger.Printf("started (swap threshold=%.2f%%, psi threshold=%.2f)", swapPctThreshold, psiThreshold)
+
+	wasCritical := false
+	ticker := time.NewTicker(pollInterval)
+	defer ticker.Stop()
+
+	for range ticker.C {
+		swapCrit, swapPct, swapOK := swapStatus(swapPctThreshold)
+		psiCrit, psiAvg10, psiOK := memoryPressure(psiThreshold)
+
+		critical := (swapOK && swapCrit) || (psiOK && psiCrit)
+
+		if critical && !wasCritical {
+			logger.Printf("TRIGGER: swap_free_pct=%.2f psi_full_avg10=%.2f", swapPct, psiAvg10)
+			if err := revealForceQuitWindow(); err != nil {
+				logger.Printf("failed to reveal force-quit window: %v", err)
+			}
+		}
+		wasCritical = critical
+	}
+}

go.mod

@@ -0,0 +1,5 @@
+module github.com/quine-global-labs/swap-watchdog
+
+go 1.27.1
+
+require github.com/gotk3/gotk3 v0.6.3 // indirect

go.sum

@@ -0,0 +1,4 @@
+github.com/gotk3/gotk3 v0.6.3 h1:+Ke4WkM1TQUNOlM2TZH6szqknqo+zNbX3BZWVXjSHYw=
+github.com/gotk3/gotk3 v0.6.3/go.mod h1:/hqFpkNa9T3JgNAE2fLvCdov7c5bw//FHNZrZ3Uv9/Q=
+github.com/gotk3/gotk3 v0.6.4 h1:5ur/PRr86PwCG8eSj98D1eXvhrNNK6GILS2zq779dCg=
+github.com/gotk3/gotk3 v0.6.4/go.mod h1:/hqFpkNa9T3JgNAE2fLvCdov7c5bw//FHNZrZ3Uv9/Q=

systemd/force-quit-gui.service

@@ -0,0 +1,14 @@
+[Unit]
+Description=Force Quit Monitor (always-running, starts minimized)
+After=graphical-session.target
+PartOf=graphical-session.target
+
+[Service]
+Type=simple
+ExecStart=%h/Code/swap-watchdog/bin/force-quit-gui
+Restart=always
+RestartSec=5
+OOMScoreAdjust=-1000
+
+[Install]
+WantedBy=graphical-session.target

systemd/swap-watchdog.service

@@ -0,0 +1,14 @@
+[Unit]
+Description=Swap/memory-pressure watchdog that reveals Force Quit Monitor when critical
+After=graphical-session.target force-quit-gui.service
+PartOf=graphical-session.target
+
+[Service]
+Type=simple
+ExecStart=%h/Code/swap-watchdog/bin/swap-watchdog
+Restart=always
+RestartSec=5
+OOMScoreAdjust=-1000
+
+[Install]
+WantedBy=graphical-session.target

uninstall.sh

@@ -0,0 +1,23 @@
+#!/usr/bin/env bash
+# Stops, disables, and removes the swap-watchdog / force-quit-gui systemd
+# user services installed by this repo. Leaves the built binaries and the
+# log at ~/.local/state/swap-watchdog/ in place.
+set -u
+
+echo "Stopping services..."
+systemctl --user stop swap-watchdog.service force-quit-gui.service 2>/dev/null
+
+echo "Disabling services..."
+systemctl --user disable swap-watchdog.service force-quit-gui.service 2>/dev/null
+
+echo "Removing unit files..."
+rm -f "$HOME/.config/systemd/user/swap-watchdog.service"
+rm -f "$HOME/.config/systemd/user/force-quit-gui.service"
+
+systemctl --user daemon-reload
+
+echo "Killing any still-running instances..."
+pkill -f "$HOME/Code/swap-watchdog/bin/swap-watchdog" 2>/dev/null
+pkill -f "$HOME/Code/swap-watchdog/bin/force-quit-gui" 2>/dev/null
+
+echo "Done. Source and binaries are untouched; re-run install steps from README.md to re-enable."