diff --git a/docs/hdd-partition-split-for-btrfs-pool.md b/docs/hdd-partition-split-for-btrfs-pool.md new file mode 100644 index 0000000..94bfcf6 --- /dev/null +++ b/docs/hdd-partition-split-for-btrfs-pool.md @@ -0,0 +1,181 @@ +# Splitting `~/hdd` (sda3) to grow the root btrfs pool + +> **Superseded as the primary fix.** A later investigation found the actual +> largest reclaimable consumer on the pool is `/var/home/ironmagma` itself +> (11G), not something that needs more pool capacity to fix — see +> [`relocate-home-to-hdd.md`](./relocate-home-to-hdd.md), which frees that +> space directly instead of growing the pool to make room for it. This doc +> is kept for reference / as a fallback if more headroom is still wanted +> later. + +## TL;DR + +The root filesystem is a bootc/Fedora Atomic (uBlue Aurora) install where `/`, +`/home`, and `/var` are separate **btrfs subvolumes on the same physical +partition** (`nvme0n1p3`, 23.5 GiB total). That pool is at ~89% full with only +~1.7 GiB free, which caused a Discord crash (SIGBUS while writing its disk +cache — see [Trigger](#trigger)). `/var` itself only holds ~169 MiB, so the +actual space pressure lives elsewhere in the shared pool (likely `/home` or +ostree deployment data), not in `/var` specifically — **that root cause +should still be identified and possibly cleaned up for free before doing any +of this**, since this plan doesn't fix the cause, it just gives the pool more +room. + +The plan: shrink the ext4 partition on the secondary HDD (`sda3`, mounted at +`~/hdd`) to free ~100 GiB at the end of the disk, create a new partition +(`sda4`) in that space, then add `sda4` as a second device to the nvme btrfs +pool (`btrfs device add` + `btrfs balance`). + +**Not yet executed** — this doc captures the plan and exact numbers as of +the date below, for when this project picks it back up. + +## Hardware / software identification + +- Distro: Aurora (uBlue/Fedora Atomic, bootc-based), KDE Plasma +- Kernel: `7.1.8-200.fc44.x86_64` +- Root filesystem layout (`/etc/fstab`): + ``` + UUID=8d071437-8c47-4669-94a5-4a669a3b3b66 / btrfs subvol=root,compress=zstd:1,ro + UUID=8d071437-8c47-4669-94a5-4a669a3b3b66 /home btrfs subvol=home,compress=zstd:1 + UUID=8d071437-8c47-4669-94a5-4a669a3b3b66 /var btrfs subvol=var,compress=zstd:1 + UUID=a45a95fa-805f-43f2-bc32-8da8970c1b91 /var/home/ironmagma/hdd ext4 defaults,nofail + /var/home/ironmagma/hdd/flatpak /var/lib/flatpak none bind,nofail + ``` + `/`, `/home`, and `/var` are three subvolumes of **one** btrfs partition + (`nvme0n1p3`), so they share a single free-space pool. +- `nvme0n1p3` (the OS btrfs pool): 23.5 GiB total, 20.9 GiB used, 1.7 GiB + free (89% full) at time of writing. +- `sda` (secondary HDD, GPT): `sda1` EFI System, `sda2` Apple APFS (leftover + from a prior macOS install — unused, not touched by this plan), `sda3` + Microsoft basic data (ext4), which is the **last** partition on the disk. +- `sda3` (`~/hdd`, ext4): 456.9 GiB total, 174.5 GiB used, 259.2 GiB free + (41% full) at time of writing. Already bind-mounted at `/var/lib/flatpak` + in fstab, so all flatpak app data already lives here. + +## Trigger + +``` +Sep 26 01:12:11 imac audit[11852]: ANOM_ABEND ... comm="CacheThread_Blo" ... sig=7 res=1 +Sep 26 01:12:11 imac systemd-coredump[20186]: Process 11852 (Discord) ... terminated abnormally with signal 7/BUS, processing... +Sep 26 01:12:12 imac systemd-coredump[20187]: Cannot store coredump of 11852 (Discord): No space left on device +``` + +Discord (Flatpak) crashed with SIGBUS on a Chromium cache thread — classic +symptom of a memory-mapped cache file hitting a full filesystem — and the +subsequent coredump couldn't even be written, confirming the pool was +nearly out of space. + +## Why not just relocate `/var` wholesale + +Considered and rejected: bind-mounting all of `/var` (not just +`/var/lib/flatpak`) onto the HDD, the same way flatpak already is. + +- `/var`'s own subvolume is only ~169 MiB — moving it wouldn't free + meaningful space, since the 89%-full number is pool-wide, not `/var`-only. +- `/var` holds state systemd needs very early in boot (NetworkManager + profiles, container/podman storage, machine state). The existing + `/var/lib/flatpak` bind mount is `nofail` and only needed well after boot; + making all of `/var` depend on an external/secondary disk being ready in + time is a much larger risk — a slow-to-init or disconnected HDD could + leave the system unable to start normal services, not just unable to + launch flatpak apps. + +Growing the existing btrfs pool (this doc's plan) avoids that: the pool +gains capacity, but nothing critical becomes dependent on the HDD being +present at early boot in a way it isn't already. + +## Plan + +### 0. Prerequisites + +- Back up anything on `sda3` you can't afford to lose. This is a resize of + an in-use partition holding 174.5 GiB of real data (including all flatpak + app data), not a no-risk operation. +- `sudo` needs an interactive password/TTY in this environment — these + steps must be run from a real terminal, not automated non-interactively. +- `/var/lib/flatpak` (on `sda3`) is held open by far more than just Discord + — confirmed via `fuser -vm /var/lib/flatpak`: Chrome, 1Password, Dolphin, + Konsole, KWrite, `xdg-desktop-portal`, and Plasmashell all had handles + into it. A live unmount from inside the desktop session isn't realistic; + do this from a plain TTY (Ctrl+Alt+F3) after logging out of the + graphical session, not from a terminal window inside it. + +### 1. Shrink `sda3` (ext4) to free ~100 GiB at the end + +From a TTY, logged out of the graphical session: + +```bash +sudo umount /var/lib/flatpak +sudo umount /var/home/ironmagma/hdd + +# required before any resize — stop here if this reports errors +sudo e2fsck -f /dev/sda3 + +# shrink filesystem well below the future partition size for safety margin +# (current usage 174.5 GiB; new partition will be ~356.9 GiB) +sudo resize2fs /dev/sda3 350G + +# shrink the partition to match, using parted's "from the end" syntax — +# sda3 is the last partition on the disk, so this leaves clean trailing +# free space +sudo parted /dev/sda resizepart 3 -100GiB +``` + +### 2. Create `sda4` in the freed space + +```bash +sudo parted /dev/sda unit GiB print +# note sda3's new End value, then: +sudo parted /dev/sda mkpart primary 100% +``` + +### 3. Verify before doing anything else + +```bash +sudo mount /var/home/ironmagma/hdd +sudo mount /var/lib/flatpak +lsblk -f +``` + +Reboot once, log back into the normal graphical session, and confirm +flatpak apps (Discord included) still launch fine before proceeding to +step 4. Don't chain step 4 onto the same session — verify this part is +solid first. + +### 4. Add `sda4` to the nvme btrfs pool + +Not yet detailed to exact commands — high-level: + +```bash +sudo btrfs device add /dev/sda4 / +sudo btrfs balance start / +``` + +Open questions to resolve before running this step: +- Balance filters, if any, to control how data gets distributed rather than + spread arbitrarily across both devices. +- Whether to convert to a specific btrfs profile (e.g. keep `single` data + profile across both devices, vs. something with redundancy) — plain + `single` means losing `sda4` (or the whole `sda` disk) can take out data + that landed there, including parts of `/`, `/home`, or `/var`. This + raises the HDD from "flatpak-only dependency" to "part of the core OS + pool" — a real increase in blast radius that should be a deliberate + choice, not a default. + +## Open risks / unknowns + +- Exact root cause of the pool being 89% full was never identified in the + session that produced this doc — only ruled out `/var` itself (169 MiB). + Should check `rpm-ostree status` / `bootc status` for stale deployments + and `du` on `/home` before assuming this resize is even necessary; some + or all of the space may be reclaimable for free. +- `sda2` (Apple APFS, unused leftover from a prior macOS install on this + disk) was not investigated as an alternative space source — reclaiming it + instead of shrinking `sda3` was not evaluated. +- Once `sda4` joins the btrfs pool, disk health/failure handling for `sda` + matters a lot more than it does today. Not yet decided whether to monitor + it differently once it's part of the OS pool. + +--- +*Written 2026-09-26, based on a live investigation of a Discord crash caused +by the pool being full. No resize has been performed yet.* diff --git a/docs/relocate-home-to-hdd.md b/docs/relocate-home-to-hdd.md new file mode 100644 index 0000000..58d5f9e --- /dev/null +++ b/docs/relocate-home-to-hdd.md @@ -0,0 +1,214 @@ +# Relocating `/var/home/ironmagma` onto the HDD (sda3) + +## TL;DR + +The nvme btrfs pool (`/`, `/home`, `/var` subvolumes sharing one ~23.5 GiB +partition) is ~92% full and caused a Discord SIGBUS crash (full disk cache +write — see [`hdd-partition-split-for-btrfs-pool.md`](./hdd-partition-split-for-btrfs-pool.md) +for that investigation). Breakdown of the ~20.9 GiB used: + +| Location | Size | +|---|---| +| `/sysroot/ostree/repo` (the OS image itself, one deployment, not staged) | 8.4G | +| `/var/home/ironmagma` (actual `$HOME` — lives in the "home" subvolume) | 11G | +| `/var` itself (logs/cache) | 169M | + +`/var/home/ironmagma` is the single biggest reclaimable chunk and is not +part of the OS image, so relocating it onto the HDD (`sda3`, mounted at +`~/hdd`, 259G free at time of writing) fixes the actual problem directly, +rather than just giving the pool more room to keep filling up +(the partition-split doc's approach). + +**Not yet executed.** This must be run from a TTY after a full logout — +Claude Code itself runs from files under `$HOME`, so it cannot perform or +supervise this live; there is no assistant available mid-procedure. Follow +this doc standalone. + +## Why this is trickier than the existing flatpak bind mount + +`/var/lib/flatpak` is already bind-mounted from the HDD +(`/var/home/ironmagma/hdd/flatpak`), which works because the *bind mount's +source* is nested a level below where the HDD itself is mounted, and none of +that nesting is itself the target of another mount. + +Relocating `$HOME` is different: the HDD (`sda3`) is currently mounted +**inside** the very directory (`/var/home/ironmagma`) we want to replace +with a bind mount. If we tried to keep that nesting and just bind-mount some +subdirectory of the HDD back onto `/var/home/ironmagma`, the new mount would +sit *on top of* the path leading to the HDD mount itself — Linux mount +shadowing means the original `sda3` mount (and the flatpak bind nested +inside it) would become unreachable via the normal path afterward, even +though it's technically still mounted underneath. **This is why the HDD's +mount point must first move to a path that is a sibling of `/var/home/ironmagma`, +not a descendant of it** (e.g. `/var/hdd` instead of +`/var/home/ironmagma/hdd`). + +## Pre-flight checks done (safe, no changes made) + +- Grepped for hardcoded references to the old `/var/home/ironmagma/hdd` + path across `~/Code` and `~/.config`: no scripts, systemd units, or build + configs depend on it. Only two trivial, non-blocking references found: + - `~/.config/spectaclerc` — Spectacle's screenshot save folder + - `~/.config/session/dolphin_dolphin_dolphin` — a remembered Dolphin tab + Both just need re-pointing after the move; neither blocks anything. +- Current relevant `/etc/fstab` lines: + ``` + UUID=a45a95fa-805f-43f2-bc32-8da8970c1b91 /var/home/ironmagma/hdd ext4 defaults,nofail 0 2 + /var/home/ironmagma/hdd/flatpak /var/lib/flatpak none bind,nofail 0 0 + ``` + +## Plan + +### Phase 0 — from the normal desktop session, before logging out + +1. **Take a real backup of `/var/home/ironmagma`** to somewhere independent + of this machine's disks (external drive, another machine, cloud). The + copy made in Phase 2 is *not* a substitute for this — it lands on the + same physical HDD this plan is otherwise trusting; a real backup is the + actual safety net if something goes wrong mid-procedure. +2. Note down anything you'd hate to lose track of that isn't obviously + under `$HOME` in the usual places (e.g. GPG/SSH agent state, browser + session you care about) so you can sanity-check it post-move. + +### Phase 1 — drop to a TTY + +Log out of the graphical session fully (this ends the current Claude Code +session too). Switch to a TTY (Ctrl+Alt+F3) and log in as `ironmagma`. + +### Phase 2 — move the HDD's mount point out from under `$HOME` + +```bash +sudo mkdir -p /var/hdd +sudo umount /var/lib/flatpak +sudo umount /var/home/ironmagma/hdd +``` + +Edit `/etc/fstab`, changing: +``` +UUID=a45a95fa-805f-43f2-bc32-8da8970c1b91 /var/home/ironmagma/hdd ext4 defaults,nofail 0 2 +/var/home/ironmagma/hdd/flatpak /var/lib/flatpak none bind,nofail 0 0 +``` +to: +``` +UUID=a45a95fa-805f-43f2-bc32-8da8970c1b91 /var/hdd ext4 defaults,nofail 0 2 +/var/hdd/flatpak /var/lib/flatpak none bind,nofail 0 0 +``` + +Then: +```bash +sudo systemctl daemon-reload +sudo mount /var/hdd +sudo mount /var/lib/flatpak +ls /var/hdd # sanity check: should show the same old contents, incl. flatpak/ +``` + +Home directory content is untouched by this phase — `/var/home/ironmagma` +still has its old (now unused) empty `hdd/` placeholder directory in it, +harmless, cleaned up later. + +### Phase 3 — copy `$HOME` onto the HDD + +Nothing is mounted inside `/var/home/ironmagma` anymore at this point, so +this is a plain, non-nested copy: + +```bash +sudo rsync -aHAX --info=progress2 /var/home/ironmagma/ /var/hdd/home-ironmagma/ +``` + +`-aHAX` preserves permissions, hardlinks, ACLs, and xattrs (including +SELinux labels stored as `security.selinux` xattrs — matters for `.ssh` +permissions and SELinux contexts). + +### Phase 4 — verify before touching the original + +```bash +sudo diff -rq /var/home/ironmagma /var/hdd/home-ironmagma +``` +Should report no differences. Do not proceed if it does. + +### Phase 5 — swap + +```bash +sudo mv /var/home/ironmagma /var/home/ironmagma.old +sudo mkdir /var/home/ironmagma +sudo chown ironmagma:ironmagma /var/home/ironmagma +``` + +Add to `/etc/fstab`: +``` +/var/hdd/home-ironmagma /var/home/ironmagma none bind 0 0 +``` + +**Deliberately no `nofail` here**, unlike the hdd/flatpak lines. If the HDD +fails to come up, we want boot to visibly stop and demand attention, not +silently hand you a freshly-empty `/var/home/ironmagma` that looks like your +whole home directory vanished. + +```bash +sudo systemctl daemon-reload +sudo mount /var/home/ironmagma +ls -la /var/home/ironmagma # should show all the familiar dotfiles/dirs +sudo restorecon -R /var/home/ironmagma # SELinux label safety net +``` + +### Phase 6 — test before rebooting + +```bash +sudo -u ironmagma touch /var/home/ironmagma/.relocate-test +ls -la /var/home/ironmagma/.relocate-test +rm /var/home/ironmagma/.relocate-test +``` + +### Phase 7 — reboot and verify for real + +Reboot, log into the normal graphical session, and check: +- SSH agent / keys work +- `git` works in `~/Code` +- Dropbox resumes syncing +- 1Password unlocks +- Browser profile intact +- Claude Code itself starts (it lives under `$HOME` — if this doc is + readable again from inside a working Claude Code session, that's a good + sign the move succeeded) +- Re-point Spectacle's save folder and check the stale Dolphin tab (see + pre-flight checks above) + +### Phase 8 — only once fully confident (a few days of normal use) + +**Nothing before this step actually frees space on the nvme pool** — until +now we've only relocated and kept a full fallback copy. This is the step +that does it: + +```bash +sudo rm -rf /var/home/ironmagma.old +``` + +This drops nvme pool usage from ~92% to roughly ~43%. + +### Optional cleanup + +- `/var/home/ironmagma/hdd` no longer exists as a mountpoint (it's `/var/hdd` + now); if you want a convenience path at the old location, a plain symlink + works fine since nothing is mounted there anymore: + ```bash + ln -s /var/hdd /var/home/ironmagma/hdd + ``` + +## Open risks / unknowns + +- Whether SELinux contexts translate perfectly onto ext4 via rsync `-X` — + `restorecon -R` in Phase 5 is the safety net, but not independently + verified here. +- Making the home bind mount required (no `nofail`) means a failed/missing + HDD blocks login entirely rather than degrading gracefully. This is a + deliberate choice (see Phase 5) but worth revisiting if the HDD turns out + to be less reliable than expected. +- The still-unexplained podman `system df` discrepancy (reports 2.412GB of + "reclaimable" images with an empty image list, real 2.3G sitting in + `~/.local/share/containers/storage/overlay`, and 9 volumes that return + `Permission denied` on inspection) was not resolved — separate from this + plan, and worth checking whether those volumes belong to a still-used + distrobox/toolbox environment before ever running `podman system reset`. + +--- +*Written 2026-09-26. No steps executed yet.*