Two-layer bug, found while diagnosing why chromium-fork's sccache/MinIO
credentials never reached the remote autoninja invocations in ci runs
#69/#70 (which use exactly this pattern: -env AWS_ACCESS_KEY_ID -env
AWS_SECRET_ACCESS_KEY -- 'export ... && autoninja ...'):
1. buildPayload() only ever built the env map inside its -script branch.
The plain -- <cmd> branch returned remotePayload{command: ...} with
env left nil -- so -env flags on a -- <cmd> invocation were silently
accepted by the CLI parser and then just... dropped. No forwarding,
and no validation that the named vars even existed locally either
(LookupEnv was never called for this path at all).
2. Even had (1) not existed, sshExec (the function -- <cmd> actually
calls) took no env parameter whatsoever -- there was nowhere for a
populated env map to go. sshExecScript (the -script path) already
prefixed `export KEY=value` lines onto its script correctly; sshExec
never did anything equivalent.
Fixed both: buildPayload now builds env unconditionally before branching
on script-vs-command, and sshExec now accepts env and prefixes `export
KEY='value'; ` onto the command string, mirroring sshExecScript.
Verified against a real box: `-env TESTVAR -- 'echo "TESTVAR=$TESTVAR"'`
printed empty before this fix, prints the real value after.
Mirror of -upload: streams a remote file straight to a local path via a
plain (non-PTY) `cat remotePath`, without buffering it in memory first --
build artifacts pulled this way can be hundreds of MB to a few GB.
Only downloads if the remote command actually succeeded (a nonzero exit
means there's nothing valid to pull), and a download failure counts as a
Go-level error for the existing cleanup-on-error path, same as upload
failures.
For retrieving Chromium build output before the ephemeral box is torn
down: hetzner-runner runs on mainframe itself, so this pulls straight to
mainframe's own disk (a mounted Hetzner Volume, /mnt/volume-hil-1) with
no intermediate storage needed.
Verified against a real box: 500KB random file round-tripped with
matching md5sum on both ends.
Two related hardenings found while investigating ci run #65's orphaned
box: the "Tear down Hetzner build box" if:always() step never got a
chance to run at all, because act_runner cancelling a step on its own
job timeout aborts the whole job immediately rather than continuing to
later steps -- SIGKILL to the exec process, no defer, no signal handler,
nothing runs.
- max-lifetime (cloud-init self-destruct) raised from 8h to 10h: with
the job timeout doubled to 6h (separate commit, same investigation),
8h left only 2h of margin -- uncomfortably close given
provisioning/retry overhead on top of the job's own 6h budget.
- exec now defers a cleanup attempt for genuine Go-level errors (upload
failure, SSH connection failure, panic) partway through a step --
distinct from an ordinary nonzero remote exit code, which is left
alone for the workflow's own separate, later "down" step (so one
failing build phase doesn't nuke a box other phases might still want).
A SIGKILL still can't be caught by anything running in-process, so
this doesn't close that specific gap, but it covers every other way
this process could die with an error instead of a clean exit -- cases
that previously fell through to nothing until the separate down step,
or the cloud-init failsafe hours later, eventually caught them.
Verified against a real box: a plain `exit 3` from the remote command
correctly leaves the server alive (exit code 3 propagated, no teardown);
an upload failure against a nonexistent local file correctly tears the
server down immediately ("tore down the server after an error in this
step", exit code 1).
Confirmed twice now (ci runs #62 and #64) that Gitea's job log/status can
desync from reality: run #62 stayed "running" forever after its process
and box had already vanished; run #64 was marked "failure" seconds after
the log's last line, mid-compile, with zero error, no OOM, nothing in
the runner service's own journal (which only ever logs one line per task
dispatch, never per-step detail). Both times, our only evidence was
Gitea's own unreliable view.
dlog() appends timestamped lines to hetzner-runner-debug.log in $HOME on
the runner host (mainframe) for every lifecycle event (up/down/exec
start, success, failure) -- independent of stdout, independent of
whether the SSH session or Gitea's log capture are working at all.
withHeartbeat() logs every 30s while a remote exec is in flight, so if
this process gets killed abruptly mid-call (no "after" to log), the last
heartbeat timestamp pins down precisely when it was still alive -- next
time this happens, `cat ~/hetzner-runner-debug.log` on mainframe gives a
real answer instead of another round of guessing from Gitea's log.
Root cause of ci run #63's stall (found via live inspection this time --
mainframe now hosts the runner, so its ephemeral SSH key is reachable):
bash was blocked in do_select() on stdin, and depot_tools hadn't been
cloned yet even though later script output (an apt-get install) had
already reached the log. The script was truncated mid-delivery, not just
missing its termination signal -- consistent with a PTY's small,
fixed-size canonical-mode input queue (historically 4KB on Linux)
silently dropping the tail of sync.sh's ~7.4KB single write() when it
arrives faster than bash drains it.
This is a different, worse failure mode than the EOF-hang bug fixed
earlier today (data loss vs. a missing signal), and the previous fix
(explicit `exit $?`) didn't address it since the trailing exit was itself
part of what got dropped.
Fix: never send script content through a PTY's stdin at all. Upload it
via the already-proven-reliable, non-PTY uploadBytes path (a plain pipe
has no such queue-size limitation), then exec it as a file over a PTY
session with no stdin of its own -- no EOF ambiguity when there was never
any stdin data to signal the end of in the first place.
Verified against a real box with the actual sync.sh (7.4KB): full
depot_tools clone, full chromium-fork checkout (500k+ files), gclient
sync invoked cleanly through all 5 retry attempts, clean "torn down" --
no truncation, no hang. (The run itself failed on an unrelated, expected
test artifact: stall-watch.py wasn't uploaded in this manual invocation.)
Previously removed the local ephemeral key unconditionally, even when
the Hetzner API call to delete the server failed -- meaning a failed
teardown could both leave the server running AND destroy the one thing
that could get back into it to inspect or manually clean up. Confirmed
missing in exactly this scenario while investigating ci run #62
(2026-08-05): server still alive, local key already gone.
Confirmed root cause of ci run #61's failure: install-build-deps.py's
dev_list() shells out to `file -L <path>` to detect ELF architecture;
this minimal Ubuntu cloud image doesn't ship it by default.
(FileNotFoundError: [Errno 2] No such file or directory: 'file')
This is a real, ordinary failure with a full clean run and proper
teardown around it -- the sshExecScript PTY-EOF fix is confirmed working
end to end.
Root cause of the "script finishes, process never returns" hangs (ci
runs #56 and #60, and a direct local reproduction): SSH channel EOF just
tells the server "not sending more data right now" -- for a plain exec
channel that maps onto closing the child's stdin pipe (clean, unconditional
EOF on read()), but for a PTY-backed session the server is managing an
interactive terminal, where a pause in input isn't supposed to mean "hang
up". It doesn't reliably translate into an EOF condition the remote
bash's read() ever sees, so bash just waits forever for a keystroke that
never comes.
This held true even after switching to Start()+StdinPipe()+explicit
Close() (the fix that did work for uploadFile, which uses a plain
non-PTY session) -- ruling out Run()'s implicit copying as the cause.
Confirmed independently outside this codebase: golang/go#6088 (the exact
same x/crypto/ssh symptom) and net-ssh/net-ssh#460 (different language,
same symptom, and confirms removing request_pty fixes it) -- this is a
general PTY/SSH-protocol interaction, not a library bug.
Fix: append `exit $?` to the script fed to bash -s, so it terminates the
instant the last command finishes, independent of whatever the PTY does
or doesn't do with EOF. Verified against a real box with a script that
prints its own completion timestamp then keeps running unless killed --
local process now returns within seconds of the remote script finishing
instead of hanging indefinitely.
ci run #57 (2026-08-05) spent 90+ minutes cycling through the retry loop
with no clear signal from outside about which attempt it was on, or
whether stall-watch.py had fired -- confirmed (again) that Gitea's log
capture lags real execution significantly on long steps, so "no new log
lines" doesn't mean "no progress" and can't be trusted as a proxy for
step state.
Two independent fixes, neither dependent on the log catching up in real
time:
- explicit start/end/duration logging per attempt, so whatever log
*does* eventually land is unambiguous about where the time went
- a hard 40-minute wall-clock budget across the whole retry loop, not
just a per-attempt stall timeout -- the loop now gives up and reports
on its own instead of being able to run indefinitely even if each
individual attempt stays under stall-watch's threshold
Also drops stall-watch's own threshold from 10 to 5 minutes now that a
real stall's signature (sleep, in the earlier smoke test) is confirmed
detected correctly -- faster feedback, still well outside anything
legitimate.
Verified the attempt/budget/max-attempts control flow in isolation
(stubbed success-on-attempt-N, always-fail, and zero-budget cases) before
pushing.
stall-watch.py wraps a command and, if it produces no output for
--stall-timeout seconds, dumps the process tree, network connections,
per-process /proc/*/wchan, and a 5s live strace sample of the stalled
process before killing it. Wired around gclient sync in sync.sh: ci run
#56 (2026-08-04) stalled there for ~3h with zero output and no diagnostic
trail (last thing logged: devtools-frontend's sync_rollup_libs.py hook).
Allocates its own PTY for the wrapped child so it doesn't reintroduce the
block-buffering the SSH PTY fix was added to avoid.
Needed a general upload capability to get stall-watch.py onto a box that
otherwise only ever receives whatever's piped to `bash -s` -- added
`exec -upload local=remote` (cat > remote over a plain SSH session).
That surfaced a real bug: sshSession() requested a PTY unconditionally,
which uploadFile inherited for its `cat > file` transfer. PTY canonical
mode needs a line-oriented EOF signal a plain channel-close doesn't
reliably deliver, so the very first real upload test hung indefinitely
(confirmed: had to manually kill the process and delete the orphaned
server). Fixed by making PTY allocation opt-in per session (still on for
sshExec/sshExecScript, off for uploadFile) and rewriting uploadFile to
use Start()+StdinPipe()+explicit Close() instead of Run()'s implicit
Stdin-reader-copying, for precise control over when EOF is signaled.
Re-verified end-to-end on a real box: checksum-clean upload, and a
simulated stall correctly captured a real process tree/strace/wchan
showing the stuck `sleep` and killed it with no orphan left behind.
Root cause of the python3_bin_reldir.txt failures (ci runs #54, #55):
depot_tools/update_depot_tools has an explicit guard --
"Running depot tools as root is sad" -- and no-ops entirely when $USER is
root, which we always are on the ephemeral build box. gclient itself
still works because it goes through vpython3 (separately bootstrapped),
but the plain python3 wrapper's bootstrap file never gets created via
that path.
Fix: call the two functions update_depot_tools invokes after its root
check (cipd_bin_setup, bootstrap_python3) directly, sidestepping the
check. Verified end-to-end against a real ephemeral box: file gets
created, `depot_tools/python3 --version` succeeds as root.
exec runs a command/script against an already-provisioned server (from a
prior up), without provisioning or tearing down -- lets the CI workflow
split a build into separately-visible steps (Sync, gn gen, Build v8, ...)
that all share one box.
sync.sh also fixes a real bug found in ci run #54: chromium's own gclient
hooks run update_depot_tools_toggle.py --disable partway through sync,
which can win a race against depot_tools' python3 wrapper's own
first-use bootstrap and leave python3_bin_reldir.txt missing for the next
phase's install-build-deps.py. Forcing the bootstrap before sync starts
sidesteps the race.
Without a PTY, ninja/gclient/python block-buffer their output when stdout
isn't a terminal, which can make a multi-hour build look stalled in the CI
log for long stretches even though it's progressing fine.
Provisions a throwaway Hetzner Cloud server sized for a Chromium build,
runs a script on it over SSH, and guarantees teardown afterward (normal
exit, remote command failure, or local interruption) plus a cloud-init
self-destruct failsafe in case this process itself never gets to run its
own cleanup.
