stall-watch.py wraps a command and, if it produces no output for
--stall-timeout seconds, dumps the process tree, network connections,
per-process /proc/*/wchan, and a 5s live strace sample of the stalled
process before killing it. Wired around gclient sync in sync.sh: ci run
#56 (2026-08-04) stalled there for ~3h with zero output and no diagnostic
trail (last thing logged: devtools-frontend's sync_rollup_libs.py hook).
Allocates its own PTY for the wrapped child so it doesn't reintroduce the
block-buffering the SSH PTY fix was added to avoid.
Needed a general upload capability to get stall-watch.py onto a box that
otherwise only ever receives whatever's piped to `bash -s` -- added
`exec -upload local=remote` (cat > remote over a plain SSH session).
That surfaced a real bug: sshSession() requested a PTY unconditionally,
which uploadFile inherited for its `cat > file` transfer. PTY canonical
mode needs a line-oriented EOF signal a plain channel-close doesn't
reliably deliver, so the very first real upload test hung indefinitely
(confirmed: had to manually kill the process and delete the orphaned
server). Fixed by making PTY allocation opt-in per session (still on for
sshExec/sshExecScript, off for uploadFile) and rewriting uploadFile to
use Start()+StdinPipe()+explicit Close() instead of Run()'s implicit
Stdin-reader-copying, for precise control over when EOF is signaled.
Re-verified end-to-end on a real box: checksum-clean upload, and a
simulated stall correctly captured a real process tree/strace/wchan
showing the stuck `sleep` and killed it with no orphan left behind.
