* fix(terminal): self-heal panes whose renderer dies while the PTY stays alive
A pane's xterm write pipeline can die while its shell keeps running: a
synchronous throw escaping an unguarded write callback wedges WriteBuffer
(issue #2836), and write() on a disposed terminal silently drops its
completion callback (verified against vendored xterm 6.1.0-beta.287 — it
does NOT throw, falsifying the output scheduler's disposed-race catch).
Every recovery path we have (dead-session reconcile #6514/#7002,
hibernation wake #7145, the allDead activation generation bump) gates on
the PTY being dead, so these panes stayed fossils: last frame painted,
every keystroke and byte of output silently dropped, delivery ack credits
leaking, until the user reloaded the window (issue #8104 class).
Detection is probe-certified, mirroring replay-guard.ts:
- a scheduler write whose completion stalls gets an empty probe write; a
probe that never parses certifies the pipeline dead (catches both the
wedged WriteBuffer and the disposed-terminal case) and credits the
queued deliveries so main's in-flight window no longer leaks
- the replay guard's existing wedged release ("pane likely needs
recovery") now actually hands the pane to recovery
- user input rejected by an unbound transport (detached during a
remount/move and never rebound) arms recovery after confirming the PTY
is alive via pty:hasPty
Recovery reuses the proven remount seam: bump the tab's generation so
TerminalPane unmounts, detach() preserves the live PTY, and the remounted
pane builds a fresh xterm that reattaches and replays the daemon
snapshot — no shell restart, capped per tab to prevent remount storms.
The new e2e spec pins both phenotypes end-to-end (wedge → recover,
dispose-under-live-bindings → recover); both fail on main and pass with
the fix, and the same arc was validated live in a pnpm dev instance.
* fix(terminal): make pane recovery strictly best-effort in timer contexts
Recovery fires from stall-watch timers, replay-guard releases, and onData —
contexts where a throw becomes an unhandled error (CI verify caught this:
pty-connection.test.ts mocks a partial store, other tests advance fake
timers past the stall window, and the certification path hit a missing
remountTerminalTabForRecovery). Guard the store action call and the
ptyIdsByTabId reads so a partial surface yields a false return, never a
throw, and pin it with a regression test.
* fix(terminal): guard pane recovery against in-flight reattach and remote liveness blind spots
Review findings on the self-heal (adversarial pass):
1. HIGH: typing during an in-flight connect/reattach (startup restore,
app-SSH) hits sendInput while the transport is legitimately unbound; an
input-undeliverable remount there destroys the unbound transport (no
ptyId yet, so unmount cannot detach), and pty-transport's destroyed
check then kills the PTY the resolving reattach returns — the live
shell recovery exists to preserve. Gate the input detector on a
transport-connect-in-flight flag (set around all three connect sites)
and on disposed, so "not deliverable YET" never remounts. The fossil
case (detached and never rebound) has no pending connect and still
recovers.
2. pty:hasPty answers null for ids the local registry does not own, which
made the liveness gate inert for remote panes: a disconnected remote
runtime would remount-churn on every cooldown window while typing.
Remote panes (connection-tagged or remote:-prefixed) now require an
authoritative true; local panes keep the lenient null-proceeds gate.
Flagged for follow-up, not changed here: pty-transport's destroyed check
kills reattached sessions without discriminating isReattach — a
pre-existing hazard that tab-close covers by killing per id anyway.
* fix(terminal): keep certification throw-proof end to end
Guard the two remaining throw paths in the certification chain — the
entry-discard callback and the recovery handler — so nothing can escape a
timer as an unhandled error, and a throwing discard cannot suppress the
recovery notification it exists to precede. Pinned by two new tests.
* fix(terminal): breadcrumb swallowed recovery failures
A store-action throw in recovery returns false without consuming budget,
so the detector retries each cooldown — an invisible loop unless it
leaves a trace. Breadcrumb it (the recorder is self-guarded and cannot
throw where recovery runs). Also invoke transport.isConnected optionally
so partial test transports fail the gate quietly instead of logging a
contained TypeError.
* fix(terminal): end the zombie-pane replay loop at its root
Root-caused the production "wedged release drip" (1,302 breadcrumbs in one
day on one machine, 4-write bursts on a fixed timer phase, idle-required):
once a pane's xterm pipeline dies while its connection lives, the delivery
watchdog's heal (60s cooldown, fires only while idle because a dead xterm
never ACKs its in-flight bytes) re-delivers restore markers, the hidden
output restore replays 3-4 chunks into the dead parser, each write arms a
replay guard destined for another wedged release — and nothing ever
learns. The loop runs forever and re-forms after app restart.
Three fixes so the loop learns:
- replayIntoTerminal/Async short-circuit on a probe-certified dead
pipeline: no more futile writes, so no more guard drips, and awaited
restore chains resolve instead of hanging.
- requestHiddenOutputRestoreIfNeeded is gated the same way, so the
watchdog heal stops refetching snapshots for a pane recovery owns.
- a window-cap recovery decline now schedules one retry for when the
budget window reopens (deduped per tab, cancelled by any successful
remount). Without it, the certified-dead latch plus the new write
silence made a capped pane a permanent zombie: nothing would ever
re-request recovery. Cooldown declines deliberately do not retry —
remounts are tab-scoped, so the just-made remount already replaced
every pane's xterm in the tab.
Still open (tracked separately): the deterministic wedge surface that
creates the dead pipeline on the release build in the first place — an
unguarded, unreported parse-path throw or silent disposed-write; zero
guard breadcrumbs fired all day, so the trigger predates the guards'
coverage.
* feat(terminal): name the silent zombie producer in breadcrumbs
The last unproven link in the zombie-pane chain is HOW a pane's xterm
dies on the release build. Field discriminators eliminated every
reporting channel: zero terminal guard breadcrumbs and zero xterm-stack
renderer_error/unhandled_rejection events across days of logs, while the
drip re-formed after a clean app restart. Every content-triggered wedge
surface would have reported; the only fully silent mechanism left is a
restore write into an already-disposed xterm instance (write() drops its
completion callback without a throw — verified against the vendored
6.1.0-beta.287).
Instrument that exact moment: a version-pinned disposal probe (its test
runs against the real vendored build so an upgrade that moves the private
field fails loudly), a terminal_restore_write_target_disposed breadcrumb
where startup scrollback restore would write into a disposed instance,
and a terminal_restore_write_failed breadcrumb replacing the fully silent
restore catch. The next zombie formation logs its own root cause.
9924b07bcc