* fix(P1-A): persist SSH consumer recovery without a sync store flush
rememberPtyConsumerRecovery ran on the live establish/reconnect path and
called flushOrThrow -> writeToDiskSync, parking the Electron main thread on
the profile-directory write. On a stalled or slow profile mount that freezes
the whole app during SSH recovery and reconnect.
Add Store.flushAsync(): same debounce-cancel and write serialization as
flushOrThrow, but awaits writeToDiskAsync instead of blocking. The consumer
recovery upsert/remove pair is now async and awaits it, and the SSH callers
await through to establish()/reconnect() so ownership is still durable before
relay setup continues. In-memory state still mutates synchronously (before the
first await), so no caller can observe a torn record and dispose() stays
synchronous.
* fix(P1-A): detach the SSH session when a connect attempt fails
Both failure exits in doConnect dropped the session from activeSessions
without calling detach(). claimSshPtyConsumerRecovery only reuses an existing
in-memory entry when detached === true, so the next connect attempt fell
through to minting a fresh clientInstanceId, discarding the remembered owner
lease and its resume identity.
Route both exits through abandonFailedSshSession(), which detaches (keeping
PTY ownership, unlike dispose()) before removing the session, and tolerates a
teardown throw so it can't mask the connect error being rethrown.
* fix(P1-A): await async lease persistence in SSH relay teardown
Failed connect attempts now wait for 'detached' leases to persist before
throwing, preventing reconnects from claiming them before cleanup completes.
Detach and dispose operations are now async and await store durability.
* fix(ssh): make session detach lease writes retryable on failure
Separate in-memory detach (identity recovery, provider cleanup) from lease
write persistence so rejected writes can be re-issued without re-running
provider teardown or re-minting the session identity. Introduce
flushDurableStateOrThrowAsync to flush only SSH-recovery state on the
live establish/reconnect path, avoiding snapshot writes of sidecars that
belong to quit/startup. Use Promise.allSettled in test reset to prevent
one rejected disposal from leaking state into the next test.
* fix(ssh): dispose mux on failed establish and propagate sync errors
- Dispose mux when session is disposed during establish to prevent resource leak
- Propagate synchronous errors in teardown via the completion promise instead of leaving completion undefined
- Add test coverage for terminated PTYs that exit mid-reattach and must stay dead
* fix(P1-B): recover system-SSH targets after a network drop
Two defects stopped a remote workspace auto-recovering after a blip.
runReconnectAttempt classified failures with isTransientError, which only
matches ETIMEDOUT/ECONNREFUSED/ECONNRESET by errno code or literal
substring. The system-SSH transport — the only transport FIDO2 and
ProxyUseFdpass targets can use — reports network failures as OpenSSH
prose ("System SSH connection timed out"), so the ladder published a
permanent 'error' on the first timeout and the target never came back
without a manual reconnect. isTransientReconnectError adds a
network-shaped prose table on top of isTransientError and is used only on
the reconnect path: connect() keeps the narrow classifier so an
unreachable host still fails fast instead of burning five 30s attempts
and five security-key touch prompts. Auth and passphrase failures stay
permanent on both paths.
runReconnectAttempt also had no generation fence, so a superseded attempt
published its cancellation as a permanent error over the winner's live
connection — reachable when a system-transport proc.onExit schedules a
reconnect while an attempt is still in flight. Cancellation now carries a
stable error name, and both connect() and runReconnectAttempt claim their
connectGeneration and stay silent when a newer attempt owns the state.
* fix(P1-B): retry a dropped watcher overflow marker on real capacity
emitWatcherOverflowToClient published the {kind:'overflow'} resync marker
with controlOverflow:'reject'. A full control queue rejects at admission
with no settlement callback, so the marker was silently discarded and the
remote File Explorer stayed stale until some later watcher event happened
to produce another one — for a quiet tree, possibly never.
The emitter now retains a rejected marker per (client, root) and
republishes it when the sink actually frees up. The existing
onLegacyPtyCapacity signal cannot drive that: it is gated on producer
retention, so it stays silent exactly under the dual-queue pressure that
caused the rejection. RelayDispatcher.onClientCapacity is an ungated
per-client capacity signal that fires on every writer settlement and
drain. It lives on the dispatcher rather than the writer so a retained
marker survives setWrite() replacing the primary sink, and setWrite
notifies capacity once afterwards so the marker does not wait on traffic
that may never arrive.
Retention is bounded to one marker per (client, root), released on
settlement and purged on client detach.
* fix(P1-B): address all review findings on SSH network recovery
Fix four issues from code review:
1. **Bug — admitted overflow markers lost on setWrite**: Retain markers when
settlement fails `ok: false`, not just on admission rejection. Prevents
desynced filesystem trees after SSH sink replacement.
2. **SSH error classification expanded**: Add missing OpenSSH patterns
(`ssh_exchange_identification`, `connection closed by remote`) and new
`isDefiniteSystemSshHostFailure()` classifier.
3. **ControlMaster retry optimization**: Skip second probe when first failure is
already definite host-level (network timeout, refused, unreachable). Saves
~30s per reconnect ladder step.
4. **Overflow flush under dual-queue pressure**: Gate pending marker retries on
control-lane headroom instead of re-attempting on every capacity notification.
Reduces thrash proportional to producer traffic.
Add regression tests for marker republish on sink replacement and validate auth
error detection against live OpenSSH credential rejection messages.
* rm random doc
* fix(P1-B): skip credential-failure retries and recover watcher markers o
- Auth and passphrase errors fail immediately without retry attempts
- Bare "System SSH probe failed (exit 255)" is transient only for reconnect
- Watcher markers survive client invalidation when switching SSH connections
- Add network error patterns: "lost connection", "remote end closed"
* fix(P1-D): bound hosted-review lookups with a detachable deadline
The `inflight` map in the hosted-review branch cache was only ever cleared
when the lookup settled, and nothing bounded how long that took. One wedged
provider call pinned its branch for the life of the process: every later poll
joined the same dead promise, so the card loaded forever with no in-session
recovery.
Each lookup now runs under a 120s deadline. Nothing below the funnel can be
cancelled, so the deadline detaches instead: the record is released, the
callers get the last known review (or a timeout error), and the branch enters
the existing failure backoff. The lookup keeps running and its answer is still
adopted if it lands, so a slow-but-alive host converges rather than failing
forever. A token identity keeps a detached lookup from evicting the record
that replaced it, and a wall-clock sweep expires records whose timer never
fired — main's timers are suspended across system sleep. `inflight` is capped
independently of the completed cache.
The failure backoff moves to its own module: it has a different lifetime from
the answer cache and is what a deadline records against.
* fix(P1-D): bound `git remote get-url` on the local/WSL path
`getRemoteUrlForRepo` ran the git child with no timeout, which is the one
unbounded step under the hosted-review lookup funnel: `git/runner.ts` only
arms its kill path when a timeout is passed, so a dead network mount or a
stalled WSL interop hangs the call and everything above it. The SSH branch is
already bounded by the relay mux's 30s request timeout, so it is unchanged.
* rm review doc
* rm review doc
* test(P1-D): add probe tests and transient-failure recovery verification
Add tests for coalesced-probe and remote-url-probe infrastructure. Add integration test verifying that transient Bitbucket API failures don't cache as a definitive no-review result, allowing recovery after cache TTL expiration.
* fix(P1-D): track lookups from start, prevent stale scope adoption
- Count unsettled lookups when they start, not after deadline expires: prevents multiple concurrent lookups for the same branch.
- Add evicted generation floor: prevents adopting stale results when scope is invalidated and evicted from the map.
- Consolidate duplicate repository reference cache logic into createRemoteRefProbeCache utility.
- Fix deadline wrapper in git config signature lookup: bound the caller's deadline only, not the coalesced probe itself.
* feat(P1-D): add remote-ref-probe-cache utility
Cache successful remote URL probes per repo/runtime to avoid duplicate work.
Skip caching transient errors and SSH failures so providers can retry on
reconnect, preventing stale scope adoption during the session.
* fix(agent-status): preserve Claude background work
* fix(agent-status): harden background task lifecycle
* fix(agent-status): narrow interruption retention
* fix(agent-status): scope background task authority
* fix(agent-status): isolate lifecycle inventories
* fix(agent-status): harden background evidence recovery
* fix(agent-status): reject ambiguous child authority
* perf(agent-status): skip lifecycle inventory scans
* refactor(agent-status): isolate task inventory parsing
* fix(agent-status): clear stale background evidence
* fix(agent-status): gate accepted remote evidence
* test(agent-status): pin session cron interrupts
* fix: harden Claude inventory tracking
* test: pin Claude cron drain authority
* refactor(agent-status): unify Claude turn-boundary predicate
Collapse the five inline copies of the Stop/StopFailure test into a single
isTurnBoundary constant and drop the reportedStateName/stateName alias, so a
future edit can't move one copy and leave the others behind.
Pin the two behaviors that unification now depends on: a non-interrupted
StopFailure keeps gating on live background work, and interrupted state does
not survive a mid-turn lead event that has no prompt submit.
Co-authored-by: Orca <help@stably.ai>
* fix(agent-hooks): gate local Claude background evidence
---------
Co-authored-by: Orca <help@stably.ai>
* fix(quit): stop durable state writes from parking the main thread
will-quit ran stats.flush() and store.flush() synchronously, before
preventDefault(). Both fsync and rename a multi-MB file on the profile
directory. When that directory sits on a stalled network mount the
syscall enters an uninterruptible wait: the app stops repainting and
stops responding to Force Quit, because a process blocked in the kernel
ignores SIGTERM and SIGKILL alike.
The existing 20s teardown deadline could not bound this. Its timer runs
on the very thread the syscall parked, so it never fires. The fix is to
make the quit path awaitable rather than to try to bound it — a quit
that is slow but responsive stays killable by the OS.
- preventDefault() now runs first, so every teardown step is free to await
- stats and state gain flushAsync() twins that use node:fs/promises
- both join the existing teardown barrier, which can now actually bound them
- the pass-2 will-quit re-entry returns early instead of re-running teardown
- quitFlushStarted makes the quit flush the last write, so a teardown step
touching the store cannot arm a debounce that races process exit
Making the swap async cost the atomicity of check-generation-then-rename:
a writer parked on await rename has already cleared the guard, so a later
synchronous flush could be clobbered by stale state. Both async writers now
claim their temp path, and the sync writers delete it, turning that swap
into a swallowed ENOENT.
Atomic temp+rename is unchanged, so a write cut short by the deadline
leaves the previous file whole — bounded loss, never corruption.
* fix(quit): harden async persistence finalization
* fix(persistence): bound best-effort flushes
* fix(ssh): gate FIDO2 system-transport on an OpenSSH binary
`ssh -G` echoes OpenSSH's built-in default identity list for every host, so
`usesDefaultPaths` was almost never true and the security-key gate returned
`!usesDefaultPaths || findSystemSsh() !== null` — forcing system transport
without checking that an `ssh` binary exists. `spawnSystemSsh()` then throws
`No system ssh binary found`, hard-failing connections that worked on ssh2.
The same flag also stopped the default scan at the first existing normal
private key, so a host that only accepts a FIDO2 key never reached system
OpenSSH when `~/.ssh/id_rsa` happened to exist.
Both decisions are independent of where an identity path came from: always
require `findSystemSsh() !== null` before forcing system transport, and scan
every candidate identity instead of stopping on the first normal key.
`shouldUseSystemSshTransport()` is untouched, so ProxyCommand / ProxyJump /
ProxyUseFdpass keep their intentional system transport.
* test(ssh): isolate connection tests from the developer's own FIDO2 keys
Transport selection now scans every default identity instead of stopping at
the first normal key, so a `~/.ssh/id_ed25519_sk` on the machine running the
suite would decide which transport the default-target tests take. Mock
`findSystemSsh` to null by default and opt the two security-key tests in.
* Add first user prompt to AI Vault session history rows
Re-parse transcripts on demand to extract and display the untruncated first
user prompt for copy/reuse. List scans omit the body (payload/perf); UI loads
it when session details expand. Grok sessions extract the typed ask from
<user_query> envelope, skipping injected <user_info> bootstrap rows. Supports
Claude, Codex, Grok, and OpenCode agents.
* fix(ai-vault): split SessionTime out to pass max-lines lint
AiVaultSessionDetails exceeded the 400-line oxlint limit after adding
first-prompt UI; move SessionTime into its own module.
* fix(ai-vault): handle corrupt transcripts and fix OpenCode prompt captur
Corrupt transcripts now resolve null instead of rejecting the IPC call, matching behavior for other unavailable cases. OpenCode SQLite parsing now correctly captures all text parts from the earliest user message only, fixing truncation of large prompts and padding of small ones. Add stale-response guard in the UI to prevent late results from overwriting the current session when tabs switch. Consolidate text slicing via `sliceAtCodeUnitLimit` to avoid surrogate-pair splits across all callers.
* test(ai-vault): add first-user-prompt UTF-16 safety tests
Ensure truncation at safety limits doesn't split UTF-16 surrogate pairs,
preventing corruption of astral characters in captured prompts.
* fix(ai-vault): key first-prompt-card by session.id
Remounting the card on session switches prevents late responses from
a previous load from writing stale data into the component's refs.
Also improves conversation-turn key stability.
* Tier GitHub PR lookup polling to prevent quota exhaustion
The selected worktree (O(1)) checks per-minute; card list (O(N)) per-15-minutes.
Introduce process-wide cache to collapse concurrent polling and gate lookups on
available rate-limit budget with exponential backoff on failure.
- Preserve last-known review during backoff
- Invalidate cache when Orca opens a PR
- Stop coordinator from double-charging
* Tier GitHub PR lookup polling to prevent quota exhaustion
- Return the latest reset time when both GitHub API buckets are rate-limited, preventing premature retries against still-blocked buckets.
- Serve the last known review on transient lookup failures, preventing reviews from blinking out on temporary errors.
- Discard in-flight lookups that predate an invalidation so stale answers cannot overwrite newly opened reviews.
* fix: give rate-limit reset tests unique titles
oxlint vitest/no-identical-title was failing static analysis because two
cases shared the same describe title.
* fix(watch/relay): bound remote watcher fan-out and read the live relay grace
Three P1 fixes from the SSH/remote freeze audit:
- Remote watchers now debounce on the same 150/500 window as local ones
(finding D), and every teardown path drops the trailing flush timer
instead of letting it fire into a dead watch. The deferred send is
wrapped so a frame disposed mid-window can't escape as a fatal
main-process exception.
- File Explorer refreshes are scheduled and concurrency-capped rather
than fanned out unbounded over expanded dirs (finding C). Local
transports use a zero window, since main already coalesced the burst.
- relay.startGrace reads ptyHandler.configuredGraceTimeMs instead of the
launch-time argv closure, so a grace raised after launch is honored.
The branch selection moves to relay-grace-branch.ts because relay.ts
has no exports and calls main() at import, making it untestable.
Consequence: a host-sleep relay holding zero PTYs now exits after the
idle cap. Pinned by test and documented in
docs/reference/relay-grace-time-reconfiguration.md.
Also drops the duplicated 150/500/5000 constants in the runtime-RPC
batcher in favor of the shared window module.
* docs(relay): correct grace-reconfiguration line numbers after the relay.ts edit
Co-authored-by: Orca <help@stably.ai>
* refactor(file-explorer): use useMemo for paths; remove relay reference
Replace manual ref-based caching with proper React hooks for content-stable path memoization. Remove outdated relay grace-time reference documentation from code review cycle.
* rm design doc
* fix(watcher/explorer/relay): coalesce POSIX paths by byte identity; pres
- Remote watcher event coalescing now keeps NFC/NFD-distinct POSIX paths separate while still folding Windows path spellings, fixing cache invalidation when recreating directories with different Unicode compositions.
- Relay grace reconfiguration now preserves shutdown-deferred state when grace is set to zero, preventing premature shutdown when the grace timer is reconfigured mid-flight.
- File explorer refresh refactored from fixed-wave batching to concurrent task execution with `forEachWithConcurrency`, batching results every N settled reads instead of every wave, and reporting whether cancellation discarded pending work.
- Watch handler now resyncs when events arrive after disposal, ensuring refresh requests aren't lost when events race cleanup during worktree switches.
* fix(watcher/explorer): resilient commit batches on callback error
Cache writes precede callbacks, so a throwing callback cannot strand
the batch with stale marks. All callbacks complete despite errors,
with the first error thrown after.
- Fix commitBatchSize calculation for empty dirs
- Add scheduler discard-report test
- Move disposed variable before handleFsChanged
- Expand batching-strategy comments
* test(activity): stabilize portal readiness latch release under CI load
The fixed 9-flip budget could miss enough MutationObserver deliveries under
shard load for sibling DOM to still report loading. Drain readiness rAF after
each flip and wait until latched unavailable is actually observed.
* fix(watcher/explorer): stop batches on commit error; verify drain settle
Enhance resilience under CI load and error conditions:
- Portal readiness drain now returns a boolean confirming settlement; test expects verify the drain completed before proceeding.
- File explorer commit batches stop executing after a callback throws, preventing stale writes after caller-observed rejection.
- Watch hook requires worktree ID upfront in effect guard, eliminating redundant checks inside loops.
* fix(relay): re-arm grace on configured change
Refactor grace reconfiguration into a dedicated decision function. startGrace
samples the configured grace at arm time, so a raised value landing mid-window
would still fire at the old deadline without re-arming. Extract the logic with
proper tests to ensure shutdown-deferred state is preserved across re-arms.
* refactor(relay): extract grace reconfiguration to testable function
Extract `applyRelayGraceTimeConfiguration` so the grace-time re-arming
logic can be tested independently. relay.ts runs `main()` on import and
exports nothing, making the call site (including retryDeferredShutdown
hand-off into startGrace) otherwise untestable.
---------
Co-authored-by: Orca <help@stably.ai>
* fix(watch/relay): bound remote watcher fan-out and read the live relay grace
Three P1 fixes from the SSH/remote freeze audit:
- Remote watchers now debounce on the same 150/500 window as local ones
(finding D), and every teardown path drops the trailing flush timer
instead of letting it fire into a dead watch. The deferred send is
wrapped so a frame disposed mid-window can't escape as a fatal
main-process exception.
- File Explorer refreshes are scheduled and concurrency-capped rather
than fanned out unbounded over expanded dirs (finding C). Local
transports use a zero window, since main already coalesced the burst.
- relay.startGrace reads ptyHandler.configuredGraceTimeMs instead of the
launch-time argv closure, so a grace raised after launch is honored.
The branch selection moves to relay-grace-branch.ts because relay.ts
has no exports and calls main() at import, making it untestable.
Consequence: a host-sleep relay holding zero PTYs now exits after the
idle cap. Pinned by test and documented in
docs/reference/relay-grace-time-reconfiguration.md.
Also drops the duplicated 150/500/5000 constants in the runtime-RPC
batcher in favor of the shared window module.
* docs(relay): correct grace-reconfiguration line numbers after the relay.ts edit
Co-authored-by: Orca <help@stably.ai>
* refactor(file-explorer): use useMemo for paths; remove relay reference
Replace manual ref-based caching with proper React hooks for content-stable path memoization. Remove outdated relay grace-time reference documentation from code review cycle.
* rm design doc
* fix(watcher/explorer/relay): coalesce POSIX paths by byte identity; pres
- Remote watcher event coalescing now keeps NFC/NFD-distinct POSIX paths separate while still folding Windows path spellings, fixing cache invalidation when recreating directories with different Unicode compositions.
- Relay grace reconfiguration now preserves shutdown-deferred state when grace is set to zero, preventing premature shutdown when the grace timer is reconfigured mid-flight.
- File explorer refresh refactored from fixed-wave batching to concurrent task execution with `forEachWithConcurrency`, batching results every N settled reads instead of every wave, and reporting whether cancellation discarded pending work.
- Watch handler now resyncs when events arrive after disposal, ensuring refresh requests aren't lost when events race cleanup during worktree switches.
* fix(watcher/explorer): resilient commit batches on callback error
Cache writes precede callbacks, so a throwing callback cannot strand
the batch with stale marks. All callbacks complete despite errors,
with the first error thrown after.
- Fix commitBatchSize calculation for empty dirs
- Add scheduler discard-report test
- Move disposed variable before handleFsChanged
- Expand batching-strategy comments
---------
Co-authored-by: Orca <help@stably.ai>
* fix(ssh,relay): stop remote connections from being killed by backoff and frame caps
Three independent connection killers found in the SSH/remote freeze audit.
FINDING A - the reconnect ladder never escalated for post-handshake drops.
scheduleReconnect() used the single published state.reconnectAttempt for both
the delay index and the give-up test, and runReconnectAttempt() zeroed it
before connecting (ssh.ts gates the relay redeploy on 0-at-connected). Every
post-handshake drop therefore re-entered at 1000ms forever, ~3600 relay
redeploys/hour, and 'reconnection-failed' was unreachable for a flapping host.
New SshReconnectLadder splits the delay index (advanced by every retry) from
the failure streak (advanced only by a failed handshake), so flaps back off
while give-up semantics stay byte-identical to shipped.
FINDING B - notify() closed the client whenever a frame exceeded the producer
frame capacity, conflating a permanently un-sendable frame with transient
backpressure. A 5000-event fs.changed is 425KB against a 49KB cap, so the
watcher flood killed the link and re-killed on every reattach+replay. notify()
now drops and logs once per generation; fs.changed is chunked to each sink's
capacity with a control-lane overflow marker as the resync fallback; agent-hook
envelopes shed lastAssistantMessage/interactivePrompt/subagents to fit.
FINDING B2 - sendResponse routed >1MB responses to a lane whose admission
ignores the frame cap and closed the client on rejection, so a large
fs.listFiles dropped the SSH host. It now substitutes a JSON-RPC error so the
request fails instead of the connection.
Also moves fs.streamEnd/fs.streamError to the control lane so a terminal frame
cannot be dropped by the producer-lane check.
Co-authored-by: Orca <help@stably.ai>
* fix(relay): stop the overflow marker from re-killing the link it protects
Round-1 review fixes on the P0 freeze work.
The control-lane overflow marker could reinstate the exact failure this P0
removes: dispatcher-client-writer closes the client when control-lane
admission fails, and admitControl is the only lane that returns an error, so
one marker per failing batch accumulated to the 256-frame/1MB bound and
dropped the link. Markers are now deduped to one outstanding per
(client, root), cleared on settle.
Chunking also defeated the renderer's per-payload directory dedupe -- events
are now stable-grouped by parent directory so one directory lands in one
chunk -- and the halving walk overshot the byte minimum ~1.7x while the fast
path paid three JSON encodes; both are fixed by publishing first and sizing
from a measured bytes-per-event estimate.
Agent-hook shedding now surrenders the blocking interactive prompt LAST
rather than first, so a degraded envelope cannot strand a pane at
state=waiting with no answerable question card.
The dropped-notification log now distinguishes over-capacity from producer
queue backpressure and no longer lets the first dropped method silence every
other producer for the life of the connection.
* fix(relay,ssh): keep status delivery and terminal frames from trading one freeze for another
Round-2 review fixes.
The round-0 change from close-on-rejection to silent drop removed the only
redelivery path for agent.hook envelopes: they are fire-and-forget and the
per-pane cache only replays on handler install, so a saturated link stranded
a pane on a stale Working spinner until reconnect. Closing used to guarantee
delivery by forcing that replay. Envelopes now publish per client and pend
for bounded latest-wins redelivery when the producer queue rejects them.
Shed fields are now named on the wire. The subagent roster is not cosmetic --
the renderer replaces rather than merges it, and hibernation gates on its
length -- so an unmarked shed could sleep a live pane.
fs.streamEnd rode the control lane because it must not be dropped, but that
lane kills rather than drops. The stream's concurrency slot is now held until
the terminal frame settles rather than until the fd closes, capping queued
terminal frames well under the control budget; overflow costs one refused
read instead of the connection.
The watcher chunk walk now stops while producer retention sits past its
reserve and degrades to a resync, so a 5000-event flood cannot fill the queue
that interactive PTY traffic shares and stall every remote terminal.
The reconnect ladder caps its flap-path delay so delay plus handshake timeout
cannot cross the relay grace floor and let the remote daemon kill live PTYs.
Also: the suppression key no longer embeds a NUL byte, which had made the
file binary to git and grep; producerEnvelopeBudget no longer reports
infinite capacity for a departed client; the drop logger no longer encodes a
frame it will not log; and an over-capacity response substitution no longer
settles as if the result had been delivered.
* fix(relay,ssh): restore relay-shed status fields and scope backpressure per client
Round 3 + 4 review fixes.
Watcher chunking is now gated on the *client's* retention reserve rather than
the dispatcher-wide one, so one stalled peer no longer forces a healthy client
into a full file-tree resync. The relay-lost redeploy ladder no longer burns its
6-attempt budget while the SSH transport itself is down: it holds at the 15s step
with a non-terminal status and rearms, so a laptop that slept past the ladder
comes back instead of landing on a terminal "give up" banner.
The shedFields wire marker had no consumer, so an agent-hook envelope whose
subagent roster was dropped to fit the frame read as "roster cleared" on the Orca
side: live child rows blanked and a done pane became hibernation-eligible while
its teammates were still running. ingestRemote now restores shed fields from the
cached payload (interactivePrompt deliberately excluded — a stale answerable
question card is worse than none).
Also: stream terminal-frame slots are counted per client, since the control queue
they protect is per client; the chunking fast path no longer logs a drop for a
batch it goes on to deliver in full; -32010 is now RelayErrorCode.ResponseOverCapacity.
Test debt from the review: pending-pane eviction, per-client stream isolation, and
the reconnect budget are now asserted rather than assumed; four fragile exact-byte
pins dropped in favour of the tier comparisons that carry the requirement.
* fix(relay,ssh): restore relay-shed status fields and scope backpressure
- Oversized relay responses now fail their request instead of closing the connection,
preventing one frame from killing every pane on the host
- Restore subagent state for correct hibernation; don't resurrect stale prose
across turns
- Account for relay re-establishment and PTY reattach time in SSH flap delay caps
- Only log drops of final unsendable envelopes, not temporary rejections during
measurement probes
- Fix watcher overflow marker release race when notification admission rejects
without settlement; use precise byte counting for event batching
* Restore relay-shed fields with digest validation and scoped backpressure
Validate that shed subagent rosters match their wire digest and turn identity before
restoration, preventing stale roster resurrection. Compact interactive prompts for waiting
states instead of dropping them. Demote control-queue overflow to non-fatal rejection so
clients can retry on capacity recovery, keeping the link alive during transient backpressure.
* fix(relay): correct ResponseOverCapacity error code
ResponseOverCapacity should use -33008 to stay in the -33xxx range
for relay protocol errors, not -32010.
* fix(relay): close client when pty.replay overflows control queue
Replay is never retried, so it uses the control lane where overflow
is fatal — the writer closes the client and reconnect reloads history
rather than stranding a short buffer.
* fix(relay): prevent infinite redeploy on flapping SSH transports
Charge reconnect attempts when connection restores mid-backoff, preventing
infinite loop on transports that flap between states. Refactor control overflow
handling to use entry property instead of WeakSet marker for clarity.
---------
Co-authored-by: Orca <help@stably.ai>
* chore: condense code comments
* chore: shorten more code comments
* clarify PTY agent session descendant cleanup behavior
Refine the comment on ptyAgentSessionIds to more accurately describe
when agent sessions sweep their descendant process trees and note the
exception on immediate Windows shutdown.
macOS anchors a notarized Developer ID app's TCC grants on identifier +
team, which is cdhash-independent and so survives an in-place update.
Without a notarization ticket there is no such stable identity, so every
hourly reads as a different client: the grant row stays but stops
matching, and file access under Documents/Desktop/Downloads fails with
EPERM and no re-prompt. `tccutil reset` fixes it until the next build —
and orca-hourly has shipped as many as 14 builds in a day.
Skipping notarization was chosen because Squirrel.Mac validates the
replacement bundle's signature, not its notarization. That is true, but
it is the wrong requirement; the in-place swap was never the problem.
Budgets grow to absorb the notary round trip (publish 2x45, job 150), and
the App token is re-minted after the build so its one-hour life starts at
the first call that uses it rather than during `pnpm install`.
* fix(terminals): make redundant tab activation idempotent (React #185)
setActiveTab always reallocated activeTabIdByWorktree, even when the tab was
already active for that worktree. Terminal's active-terminal repair effect
depends on that map, so when the repair cannot converge activeTabId -- which
happens when an earlier-scanned worktree reuses the tab id -- the effect
re-triggers itself every commit until React throws #185.
Crash cluster A: 12 reports, boundary terminal.workbench, 1.4.162/1.4.163.
Co-authored-by: Orca <help@stably.ai>
* fix(terminals): converge activeTabId when a tab id is owned by two worktrees
Prefer the active worktree when resolving a terminal tab's owner. First-match
ownership left activeTabId permanently unconvergeable under a duplicated tab
id, so the active-terminal repair effect re-triggered itself into React #185.
Breadcrumb the duplicate-ownership state (once per tab id) so a crash bundle
can prove or kill the production origin of the precondition.
Co-authored-by: Orca <help@stably.ai>
* fix(crash-reporting): coalesce the duplicate-tab-owner breadcrumb
Its renderer guard is once-per-tab-id, so the stale worktree map it exists to
diagnose duplicates every tab id at once and could evict the whole 30-entry
ring. Also drops two keyed re-reads of tabsByWorktree that would throw for a
prototype-named worktree id, and pins the activeTabIdByWorktree guard with a
test that fails without it.
Co-authored-by: Orca <help@stably.ai>
* fix(crash-reporting): key the duplicate-tab-owner crumb on its convergence flag
Name-only coalescing keeps only the newest payload, so a resolvedToActiveWorktree
false sample — the one value saying the activation still could not converge — was
erased by any later benign true in the same 30s window. Keys on the flag instead,
mirroring the WebGL name:kind branch; two keys still bound the burst.
Also: the previous coalescing commit had no test at all (removing the name from
both sets broke zero of 2701 tests), the resolver's activeWorktreeId truthiness
check was a hole rather than a guard for a '' active id, and the resolver test
file failed oxfmt --check.
Co-authored-by: Orca <help@stably.ai>
* fix(crash-reporting): keep the non-converging duplicate-tab verdict
The two earlier commits contradicted each other. Splitting the coalesce
key existed so a `false` verdict could not be erased by a later benign
`true` — but the renderer guard was keyed on the tab id alone, so for
any one id only the first verdict was ever emitted.
A duplicated id that first resolves benignly, then stops converging when
the user switches worktrees, dropped the `false` sample at the source.
That sample is the whole reason the breadcrumb exists: it is the only
value saying the activation could not converge activeTabId.
Key the guard on id plus verdict. At most two crumbs per tab id, and
the main process still folds each verdict into its own ring entry, so
the flood bound is unchanged.
* fix(crash): correct the duplicate-tab verdict rationale, pin and cap the guard
Three comments said `false` is the verdict that matters because it is the
only one showing the activation could not converge. That is backwards.
The repair effect activates a tab drawn from tabsByWorktree[active], so
the React #185 path can only ever emit `true`; `false` is what a
deliberate jump-to-agent into a background worktree emits from a fully
converged state. A reader of the next bundle would have discarded the
exact sample the breadcrumb exists to capture.
The mechanism was right, only its stated reason was wrong: the real
justification for keying on the verdict is symmetric, since coalescing
keeps only the newest payload and either verdict would erase the other.
The suite also did not pin the "at most 2 per tab id" bound - a guard
keyed on `${tabId}:${activeWorktreeId}` passed all 11 tests while
emitting once per worktree, the storm the guard exists to prevent. Adds
a count-pinning test that kills it.
Caps the never-pruned guard set at 256 distinct verdict keys (~85KB),
mirroring MAX_COALESCE_KEYS. Measured 330 B/entry; a realistic thousand
duplicated tab ids is ~0.6MB, negligible but unbounded in principle.
* perf(crash): scan worktree tabs by key, and soften the verdict rationale
Round 6 corrected my own round-5 comment. I had written that `true` is the
repair-loop signature and `false` covers a deliberate background activation. The
repair effect can emit `false` too: its closure holds the worktree from its render
while the guard runs against live state, so a worktree switch landing in between
reattributes the tab. The verdict hints at the caller; it does not prove it, and
neither value should be discarded. Comment-only.
Also take the free scan win the perf review measured: Object.entries allocates a
pair array per worktree on a path that runs per tab activation. Own keys are safe
to index by, so Object.keys plus an indexed read is behaviour-identical
(16.1us -> 5.0us at 170 worktrees x 10 tabs).
* fix(terminal): keep a duplicated tab id from re-sorting the active worktree
setActiveTab now prefers the active worktree when a tab id is held by more
than one, but terminals.ts has a second, older owner resolver:
getTerminalTabOwnerWorktreeId, a memoized map built last-writer-wins. Two of
its callers — setRuntimePaneTitle and clearRuntimePaneTitle — use the result
for the same "is this pane in the active worktree" gate, so under a duplicate
the two resolvers disagree: the cache names whichever worktree it saw last,
which can be a background one for a pane the user is looking at. The gate then
fails open and every classified OSC title frame bumps sortEpoch, reinstating
the click-driven sidebar re-sort #209 removed — 20 title frames measured 20
bumps, each one a store write that re-renders every sortEpoch subscriber.
isTabInActiveWorktree answers from the active worktree's own tab list instead
of a tie-break. It stays behind the cheap id equality so the common
non-duplicated path is unchanged, and it is a hasOwn lookup plus one scan of
that worktree's tabs rather than resolveActiveTabOwnerWorktreeId, whose full
scan would run per title frame and whose breadcrumb would fold a second caller
into one verdict.
Leaves updateTabTitle and clearTabLaunchAgent on the cache: they pick which
copy of a duplicated tab to mutate, where no answer is defensible until the
duplication itself is fixed.
* test(terminal): pin the SSH-hydration origin of the duplicate tab id
Drives the duplicate from real hydration rather than constructing it: a
direct-SSH snapshot is keyed by worktree path, so renaming the worktree on
the host (or re-adding the repo, which mints a fresh id) re-resolves it to a
new worktree id while replaceHydratedRecordKeys retains the old key verbatim.
Nothing de-dupes across keys.
Fails on unfixed origin/main with converged=false after 200 passes; the two
precondition assertions pass on both sides, so the red is the non-convergence
itself and not a setup divergence.
The reconnectPersistedTerminals stub is load-bearing and marked as such: with
no registered PTY the orphan sweep cleans the duplicate up before the repair
effect sees it.
* docs(test): record the end-to-end #185 reproduction method on the regression test
Co-authored-by: Orca <help@stably.ai>
* test(terminal): pin the null-active-worktree guard in isTabInActiveWorktree
Dropping the `activeWorktreeId === null` early return was killed by nothing:
`Object.hasOwn(map, null)` coerces to the string key 'null', so a worktree
literally named 'null' would answer for "no active worktree". An untested
guard reads as dead code and gets deleted.
* rm triage context
* rm triage context
* rm context files
* refactor(terminal): extract repair logic into reusable hook and guard ag
Extract the active terminal repair effect from Terminal.tsx into `useActiveTerminalRepair`
hook to enable reuse in tests and clarify responsibilities. Replace falsy coercion guards
(`obj[id] ?? []`) with explicit `Object.hasOwn()` checks to handle edge cases: empty-string
worktree ids (valid but falsy), prototype-named ids like 'toString', and duplicated tab ids
across worktrees. Remove the now-unused `isTabInActiveWorktree` helper. Simplify the
isActive logic in terminals.ts to rely solely on owner-equality since the repair now uses
proper membership checks.
---------
Co-authored-by: Orca <help@stably.ai>
Co-authored-by: Jinjing <6427696+AmethystLiang@users.noreply.github.com>
Recovery logic strengthened with replacement-policy tiers (reuse/prefer-replacement/require-replacement) and bounded same-handle end cycles. Prevents infinite flapping by capping reuse attempts and inventory wait windows. Tracks ready evidence to reattach from prior snapshots when inventory becomes unavailable.
* fix(win32): suppress Command Prompt window on IDE launches
- Prefer JetBrains GUI executables (`*64.exe`) over `.cmd` shims to avoid
console allocation (STA-3040).
- Use `start "" /B` when launching GUI apps via batch scripts; shims chain
through console helpers that allocate a visible prompt even with
`windowsHide`. `start /B` returns immediately, preventing the lingering window.
* fix(win32): suppress Command Prompt window on IDE launches
Prevent lingering Command Prompt windows when launching JetBrains IDEs
on Windows. Use `start "" /B cmd /d /c` so the nested shell exits with
the batch script, but only for JetBrains shims—VS Code and Cursor keep
the waiting form because `start` re-parses arguments and breaks remote
paths with spaces. Prefer colocated `*64.exe` executables beside the
resolved `.cmd` shim over PATH lookups to avoid stale installations.
* fix(win32): extend IDE launcher console suppression to direct paths
Support IDE paths stored directly in settings (e.g., idea.exe,
webstorm.cmd). Detect console idea.exe stubs alongside batch shims
for upgrade to GUI *64.exe. Fix start command title escaping: use
empty string instead of '""' to prevent libuv re-quoting.
* fix(runtime): drop stale local agent rows from worktree.ps after tab close
attachAgentRowsToSummaries attached every hydrated hook row by worktreeId
with no check that the pane/tab still exists, so agents from closed tabs
(last-status.json hydrates for days) kept showing on mobile as current
activity. Local rows now require the tab in a session/runtime graph or a
connected PTY; remote rows are exempt since their tabs may only exist on
the remote host.
Fixes#6072
* fix(runtime): resolve legacy numeric pane keys through the stale-row filter
Non-UUID leaves produce tabId:paneRuntimeId keys with no tabId field;
without parsing them the stale filter was bypassed entirely for such rows.
* fix(runtime): filter stale WSL agent rows
* fix(runtime): ignore persisted tabs for agent liveness
* fix(runtime): restore session-tab liveness and thread OSC transport through the stale-row filter
Review loop pass 1 (3 independent same-model reviewers, findings converged):
- Revert a829e8f9cf's `!this.tabs.has(tabId)` to `mirroredWorktreeId ===
undefined`. The renderer graph is structurally empty under headless serve
(index.ts publishes {tabs: [], leaves: []}), is cleared by
markGraphUnavailable, and omits unvisited/cold-parked workspaces, so
graph-only existence dropped live agent rows in all those states and broke
worktree.ps/session.tabs.list parity. Every close path prunes the persisted
tab, so session tabs remain valid liveness evidence; the stale-persisted-tab
premise did not survive tracing.
- Restore the rename and legacy-pane-key tests to their session-only fixtures
(the graph syncs added with the flipped predicate masked the contract
change) and pin the restored contract in a named test.
- Thread the pane's connectionId through RuntimeAgentRowSnapshot so
OSC-retained rows keep the SSH exemption; previously a fresher OSC ping
hardcoded null and stripped it.
- Pin each rescue conjunct individually (paneKey-only, tabId-only, ptyId after
binding clear), the WSL keep direction, the unresolvable-paneKey guard, and
row presence in the freshness cases.
* fix(runtime): carry the OSC-observed ptyId when a hook row wins the freshness race
Hook payloads have no ptyId field, so overwriting the rowSources entry
discarded the OSC-observed one and the connected-PTY ptyId rescue went dead
for hook-fresh panes during a binding-clear window (pass-2 review P3). Also
corrects the incarnation-change comment on the OSC rescue test.
* batch remote watcher events and defer File Explorer refreshes
Remote filesystem watcher events now batch with the shared 150ms trailing and 500ms
max-wait window, coalescing per-path like local events. File Explorer tree and
directory refreshes are scheduled with debounce and transport-aware concurrency caps
(16 local, 8 runtime, 4 SSH). Stale directory cache tracking prevents trusting
collapsed listings skipped by full refresh; they are re-read on re-expansion. Relay
implements a 15-minute idle-only grace cap for zero-PTY relays via PTY pool
lifecycle tracking, independent of explicitly configured grace time.
* fix(watch/relay): bound remote watcher fan-out and read the live relay grace
Three P1 fixes from the SSH/remote freeze audit:
- Remote watchers now debounce on the same 150/500 window as local ones
(finding D), and every teardown path drops the trailing flush timer
instead of letting it fire into a dead watch. The deferred send is
wrapped so a frame disposed mid-window can't escape as a fatal
main-process exception.
- File Explorer refreshes are scheduled and concurrency-capped rather
than fanned out unbounded over expanded dirs (finding C). Local
transports use a zero window, since main already coalesced the burst.
- relay.startGrace reads ptyHandler.configuredGraceTimeMs instead of the
launch-time argv closure, so a grace raised after launch is honored.
The branch selection moves to relay-grace-branch.ts because relay.ts
has no exports and calls main() at import, making it untestable.
Consequence: a host-sleep relay holding zero PTYs now exits after the
idle cap. Pinned by test and documented in
docs/reference/relay-grace-time-reconfiguration.md.
Also drops the duplicated 150/500/5000 constants in the runtime-RPC
batcher in favor of the shared window module.
* docs(relay): correct grace-reconfiguration line numbers after the relay.ts edit
Co-authored-by: Orca <help@stably.ai>
* refactor(file-explorer): use useMemo for paths; remove relay reference
Replace manual ref-based caching with proper React hooks for content-stable path memoization. Remove outdated relay grace-time reference documentation from code review cycle.
* rm design doc
* fix(remote-watcher): prevent stranded timer after close
An in-flight provider receive can land after the batch is torn down.
Without a guard, pushing events to a closed batch would re-arm a timer
that would never be cleared, stranding the task indefinitely. Track the
closed state and skip pushes after close().
Relay.ts comment clarifies why pool watches remain registered during
grace-period shutdown deferral — the socket server stays listening so
a reconnecting client can cancel the grace and resume.
---------
Co-authored-by: Orca <help@stably.ai>
Use async profile probes and serialize the current-time due decision plus mutations under one async owner so sync flushes and detached writers cannot double-shift or miss the recovery interval. Add syscall, timer-liveness, parity, and held-I/O interleaving coverage.
* fix(relay): chunk fs.changed notifications to fit the client frame capacity
emitRelayWatcherEvents published an entire watcher batch as one
fs.changed notification on the ordinary lane. A batch runs routinely
tens of times over the per-frame producer capacity (12,288 bytes against
a Node <=21 socket, 49,152 against Node >=22), and an over-capacity
frame there is not queued or trimmed -- notify() fails admission and
closes the client, tearing down every in-flight request on that
connection. Regressed in 5f7807497e (#11005).
The emitter now sizes chunks against the tightest capacity across
attached clients, measured in encoded bytes through the same envelope
admission measures, and preserves event order. A single event too large
for an empty envelope cannot be chunked, so the ordinary-lane close
remains its backstop: a silent drop on this lane has no resync contract.
* chore(relay): shorten watcher chunking comments
Swap the worktree-name emoji picker from emojibase-data's `github` shortcode preset to `emojibase`, which carries both `flag_kr` and `south_korea` style flag names, and drop the hand-maintained `kr` entry that patched around the gap. Filter skin-tone aliases so they neither crowd the suggestion list nor clobber base-emoji branch names.
Search now matches anywhere in the shortcode, ranked exact > prefix > word-start > substring, so `:korea` surfaces both Koreas.
Emoji-derived branch names now prefer spelled-out aliases: flags use country names (japan, germany, south-korea) and cryptic stubs are skipped (thumbsdown over no, victory over v).
Keep only the durable docs already allowlisted for tracking
(STYLEGUIDE, assets, localized readme, and reference compatibility
guides). Drop feature design notes, plans, and repro artifacts that
were force-added past the existing docs ignore rules.
Remove one-off incident docs and committed test-results noise, move
dev/repro/bench tools under tests/tools, and relocate i18next config
into config/ so the GitHub root scrolls to the description faster.
* perf(mobile): avoid unchanged worktree catalog payloads
* fix(mobile): isolate catalog snapshots by limit
* review: reassert host truth on unchanged polls; content-address snapshots
Client — the `changed` gate meant an unchanged poll skipped setWorktrees /
setLastKnownWorktrees / setCachedWorktrees, so optimistic local edits
(togglePin, handleDeleteWorktree's failure re-add) and the #8498 cache guard
were no longer repaired while the host catalog was stable. The gate bought
nothing: setCachedWorktrees is an in-memory Map write and areWorktreeListsEqual
already ran every poll, so the steady state still short-circuits on array
identity. All wire savings are unaffected. admit() now just returns the
confirmed rows and HostScreen applies them exactly as it did pre-PR.
Also on the client:
- a stale response from a superseded client/host no longer clears the token the
current client/host just established
- discriminate on `worktrees` rather than on `'unchanged' in response`, so a
future catalog field named `unchanged` can't reclassify a full response
- useRef over useMemo for the snapshot client; React may discard memoized values
- hoist WORKTREE_PS_FULL_LIMIT so the truncates-at-200 rationale travels with it
Host — replace the per-limit snapshot cache with a content-addressed id (ETag
semantics). Ownership lives in the id, so concurrent clients, differing limits,
and runtime restarts are correct by construction; this drops the LRU, the
eviction policy, the per-runtime WeakMap, and the retention of up to 8 full
catalogs. The remaining cache is a pure memo: because ids derive from content,
dropping or thrashing it costs CPU and nothing else. Keeping the memo also
keeps the measured steady-state cost — hashing every poll instead measured
2.24ms vs 0.75ms for the compare on a 310KB catalog.
Verified: mobile 2784 passed / 3 skipped, src/main/runtime/rpc 1064 passed,
node + mobile typechecks, oxlint, oxfmt, max-lines ratchet.
* fix(runtime): isolate catalog snapshot memo
---------
Co-authored-by: Brennan Benson <79079362+brennanb2025@users.noreply.github.com>
OrcaWin
Neil
Jinjing
github-actions[bot]
闲人
Brennan Benson
Jinwoo Hong