fix(mobile): harden WebSocket accept path against socket overload (#9247)

* fix(mobile): make the :6768 WebSocket accept loop leak-proof and self-healing

The desktop runtime's mobile WebSocket server on :6768 could wedge with the
socket in LISTEN but the TCP accept loop stalled — new connections piling up in
SYN_RCVD with zero ESTABLISHED, even from localhost, with no self-heal. Since
Orca mobile has no APNs (the persistent WS doubles as the push channel), a
wedged accept loop silently drops every notification while the app still shows
cached UI.

Root cause: connections were only capped at the WebSocket-upgrade layer
(MAX_WS_CONNECTIONS), after the socket is already accepted. TCP-level sockets
were unbounded, so leaked/half-open sockets (backgrounded phones, flaky relay,
reconnect storms) could grow until the process ran out of file descriptors and
accept() started failing with EMFILE.

- Bound TCP sockets via httpServer.maxConnections (2x the WS cap). At the cap
  Node accepts-then-closes, so the accept loop always keeps draining and the
  EMFILE/SYN_RCVD wedge is structurally impossible on this listener.
- Handle accept-level errors: ws forwards httpServer 'error' onto the
  WebSocketServer; with no listener Node rethrows it as an uncaught exception.
  Swallow + log and keep listening.
- Force-terminate over-capacity sockets: a bare ws.close(1013) left half-open
  phones lingering at >cap forever, rejecting everyone after them. Add an error
  handler + a 1s terminate fallback so the descriptor is always freed.
- Log heartbeat reaping / near-cap live counts so the leak is observable.

* fix(mobile): avoid masking fatal accept failures
This commit is contained in:
Brennan Benson 2026-07-17 17:12:27 -07:00 committed by GitHub
parent 72a5c6f199
commit 49e625d41a
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
2 changed files with 75 additions and 1 deletions

View File

@ -327,6 +327,46 @@ describe('WebSocketTransport', () => {
liveClient.close()
})
it('bounds raw TCP sockets above the WebSocket connection budget', async () => {
// Why: the WebSocket cap applies only after upgrade, so raw sockets need a
// finite independent bound without reducing the 128 legitimate WS slots.
const { transport } = await createTransport()
await transport.start()
const httpServer = (transport as unknown as { httpServer: { maxConnections: number } })
.httpServer
expect(httpServer.maxConnections).toBe(256)
})
it('force-terminates an over-capacity socket that ignores the close frame', async () => {
// Why: a backgrounded/half-open phone may never ack the 1013 close, so a
// bare ws.close() retains its descriptor until the heartbeat. A reconnect
// flood can fill the TCP headroom during that window, so rejection must
// hard-close on a short fixed deadline.
const { transport } = await createTransport()
await transport.start()
const ws = await connectWs(transport)
const wss = (transport as unknown as { wss: { clients: Set<WebSocket> } }).wss
const serverSocket = Array.from(wss.clients)[0]
expect(serverSocket).toBeDefined()
const terminateSpy = vi.spyOn(serverSocket!, 'terminate')
vi.useFakeTimers()
try {
;(transport as unknown as { rejectOverCapacity(ws: WebSocket): void }).rejectOverCapacity(
serverSocket!
)
vi.advanceTimersByTime(1_000)
} finally {
vi.useRealTimers()
}
expect(terminateSpy).toHaveBeenCalled()
terminateSpy.mockRestore()
ws.close()
})
it('is idempotent on double start', async () => {
const { transport } = await createTransport()

View File

@ -14,6 +14,11 @@ const MAX_WS_MESSAGE_BYTES = 1024 * 1024
// streams (session tabs, terminals, file watches, browser streams). Keep the
// cap high enough that leaked/stale streams do not starve short control RPCs.
const MAX_WS_CONNECTIONS = 128
// Why: hard-bound this listener's descriptor use above the WS-upgrade cap.
// Node accepts then drops sockets beyond maxConnections, so raw/pre-upgrade
// clients cannot grow without bound while the existing WS budget remains
// available to legitimate long-lived streams.
const MAX_TCP_CONNECTIONS = MAX_WS_CONNECTIONS * 2
const PRE_AUTH_TIMEOUT_MS = 10_000
type WebSocketMessagePayload = string | Uint8Array<ArrayBufferLike>
type WebSocketMessageHandler = {
@ -224,6 +229,10 @@ export class WebSocketTransport implements RpcTransport {
})
})
// Why: the WS cap applies only after upgrade. A separate TCP cap prevents
// raw and pre-upgrade sockets from consuming an unbounded descriptor budget.
httpServer.maxConnections = MAX_TCP_CONNECTIONS
const wss = new WebSocketServer({
server: httpServer,
maxPayload: MAX_WS_MESSAGE_BYTES
@ -231,7 +240,7 @@ export class WebSocketTransport implements RpcTransport {
wss.on('connection', (ws) => {
if (wss.clients.size > MAX_WS_CONNECTIONS) {
ws.close(1013, 'Maximum connections reached')
this.rejectOverCapacity(ws)
return
}
this.handleConnection(ws)
@ -242,6 +251,22 @@ export class WebSocketTransport implements RpcTransport {
this.startHeartbeat()
}
// Why: over the WS-upgrade cap, request a graceful 1013 close but force the
// socket down shortly after. A backgrounded/half-open phone may never ack the
// close frame, so a bare ws.close() can retain the descriptor until the next
// heartbeat. Under a reconnect flood, shortening that window bounds the
// number of rejected sockets that can accumulate behind the WS cap. The
// 'error' listener prevents a reset while closing from becoming unhandled.
private rejectOverCapacity(ws: WebSocket): void {
ws.on('error', () => {})
ws.close(1013, 'Maximum connections reached')
// Why: give the 1013 close a brief window, then hard-terminate so the
// descriptor is freed even if the client never acks the close frame.
const terminateTimer = setTimeout(() => ws.terminate(), 1_000)
terminateTimer.unref?.()
ws.once('close', () => clearTimeout(terminateTimer))
}
// Why: ping every live socket on a fixed cadence and terminate any that
// didn't pong since the previous tick. This is the only thing that
// reliably reaps half-open mobile sockets stranded by background
@ -255,12 +280,14 @@ export class WebSocketTransport implements RpcTransport {
if (!wss) {
return
}
let reaped = 0
for (const ws of wss.clients) {
if (!this.wsAlive.has(ws)) {
// Why: terminate() (vs close()) skips the close handshake and
// immediately fires the 'close' event, freeing the slot. close()
// on an already-dead socket can hang for the OS-level TCP timeout.
ws.terminate()
reaped++
continue
}
this.wsAlive.delete(ws)
@ -271,6 +298,13 @@ export class WebSocketTransport implements RpcTransport {
// close handler will run regardless, so swallow the throw.
}
}
// Why: steady reaping or a client count riding the cap are early overload
// signals; surface them without logging on healthy heartbeat ticks.
if (reaped > 0 || wss.clients.size >= MAX_WS_CONNECTIONS) {
console.warn(
`[ws-transport] heartbeat reaped ${reaped}; ${wss.clients.size} tracked sockets`
)
}
}, this.heartbeatIntervalMs)
if (typeof this.heartbeatTimer.unref === 'function') {
this.heartbeatTimer.unref()