wp-durable · Server v2 handover (Change 3, 2026-10-10)

Status: ACCEPTED by piinchrome-1c on 2026-10-11 on deployment 18c9c20a (image of a69e0309 plus ANTHROPIC_API_KEY). See Acceptance.

Brief: docs/SERVER-V2-BRIEF.md (approved by Steven). Caller documentation: docs/API-GUIDE.md (also on https://wp-durable-docs.pages.dev). Earlier changes: STAGE1-HANDOVER.md (Stage 1 and Change 2).

Deployment

What changed

Image programs (apt, Debian bookworm, as built 2026-10-11)

bash 5.2.15-2+b13 · coreutils 9.1-1 · findutils 4.9.0-4 · grep 3.8-5 · sed 4.9-1+deb12u1 · gawk 1:5.2.1-2 · tar 1.34+dfsg-1.2+deb12u1 · gzip 1.12-1 · xz-utils 5.4.1-1+deb12u2 · unzip 6.0-28+deb12u1 · procps 2:4.0.2-3 · curl 7.88.1-10+deb12u15 · jq 1.6-2.1+deb12u2 · git 1:2.39.5-0+deb12u3 · ca-certificates 20250419~deb12u1 · python3 3.11.2-1+b1 · python3-venv 3.11.2-1+b1 · python3-pip 23.0.1+dfsg-1 · util-linux 2.38.1-5+deb12u3 (setpriv, prlimit). From the base image: node v22.23.3, npm 10.9.9. Versions move with Debian security updates at each build.

Isolation (brief §6)

1. Process environment: WP_DURABLE_TOKEN, ANTHROPIC_API_KEY, ANTHROPIC_AUTH_TOKEN, ANTHROPIC_OAUTH_TOKEN are read and deleted from process.env before anything else runs; pi-ai gets them through createModels({ authContext }); redact() masks the captured values.
2. Command environment: only HOME, TMPDIR, PATH, LANG, WP_CONVERSATION_ID (inheritEnv: false, enforced in the environment's exec).
3. Privilege separation: every command runs as prlimit --nproc=256 --fsize=2147483648 --core=0 -- setpriv --reuid --regid --init-groups --inh-caps=-all --no-new-privs -- …. Volume modes set at every start: /data 0711 root, agent.sqlite* and credentials.json 0600 root, /data/work 0755 root, /data/work/<id> 0700 agent, /data/.spill 0700 root.
4. File confinement: a canonical-path check gives the "outside the work directory" message; reads then check the opened descriptor (a regular file owned by agent); every write into a work directory runs as agent through a short helper command, so the kernel enforces it. Invariant: every file under /data/work is owned by agent; the Server never reads a file that is not owned by agent.

Startup self-test (fail closed: any failing check turns all five tools off; the report is in /v1/status): commands run as agent (uid and name); no secret names in the command env; no capabilities and NoNewPrivs; resource limits; the volume modes; work directories owned by agent; the wrapper and the descriptor check refuse root files; bash cannot read a root file or credentials.json or list /data; the agent cannot signal the Server; the agent helper cannot write a root file. A failing check with EAGAIN names the likely cause (a uid collision).

Why uid 61337 and not the brief's 1001: Railway runs containers without a user namespace (uid_map 0 0 4294967295), so a uid in the container is the host uid, and RLIMIT_NPROC counts that uid's processes on the whole host across tenants. With uid 1001 the host already had more than 256 such processes, so exec after setpriv failed with EAGAIN and the self-test turned the tools off on ebaa09b9. An unusual uid keeps the 256-process limit a real bound.

Tests

npm test runs the unit, faux e2e, and mock suites. The e2e suites cover: every §4 route; streams (events, watch, a 1.5 KB answer streamed in many text events); images; busy handling, withdraw, abort, long poll; reset, compact, fork; exactly-once across a restart; tool rounds for all five tools; refusals (outside paths, symlinks, blocked targets, redirects to loopback); timeouts, spill files, abort of a running command; the work-directory routes; callbacks (delivery, retries, giving up, refusal, background); resume after a kill in the middle of a bash call (interrupted result with the output so far, an answer, one callback, orphaned processes killed) and of an http_fetch POST (not sent again).

Deviations from the brief

M1 (accepted)

1. A conversation's usage stays the full pi.usage document (Stage 1, Change 2); the contract's {tokens, cost} is usageSummary.
2. Error bodies keep { error: message }, plus code on 409s (busy, request_id_conflict, already_placed, settled) and provider on the credential 409.
3. Times: the Engine stores none. Conversation createdAt is stamped by the conversationCreated hook (older conversations fall back to the oldest message time); entry createdAt is the model message timestamp (entries without a message have none); submission createdAt comes from wp.request, settledAt from a commit-listener follow-up commit. No placedAt.
4. requestId stays required; type defaults to input on the conversation route; /v1/submit takes input only; unknown keys on submit bodies are ignored (Stage 1 behaviour, both submit routes).
5. /v1/tasks/:id shows the Engine-stamped startedAt/endedAt; input and checkpoint are not exposed.
6. Watch: a slow client gets one larger diff patch instead of a new snapshot frame.
7. The submission stream forwards deltas from attach time on (see M3 item 7).
8. Faux mode only: retries off, compaction keepRecentTokens 1.
9. Limit overrides are named WP_DURABLE_LIMIT_<NAME>.

M2 (accepted)

1. bash keeps the last 64 KB (the Engine keeps head or tail, not both); the full output is in the spill file.
2. Spill files stay Server-private (/data/.spill, root 0700) while the command runs, then are copied into <workdir>/.tmp as agent.
3. Directory listings and file info stay in the Server process (names and sizes only).
4. Fail closed turns off all five tools, http_fetch included (its downloads use the agent helper).
5. http_fetch refuses URLs with credentials, drops host and hop-by-hop headers, decompresses gzip/deflate/br under the size cap, and also blocks 100.64/10, 192.0.0/24, 198.18/15, 240/4, NAT64 with the embedded IPv4 checked, and the IPv6 documentation and discard ranges.
6. read is replay safe; http_fetch is declared replay safe and marks non-GET/HEAD requests with a durable memo, so a rerun returns interrupted without sending again.
7. The self-test uses /data/work/0 and /data/.isolation-probe and removes both.
8. POST /v1/conversations/:id/submit stays lenient on unknown keys, like /v1/submit.
9. Laptop tests use WP_DURABLE_TEST_UNISOLATED=1 (faux only).
10. The Dockerfile has a test stage, so the deploy archive includes test/; the runtime image never contains it.
11. Agent uid/gid 61337 instead of 1001 (cause above). Work directories of an earlier agent uid are handed to the current one at start (logged); root-owned entries under /data/work fail the self-test.

M3

1. The callback's exactly-once key is the request record's callbackTaskId (wp.request), set in the commit that creates the task; the Engine has no request id for tasks.
2. The task is created in the follow-up commit that stamps the settlement (a commit listener), not in the settling commit itself. A crash between the two is repaired at start from the wp.callbacks pending list (written with the request record before admission).
3. Four attempts in all (one plus three retries); redirects are not followed; a restart in the middle of an attempt repeats that attempt with the same x-wp-durable-attempt (at-least-once per attempt).
4. Callback URLs with credentials are refused; the stored URL is shown without path and query.
5. The callback body is the submission view at delivery time (its callback field shows the delivery in progress).
6. At start, the Server kills leftover processes of the agent user (pkill -u) before resuming, so an interrupted command cannot keep writing. On Railway the container is new after a restart anyway.
7. Fix of an M1 defect in the submission stream: the opening content of each message (carried by message_start) and the partial already written at attach time were dropped, so a short answer arrived as one text event at the end. Now the concatenated text deltas always equal the answer.
8. Test-only switches WP_DURABLE_FAUX_SCRIPT (responses scripted at start, for the kill-and-restart test) and WP_DURABLE_TEST_CALLBACK_DELAYS_MS.
9. scripts/build-site.py (outside the brief's file list) publishes this handover on the docs site.
10. The container runs the e2e files one after another (a starting Server kills the agent user's processes).

Acceptance

piinchrome-1c, production, real models, 2026-10-11:

Logs of a69e0309 and 18c9c20a (wpDurable scan, up to 5000 lines each): no token, no API key or key fragment, no sk-ant, no bearer header.

Known incidents

• 2026-10-11 · Long polls of 30 s or more were reset on the way to the client (found by piinchrome-1c in the regression on 7626a6a9). GET /v1/submissions/:id/wait held a silent response up to timeoutMs (default 30 s, at most 60 s). Railway's edge resets a response that carries no bytes for 30 s: the Server and the edge logged 200 at 30001–30019 ms, and the client saw a TCP reset at 30.4–30.5 s; timeoutMs=60000, as the guide showed, never reached the client. Fix: the Server holds a long poll at most 25 s (WP_DURABLE_LIMIT_WAIT_MS, default 25000), then answers timedOut: true with the current status; the default timeoutMs is 25000; the accepted range stays 0–60000. The SSE routes send : hb every 15 s and the bridge's WebSocket pings every 15 s, so they were never affected; the SSE heartbeat is now kept at most at the long-poll hold. Rule for any new long-lived route: send a byte at least every 15 s, or end the response before 25 s. Deployed as dce3cd62-c88b-490e-b80c-335bf020eb6e (main 9b43360) during a running bash job: the resume test passed 8/8 (the job ended interrupted, the run answered, the callback was delivered once) and piinchrome-1c's final regression was all green, the wait-cap probe 6/6 (timeoutMs=60000 → 200 after about 25 s with timedOut: true; 70000 → 400).

• 2026-10-11 · A conversation without tools looped on fake tool calls (fixed by Change 5). A conversation created with agent.tools: [] still received the environment section, which named bash and http_fetch. The model (claude-fable-5-1, through the /fable mod) believed it had them, wrote <invoke name="http_fetch"> as text, got nothing back, and repeated it until the model's 128K output cap: one generation of 26 min, 128,006 output tokens, $6.41. Fix: the environment section names only the tools the request offers and is left out when it offers none of them; a request without tools gets a no-tools section; every model request carries the provider's max-tokens parameter (default 32000, WP_DURABLE_LIMIT_OUTPUT_TOKENS, per submission limits.outputTokens). Cost facts for Steven, as information only: the cap bounds one request's output (a fixed thinking budget comes on top; adaptive thinking counts within it); at claude-fable-5-1's output price the same generation would now stop near 32000 tokens, about a quarter of the incident's cost; openai-codex models get no cap (the ChatGPT backend has no such parameter). By Steven's decision the Server has no spend or turn limit; spend limits are set on the provider platforms. Deployed as e19d853f-2ae1-49c5-a913-69b8ee4de23b (main 22ac962). Production check (claude-haiku-5-5, conversations 671 and 684, $0.00007): limits.outputTokens: 40 → stopReason: "length" after 40 output tokens (all thinking, empty answer); a tools: [] conversation got <no-tools> and no environment section, and the model said it cannot run tools. Accepted by piinchrome-1c on e19d853f, 12/12: the original scenario (claude-fable-5-1, thinking max, tools [], "今天都有些什么新闻？", limits.outputTokens 4000) now answers honestly in 6 s for $0.013 with stopReason: "stop" and no fake tool syntax (conversation 697); gpt-6-sol with tools [] answers in one sentence (710); an all-tools haiku conversation names its tools, and outputTokens 40 settles done with stopReason: "length" (723); limits.costUsd → 400; status shows limits.outputTokens 32000; no token in entries.

Known limitations

• One token, one OS user for all conversations: a conversation can read another conversation's work directory. Never put secrets in a work directory.
• No network restriction for bash (Railway has no egress control); only http_fetch and callbacks apply the address rules.
• No budget cap; Anthropic calls bill the key.
• A file swapped by a racing process can expose a name or size through a directory listing (never content).
• Callback attempts are at-least-once across a restart; receivers should be idempotent on x-wp-durable-submission.
• Railway warns that railway.json config-as-code is deprecated; it keeps working until 2026-12-01.
• A request in flight while the Server shuts down for a restart can answer 500 "Session is closed" (seen once: a long poll during the resume test). Retry after the restart.

Open items

None. ANTHROPIC_API_KEY is set (2026-10-11); rotate it by replacing .secrets/anthropic-api-key and running the command of STAGE1-HANDOVER.md § Change 2, then a redeploy.