Base incident runbook

This runbook applies after an approved Base rollout. It does not claim current deployment. Begin with operations and rollback and the Base contract boundary.

First response

Record UTC, PoP (EU/Dedic or US), candidate binary/config hashes, alert labels, and the last known healthy time. Do not paste credentials or keyed URLs into the incident record.
Check proxy health, Base readiness, journal errors, systemd memory events, and Prometheus scrape health. US Prometheus is loopback 127.0.0.1:9091; access requires the approved tunnel.
Determine whether the failure is edge routing, auth/tier, provider-budget store, upstream identity/TLS/head, feed decode/continuity, or resource pressure. Unknown and key_required capabilities are not outages.
If customer impact or route ownership is ambiguous, execute the no-Base rollback dry-run. Apply only with release authority; legacy QN Base remains off during rollback.

Alert inventory

These names exist in the accepted Base alert rules:

Alert	Meaning	Immediate checks
`BaseChainProberDown`	Mounted Base chain status is down for 5 minutes.	Registry mount, auth loopback, `eth_chainId=0x2105`, upstream health.
`BaseChainProberDegraded`	Chain status is degraded for 5 minutes.	Per-capability readiness and provider groups; do not promote a fallback.
`BaseRPCAvailabilityLost`	An admitted RPC family is unavailable.	TLS/identity/head, shared budget, catalog/category, zero forbidden dials.
`BaseWSAvailabilityLost`	An admitted WS family is unavailable.	Upgrade stayed on portal, auth/tier before dial, ACK plus real delivery.
`BaseFeedAvailabilityLost`	Admitted normalized feed is unavailable.	Decoder gate, source/version, last sequence, raw-feed lease, queue bounds.
`BaseAuctionAvailabilityLost`	Admitted auction feed is unavailable.	Commercial grant, secret resolver, budget store; never print the key.
`BaseFeedGapObserved`	At least one measured continuity gap occurred.	Preserve sequence/parent/reorg evidence; do not call replay data fresh.
`BaseProviderBudgetUnavailable`	Shared provider-budget admission cannot decide.	PostgreSQL availability and policy consistency across PoPs; fail closed.

The dashboard UID is triport-base. Sentinel definitions are base, base-ws, base-feed, and disabled base-auction in the accepted sentinel config. base-auction must not be enabled merely to silence missing-key state.

Routing diagnosis

Expected external edge behavior:

Request	Target/result
non-upgrade `GET`/`HEAD /rpc/base`	Next SEO profile
`POST /rpc/base`	portal/native JSON-RPC
WebSocket Upgrade on `/ws/base*`	portal protocol handler
WebSocket Upgrade overlapping `/rpc/base`	portal; never Next

Expected direct-origin behavior is stricter: exact /rpc/base never enters the SEO/static fallback. Non-POST returns 405 plus Allow: POST; POST reaches the native router if mounted. A wrong response service indicates edge/Caddy or origin middleware drift, not an upstream provider failure.

Resource and continuity incidents

If memory approaches MemoryHigh, inspect Base feed queue bytes/depth, decode errors, slow consumers, and Go heap before changing limits.
MemoryMax termination should be visible as a systemd/cgroup event followed by supervised restart. Repeated restarts require rollback; do not raise the cap during the incident without capacity evidence.
ACK/pongs with no events do not prove a healthy WS/feed. Check delivery age, head progression, gaps, decode errors, queue drops, and provider-budget availability together.
A stale head whose wall clock advances remains stale. Verify chain height and chain ID, not only process liveness.

No-Base rollback checks

Use the staged rollback command from operations. A valid dry-run proves native false, legacy false, no Base in the enabled QN CSV, and the previous-eight QN map intact. Apply mode uses cp, not mv, so a Docker bind mount continues to reference the updated inode.

After rollback, verify /rpc/base is unavailable at origin, Base is absent from the mounted runtime registry, the eight prior networks still pass their read-only smokes, and non-Base dashboards/alerts have not regressed. Preserve the failed candidate hashes and evidence for review; do not claim preprod_verified or deployed_verified from this procedure alone.