Which code this compares
Three things carry the name "relay v5". Only one of them is the release. Get the baseline wrong and every comparison below is wrong with it.
| Item | V4 (shipping) | V5 (next release) |
|---|---|---|
| Repository | mte-relay-server-go, branch master | unified-mte-relay, branch develop |
| Version | 4.7.0 (MARversion.txt and MRSversion.txt) | 5.0.0-beta.4; SocketX Server 2.0.0-beta.2; TCPQ 1 |
| Wire protocol | Custom headers plus Base64 ciphertext in the URL and a header | Frame version 2, one binary frame layout for HTTP, WebSocket, TCP |
| Browser SDK | mte-relay-browser 4.4.1 | mte-relay-browser 5.0.0-beta.12, public client 5.0.0-beta.10 |
| Relay errors | HTTP 559 to 569 | HTTP 470 to 490 with reason tokens, plus 500, 502, 503 |
| Kyber | 512, hardwired | 1024 default; 512, 768, or 1024 per deployment |
mte-relay is the unreleased frame version 1 prototype; even its main branch reads 5.0.0-beta.1. The design site at misc.trevorblackman.dev/relay-v6 calls the rewrite "v6" and the prototype "v5". Product naming is now V5 for the rewrite. Every published benchmark compares the prototype with the rewrite, not V4.What V4 already has
- Go rewrite (4.5),
DOMAIN_MAPmulti tenant routing (4.5), MKE streaming and SSE flushing (4.5), Kyber pairing, Redis shared state, correlation ids, zerolog events. - Cloud marketplace billing strategies with on premises and AWS immutable images (4.6), an AKS extension (4.6),
X-Forwarded-Hoststamping (4.7). - An outbound (MAR) pair pool that already grows on demand: 1,024 initial pairs per origin, grow by 256, cap 8,192, 5 second acquire timeout.
- Opt in pprof and per stage perf stats.
One binary frame on every transport
V4 spreads one protected request across the URL, two custom headers, and the body, with Base64 on two of the pieces. V5 sends frames: a 12 byte envelope, then a kind specific payload. The same envelope carries HTTP, WebSocket, and TCP.
| Envelope field | Size | Rule |
|---|---|---|
| Signature | 3 | 0x89 0x4D 0x54. The V1 signature MTE is recognised forever and answered with 478 naming the minimum SDK versions. |
| Version | 1 | 2. Checked before any width dependent field, so a second layout can coexist later. |
| Kind | 1 | REQUEST 0, RESPONSE 1, DATA 2, ERROR 3, OPEN 4, OPEN_ACK 5, CLOSE 6, PING 7, PONG 8, GOAWAY 9, REKEY_REQUEST 10. Unknown kinds are rejected. |
| Flags | 1 | END (HTTP), MORE and TEXT (WebSocket), HALF (TCP). Bits 4 to 7 reserved and rejected when set; bit 6 is held for an authenticated control channel. |
| Reserved | 2 | Must be zero. |
| Length | 4 | Payload bytes. Compared to the advertised maxFrameBytes before any allocation; over the bound is 481 on HTTP and fatal on a connection. |
HTTP mapping
POST / with application/octet-stream. The request body is one REQUEST then zero or more DATA. The response is one RESPONSE then zero or more DATA, chunked, never with Content-Length. Every framed response carries Cache-Control: no-store, no-transform, private, nosniff, X-Accel-Buffering: no, and X-MTE-Relay-Request-Id.
What left the wire
Base64 on the data path, URL escaping of ciphertext, the x-mte-relay and x-mte-relay-eh headers, the 32 character text pair id, and any identifier on responses. V5 still uses Base64 in the pairing JSON and saved codec state, which is fine and is not the data path.
Verify first, release second
The V4 streaming path writes each DecryptChunk result onward immediately and checks the integrity tag only at FinishDecrypt. V5 makes every frame one complete operation that is verified before a single plaintext byte leaves.
Method, status, and flags are bound
The request metadata must carry v and method; the response metadata must carry v and status. The server compares them to the plaintext envelope; a mismatch is 483 header_mismatch, pair intact, logged as a security event. Every DATA plaintext starts with a copy of the flags byte, so a forged END, MORE, TEXT, or HALF is detected. In V4 the method and status ride outside the encryption.
Unframed is never data
A response that is not a valid frame is a transport error on every transport, whatever the HTTP status or content type. A CDN error page, a WAF block, an ingress 404, or injected plaintext lands in one bucket, MteRelayTransportError, with the status attached for logging.
Completion is explicit
On HTTP, EOF before END is 477 eof_before_end and poisons the pair. A failure after RESPONSE has started ends with an in band ERROR frame instead of a truncated body. A bodiless answer (204, 304, 1xx, HEAD, Content-Length: 0) is one RESPONSE frame with END and one operation, which retires the "browser skips 204 and desyncs" bug class.
Length 0 DATA rule
A zero length DATA performs no operation, so it may carry no flags. A direction that ends with no bytes sends a DATA whose plaintext is the flags byte alone. Without this, an on path party could append an unauthenticated empty END and turn a truncation into a clean completion.
Tests in unified-mte-relay/tests: TestCorruptResponseDataDeliversNoPlaintext, TestCorruptRequestDataNeverReachesTheOrigin, TestMethodByteRewriteIs483, TestMethodMismatchIs483, TestAnswerThatIsNotAFrameIsATransportError, TestTruncatedResponseFrameIsATransportError, TestBodilessAnswersAreOneFrame, TestServerRejectsFrameShapesBeforeDecode.
Canonical metadata, one operation
V4 encrypts the path and the headers JSON as two separate operations. V5 encrypts one canonical JSON object. Canonical means every platform produces the same bytes, so byte exact test vectors are possible.
| Member | Where | Rule |
|---|---|---|
v | required everywhere | The frame version, compared to the envelope. |
method | required on requests | RFC 9110 token, at most 32 bytes. Authoritative when the method byte is 255, so WebDAV and other methods work instead of silently becoming GET. |
status | required on responses | 100 to 599, compared to the RESPONSE status. |
path | requests | Path plus query, no leading slash, percent encoded as sent, at most 16 KiB. Readers refuse // and .. segments. |
headers | optional | Lowercase names, one key per name, a string or an array of strings when repeated. Limits: 256 names, 64 values per name, 8 KiB per value. |
target, port | TCP | An alias name from configuration, never a host and port; an optional port inside an allowed range. |
id | optional | Client correlation, unsigned integer. |
Canonical form
Keys sorted in byte order at every level, no insignificant whitespace, no HTML or slash escaping, integers in plain decimal. Readers reject duplicate keys; on a request that is 482 with the decoder state saved.
Repeated headers survive
Arrays keep every Set-Cookie, Vary, Link, and WWW-Authenticate value through a relay chain. V4's encrypted response headers kept only the first value.
Bounds are advertised, not hardcoded
maxMetadataBytes is enforced per MTE type (defaults 61,439 for MKE and 7,168 for standard MTE) and published in discovery. V4's 65,535 byte header ceiling and the V1 hardcoded constants are gone.
Pairing, tokens, and abuse limits
The pairing JSON and the cross wired Kyber exchange are unchanged in shape. What changed is what the server checks before it does any expensive work, and what it tells the client so both ends build identical codecs.
Kyber 1024 and a build profile
KYBER_STRENGTH defaults to 1024 (V4: MteKyberInit(512), hardwired). Discovery publishes mteProfile: library version, TOKBYTES, DRBG, cipher, hash, verifiers, Kyber strength, and a probe digest over two known encodes. A client compares before pairing; the pair request repeats it so the server can refuse with 490 profile_mismatch. A client build against a server build interoperates; an ANY build against either does not, and V5 says so at auth instead of failing on the first decode.
The 40 byte token
16 byte client id, 8 byte issued at, and 16 bytes of HMAC SHA256 under the domain secret. Text form is 54 characters of base64url on auth, pair, keepalive, and OPEN; the REQUEST frame carries the raw 40 bytes. Max age (default 24 hours, advertised) is checked at pair, keepalive, and the first OPEN; the client refreshes proactively by calling auth with its current token, keeping its id and pool. V4's id is a UUID plus 64 hex characters of HMAC, echoed on both directions.
Pair ids and personalization
Pair ids are 16 random bytes, unique per client, collisions rejected, never re paired while live. Personalization strings follow one grammar: mte-relay/2/http/, 22 or more characters of randomness, then the relay host, at most 1,024 bytes, checked on both strings. That blocks pair reset and cross transport replay.
Rate limits and replay
Auth and pair are limited per source, pair and keepalive per client id, never the data path. Defaults: auth 600 per minute, pair 600, keepalive 120, answered 488 rate_limited with Retry-After. Replay to the same relay is 479 replay_rejected with the pair intact and a security event. sequenceWindow (minus 63) and timeWindow (1000) are pairing inputs taken from discovery, because a mismatch is no pair at all. V4 has no rate limiting.
An error block infrastructure leaves alone
V4 answers relay faults with 559 to 569. Envoy and Istio outlier detection eject a pod after five consecutive 5xx, and two of those codes are reachable by anyone with curl. Cloudflare, Front Door, and App Gateway replace 5xx bodies with their own pages. V5 moves relay generated errors to 470 to 490, which is clean against every load balancer and CDN checked.
| Code | Reason | Client action |
|---|---|---|
| 470 to 474 | pair_not_found, decoder_restore_failed, encoder_restore_failed, decode_failed, encode_failed | Replace that one pair. |
| 475 | invalid_client_id, invalid_token, unknown_key | Full repair at most once per 30 seconds; re auth; on unknown_key stop and surface to the operator. |
| 476 | pool_exhausted | Back off. |
| 477 | invalid_frame, eof_before_end | Surface with pair intact; on eof_before_end replace the pair. |
| 478 | frame_version_unsupported | Surface; the message names the accepted versions and minimum SDKs. |
| 479 | replay_rejected | Surface, pair intact, log a security event. |
| 480 | no_domain_for_host | Surface; the body names the Host the relay saw. |
| 481 | frame_too_large, body_too_large, message_too_large | Surface, pair intact on HTTP; close and reopen on a connection. |
| 482, 483 | metadata_invalid, header_mismatch | Surface, pair intact; 483 is a security event. |
| 484, 489 | target_forbidden, port_out_of_range, authz_revoked, target_reset | Connections only; reset the local socket; stop retrying a forbidden or revoked alias. |
| 485, 486 | upstream_capacity, pair_busy, not_owner | Back off; retry once then replace; owner mode backoff. |
| 487, 488 | rekey_overdue, connection_limit, rate_limited | Reconnect with a new OPEN; back off with jitter and honor Retry-After. |
| 490 | invalid_request, pair_exists, pair_limit, profile_mismatch | Validation on the JSON endpoints; profile_mismatch is a fatal setup error. |
| 500, 502, 503 | internal, configuration; upstream_connect_failed (nothing executed), upstream_response_failed (executed); state_store_unavailable, state_store_write_failed | True faults stay 5xx on purpose. The 502 split tells the caller whether the origin ran the request. 503 unavailable retries after Retry-After; write failed replaces the pair. |
Two carriers
The body of every relay generated response is an ERROR frame: a uint16 code and canonical JSON with reason, optional message, and an optional nested cause from a MAR hop. The same code and reason travel in X-MTE-Relay-Error in case a proxy strips the body. A body under a relay status that is not a frame is treated as a replaced page.
Registry driven clients
Every code has a registered default action. A receiver accepts an unknown reason of registry shape under a known code and applies the code's action, so a newer relay does not break an older client. The 559 to 569 block is retired and recorded with its old meanings so it can never be reassigned.
Connections
ERROR is sent in band and the connection closes. On WebSocket the close code is 4000 plus the code minus 400, with the reason token as the close reason, and 1000 to 1015 have their own mapping for library and load balancer closes.
Clients read the server instead of hardcoding it
V4's HEAD /api/mte-relay returns a signed client id in a header and nothing else. V5's GET /api/mte-relay?frameVersions=2 negotiates the frame version before any Kyber work and returns a discovery document. The query parameter, not a header, keeps a cross origin fetch free of a preflight.
What is advertised
discoverySchema, buildVersion, frameVersions, features, mteProfile, configVersion, kyberStrength, sequenceWindow, timeWindow, maxFrameBytes, maxMessageBytes, maxMetadataBytes, maxPairsPerBatch, maxPairsPerClient, maxPairBytes, maxPairSeconds, maxRequestSeconds, maxResponseSeconds, idleTimeoutSeconds, pingIntervalSeconds, drainSeconds, sessionTimeoutSeconds, tokenMaxAgeSeconds, transports, stateMode, routeHeader, maxRequestBodyBytes, passThroughRoutes, forwardPlainHeaders, forwardBrowserCookies.
What SDKs do with it
Size DATA chunks from maxFrameBytes. Tune keepalive from sessionTimeoutSeconds. Refuse a body over maxRequestBodyBytes before uploading it. Send pass through routes as plain fetches. Clamp the pool to maxPairsPerClient. Refuse to pair on an mteProfile mismatch. Key behavior on features, never on buildVersion. Reread discovery when a response carries a higher X-MTE-Relay-Config-Version.
http, keepalive, plain-headers, browser-cookies, pass-through, and relay-error-header, with transports: ["http"]. The ws, rekey, goaway, and tcp names are registered for SocketX and TCPQ, which are separate binaries.Keepalive, capacity, and per pair recovery
V4 has no keepalive endpoint. Pair state expires after 1 hour in memory or 24 hours in Redis with no way to refresh it, so an idle client pairs again from scratch and a fleet waking together stampedes the relay with Kyber work.
| Setting | V4 (4.7.0) | V5 |
|---|---|---|
| Keepalive | None | POST /api/mte-keepalive with {clientId, drop}: one call refreshes every pair of the client and deletes the dropped ones. Rate limited per client. |
| Idle eviction | 1 h memory, 24 h Redis, no refresh | SESSION_TIMEOUT_MINUTES 120, refreshed by traffic or keepalive; advertised so SDKs tune their interval. |
| Server codec pool | POOL_SIZE 600, maximum 1,000 | DEFAULT_POOL_SIZE 250, grows on demand to MAX_POOL_SIZE_LIMIT 100,000 |
| Browser SDK pairs | 5 at init (4.4.1) | 1 at init, 4 minimum, 8 maximum, filled in the background after the first response |
| Go client and MAR outbound pool | 1,024 initial per origin, grow 256, cap 8,192, 5 s acquire wait | 25 warm minimum, background refill at 10 free, cap 100, 250 ms bounded wait, pairs reserved for open streams |
| Pairing caps | None | 100 pairs per batch, 200 per client, both advertised |
| Recovery | Clients typically wipe the whole pool on any 559 to 569 | Registered action per code: replace one pair, full repair (rate limited), back off, or surface |
Failures poison the right thing
A failed encoder or decoder state save poisons that pair and answers 471 or 472 instead of leaving stale keystream state in the store. Any local decoder error, abort, early close, or EOF before END on a pool pair replaces that pair. Test: TestStoreWriteFailureReplacesThePair, TestSurfacedErrorLeavesTheOnePairUsable.
No transparent retry
The client never re issues a failed application request on its own. It returns a typed error that says whether the request is retryable; the caller, who knows whether the request was idempotent, decides. Go's Do retries only when GetBody is set or the body is empty.
Frames as the unit of transfer
V4 buffers whole bodies in memory under standard MTE or when streaming is off (io.ReadAll in proxy.go), and streams 16 KiB chunks under MKE. V5 moves every body as bounded frames on the server, the Go client, and the browser upload path.
DATA budget
One function, wire.DataBudget, sizes every chunk: under MKE the advertised bound less a 1,024 byte reserve; under standard MTE (maxFrameBytes - 16) / 8, because a plaintext of n bytes encodes to 8n plus 4. No floor, no separate 64 KiB cap in five places in three languages.
64 KiB by measurement
MAX_FRAME_BYTES defaults to 65,536 (floor 4,096). On a 1 MiB body with everything else fixed, 64 KiB frames ran 45 percent faster than the earlier 1 MiB default and 3 percent faster than the prototype, at 7,445 extra bytes per round trip (0.36 percent). 16 KiB was slower than 1 MiB, so the curve has a floor and 64 KiB sits in it.
Response coalescing
RESPONSE_COALESCE_MS (default 2, per domain response_coalesce_ms, 0 turns it off) holds a response frame open after its first byte so a paced source of small writes shares frames. At 1,000 events per second with 100 byte events: 392 bytes per event at 0 ms, 210 at 2 ms, 143 at 10 ms. At 50 events per second the default does nothing; set 50 ms and pay that much latency per event.
Headers first
The server writes RESPONSE the moment the origin's status and headers arrive, without END, so the client sees them before the origin sends a body byte. END rides on the last data carrying frame. Test: TestFirstEventArrivesBeforeTheSecondIsSent.
Silence deadlines
The request read deadline is extended per frame, so a slow upload is bounded by silence, not by a fixed wall clock. Request, response, idle, and ping timeouts are per domain and advertised. V4's 30 second client timeout kills a slow upload outright.
Memory
1 GiB upload, measured against the prototype: 277 MiB/s at 44 MiB peak RSS, against 193 MiB/s at 1,050 MiB for the prototype's single chunk session. The browser SDK uses replayable blobs and bounded encryption batches, so a large upload never materializes whole plaintext and ciphertext buffers.
The control plane: UI and API for the whole fleet
This is the feature customers who have previewed V5 rank first. V4 is configured by environment variables and a restart. V5 ships an optional control plane in the same image: MODE=control runs the panel, the default MODE=relay runs the data plane. The API is the product; the panel is one client of it.
The UI
Dashboard
Workers online, in sync count, current config version, proxy count, and quick links. Refreshes every 5 seconds.
Fleet
One row per worker: hostname, id, address, version, uptime, last seen, a status badge (in sync, behind vN, stale, config rejected, partial), pending restart badges per field, and the rejection message when a worker refused an override set. Expand a row for the effective configuration it is running, with the origin of every value and secrets shown as keyed fingerprints. A restart guidance card renders the restart, scale, status, and logs commands for Kubernetes, OpenShift, Swarm, Compose, ECS, or systemd with your names filled in.
Proxies
A structured editor over DOMAIN_MAP, one page per host: routing (host, upstream, outbound toggle, personalization string, pass through routes), custom headers, security (client id secret with generate and reveal, a warning when a fleet is running auto generated secrets), and CORS (origins, methods, allowed and exposed headers, preflight max age, credentials, preflight passthrough). Add, rename, delete, an unsaved changes blocker, and a raw JSON fallback.
Configuration
Every schema field grouped, each badged live or restart, with description, default, reset per field, reset all, reveal on sensitive fields, a sticky save bar, and a dry run before every save whose warnings never block a write. 39 overridable fields in 7 groups; 5 apply live today: LOG_LEVEL, DOMAIN_MAP, MAX_REQUEST_BODY_MIB, MAX_CONTROL_BODY_MIB, FORWARD_HEADERS.
History
The last 50 versions with time, who, and which keys changed. Open any version, diff it, and roll back; a rollback installs the old snapshot as a new version, so the counter only moves forward and nothing is lost.
Service tokens and security
Named, scoped (read or config), fixed lifetime from 1 hour to 90 days, shown once, stored as a hash, revocable immediately, and attributed in the audit log as token:name. Admin sessions have a 15 minute sliding idle window and an 8 hour cap; the session signing secret rotates from the UI. Light, dark, and system themes; a prerendered login page; CSP hashes derived from the embedded bundle.
The API
| Route | Scope | Purpose |
|---|---|---|
GET /api/schema | read | Field schema: key, group, type, enum, dynamic or not, sensitive, default, description. Drives the whole Configuration page. |
GET and PUT /api/config | read, config | The sparse override document with its version as an ETag. Writes need If-Match; ?dry_run=true validates and reports the diff without writing. |
/api/config/domains/{host} | read, config | Per host DOMAIN_MAP reads, creates, replaces, moves, and deletes without touching other hosts. |
/api/config/history, /api/config/rollback | read, config | Version metadata, one stored version's document, and rollback by version number. |
GET /api/fleet, GET /api/effective-config | read | Every worker's last heartbeat; one worker's self reported running configuration with fingerprints for sensitive values. |
/api/tokens, /api/rotate-admin-secret | admin | Create, list, revoke service tokens; rotate the session secret. |
GET /api/poll, POST /api/report | worker | The worker poll with heartbeat headers, and the effective configuration report the panel requests with X-Effective-Report: wanted. |
Correctness machinery
Optimistic concurrency with 412 on a stale write. Fail safe apply: a worker that cannot validate an override set keeps its last good config, does not advance its applied version, and reports the error in its heartbeat. Boot sync: workers fetch overrides before validation and codec construction, so restart fields really take effect on restart. Live fields land in an immutable snapshot behind one atomic pointer load, so the hot path pays nothing. A live DOMAIN_MAP swap keeps each host's generated client_id_secret unless the override sets one, so an edit never invalidates active pairings.
Operations
Persistence is overrides.json, admin_jwt_secret, agent_tokens.json, and history/v<N>.json under CONTROL_PLANE_DATA_DIR, written atomically, so a copy at any moment is a valid backup. The panel is a deliberate singleton. Mixed version fleets degrade gracefully: unknown override keys are ignored with a warning and reported per worker. The API only grows within 5.x. An end to end suite deploys the panel, 3 workers, and 2 upstreams on minikube and asserts propagation of every live field, the fail safe path, and the restart path.
What changed in configuration
The single upstream settings V4 marked "deprecated, removed in v5" are gone. Everything routes through DOMAIN_MAP, and each domain entry carries far more than V4's six fields.
| Per domain field | V4 | V5 |
|---|---|---|
upstream, cors_origins, cors_methods, client_id_secret | yes | yes |
pass_through_routes | regex patterns | exact paths or /prefix/*; /api/mte- can never be pass through |
outbound_token plus x-mte-upstream request headers | yes | replaced by outbound: true and personalization_string per domain |
cors_allow_credentials, cors_allowed_headers, cors_exposed_headers, cors_max_age_seconds, cors_preflight_passthrough | no | yes; the relay's own X-MTE-Relay-* headers and Retry-After are always exposed |
headers (custom upstream headers) | global HEADERS only | per domain |
preserve_host, trust_forwarded_headers | no | yes, each with a documented caveat; trust_forwarded_headers only on relay fed domains |
forward_browser_cookies, forward_plain_headers | no | yes; the frame wins on a name collision; a hard exclude list can never be forwarded from the plain hop |
request_seconds, response_seconds, idle_seconds, ping_seconds, response_coalesce_ms | no | yes, advertised in discovery |
allow_plain_upstream | implicit | explicit opt in; an http:// outbound upstream is refused by default |
Environment variables
| V4 | V5 | Note |
|---|---|---|
UPSTREAM, CLIENT_ID_SECRET, CORS_ORIGINS, PASS_THROUGH_ROUTES, OUTBOUND_TOKEN, HEADERS | removed | All per domain in DOMAIN_MAP. |
POOL_SIZE (600, max 1,000) | DEFAULT_POOL_SIZE 250, MAX_POOL_SIZE_LIMIT 100,000 | Pools grow on demand. |
OUTBOUND_PAIR_COUNT, OUTBOUND_PAIR_GROW_COUNT, OUTBOUND_MAX_PAIR_COUNT, OUTBOUND_ACQUIRE_TIMEOUT | CLIENT_MIN_PAIRS 25, CLIENT_MAX_PAIRS 100, CLIENT_REFILL_THRESHOLD 10, CLIENT_PAIR_ACQUIRE_WAIT_MS 250 | Warm minimum, background refill, stream reservations. |
FORWARD_HOST_HEADER | FORWARD_HEADERS | Stamps all three forwarding headers; live via the control plane. |
RESEED_THRESHOLD | removed from the server | The browser SDK retires a pair at 90 percent of the DRBG reseed interval. |
| none | MAX_FRAME_BYTES 65536, RESPONSE_COALESCE_MS 2, SESSION_TIMEOUT_MINUTES 120, MAX_REQUEST_BODY_MIB 0 (inclusive cap, 481 over it), MAX_CONTROL_BODY_MIB 4, KYBER_STRENGTH 1024, SEQUENCE_WINDOW, TIME_WINDOW, USE_MKE, USE_STREAMING, HTTP_CLIENT_TIMEOUT_SECONDS | New protocol and limit knobs. |
| none | MODE, CONTROL_PLANE_URL, CONTROL_PLANE_TOKEN, CONTROL_PLANE_POLL_SECONDS, CONTROL_PLANE_DATA_DIR, ADMIN_USERNAME, ADMIN_PW, ADMIN_JWT_SECRET, RELAY_RUNTIME | Control plane. Change ADMIN_PW before exposing the panel. |
REDIS_URL | REDIS_URL plus pool, timeout, and retry tuning; LOCAL_STATE_CACHE; STATE_MODE (shared only today) | Redis is still required for 2 or more replicas. The cache is correct only behind a hash proxy on X-MTE-Relay-Route. |
BILLING_STRATEGY, BILLING_PROVIDER, BILLING_PRODUCT | not present in the V5 relay tree | SocketX V2 images carry the marketplace metering providers; the Relay images do not yet. |
WebSocket on the same frame
SocketX V1 speaks a 7 byte plaintext prefix with no length field, no token, no pair id, and no error frame, with a three step handshake on /. V2 deletes that protocol rather than deprecating it; / answers 410 and names the real endpoints.
Endpoints and open sequence
GET /api/mte-relay?frameVersions=2 issues a token and the discovery document. GET /api/mte-ws is the socket, subprotocol mte-relay.v2, one frame per binary message, permessage-deflate refused because compression makes ciphertext length a function of plaintext. First message OPEN (token, host, one pair request with two Kyber public keys), then OPEN_ACK, then REQUEST and RESPONSE 101 carry the upgrade path and headers inside the ciphertext. Two round trips and about 9.5 KB of Kyber material once per connection. Validation order is normative: token HMAC, frame bound, connection caps and open budgets, then Kyber, so a stranger never makes the server do the expensive part.
Connection semantics
One codec pair per connection, in memory only, never in a pool or a store, discarded at CLOSE. Fragmentation with MORE and a constant TEXT bit per run, bounded by maxMessageBytes, 1,024 fragments, and a wall clock deadline. Rekey is an encoded OPEN at 80 percent of maxPairBytes or maxPairSeconds; the server enforces 487 at the limit. PING and PONG keep the pair and every load balancer idle timer alive; GOAWAY drains over drainSeconds. Repeated headers keep every value, so a repeated Set-Cookie from the upstream survives the hop back.
Operational surface
Per source and per client id connection caps, open rate limits with burst, origin validation before the upgrade (absent origin accepted on purpose, present but disallowed refused), host matching most specific first, preserveHost, forwarded headers stamped by the server and refused from the client's metadata, /healthz, /readyz, and an echo endpoint. Billing per completed application message on AWS and Azure, failing open with /readyz reporting degraded.
Client
SocketXClient carries the standard WebSocket interface, so a caller swaps the constructor. Headers travel inside the ciphertext. send before open throws rather than buffering plaintext. Relay errors arrive as close events; relayCodeFromClose and defaultAction map them back to the registry. The client owns its reconnect policy.
TCP applications without an SDK
tcpq-agent runs beside the application, holds the client key, and turns each accepted local connection into one tunnel. tcpq-relay runs beside the destinations, exposes a frame listener and an HTTPS control plane, and dials only the alias the client named.
Authorization
A gateway cannot mint identities freely the way the HTTP relay does, because the relay itself is the authorization boundary. Every client holds a preshared key issued by the operator; the client id in its token derives from the key id, so a token proves which key minted it with no store. Token refresh presents the key again, gateway tokens live minutes, revoking a key ends live tunnels within the policy refresh interval, and the relay refuses to start with an alias whose allow list is empty.
Aliases
A client names an alias, never a host and port. An alias carries host, port, allow, tls with sni and caFile (no insecure skip), an optional allowPorts range, allowPrivateNetwork off by default (which keeps an alias from reaching the cloud metadata endpoint), pinAddresses, resolveTtlSeconds, proxyProtocol v2, per alias and per client connection caps, and coalesceMs, off by default and never on anything a person types into.
Data path
OPEN, OPEN_ACK, REQUEST with method CONNECT and target, RESPONSE 200 after the relay has connected, then DATA both ways. Backpressure is normative: each direction is one goroutine that reads, converts, and writes before it reads again, so a slow target slows its sender instead of growing a buffer. HALF is a FIN in that direction. A target reset is 489 and the agent resets the local socket, so a truncated stream is never delivered as complete. Draining: fail readiness, wait for the balancer, SIGTERM sends GOAWAY and closes at drainSeconds.
Named plainly
Rekey thresholds and the server side rekey budget (a tunnel runs on one pair for its life, bounded by maxLifetimeSeconds); SOCKS5 on the agent; the FTP control channel helper; a control plane push for revocation (today: edit the file and restart); per client banning; UDP; server initiated streams. Control frames are plaintext in version 2, so run the tunnel inside TLS, which the relay enforces. Tunnels do not survive a relay upgrade.
One spec, one registry, byte exact vectors
Normative text
spec/frame-v2.md, frozen at release candidate 1 with errata only after that. V4 and the prototype had a descriptive protocol.md duplicated per repository and no shared fixtures.
Registry
spec/registry.md reserves every value: versions, signatures, kinds, flag bits, MTE types, method bytes, codes and reasons, retired codes, metadata members, discovery fields, feature names, headers, paths, subprotocols, personalization prefixes, close reasons, and WebSocket close codes. A value once shipped is never reassigned. vectors/registry.json is the machine readable authority and both language tables are tested against it.
Vectors and conformance
spec/vectors/ holds envelope, request, response, data, error, control, metadata, path, personalization, discovery, probe, and WebSocket fixtures, byte identical between Go and the TypeScript engine. An implementation that disagrees with a vector is wrong. The metadata vectors include the four platform traps that produce valid but non canonical bytes.
Versioning policy
Three independent spaces: the frame version byte, discoverySchema, and feature names. A server accepts N and N minus 1 for 18 months after N ships; deprecation is announced in discovery at least 12 months ahead; after sunset the specific 478 stays for 12 more months; the MTE signature is detected forever. Each MAR hop negotiates independently and the effective feature set on a chain is the intersection.
Test origin
testorigin/ answers HTTP fixtures and a WebSocket echo in one process. Every suite proxies to it; it is published as an image so customer integrations can be validated against the same origin.
Product separation
One module, three products that share wire, conn, shared, spec, and mte and nothing else. A customer running one product gets a binary that contains only that one; the build fails otherwise.
What can be quoted, and how
Two kinds of numbers exist: an estimate derived from the two wire formats, and measurements of the rewrite against the unreleased prototype. No measurement of released V4 against V5 exists yet.
Estimated protection overhead per round trip
Assumptions: a TOKBYTES 8 MKE build (260 bytes per operation); V4's 101 character signed client id plus 32 character pair id echoed on request and response; standard Base64 with about 6 percent URL escaping on the path; the same header bytes encrypted in both versions; V5's route header counted; request id and correlation id headers treated as equal; TLS and HTTP framing excluded. Base64 alone turns 3,072 ciphertext bytes into 4,096 characters.
Measured: rewrite against the prototype
| Measurement | Prototype (frame v1) | V5 (frame v2) |
|---|---|---|
| 1 GiB upload, throughput and peak RSS | 193 MiB/s at 1,050 MiB | 277 MiB/s at 44 MiB |
| 1 GiB download, wire overhead | 0.049 percent | 0.028 percent |
| 256 MiB download from a 10 MiB/s origin | 0.051 percent | 0.670 percent at 2 ms, 0.049 percent at 50 ms |
| Paced SSE, 100 B events at 1,000 per second, bytes per event | 118 | 392 at 0 ms, 210 at 2 ms, 143 at 10 ms |
| HTTP round trip, 1 KiB, loopback | 0.604 ms | 0.689 ms at 64 KiB frames |
| HTTP round trip, 1 MiB, loopback | 9.888 ms | 9.617 ms at 64 KiB frames |
| Bytes per round trip, 1 KiB payload | 4,394 | 4,261 |
| Syscalls per round trip (write, read, epoll) | 13.3, 13.7, 15.3 | 5.1, 9.5, 8.3 |
Absolute costs on loopback
Below 1 KiB the relay adds a flat cost per round trip: about 0.51 ms and 1,880 bytes on HTTP, about 0.23 ms and 546 bytes on SocketX and TCPQ (two 273 byte DATA frames). Above 100 KiB all three transports converge on 130 to 165 MiB/s per connection, which is the codec, not the carrier. Connection setup with Kyber: about 10 ms and 14,860 bytes on HTTP, 8.3 ms and 18,983 bytes on SocketX, once per connection.
Caveats
The flat 0.085 ms small payload gap against the prototype is a client serialization artifact (the V5 client encodes REQUEST and DATA into one buffer instead of overlapping them), profiled and recoverable, not the design. Standard MTE mode is not a usable setting: 2,832 ms at 100 KiB against 2.03 ms for MKE. No WAN figures, single connection latency only, WSL2 loopback. Concurrency and soak suites exist but have not been run.
.bench and .reports/relay-v6/benchmarks is reusable and needs a V4 arm.Checklist for a V4 to V5 cutover
- Inventory SDKs and pin versions. Browser
5.0.0-beta.12, public browser client5.0.0-beta.10, the Go client in tree, the SocketX TypeScript client (vendored, not on npm yet), a C#MteHttpClientthat is a solution skeleton today, mobile SDKs still to be built from the frozen spec. A V4 SDK against a V5 relay gets a 478 naming the minimum version. - Stand V5 up beside V4. New hostname or path, its own pool, its own Redis keys. Never mix protocol generations behind one upstream pool.
- Move configuration. Drop
UPSTREAMand friends forDOMAIN_MAP; convert regex pass through routes to exact or prefix; set an explicitclient_id_secretper domain so every replica shares it; rename the pool and outbound variables. - State store. Keep
REDIS_URLfor 2 or more replicas. EnableLOCAL_STATE_CACHEonly behind a consistent hash proxy onX-MTE-Relay-Route. V4 Redis data is not migrated; clients pair again. - WAF, ingress, CORS. Allow
application/octet-streamrequest bodies (CRS rule 920420 blocks them by default on Azure WAF and ModSecurity). Disable automatic retries and response buffering on encrypted routes. If a domain customizes its CORS lists, includeX-MTE-Relay-Routein both allowed and exposed headers. - Observability. Move alerts from 559 to 569 to the 470 to 490 block and the
X-MTE-Relay-Errorheader. IngestX-MTE-Relay-Request-Id. Keep 500, 502, and 503 as the true fault signals. - Control plane, if wanted. Run one
MODE=controlinstance on a volume, setCONTROL_PLANE_URLand the shared token on workers, changeADMIN_PW, create scoped service tokens for CI, and take a backup ofoverrides.jsonbefore the first big edit. - Cut over per application. Ship the matching SDK and server together in a short window; expect sessions to re pair; keep V4 running until the observation period ends.
- SocketX and TCPQ separately. Treat SocketX V2 as a protocol migration even though the product keeps its name; an image only update breaks V1 clients. Pilot TCPQ with the alias list, the key issuance flow, and the reconnect story agreed first.
Claims to keep out of announcements
- "Quantum resistance is new." V4 already pairs over Kyber. The strength moved from 512 to 1024.
- "Streaming, SSE, or dynamic outbound pools are new." V4.5 added streaming; V4.7 grows outbound pools on demand. V5 verifies per frame and manages capacity better.
- "No Base64 anywhere." Pairing JSON and saved state still use it. The data path does not.
- "Frames decode independently of prior state." Frames are individually verified, but the codec is stateful. No random access, no out of order decode.
- "Control frames are authenticated." OPEN, PING, PONG, CLOSE, GOAWAY, REKEY_REQUEST, and zero length DATA are plaintext in version 2; a forged one costs a reset. Bit 6 is reserved for the fix.
- "Owner mode, fenced leases, encrypted state at rest." Designed and reviewed, not implemented.
STATE_MODEexists as a knob marked phase 7; the request path has no lease and no state encryption. - "The HTTP relay upgrade includes WebSocket and TCP."
transports: ["http"]. SocketX and TCPQ are separate builds. - "V5 is X times faster than V4." Not measured. See the numbers section.
- "Marketplace metering is unchanged." The V5 Relay images do not carry the billing providers yet; SocketX V2 images do.