fix(mcp): make consumer-mcp-key vault a best-effort cache (stop /mcp 502s)

@kaavee315checks n/achecks…fix/mcp-consumer-key-best-effort-vault → main8 files · +389 −19review: @shamsharoonupdated 1w ago

GitHub

▸Description· 1 comment · 1 noise

Description

Symptom (found via Datadog). POST /mcp on composio-dashboard returns 502 to users. Every failure is [/mcp] consumer-mcp-key-resolution-failed → cause_tag=VaultWriteError, cause_inner=UnprocessableEntityException (HTTP 422), across many orgs.

Root cause. getOrMintConsumerMcpApiKey mints a valid 7-day key (succeeds), then persists it to the vault. Today:

HCP Vault write fails ~180×/day (5s jwt-login timeouts + other HCP errors) → runtime fallback to WorkOS Vault.
WorkOS createObject then fails with 422 (UnprocessableEntityException, non-conflict — a WorkOS-side limit not surfaced in detail).
That VaultWriteError propagates → ApiKeyResolveError → badGateway() = 502, discarding a perfectly good minted key.

This is a regression in disguise: the vault is only a cache, but a cache-write failure was treated as a hard dependency. (A larger spike of the same class hit on 2026-05-27 — ~34k failures in ~12h, version 0d31338 — under the WorkOS-only code; the dual-backend migration reduced but did not eliminate it.)

Fix. Make the vault best-effort on both legs in consumer-mcp-key.ts:

mintAndPersist: a persist failure is logged (error, step=persist-best-effort-failed) and swallowed — the freshly-minted key is still returned. The next request re-mints.
getOrMintConsumerMcpApiKey: a hard vault read failure (anything that isn't already-handled NotFound) is logged (warn, step=read-best-effort-failed) and treated as a cache miss → mint.
ResolveError narrowed to the mint-only errors that can still surface.

Only an inability to mint remains fatal. No security change: HCP per-user JWT scoping and backend project-membership verification (which the mint re-checks against the caller's token) are untouched — we never serve another user's key.

Also widens the mcp-auth E2E trigger to cover consumer-mcp-key.ts / hashicorp-vault.ts / workos-vault.ts.

Note: this does not fix why HCP/WorkOS writes fail (HCP jwt-login timeouts; WorkOS 422 limit) — those are tracked separately. This stops the user-facing 502s and keeps the cache best-effort regardless of which vault layer is flaky.

How did I test this PR

New unit suite tests/unit/consumer-mcp-key.test.ts (+ vitest.config.unit.ts, pnpm test:unit, and a CI: Unit Tests workflow). Cases:
- cache hit → returns cached key, no mint, no write;
- persist fails (422) → still returns the minted key (the exact 502 regression) + logs persist-best-effort-failed;
- hard read failure → treated as miss → mints + logs read-best-effort-failed;
- forceMint → mints, skips read, tolerates write failure;
- mint failure → stays fatal.
Could not run typecheck/lint/unit tests locally: the install needs the org-private @composio/db (requires NPM_TOKEN), and doppler isn't installed on this machine. I verified the diff by hand (Effect error-channel types, no unused imports, tests/ is excluded from tsgo). CI will execute: Typecheck, the new Unit Tests job, and the MCP Auth E2E (now triggered by this path, exercising the real /mcp consumer-key flow against a Vercel preview).

@kaavee3151w ago

Added: vault failure diagnostics (commit 48df38d6)

The 502 fix makes vault failures non-fatal, but the causes were nearly invisible. This commit closes the blind spots (logging only, no behavior change):

HCP (hashicorp-vault.ts) — the three silent !res.ok paths now log status + Vault error body:

step=jwt-login-rejected — the dominant current failure (~182/day VaultJwtLoginError); logs status, addr, jwtAuthPath, namespace, used_dashboard_token, body.
step=kv-read-failed, step=kv-write-failed.

WorkOS (workos-vault.ts) — describeWorkosError() surfaces request_id / code / errors[] / rawData (which flattenCauseChain dropped) on the opaque 422, plus a logged updateObject failure path.

Datadog queries to use once deployed

service:composio-dashboard env:production @step:jwt-login-rejected
service:composio-dashboard env:production @feature:workos-vault "createObject failed"   # → @workos_error.request_id

Root-cause summary (infra-side, not in this PR)

Cause	Evidence	Owner action
05-27 502 spike	15,244 HCP `getaddrinfo ENOTFOUND`, all in 05-27 00:00–12:00; DNS resolves now	Resolved (cluster DNS propagation). Monitor.
Ongoing HCP fallbacks	~182/day `VaultJwtLoginError` (Vault rejecting the JWT), not DNS/timeout	Fix JWT-auth role: confirm `HCP_VAULT_USE_DASHBOARD_TOKEN`, Vault trusts the dashboard JWKS/issuer, role `bound_audiences`/`user_claim`, and the KV policy template matches the path. The new `jwt-login-rejected` body will show which.
WorkOS 422 (broken fallback)	All 422, successful updates → fails on object creation; started 05-27 11:03

▸ 1 bot/status comment hidden

@vercel[bot]1w ago

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
dashboard	Error		May 30, 2026 2:52pm

loading diff…

@kaavee3151w ago

Added: vault failure diagnostics (commit 48df38d6)

The 502 fix makes vault failures non-fatal, but the causes were nearly invisible. This commit closes the blind spots (logging only, no behavior change):

HCP (hashicorp-vault.ts) — the three silent !res.ok paths now log status + Vault error body:

step=jwt-login-rejected — the dominant current failure (~182/day VaultJwtLoginError); logs status, addr, jwtAuthPath, namespace, used_dashboard_token, body.
step=kv-read-failed, step=kv-write-failed.

Datadog queries to use once deployed

service:composio-dashboard env:production @step:jwt-login-rejected
service:composio-dashboard env:production @feature:workos-vault "createObject failed"   # → @workos_error.request_id

Root-cause summary (infra-side, not in this PR)

Cause	Evidence	Owner action
05-27 502 spike	15,244 HCP `getaddrinfo ENOTFOUND`, all in 05-27 00:00–12:00; DNS resolves now	Resolved (cluster DNS propagation). Monitor.
Ongoing HCP fallbacks	~182/day `VaultJwtLoginError` (Vault rejecting the JWT), not DNS/timeout	Fix JWT-auth role: confirm `HCP_VAULT_USE_DASHBOARD_TOKEN`, Vault trusts the dashboard JWKS/issuer, role `bound_audiences`/`user_claim`, and the KV policy template matches the path. The new `jwt-login-rejected` body will show which.
WorkOS 422 (broken fallback)	All 422, successful updates → fails on object creation; started 05-27 11:03