Description
Incident: On 2026-05-22 ~01:23 UTC, all outbound webhook deliveries started failing fleet-wide (59+ projects in a 1-min Datadog sample). apollo logged Failed to send external proxy request → Request failed with status code 500 for every delivery, and customers saw Error occurred while queuing webhook. Onset matched an external-proxy deploy exactly.
Root cause: The Sync secrets from Doppler to Cloudflare step fetched secrets inside jq command substitutions:
jq -n --arg cf_admin "$(doppler secrets get EXTERNAL_PROXY_ADMIN_KEY --plain)" ... | npx wrangler secret bulk ...
The pipeline's DOPPLER_TOKEN was invalid (Doppler Error: Invalid Auth token). On auth failure, doppler secrets get writes to stderr and returns an empty string. Because the failure is hidden inside a $() used as a jq argument, jq still exits 0, so wrangler secret bulk uploaded CF_ADMIN_API_KEY="" / APOLLO_CALLBACK_ADMIN_TOKEN="" and the step went green.
An empty CF_ADMIN_API_KEY is falsy, so it trips the worker's auth guard (if (!adminApiKey) return 500 in src/lib/auth.ts) → HTTP 500 on every proxy request → all webhook deliveries fail. (A wrong caller key would be 401; the 500 confirms the worker's own secret was blank.)
Fix to .github/workflows/external-proxy-deploy.yaml:
set -euo pipefail and fetch each secret in its own assignment, so a non-zero Doppler exit aborts the step via set -e (a failing $() inside a jq arg does not).- Reject empty/whitespace-only values before
wrangler secret bulk, with a ::error:: annotation pointing at DOPPLER_TOKEN_HERMES_*.
Net effect: a bad/expired Doppler token now red-fails the sync step (firing the existing if: failure() Slack alert) before the deploy step runs, instead of silently overwriting live Cloudflare secrets with blanks.
Note: this prevents recurrence. Restoring the currently-blank prod secrets (re-sync with a valid Doppler token / wrangler secret put) and rotating the expired DOPPLER_TOKEN_HERMES_PRD are separate operational follow-ups.
How did I test this PR
Reproduced both failure modes locally against the new shell logic:
- Doppler exits non-zero (the actual
Invalid Auth token case): a stubbed doppler returning 1 aborts at the variable assignment under set -e — the upload line is never reached (exit=1).
- Doppler exits 0 but returns blank: the empty-value guard prints
::error:: and exits 1 before wrangler secret bulk.
- Happy path (both values non-empty): passes the guard and proceeds to
jq | wrangler secret bulk unchanged.
bash -n syntax check passes; YAML structure verified.
@anshugarg15