PR Custody

PRPR Custody

fix(ci): fail embeddings bundle workflow on missing cache and other improvements

@wjayeshchecks n/achecks…fix/embeddings-bundle-fail-loud → master20 files · +987 −118updated 3w ago

GitHub

▸Description

Summary

Fail loudly when the canonical cache_bundle.json cannot be downloaded from R2; network/integrity failures are no longer treated as cache misses.
Add adaptive AWS retries only around large R2 copy operations, while keeping credential/probe calls fast.
Snapshot the current canonical bundle to a dated daily backup before overwriting it, and serialize workflow runs with concurrency so shared R2 keys do not race.
Fail explicitly on canonical bundle upload errors.
Share the built cache bundle between workflow jobs with short-lived GitHub artifacts to avoid repeated 2.7 GiB R2 downloads.
Alert Slack for staging/scheduled failures as well as production failures.
Use the dedicated EMBEDDINGS_OPENAI_API_KEY for embeddings metadata, related-tools generation, and OpenAI artifact embeddings.
Make full bundle builds fail on critical pipeline failures instead of saving/uploading sparse bundles.
Enforce slug-delta metadata generation: existing tool metadata/use cases are reused, new tool slugs may generate, and missing generated data for an existing slug is treated as corruption.
Allow missing vectors to regenerate from existing generated metadata/use cases, including OpenAI artifact embeddings.
Read existing tool slugs from TurboPuffer row attributes correctly (row.attributes.tool_slug) and cover it with a regression test.
Treat only structured TurboPuffer 404s as missing namespaces; ambiguous routing/auth/API errors now fail loudly instead of degrading to empty namespace.
Propagate related-tools provider/model/schema failures instead of silently writing empty related-tool sets.
Update toolkit-scoped Thermos refresh so newly refreshed tools are classified as new after the merge.
Add design notes for future candidate promotion and fingerprint-based invalidation.

Why

Recent scheduled runs of Build Embeddings Bundle (R2) were cancelled at the 3h timeout or failed after long rebuild attempts. The trigger was a transient R2 connection drop on the ~2.7 GiB cache_bundle.json download (IncompleteRead(...)). The old workflow masked that as a missing cache, rebuilt from partial/no prior state, and uploaded a much smaller bundle that poisoned later runs.

This PR hardens the whole read-transform-write loop: source bundle reads must be real, critical provider/cache/Thermos failures stop the run before canonical upload, and the previous canonical bundle is snapshotted before overwrite.

It also prevents accidental full metadata regeneration. Generated metadata/use cases are currently slug-delta driven: new slugs can generate, existing slugs must reuse cached generated data, and missing existing generated data fails loudly. A separate doc captures the future fingerprint-based invalidation design for intentionally regenerating changed tools.

Notes

R2 remains the source of truth. GitHub artifacts are same-run transfer optimizations only.
cache-bundle is produced by build_bundle and consumed by downstream jobs instead of re-downloading from R2.
The daily backup keeps the first successful snapshot for a UTC date; later runs leave it untouched. The existence check uses exact-key list-objects-v2 rather than load-bearing stderr parsing.
Post-upload size equality verification was intentionally removed. S3 multipart completion is atomic; the real failure mode was a locally sparse/corrupt bundle being uploaded successfully, not a torn upload.
--from-scratch is removed from the canonical path. Future hash/fingerprint invalidation should replace slug-delta only in a separate PR.

Test plan

actionlint .github/workflows/embeddings_bundle_r2.yaml
git diff --check
bun test ./embeddings/src/pipeline/turbopuffer.test.ts
bun test ./embeddings/src/utils/generate_related_tools.test.ts
bun test ./embeddings/src/pipeline/tool_artifact_targets.test.ts ./embeddings/src/pipeline/turbopuffer.test.ts ./embeddings/src/utils/generate_related_tools.test.ts
Manual staging workflow_dispatch against this branch with namespace=composio-tools-pr-23551-smoke, use_last_checked_sha=false, reconcile_redis_deletes=false.
- Run: https://github.com/ComposioHQ/mercury/actions/runs/25866671749
- Validated cached R2 download, build, related-tools generation, daily backup, R2 upload, artifact upload/download, and related-tools backfill flow.
Not force-tested: R2 download failure, artifact storage outage fallback, serialized concurrency queueing, ambiguous backup lookup errors, Slack failure rendering.

loading diff…