fix(rube/ingestion): add backoff to toolkit resolver retries

@zen-agentchecks n/achecks…zen/fix-toolkit-resolve-backoff-huibgb → master2 files · +155 −0updated 1mo ago

▸Description· 1 comment

Description

A single transient Composio API blip (<1s) was wiping out an entire ingestion plan group. The retry loop in get_toolkit_for_tool reattempted 3 times with no sleep, so all attempts failed within milliseconds and the function gave up; get_toolkits_from_tools then raised RuntimeError, which propagated through run_with_concurrency_limit and crashed the whole plan group:

2026-04-27T11:49:25Z ❌ Plan group processing failed with exception: Failed to resolve toolkit for tool slug 'HUBSPOT_CREATE_CONTACTS' (API error)
                       RuntimeError: Failed to resolve toolkit for tool slug 'HUBSPOT_CREATE_CONTACTS' (API error)
                       Traceback (most recent call last):
                       ❌ Task failed for item=<unknown>: ... (API error)

HUBSPOT_CREATE_CONTACTS is a real, current tool (verified via /api/v3/tools), so the failure is purely transient retry exhaustion — not a stale slug. The fix adds exponential backoff with jitter (1s, 2s + up to 1s jitter) between attempts so a quick blip doesn't burn every retry.

Existing semantics are preserved: 200 caches the toolkit, 404 caches None, exhausted retries return None without caching so the next call retries.

How did I test this PR

Added focused unit tests in app_tester/rube_learning/ingestion_pipeline/tests/test_toolkit_resolver_backoff.py covering: success on first try, 404 short-circuits, transient 5xx then success sleeps between attempts, request-exception retry, and all-attempts-fail returns None without caching.
Ran the new suite locally: pytest app_tester/rube_learning/ingestion_pipeline/tests/test_toolkit_resolver_backoff.py -v → 5 passed.
ruff check and ruff format --check clean for both touched files.
CI: Test - Learning Pipeline ✅ passed. Lint - Integrator is failing on master for pre-existing reasons (mypy errors in pii_mask.py, test_search_turbopuffer_optimized.py, etc. — files this PR does not touch); confirmed via gh api repos/ComposioHQ/integrator/actions/runs?branch=master.

Origin: cron-5d55c321e47a / zen-cron-3b1582bd7196 Triggered by: dhawal@composio.dev | Source: cron Session: https://zen-api-production-4c98.up.railway.app/dashboard/#/chat/zen-cron-3b1582bd7196

@zen-agent1mo ago

Post-PR status update

✅ Codex review (/codex-review-loop): "The change adds retry backoff behavior and accompanying tests without introducing a clear functional regression in the modified code paths." No actionable issues — LGTM.
✅ Test - Learning Pipeline CI: passed.
⚠️ Lint - Integrator CI: failed for pre-existing reasons. Same 11 mypy errors in 5 files (pii_mask.py, test_search_turbopuffer_optimized.py, test_workflow_search_no_truncate.py, test_tool_consolidation.py, test_tool_consolidation_skip_on_llm_failure.py) plus the same isort/black/flake8 failures fail on master HEAD. None of those files are touched by this PR. Verified via gh run view on both this PR's run and the latest master run.
✅ Unit tests for changed behavior: pytest app_tester/rube_learning/ingestion_pipeline/tests/test_toolkit_resolver_backoff.py — 5 passed locally.
⏭️ E2E/runtime checks: not feasible — get_toolkit_for_tool is a leaf helper called inside the ingestion-pipeline polling worker, which is not runnable from the sandbox (it pulls from prod ClickHouse, makes Composio API calls, writes to TurboPuffer). Local unit tests + staging deploy is the verification path.
✅ PR review comments: none filed yet.

Ready for review.

loading diff…

@zen-agent1mo ago

Post-PR status update

✅ Codex review (/codex-review-loop): "The change adds retry backoff behavior and accompanying tests without introducing a clear functional regression in the modified code paths." No actionable issues — LGTM.
✅ Test - Learning Pipeline CI: passed.
⚠️ Lint - Integrator CI: failed for pre-existing reasons. Same 11 mypy errors in 5 files (pii_mask.py, test_search_turbopuffer_optimized.py, test_workflow_search_no_truncate.py, test_tool_consolidation.py, test_tool_consolidation_skip_on_llm_failure.py) plus the same isort/black/flake8 failures fail on master HEAD. None of those files are touched by this PR. Verified via gh run view on both this PR's run and the latest master run.
✅ Unit tests for changed behavior: pytest app_tester/rube_learning/ingestion_pipeline/tests/test_toolkit_resolver_backoff.py — 5 passed locally.
⏭️ E2E/runtime checks: not feasible — get_toolkit_for_tool is a leaf helper called inside the ingestion-pipeline polling worker, which is not runnable from the sandbox (it pulls from prod ClickHouse, makes Composio API calls, writes to TurboPuffer). Local unit tests + staging deploy is the verification path.
✅ PR review comments: none filed yet.

Ready for review.