fix(rube/ingestion): skip plan upsert when all consolidated tool slugs 404

@zen-agentchecks n/achecks…zen/skip-empty-toolkits-2wmagw → master4 files · +269 −1updated 1mo ago

▸Description· 2 comments

Description

When LLM tool consolidation produces a tool-slug list whose toolkits all 404 in the catalog (deprecated/removed tools), get_toolkits_from_tools returns []. Previously this still proceeded to:

The expensive plan-consolidation LLM call.
A TurboPuffer upsert that always fails validateEntry with Toolkits field is empty, logging a confusing 5-line ⚠️ Validation FAILED + ❌ Failed to update TurboPuffer block per plan.

In the last 24h we saw 25 such failures (up from 13 the run prior) on real cached plans whose stored tool slugs have since been renamed/removed — e.g. LINEAR_CREATE_LINEAR_ATTACHMENT, MICROSOFT_TEAMS_GET_TEAM, JIRA_GET_ISSUE_EDIT_META, HUBSPOT_RETRIEVE_EXISTING_OBJECT_SCHEMA. The catalog API correctly returns 404 for these (and get_toolkit_for_tool caches the 404), so when LLM consolidation re-emits an old slug, no toolkit is resolved.

Both code paths now short-circuit immediately after toolkit resolution when the consolidated tool list is non-empty but no toolkit resolved:

cached_plan_processor.process_cached_plan_group: marks the group skipped=True, leaves the existing TurboPuffer entry untouched.
unmatched_cluster_processor.process_unmatched_cluster: returns None so the cluster is skipped (matches the existing tool-consolidation-LLM-failure contract).

This saves one paid LLM call per failing plan and removes the noisy "Failed to update TurboPuffer" log lines from the ingestion dashboard.

Partial-resolution cases (some toolkits resolved, some 404'd) keep their current behavior — a single resolved toolkit is enough to continue.

How did I test this PR

Added tests/test_skip_empty_toolkits.py with 3 cases:
- cached-plan path: all tools 404 → skip, no LLM call, no upsert, log mentions "unresolvable" and "Existing TurboPuffer entry left untouched"
- cached-plan path: partial resolution → normal flow continues
- unmatched-cluster path: all tools 404 → returns None, no LLM call

Ran new tests + existing related tests:

pytest app_tester/rube_learning/ingestion_pipeline/tests/test_skip_empty_toolkits.py app_tester/rube_learning/ingestion_pipeline/tests/test_tool_consolidation_skip_on_llm_failure.py
# 6 passed in 5.03s

Lint clean: ruff check, black --check, isort --check, flake8 --config tox.ini all pass on the four touched files.
Updated tests/test_cached_plan_processing.py integration test mock to return a non-empty toolkit list, so the new skip path doesn't short-circuit the integration test.

cc @jayeshparmar

Origin: cron-5d55c321e47a / zen-cron-7935eecdc695 Triggered by: dhawal@composio.dev | Source: cron Session: https://zen-api-production-4c98.up.railway.app/dashboard/#/chat/zen-cron-7935eecdc695

@zen-agent1mo ago

Post-PR status (automated):

✅ /build-checks — make fmt PASS. make chk flagged 1 flake8 + 11 mypy errors, but all 12 are pre-existing on master (verified by checking out master and re-running). The 2 mypy errors that originated from this PR (test_skip_empty_toolkits.py SimpleNamespace → UsecaseIngestionConfig arg-type) were silenced with # type: ignore[arg-type] in 90631001 — same pattern carried by the existing test_tool_consolidation_skip_on_llm_failure.py neighbor.
✅ /codex-review-loop — Iteration 1 LGTM, no actionable issues. Codex described the change as: "adds an early skip for consolidated tool lists that cannot resolve to any toolkit, avoiding a known validation failure path without affecting successful partial-resolution flows. The accompanying tests cover the new cached-plan and unmatched-cluster skip behavior."
⏳ /gh-fix-ci-loop — Test - Learning Pipeline ✅ green; Lint - Integrator still running on the latest commit (90631001). Will report back if it goes red.
✅ Testing checklist — 3 new unit tests + 3 existing related tests pass. The behavior is purely a logging/skip change inside the cached_plan_processor / unmatched_cluster_processor async helpers; there is no Apollo/Thermos endpoint to curl. Live runtime verification will happen on the next ingestion cycle once merged (no API surface to hit pre-merge).
✅ /address-pr-comments — only 1 inline comment, from github-advanced-security[bot] flagging .github/workflows/test-learning.yml:34 for missing GITHUB_TOKEN permissions. That file was added in #1404 and is pre-existing on master — not part of this PR's diff. Out of scope.

@zen-agent1mo ago

CI update — Lint failure is pre-existing on master:

Lint - Integrator failed on this PR's commit. Verified the failure is :

loading diff…

Description

When LLM tool consolidation produces a tool-slug list whose toolkits all 404 in the catalog (deprecated/removed tools), get_toolkits_from_tools returns []. Previously this still proceeded to:

The expensive plan-consolidation LLM call.

A TurboPuffer upsert that always fails validateEntry with Toolkits field is empty, logging a confusing 5-line ⚠️ Validation FAILED + ❌ Failed to update TurboPuffer block per plan.

Both code paths now short-circuit immediately after toolkit resolution when the consolidated tool list is non-empty but no toolkit resolved:

cached_plan_processor.process_cached_plan_group: marks the group skipped=True, leaves the existing TurboPuffer entry untouched.

unmatched_cluster_processor.process_unmatched_cluster: returns None so the cluster is skipped (matches the existing tool-consolidation-LLM-failure contract).

This saves one paid LLM call per failing plan and removes the noisy "Failed to update TurboPuffer" log lines from the ingestion dashboard.

Partial-resolution cases (some toolkits resolved, some 404'd) keep their current behavior — a single resolved toolkit is enough to continue.

How did I test this PR

Added tests/test_skip_empty_toolkits.py with 3 cases:

cached-plan path: all tools 404 → skip, no LLM call, no upsert, log mentions "unresolvable" and "Existing TurboPuffer entry left untouched"
cached-plan path: partial resolution → normal flow continues
unmatched-cluster path: all tools 404 → returns None, no LLM call

Ran new tests + existing related tests:

pytest app_tester/rube_learning/ingestion_pipeline/tests/test_skip_empty_toolkits.py app_tester/rube_learning/ingestion_pipeline/tests/test_tool_consolidation_skip_on_llm_failure.py
# 6 passed in 5.03s

Lint clean: ruff check, black --check, isort --check, flake8 --config tox.ini all pass on the four touched files.

Updated tests/test_cached_plan_processing.py integration test mock to return a non-empty toolkit list, so the new skip path doesn't short-circuit the integration test.

cc @jayeshparmar

@zen-agent1mo ago

Post-PR status (automated):

✅ /build-checks — make fmt PASS. make chk flagged 1 flake8 + 11 mypy errors, but all 12 are pre-existing on master (verified by checking out master and re-running). The 2 mypy errors that originated from this PR (test_skip_empty_toolkits.py SimpleNamespace → UsecaseIngestionConfig arg-type) were silenced with # type: ignore[arg-type] in 90631001 — same pattern carried by the existing test_tool_consolidation_skip_on_llm_failure.py neighbor.
✅ /codex-review-loop — Iteration 1 LGTM, no actionable issues. Codex described the change as: "adds an early skip for consolidated tool lists that cannot resolve to any toolkit, avoiding a known validation failure path without affecting successful partial-resolution flows. The accompanying tests cover the new cached-plan and unmatched-cluster skip behavior."
⏳ /gh-fix-ci-loop — Test - Learning Pipeline ✅ green; Lint - Integrator still running on the latest commit (90631001). Will report back if it goes red.
✅ Testing checklist — 3 new unit tests + 3 existing related tests pass. The behavior is purely a logging/skip change inside the cached_plan_processor / unmatched_cluster_processor async helpers; there is no Apollo/Thermos endpoint to curl. Live runtime verification will happen on the next ingestion cycle once merged (no API surface to hit pre-merge).
✅ /address-pr-comments — only 1 inline comment, from github-advanced-security[bot] flagging .github/workflows/test-learning.yml:34 for missing GITHUB_TOKEN permissions. That file was added in #1404 and is pre-existing on master — not part of this PR's diff. Out of scope.