fix(rube/ingestion): persist deterministic synthetic_id on unmatched-cluster plans

@zen-agentchecks n/achecks…zen/fix-unmatched-dup-plan-id-ismct6 → master3 files · +300 −16updated 3w ago

▸Description· 1 comment

Description

The unmatched-cluster processor computes a deterministic synthetic_id from the representative usecase (generate_plan_id = sha256[:16] of lowercased+stripped usecase) and then immediately overwrites best_plan_copy.cached_plan_id to None before returning. That triggers the uuid.uuid4().hex fallback in build_turbopuffer_rows_from_plans, so re-ingesting an identical cluster on a later run writes a new TurboPuffer row instead of upserting the existing one.

Production impact (verified via TurboPuffer dump on 2026-05-16): 1,987 rows across 585 byte-identical-usecase clusters in usecase_plans_prod_new_v1. Top dup clusters:

30 rows: search web stock market news on a specific date (composio_search)
26 rows: execute python script in remote sandbox (e2b)
22 rows: read google docs document content by URL or document ID (googledocs+googledrive)
22 rows: evaluate simple arithmetic expression (codeinterpreter)
21 rows: download gmail email attachment by attachment id and message id (gmail+googledrive)

The trickle is still active post-Apr-23 fixes: 4 new "stock-market" duplicates landed in May 2026 (latest 2026-05-15), 7 new gmail+googledrive attachment dupes, 8 new composio_search news dupes. ClickHouse usage data also shows many duplicates are actively surfaced by COMPOSIO_SEARCH_TOOLS (e.g. plan_id 164ea2f68d9e4e6998a9c147cf66ccb4 was surfaced 95 times in the last 30 days), so this bug is splitting cache traffic across functionally identical plans.

Fix: assign best_plan_copy.cached_plan_id = synthetic_id instead of None. The TurboPuffer write becomes idempotent across re-ingest runs — repeated ingestion of the same cluster will upsert the same row, not duplicate it.

Out of scope (follow-up): Apollo vector_search.ts:50 still uses an exact-Eq filter on the comma-joined toolkits string, which causes the matcher's read-side join to miss cached rows when toolkit ordering or membership changes. Fixing the write side alone is sufficient to stop the duplicate-row trickle this PR targets; the read-side filter fix is a separate change.

How did I test this PR

New test file app_tester/rube_learning/ingestion_pipeline/tests/test_unmatched_synthetic_id.py covers two guards:
1. generate_plan_id is deterministic and produces a 16-char sha256 prefix (re-derived independently inside the test).
2. A source-level guard that the consolidation tail in unmatched_cluster_processor.py assigns cached_plan_id = synthetic_id and does NOT contain cached_plan_id = None. Same shape as test_tool_consolidation_skip_on_llm_failure.py — full async-flow tests would require the existing extensive LLM/HTTP mock surface that wasn't present in the original broken commit either.
Local lint: ruff check passes, ruff format --check passes on both files.
Sandbox pytest collection trips on pre-existing missing tiktoken/composio deps in this Zen sandbox env (same failure on the existing test_tool_consolidation_skip_on_llm_failure.py), so CI will run the new test.
Manual verification of the bug in production data: see "Production impact" above; full audit data persisted in cron memory.

Origin: cron-fe3a7d9c5460 / zen-cron-9ca25f9ee0f6

Triggered by: dhawal@composio.dev | Source: cron-fe3a7d9c5460 Session: https://zen-api-production-4c98.up.railway.app/dashboard/#/chat/zen-cron-9ca25f9ee0f6

@zen-agent3w ago

Post-PR status update

Codex review loop: 4 iterations, 5 findings, all addressed:

Toolkit-group collision → include toolkit_key in synthetic_id
Known-fields collision → include representative_known_fields
Empty/"None"/"null" canonicalization for no-known-fields clusters
Test decoupled from heavy ingestion deps (sha256[:16] re-derived in test)
Resume mode: _load_resume_data now preserves toolkit_key and representative_known_fields; processor falls back to toolkits_str derived from cluster_plans when toolkit_key is absent

Tests: 4 standalone tests all pass (test_synthetic_id_distinguishes_toolkit_groups, ..._distinguishes_known_fields, ..._persists_synthetic_id_on_returned_plan, ..._empty_none_null_known_fields_collapse_to_same_id). Source-level guards lock in the contract.

Local lint on changed files (black 23.3.0 + isort 5.12.0 + flake8 6.0.0): all clean.

CI:

✅ test-learning (Test - Learning Pipeline) — passing
⚠️ run-lint (Lint - Integrator) — pre-existing failure on master since 2026-04-21 (last master green run was commit e2c0971f on 2026-04-20). Pre-existing issues are in unrelated files: app_tester/rube_learning/common/pii_mask.py, ..common/untrusted_input.py, ..ingestion_pipeline/tests/test_search_vector_cache_raises.py, ..ingestion_pipeline/tests/test_search_turbopuffer_optimized.py, ..ingestion_pipeline/tests/test_tool_consolidation_skip_on_llm_failure.py, ..ingestion_pipeline/tool_consolidator.py, ..tests/test_untrusted_input.py, plus mypy errors throughout. My changed files (unmatched_cluster_processor.py, the new test file, and the 1-block _load_resume_data edit in usecase_ingestion_service.py) pass black/isort/flake8 locally.

Manual cleanup follow-up (not in this PR): the 1,987 existing dup rows in TurboPuffer were created under the old uuid4 fallback and won't be reaped by this fix on their own — they need a one-time backfill that re-keys each dup row's content to its new deterministic ID. Happy to file a separate PR or one-off script for that on request.

loading diff…

@zen-agent3w ago

Post-PR status update

Codex review loop: 4 iterations, 5 findings, all addressed:

Toolkit-group collision → include toolkit_key in synthetic_id
Known-fields collision → include representative_known_fields
Empty/"None"/"null" canonicalization for no-known-fields clusters
Test decoupled from heavy ingestion deps (sha256[:16] re-derived in test)
Resume mode: _load_resume_data now preserves toolkit_key and representative_known_fields; processor falls back to toolkits_str derived from cluster_plans when toolkit_key is absent

Local lint on changed files (black 23.3.0 + isort 5.12.0 + flake8 6.0.0): all clean.

CI:

✅ test-learning (Test - Learning Pipeline) — passing
⚠️ run-lint (Lint - Integrator) — pre-existing failure on master since 2026-04-21 (last master green run was commit e2c0971f on 2026-04-20). Pre-existing issues are in unrelated files: app_tester/rube_learning/common/pii_mask.py, ..common/untrusted_input.py, ..ingestion_pipeline/tests/test_search_vector_cache_raises.py, ..ingestion_pipeline/tests/test_search_turbopuffer_optimized.py, ..ingestion_pipeline/tests/test_tool_consolidation_skip_on_llm_failure.py, ..ingestion_pipeline/tool_consolidator.py, ..tests/test_untrusted_input.py, plus mypy errors throughout. My changed files (unmatched_cluster_processor.py, the new test file, and the 1-block _load_resume_data edit in usecase_ingestion_service.py) pass black/isort/flake8 locally.