PR Custody

feat(watchdog): gate learning pipeline on per-project config flag

@zen-agentchecks n/achecks…zen/learning-pipeline-gate-w94flw → next12 files · +941 −4updated 1mo ago

▸Description· 1 comment

Why

The watchdog learning pipeline auto-triggers fixer workflows on tool execution failures across every project. Some projects need to opt out (sensitive workloads, projects pinned to specific tool versions, projects where automated PRs aren't desirable). Apollo just gained a learningPipelineEnabled project-config flag — this PR makes watchdog actually respect it.

What

New Apollo HTTP client at watchdog/clients/apollo/project_config.py with a small in-process TTL cache (default 5 min, env-overridable). Reads the flag from the new admin endpoint GET /api/v3/admin/projects/{nanoId}/learning_pipeline_status using the existing COMPOSIO_ADMIN_TOKEN.
- Fail-open: any error (network, auth, 5xx, malformed) returns True, so a transient Apollo outage cannot silently disable the fixer fleet-wide.
- 404 is cached as True — better than blocking the fixer for a project we just haven't synced yet.
- 5xx is not cached — we want to retry on the next call.
RubeErrorItem now carries project_id, populated from the enriched ClickHouse tool_execution_logs row when available.
RubePipeline._trigger_fixer consults the flag for each item. When the project has opted out:
- Records block_reason=\"learning_disabled_for_project\",
- Increments stats.skipped_learning_disabled_for_project,
- Never calls the integrator fix-action API.
ErrorProcessor.process_tool_error (shared by tool_failure and batch_poller pipelines) extracts project IDs from pattern.logs[*].projectId. If every sample log on the pattern comes from opted-out projects, skip the LLM call entirely — avoids wasted LLM spend on errors we could never act on. If at least one project on the pattern still has the gate enabled, we proceed normally.
Patterns with no projectId on any log behave exactly as before (the gate is never consulted).

How to Test

Unit tests: cd watchdog && pytest tests/test_apollo_project_config.py tests/test_rube_pipeline.py::TestLearningPipelineGate tests/test_processor.py::TestLearningPipelineGate -v. 14 new tests added covering: cache hit/miss/expiry, 404/5xx/network failure handling, single-project vs mixed-project gating in the processor, and rube pipeline skip-on-disabled.
Full unit suite passes (48 in the directly affected files; 13 pre-existing failures in unrelated tests on next are not introduced by this PR).
make chk — clean (ruff format, ruff lint, ruff import order, pyrefly).
E2E gating against the Apollo PR: hit the new learning_pipeline_status endpoint with a project where the flag is false → client returns False → next call within TTL serves from cache → _trigger_fixer short-circuits.

Pre-Review Checklist

I have self-reviewed this PR

Notes

Companion Apollo PR: https://github.com/ComposioHQ/hermes/pull/9811. The integrator side will not gate anything until that PR is merged — without it, the admin endpoint returns 404 and we fail-open to True (existing behavior preserved).
Env vars added: APOLLO_BASE_URL (default https://backend.composio.dev), APOLLO_PROJECT_CONFIG_CACHE_TTL_SECONDS (default 300). Reuses existing COMPOSIO_ADMIN_TOKEN (or APOLLO_ADMIN_TOKEN).
The schema_validation, weak_schema, and fixer_regression pipelines call IntegratorAPIClient.fix_action directly without going through ErrorProcessor. They are intentionally not gated in this PR — they rely on Datadog-driven discovery rather than per-tool-call ClickHouse logs, so the project nanoId isn't readily available without additional plumbing. Follow-up if needed.

Triggered by: karan@composio.dev | Source: slack Session: https://zen-api-production-4c98.up.railway.app/dashboard/#/chat/zen-8254d97fe7ac

🤖 Generated with Claude Code

@zen-agent1mo ago

Post-PR status update

/build-checks: PASS — make fmt clean, make chk clean (ruff format / lint / import-order / pyrefly).
Codex review (/codex-review-loop): 3 iterations, 5 findings — all fixed:
- Round 1, P3 (real): missing skip_reason on the processor opt-out branch caused the new skip path to be bucketed as skipped_dedup by the tool-failure / batch-poller callers and emit a spurious pattern_deduped metric. Added SkipReason.LEARNING_DISABLED_FOR_PROJECT + PipelineStats.skipped_learning_disabled and updated both callers. (fb76ef82)
- Round 1, P1 (false positive but worth clarifying): clarified the module docstring to spell out that the admin endpoint returns snake_case learning_pipeline_enabled (the underlying Zod / Postgres field is learningPipelineEnabled). (fb76ef82)
- Round 2, P2(a): staging ECS sets ENV=staging, not APOLLO_ENV. Added APOLLO_ENV/ENV resolution mirroring common/composio_client.py so staging picks staging-backend.composio.dev + COMPOSIO_STAGING_ADMIN_TOKEN. (8eea66f6, 20cc2e16)
- Round 2, P2(b): bool(None) → False could silently disable the fixer. Strict isinstance(_, bool) check; non-bool / null / missing payloads fail open and are not cached. (8eea66f6)
- Round 3, P2(a): mixed pattern with one opted-out projectId and one log missing projectId was being skipped; that's not safe (the unknown row might come from an opted-in project). Now requires every sample log to carry a known projectId before skipping. (20cc2e16)
- Round 3, P2(b): Apollo lookup failures only logged at WARN; switched to logger.exception() per repo guidelines so an outage that bypasses the gate surfaces in Datadog. (5dc0a428)
Cursor BugBot: 1 finding (same as Codex round 1 P3, on the original commit) — already fixed in .

loading diff…

Why

What

New Apollo HTTP client at watchdog/clients/apollo/project_config.py with a small in-process TTL cache (default 5 min, env-overridable). Reads the flag from the new admin endpoint GET /api/v3/admin/projects/{nanoId}/learning_pipeline_status using the existing COMPOSIO_ADMIN_TOKEN.

Fail-open: any error (network, auth, 5xx, malformed) returns True, so a transient Apollo outage cannot silently disable the fixer fleet-wide.
404 is cached as True — better than blocking the fixer for a project we just haven't synced yet.
5xx is not cached — we want to retry on the next call.

RubeErrorItem now carries project_id, populated from the enriched ClickHouse tool_execution_logs row when available.

RubePipeline._trigger_fixer consults the flag for each item. When the project has opted out:

Records block_reason=\"learning_disabled_for_project\",
Increments stats.skipped_learning_disabled_for_project,
Never calls the integrator fix-action API.

ErrorProcessor.process_tool_error (shared by tool_failure and batch_poller pipelines) extracts project IDs from pattern.logs[*].projectId. If every sample log on the pattern comes from opted-out projects, skip the LLM call entirely — avoids wasted LLM spend on errors we could never act on. If at least one project on the pattern still has the gate enabled, we proceed normally.

Patterns with no projectId on any log behave exactly as before (the gate is never consulted).

How to Test

Unit tests:

cd watchdog && pytest tests/test_apollo_project_config.py tests/test_rube_pipeline.py::TestLearningPipelineGate tests/test_processor.py::TestLearningPipelineGate -v

. 14 new tests added covering: cache hit/miss/expiry, 404/5xx/network failure handling, single-project vs mixed-project gating in the processor, and rube pipeline skip-on-disabled.

Full unit suite passes (48 in the directly affected files; 13 pre-existing failures in unrelated tests on next are not introduced by this PR).

make chk — clean (ruff format, ruff lint, ruff import order, pyrefly).

E2E gating against the Apollo PR: hit the new learning_pipeline_status endpoint with a project where the flag is false → client returns False → next call within TTL serves from cache → _trigger_fixer short-circuits.

Notes

Companion Apollo PR: https://github.com/ComposioHQ/hermes/pull/9811. The integrator side will not gate anything until that PR is merged — without it, the admin endpoint returns 404 and we fail-open to True (existing behavior preserved).

Env vars added: APOLLO_BASE_URL (default https://backend.composio.dev), APOLLO_PROJECT_CONFIG_CACHE_TTL_SECONDS (default 300). Reuses existing COMPOSIO_ADMIN_TOKEN (or APOLLO_ADMIN_TOKEN).

The schema_validation, weak_schema, and fixer_regression pipelines call IntegratorAPIClient.fix_action directly without going through ErrorProcessor. They are intentionally not gated in this PR — they rely on Datadog-driven discovery rather than per-tool-call ClickHouse logs, so the project nanoId isn't readily available without additional plumbing. Follow-up if needed.

🤖 Generated with Claude Code

Post-PR status update

/build-checks: PASS — make fmt clean, make chk clean (ruff format / lint / import-order / pyrefly).

Codex review (/codex-review-loop): 3 iterations, 5 findings — all fixed:

Round 1, P3 (real): missing skip_reason on the processor opt-out branch caused the new skip path to be bucketed as skipped_dedup by the tool-failure / batch-poller callers and emit a spurious pattern_deduped metric. Added SkipReason.LEARNING_DISABLED_FOR_PROJECT + PipelineStats.skipped_learning_disabled and updated both callers. (fb76ef82)
Round 1, P1 (false positive but worth clarifying): clarified the module docstring to spell out that the admin endpoint returns snake_case learning_pipeline_enabled (the underlying Zod / Postgres field is learningPipelineEnabled). (fb76ef82)
Round 2, P2(a): staging ECS sets ENV=staging, not APOLLO_ENV. Added APOLLO_ENV/ENV resolution mirroring common/composio_client.py so staging picks staging-backend.composio.dev + COMPOSIO_STAGING_ADMIN_TOKEN. (8eea66f6, 20cc2e16)
Round 2, P2(b): bool(None) → False could silently disable the fixer. Strict isinstance(_, bool) check; non-bool / null / missing payloads fail open and are not cached. (8eea66f6)
Round 3, P2(a): mixed pattern with one opted-out projectId and one log missing projectId was being skipped; that's not safe (the unknown row might come from an opted-in project). Now requires every sample log to carry a known projectId before skipping. (20cc2e16)
Round 3, P2(b): Apollo lookup failures only logged at WARN; switched to logger.exception() per repo guidelines so an outage that bypasses the gate surfaces in Datadog. (5dc0a428)

Cursor BugBot: 1 finding (same as Codex round 1 P3, on the original commit) — already fixed in .

Iteration	Issues Found	Issues Fixed	Status
1	2	2	fixed
2	2	2	fixed
3	1	1	fixed
Total	5	5	complete