PR Custody

feat(test-and-fix): circuit breaker for ConnectedAccountNotFoundError

@AgentWrapperchecks n/achecks…feat/INT-1433 → next4 files · +354 −1updated 3mo ago

▸Description· 1 comment · 1 noise

Why

When ConnectedAccountNotFoundError is raised during parallel test-and-fix runs (connection 404/410 or missing auth headers), ALL remaining parallel agents continue running and hit the same error — wasting significant LLM spend. Investigation documented up to $1,597 lost in a single run (INT-1433).

What

Circuit breaker (threading.Event) added to TestAndFixAppConfig; starts unset
On first ConnectedAccountNotFoundError in any parallel task, the circuit breaker is set and the failing action is marked FAILED with is_auth_issue=True
Every task checks the circuit breaker before claiming the action; if tripped it returns immediately with a descriptive FAILED result — no agent spin-up
_handle_error() now re-raises ConnectedAccountNotFoundError instead of absorbing it, so it reaches the parallel orchestrator where the circuit breaker is tripped
test_app() logs a circuit-breaker event at ERROR level for observability
9 new unit tests cover: re-raise, trip, skip-on-pretripped, generic-error isolation, independent-event-per-config, default state

How to Test

Mock a ConnectedAccountNotFoundError on the first action in a multi-action test_app() run
Observe that subsequent actions are immediately marked FAILED with is_auth_issue=True and "circuit breaker" in the result summary — no agent is spawned for them
Run pytest cortex/tests/test_agents/test_test_and_fix_circuit_breaker.py -v — all 9 tests pass

Pre-Review Checklist

I have self-reviewed this PR

Notes

Tasks that are already running when the first error fires are not interrupted (Python threads can't be safely killed). They will encounter the same ConnectedAccountNotFoundError themselves and also trip the (already-set) breaker. Only not-yet-started tasks are skipped.
The circuit breaker is scoped to one TestAndFixAppConfig instance — independent runs are unaffected.
Pre-existing pyrefly failure on next branch (exit 1 with no diagnostics) is not caused by this PR.

@AgentWrapper3mo ago

Addressed bugbot review comment in fa7d6352c:

Problem: _handle_error re-raised ConnectedAccountNotFoundError, which worked for the parallel path (_test_single_action has a handler) but broke the non-parallel path (create_and_run had no handler — exception propagated uncaught, losing the is_auth_issue flag).

Fix:

_handle_error no longer re-raises — it's now a pure error-to-result converter
_run_common catches ConnectedAccountNotFoundError separately (before generic except Exception) and re-raises, so callers decide how to handle it
create_and_run (non-parallel path) now catches it and returns ActionTestResult(is_auth_issue=True)

Added test test_create_and_run_returns_auth_result that validates the non-parallel path handles this correctly.

▸ 1 bot/status comment hidden

@linear[bot]3mo ago

On first ConnectedAccountNotFoundError, cancel all remaining parallel tests. Prevents waste scenarios (up to $1,597 observed).

See investigation: agent/workbooks/investigation-create-app-failures.md

loading diff…

@AgentWrapper3mo ago

Addressed bugbot review comment in fa7d6352c:

Fix:

_handle_error no longer re-raises — it's now a pure error-to-result converter
_run_common catches ConnectedAccountNotFoundError separately (before generic except Exception) and re-raises, so callers decide how to handle it
create_and_run (non-parallel path) now catches it and returns ActionTestResult(is_auth_issue=True)

Added test test_create_and_run_returns_auth_result that validates the non-parallel path handles this correctly.