PR Custody

feat(config-generator): enforce decision-tree + Tier-1 lint rules for auth modes

@AgentWrapperchecks n/achecks…feat/plen-2462-rube-auth-scheme → next17 files · +2277 −120updated 3w ago

▸Description · 1 noise

Why

PLEN-2462 tracks rube/config-generator misclassifying server-to-server OAuth providers as redirect OAUTH2. The audit's smoking gun is that rube often correctly writes "client_credentials" / "Server-to-Server" in PR notes while emitting mode: 'OAUTH2'. This PR makes auth-mode selection a first-class, cited decision record and adds deterministic generation-time gates so invalid OAuth/S2S shapes do not ship.

Design doc: docs/config-generator-auth-workflow-redesign.html

What

Adds an Auth Decision Record workflow: Q0-Q4 with cited evidence, explicit verdict, template, confidence, and summary for each distinct auth method.
Expands parent/researcher prompts with a full auth-method inventory tree: API_KEY, BASIC, BEARER_TOKEN, NO_AUTH, OAUTH2, S2S_OAUTH2, GOOGLE_SERVICE_ACCOUNT, SERVICE_ACCOUNT, SAML, OAUTH1/OAUTH1A, BASIC_WITH_JWT, and provider-specific/custom modes; includes S2S templates, scope/region/token-auth guidance from PR #1417, provider-specific gotchas, and templating pitfalls.
Adds explicit multi-auth-method handling: apps can emit OAUTH2 + S2S_OAUTH2 (or API_KEY/BASIC/etc.) as separate schemes when docs support both, with per-method decision records and PR-review tests.
Adds an independent auth_reviewer subagent for adversarial review before final success.
Adds deterministic Tier-1 auth lints for:
- duplicate auth-scheme modes in one toolkit (collapse variants or human-review)
- OAUTH2 + client_credentials
- OAUTH2 with identical token/authorization URLs
- S2S_OAUTH2 with connection_field
- S2S_OAUTH2 missing auth_method
- S2S_OAUTH2 empty token params
- S2S_OAUTH2 default scopes not forwarded to token_config.params.scope
Adds validate_generated_config, which composes configToJson, make validate-config, Tier-1 auth lints, and token URL liveness probes.
Makes provider credentials optional; absence of credentials is recorded as credential-blocked, not treated as a config-generation failure.
Moves live connection/token verification to a required PR-review gate after the PR branch/config is deployed to staging.
Adds a bounded post-agent repair loop: validation failures are converted into repair feedback and the agent reruns instead of silently producing a bad PR.
Extends generated PR bodies with auth-decision tables, validation-gate summaries, deferred follow-ups, and draft/needs-human-review support.

How to Test

make fmt
make chk (passed from /tmp/integrator-check, a non-hidden physical checkout copied from this worktree; direct AO worktree run fails before type-checking because Pyrefly's default **/.[!/.]*/**/* exclude matches /Users/equinox/.agent-orchestrator/...)
uv run pytest cortex/tests/test_agents/test_config_generator_auth_validation.py cortex/tests/test_agents/test_config_generator_prompts.py -q

Testbed verification table

App	Correct mode/shape gate	`make validate-config` gate	PR-review live verification
paychex	Covered: `OAUTH2_CLIENT_CREDENTIALS` + identical URL lint forces S2S template	Covered by `validate_generated_config`; not re-run without Mercury workflow inputs	Required after staging deploy with provider client credentials
vanta	Covered: S2S template + scope-forwarding lint	Covered by `validate_generated_config`	Required after staging deploy with provider client credentials
anthology_student	Covered: decision tree forces client_credentials/no-user-redirect to S2S	Covered by `validate_generated_config`; note connection-field nuance must be reviewed if dynamic base URL is required	Required after staging deploy with provider client credentials
airship	Covered: decision tree + S2S connection-field lint; JWT/assertion framework gap escalates	Covered by `validate_generated_config`	Required after staging deploy; may be framework-blocked for JWT/assertion shape
adobe_document_generation_api	Covered: S2S template + token URL liveness probe catches wrong-domain class	Covered by `validate_generated_config`	Required after staging deploy with Adobe IMS credentials
pinecone	Covered: contradictory auth-code/client_credentials params rejected	Covered by `validate_generated_config`	Required after staging deploy with service-account credentials
gagelist	Covered: S2S minimum viable shape requires `auth_method` and non-empty params	Covered by `validate_generated_config`	Required after staging deploy with provider client credentials

Pre-Review Checklist

I have self-reviewed this PR

Notes

Deferred follow-ups:

Browser automation for full 3-legged OAuth consent testing.
Automated retrieval/ranking of closest shipped Mercury analogs before template fill. This PR adds prompt-level Mercury analog cross-checking, but not a deterministic analog-ranker tool yet.
Framework support for assertion/JWT S2S edge cases where current S2S_OAUTH2 cannot express provider-specific token shapes.
Broader replay harness for the seven historical Cortex workflow IDs once provider test credentials are available.
PR-review automation/checklist support that records staging deploy + auth config + connection creation + create-app workflow results before merge.

Rule nuance:

The audit says S2S_OAUTH2 + any connection_field should reject because S2S has no end-user connection surface. Some historical fix PRs still have dynamic baseUrl fields under S2S. This PR keeps the strict rule, but the prompt instructs the agent to flag a documented framework exception as needs_human_review rather than silently dropping the lint.

▸ 1 bot/status comment hidden

@linear-code[bot]1mo ago

Problem

The config-generation agent (rube) misclassifies the auth scheme for new apps. Specifically, providers that should be S2S_OAUTH2 (2-legged, client_credentials, admin-managed credentials) are being emitted as OAUTH2 (3-legged, authorization_code with a user redirect). This was flagged by the PLEN-2215 audit and confirmed by the 9 broken-on-master schemes the audit identified.

The single biggest smoking gun: in 4 of 7 documented misclassifications, rube's own notes in the PR body correctly described the flow as "client credentials" or "Server-to-Server" — but the emitted mode field is still OAUTH2. The mode-selection step is decoupled from rube's own classification of the docs.

Goal

Improve the agent so that the seven known-misclassified cases (testbed in the attached HTML) all produce the correct mode + shape when re-run with the same workflow inputs.

Testbed — known-misclassified PRs

See the attached oauth2-decision-tree.html for the full visual decision tree, the rule set, and the testbed table with each case's:

Origin PR + Cortex workflow ID + branch + commit (so the agent can be re-invoked with the same inputs)
Author / approver / merge date
Excerpt from rube's own notes showing the smoking gun
The mode it picked vs the correct mode
Which decision-tree node it flunked
Link to our fix PR

Summary of cases:

App	Workflow ID	Picked → Correct	Tree node missed	Fix PR
paychex	`awd1qin2`	OAUTH2 → S2S_OAUTH2	Q3 + URL-dedup	#23207
vanta	`73timakv`	OAUTH2 → S2S_OAUTH2	Q3 + scope-forwarding	#23209
anthology_student	`up7amdvu`

loading diff…

@linear-code[bot]1mo ago

Problem

Goal

Improve the agent so that the seven known-misclassified cases (testbed in the attached HTML) all produce the correct mode + shape when re-run with the same workflow inputs.

Testbed — known-misclassified PRs

See the attached oauth2-decision-tree.html for the full visual decision tree, the rule set, and the testbed table with each case's:

Origin PR + Cortex workflow ID + branch + commit (so the agent can be re-invoked with the same inputs)
Author / approver / merge date
Excerpt from rube's own notes showing the smoking gun
The mode it picked vs the correct mode
Which decision-tree node it flunked
Link to our fix PR

Summary of cases:

App	Workflow ID	Picked → Correct	Tree node missed	Fix PR
paychex	`awd1qin2`	OAUTH2 → S2S_OAUTH2	Q3 + URL-dedup	#23207
vanta	`73timakv`	OAUTH2 → S2S_OAUTH2	Q3 + scope-forwarding	#23209
anthology_student	`up7amdvu`

feat(config-generator): enforce decision-tree + Tier-1 lint rules for auth modes

Why

What

How to Test

Testbed verification table

Pre-Review Checklist

Notes

Problem

Goal

Testbed — known-misclassified PRs

Why

What

How to Test

Testbed verification table

Pre-Review Checklist

Notes

Problem

Goal

Testbed — known-misclassified PRs

Pass criteria

Proposed improvements (in order of leverage)

Attachment

Related