PR Custody

feat: add 5% multi-model A/B experiment routing (Kimi K2.5, GLM-5, GPT-5.3 Codex)

@AgentWrapperchecks n/achecks…feat/INT-1337 → next4 files · +216 −1updated 3mo ago

▸Description · 1 noise

Why

To validate the performance of alternative LLM models (Kimi K2.5, GLM-5, GPT-5.3 Codex) against our default model in production workflows. This A/B experiment will help us identify if these models can provide better or more cost-effective results for agent tasks.

Reference: INT-1337

What

Add cortex/generic/experiment.py with:
- 5% rollout enrollment logic (configurable via EXPERIMENT_ROLLOUT_PCT env var)
- Model configurations for Kimi K2.5 (Fireworks), GLM-5 (ZAI), GPT-5.3 Codex (OpenAI)
- Excluded agents list (mission-critical agents that should not be experimented on)
- "do-not-merge" label injection for experiment PRs
Modify cortex/generic/agent.py to override eligible agent models when experiment is active
Modify cortex/generic/agent_provider/openai.py to add fallback tool conversion for file operations
Modify cortex/generic/workflow.py to enroll workflows in the experiment and inject labels

How to Test

Local testing (dry run):

# Set experiment rollout to 100% for testing
export EXPERIMENT_ROLLOUT_PCT=100

# Set API keys for test models
export FIREWORKS_API_KEY=<key>
export GLM_API_KEY=<key>
export OPENAI_API_KEY=<key>

# Run a test workflow
uv run pytest cortex/tests/test_generic_workflow.py -v

Staging verification:

Deploy to staging with EXPERIMENT_ROLLOUT_PCT=5
Trigger multiple workflow runs (20+) to capture experiment enrollments
Check Datadog logs for experiment enrollment messages:
- Experiment enrolled: model={model_name}, rollout_pct=5
- Experiment: agent {name} using model {model_key}
Verify experiment PRs have "do-not-merge" label
Compare success rates and error patterns between experiment and control groups

Expected behavior:

~5% of workflows should show "Experiment enrolled" logs
Enrolled workflows should use experiment models instead of defaults
PRs from experiment runs should have "do-not-merge" label
Excluded agents (test_and_fix_agent, fix_action_agent, response_schema_*) should not use experiment models

Pre-Review Checklist

I have self-reviewed this PR

Notes

Experiment is controlled by EXPERIMENT_ROLLOUT_PCT environment variable (default: 5)
Set to 0 to disable the experiment
Experiment PRs are automatically labeled "do-not-merge" to prevent accidental production impact
Mission-critical agents are excluded from the experiment for safety
Models are selected randomly on enrollment for balanced distribution
API keys must be set in environment for experiment models to work

▸ 1 bot/status comment hidden

@linear[bot]3mo ago

Objective

Run 5% of all workflows on multiple alternative models alongside the default model (Claude Sonnet 4.5) to compare quality, latency, and cost. All experiment PRs should be marked do-not-merge.

Models to Test

Model	Provider	Input $/M	Output $/M	Notes
Claude Sonnet 4.5 (baseline)	Anthropic	$3.00	$15.00	Current default
Kimi K2.5	Fireworks (OpenAI-compatible)	$0.60	$2.50	1T params, 262K context, ~80% cheaper than Claude
GLM-5	ZAI	$1.00	$3.20	744B MoE (40B active), 200K context, MIT license
GPT-5.3 Codex	OpenAI	$1.75	$14.00	SOTA on SWE-Bench Pro & Terminal-Bench 2.0, 400K context

Implementation Approach

5% traffic routing: For each workflow run, randomly select 5% to also run on an alternative model
Shadow experiment: Both default and alternative model run; alternative model PRs tagged do-not-merge
No production impact: Experiment runs are purely for comparison, not replacing production output
Metrics collection: Log model used, cost, latency, and quality scores for each run

Scope

All workflows (create-app, fix-action, test-and-fix-action, finder, etc.)
All GenericAgent instances within those workflows
Exclude: test_and_fix_agent and fix_action_agent from model swap (mission-critical, per Samvit)
Exclude: response_schema_* agents (separate optimization track)

loading diff…

What

Add cortex/generic/experiment.py with:

5% rollout enrollment logic (configurable via EXPERIMENT_ROLLOUT_PCT env var)
Model configurations for Kimi K2.5 (Fireworks), GLM-5 (ZAI), GPT-5.3 Codex (OpenAI)
Excluded agents list (mission-critical agents that should not be experimented on)
"do-not-merge" label injection for experiment PRs

Modify cortex/generic/agent.py to override eligible agent models when experiment is active

Modify cortex/generic/agent_provider/openai.py to add fallback tool conversion for file operations

Modify cortex/generic/workflow.py to enroll workflows in the experiment and inject labels

How to Test

Local testing (dry run):

# Set experiment rollout to 100% for testing export EXPERIMENT_ROLLOUT_PCT=100 # Set API keys for test models export FIREWORKS_API_KEY=<key> export GLM_API_KEY=<key> export OPENAI_API_KEY=<key> # Run a test workflow uv run pytest cortex/tests/test_generic_workflow.py -v

Staging verification:

Deploy to staging with EXPERIMENT_ROLLOUT_PCT=5

Trigger multiple workflow runs (20+) to capture experiment enrollments

Check Datadog logs for experiment enrollment messages:

Experiment enrolled: model={model_name}, rollout_pct=5
Experiment: agent {name} using model {model_key}

Verify experiment PRs have "do-not-merge" label

Compare success rates and error patterns between experiment and control groups

Expected behavior:

~5% of workflows should show "Experiment enrolled" logs

Enrolled workflows should use experiment models instead of defaults

PRs from experiment runs should have "do-not-merge" label

Excluded agents (test_and_fix_agent, fix_action_agent, response_schema_*) should not use experiment models

Notes

Experiment is controlled by EXPERIMENT_ROLLOUT_PCT environment variable (default: 5)

Set to 0 to disable the experiment

Experiment PRs are automatically labeled "do-not-merge" to prevent accidental production impact

Mission-critical agents are excluded from the experiment for safety

Models are selected randomly on enrollment for balanced distribution

API keys must be set in environment for experiment models to work

Models to Test

Model

Provider

Input $/M

Output $/M

Notes

Claude Sonnet 4.5 (baseline)

Anthropic

$3.00

$15.00

Current default

Kimi K2.5

Fireworks (OpenAI-compatible)

$0.60

$2.50

1T params, 262K context, ~80% cheaper than Claude

GLM-5

ZAI

$1.00

$3.20

744B MoE (40B active), 200K context, MIT license

GPT-5.3 Codex

OpenAI

$1.75

$14.00

SOTA on SWE-Bench Pro & Terminal-Bench 2.0, 400K context

Implementation Approach

5% traffic routing: For each workflow run, randomly select 5% to also run on an alternative model

Shadow experiment: Both default and alternative model run; alternative model PRs tagged do-not-merge

No production impact: Experiment runs are purely for comparison, not replacing production output

Metrics collection: Log model used, cost, latency, and quality scores for each run

feat: add 5% multi-model A/B experiment routing (Kimi K2.5, GLM-5, GPT-5.3 Codex)

Why

What

How to Test

Pre-Review Checklist

Notes

Objective

Models to Test

Implementation Approach

Scope

Why

What

How to Test

Pre-Review Checklist

Notes

Objective

Models to Test

Implementation Approach

Scope

Implementation Details

Model Details

Kimi K2.5 (Moonshot AI)

GLM-5 (Zhipu AI)

GPT-5.3 Codex (OpenAI)

Success Criteria

Cost Context (2-week avg, excl response-schema)

References