feat: add tool router eval framework
@wjayeshchecks…feat/tool-router-eval → master83 files · +21072 −9164updated 1mo agoloading diff…
@wjayeshchecks…feat/tool-router-eval → master83 files · +21072 −9164updated 1mo ago@wjayeshchecks…feat/tool-router-eval → master83 files · +21072 −9164updated 1mo agoThis PR introduces a comprehensive benchmark framework for evaluating Tool Router configurations and features. The framework enables systematic comparison of different Tool Router paths (configurations) across a fixed set of usecases, leveraging the existing Learning Pipeline to assess session quality. There's a design doc updated with the latest changes, that you can refer to.
vectorSearchEnabled flag per projectmode="benchmark")python -m tool_router_benchmark run --config path/to/config.yamlpython -m tool_router_benchmark run --config-dirpython -m tool_router_benchmark report --run-id <id>tool_router_benchmark/ - Complete benchmark framework module
__main__.py - CLI entry pointrunner.py - Benchmark orchestrationagent_runner.py - OpenAI Agents SDK integration with Composio MCPsession_builder.py - Session construction from logsreport.py - HTML report generation (1131 lines)config.py - Configuration models and validationmodels.py - Data models for benchmarksloaders.py - YAML config loadingld_client.py - LaunchDarkly client integrationeval_configs/ - Usecase and path definitionspaths:
- name: "planning_enabled"
tool_router_project_id: "proj_xxx"
model: "gpt-4o"
description: "Planning enabled via vector search"
usecases:
- name: "send_email"
difficulty: "EASY"
prompt: "Send an email to..."
connected_accounts:
gmail: "acc_123"
concurrency:
agents: 5
pipeline: 10
timeout_seconds_per_session: 600
enabled: true
# Run single benchmark config
python -m tool_router_benchmark run --config eval_configs/benchmark_configs/planning_comparison.yaml
# Run all enabled configs
python -m tool_router_benchmark run --config-dir
# Dry run (validate without executing)
python -m tool_router_benchmark run --config config.yaml --dry-run
# Generate report from existing results
python -m tool_router_benchmark report --run-id planning_comparison_20251208_140000
uv.lock with new dependenciesSee tool_router_benchmark/DESIGN_DOC.md for comprehensive architecture and design details.