feat(observability): log project_id so Datadog @project_id surfaces Mercury logs

@zen-agentchecks n/achecks…zen/mercury-log-project-id-z1ls89 → master6 files · +136 −8updated 1mo ago

▸Description· 1 comment

Description

Datadog queries filtering by @project_id (e.g. @project_id:pr_Gv-uXCTsDSJI) return zero Mercury hits today because the Mercury Datadog sink only emits @request_id and @client_email. Filtering by @request_id works (correlation succeeds), but agents and oncall can't pivot from a project to the Mercury logs that belong to it.

This change extends request_ctx to carry an optional project_id and threads it into the Datadog sink's extra payload only when present, so logs without it look identical to today. Three ingress paths populate it:

Lambda entrypoint (serverless/handler.py): new _extract_project_id(event) reads event["project_id"] (fast path for non-tool ops) or event["data"]["metadata"]["__project_id"] (per-tool-execution path). Runs on every invocation.
Tool-execution path (serverless/base.py): when Thermos forwards __project_id inside the execute_action metadata dict (same shape as the existing __is_latest_version flag), promote it into request_ctx so per-tool logs get tagged.
HTTP middleware (observability/middleware.py): reads an optional x-project-id header for the local lambdad-on-HTTP path.

Backwards-compatible: when none of the three sources populate project_id, the Datadog payload is identical to today (no @project_id field).

Companion PR: ComposioHQ/hermes threads project_id through Apollo → Thermos → the __project_id metadata key so this field actually gets populated in production.

How did I test this PR

Added tests/test_observability/test_project_id_logging.py:
- _extract_project_id covers top-level / metadata-embedded / top-level-wins / missing cases.
- LogSink tests confirm @project_id is emitted when the context carries it and omitted when it doesn't.
pytest tests/test_observability/ tests/test_serverless/test_base.py tests/test_serverless/test_functions.py → 93 passed.
ruff check + ruff format --check on changed files → clean.
mypy on changed files → no new errors (only pre-existing third-party stub issues in unrelated mercury/secrets.py).

@zen-agent1mo ago

Status update

Step	Result
`/build-checks`	`ruff check`, `ruff format --check` clean on changed files; `mypy` clean (no new errors from this diff)
`/codex-review-loop`	1 iteration — `No actionable correctness issues were identified in the changes` (LGTM)
`/gh-fix-ci-loop`	13/14 checks green; only `Analyze (python)` (CodeQL scanner) still running — long-running scan, not code-related. Background monitor will notify when complete
Unit tests	New `tests/test_observability/test_project_id_logging.py` (6 tests): covers `_extract_project_id` precedence + missing cases, and `LogSink` emit-on-set / omit-on-unset. Broader sweep `pytest tests/test_observability/ tests/test_serverless/test_base.py tests/test_serverless/test_functions.py` → 93 passed
E2E vs local servers	Not feasible — Mercury runs as AWS Lambda and isn't exercised locally. E2E verification plan documented in PR description: post-merge, fire a tool execution and verify `@project_id:pr_*` Datadog query surfaces Mercury logs
PR comments	None on this PR

No remaining blockers. Hermes companion PR: ComposioHQ/hermes#9965.

loading diff…

@zen-agent1mo ago

Status update

Step	Result
`/build-checks`	`ruff check`, `ruff format --check` clean on changed files; `mypy` clean (no new errors from this diff)
`/codex-review-loop`	1 iteration — `No actionable correctness issues were identified in the changes` (LGTM)
`/gh-fix-ci-loop`	13/14 checks green; only `Analyze (python)` (CodeQL scanner) still running — long-running scan, not code-related. Background monitor will notify when complete
Unit tests	New `tests/test_observability/test_project_id_logging.py` (6 tests): covers `_extract_project_id` precedence + missing cases, and `LogSink` emit-on-set / omit-on-unset. Broader sweep `pytest tests/test_observability/ tests/test_serverless/test_base.py tests/test_serverless/test_functions.py` → 93 passed
E2E vs local servers	Not feasible — Mercury runs as AWS Lambda and isn't exercised locally. E2E verification plan documented in PR description: post-merge, fire a tool execution and verify `@project_id:pr_*` Datadog query surfaces Mercury logs
PR comments	None on this PR

No remaining blockers. Hermes companion PR: ComposioHQ/hermes#9965.