PR Custody

Add self-hosted plan artifact targets

@wjayeshchecks n/achecks…pipeline-self-hosted-plan-artifacts → master7 files · +1938 −398updated 3w ago

▸Description

Summary

Adds self-hosted plan artifact targets to the Rube plan ingestion pipeline.

The managed plan path remains the source of truth: ingestion still writes Voyage embeddings to the primary TurboPuffer namespace. When ENABLE_SELF_HOSTED_PLAN_ARTIFACTS=1 is set, each run reconciles provider-specific artifact namespaces from that managed namespace. OpenAI is the only implemented artifact embedding provider in this PR.

Details

Adds EmbeddingProvider abstractions for Voyage and OpenAI embeddings.
Adds VectorStoreWriter abstraction with a TurboPuffer implementation.
Adds plural artifact provider config via SELF_HOSTED_PLAN_EMBEDDING_PROVIDERS.
Defaults the OpenAI artifact namespace to {managed_namespace}_openai, with SELF_HOSTED_PLAN_TP_NAMESPACE_OPENAI override support.
Treats artifact namespaces as derived mirrors of managed state: list managed rows, re-embed every managed row with the artifact provider, upsert the full artifact set, and delete artifact rows whose IDs are no longer present in managed.
Skips artifact reconciliation in dry-run mode.
Fails the overall run if any enabled artifact embedding/write/delete fails.
Adds SELF_HOSTED_PLAN_ARTIFACTS.md with the pipeline understanding, config, extension process, and current scope.

Validation

MERCURY_PATH=/Users/wjayesh/apps/mercury ZAPIER_PATH=/tmp OPENAI_API_KEY=test PYTHONPATH=/Users/wjayesh/apps/mercury:/Users/wjayesh/apps/integrator-plan-artifacts /Users/wjayesh/apps/integrator/.venv/bin/pytest app_tester/rube_learning/ingestion_pipeline/tests/test_plan_artifact_targets.py -q
/Users/wjayesh/apps/integrator/.venv/bin/ruff check app_tester/rube_learning/ingestion_pipeline/embedding_providers.py app_tester/rube_learning/ingestion_pipeline/vector_store_writers.py app_tester/rube_learning/ingestion_pipeline/turbopuffer_ingestion.py app_tester/rube_learning/ingestion_pipeline/tests/test_plan_artifact_targets.py
/Users/wjayesh/apps/integrator/.venv/bin/python -m compileall app_tester/rube_learning/ingestion_pipeline/embedding_providers.py app_tester/rube_learning/ingestion_pipeline/vector_store_writers.py app_tester/rube_learning/ingestion_pipeline/turbopuffer_ingestion.py app_tester/rube_learning/ingestion_pipeline/tests/test_plan_artifact_targets.py
git diff --check

loading diff…