PR Custody

Add ingestion retry visibility

@wjayeshchecks n/achecks…feature/ingestion-tp-retries-and-failure-visibility → pipeline-self-hosted-plan-artifacts9 files · +322 −22updated 3w ago

GitHub

▸Description

Summary

add transient-only retries around TurboPuffer vector-store HTTP calls
add local per-cycle Slack rollups for cached/new plan write failures and cluster match/search failures
keep durable Datadog metric names unchanged while surfacing failures in existing ingestion Slack summaries
add focused tests for retry classification and failure summary rendering

Validation

OPENAI_API_KEY=test uv run pytest app_tester/rube_learning/ingestion_pipeline/tests/test_vector_store_writer_retries.py app_tester/rube_learning/ingestion_pipeline/tests/test_ingestion_failure_stats.py app_tester/rube_learning/ingestion_pipeline/tests/test_plan_artifact_targets.py app_tester/rube_learning/ingestion_pipeline/tests/test_search_turbopuffer_optimized.py app_tester/rube_learning/ingestion_pipeline/tests/test_search_vector_cache_raises.py
uv run black --check app_tester/rube_learning/ingestion_pipeline/vector_store_writers.py app_tester/rube_learning/ingestion_pipeline/ingestion_utils.py app_tester/rube_learning/ingestion_pipeline/retryable_operations.py app_tester/rube_learning/ingestion_pipeline/ingestion_stats_counter.py app_tester/rube_learning/ingestion_pipeline/cached_plan_processor.py app_tester/rube_learning/ingestion_pipeline/unmatched_cluster_processor.py app_tester/rube_learning/ingestion_pipeline/usecase_ingestion_service.py app_tester/rube_learning/ingestion_pipeline/tests/test_vector_store_writer_retries.py app_tester/rube_learning/ingestion_pipeline/tests/test_ingestion_failure_stats.py
uv run ruff check app_tester/rube_learning/ingestion_pipeline/vector_store_writers.py app_tester/rube_learning/ingestion_pipeline/ingestion_utils.py app_tester/rube_learning/ingestion_pipeline/retryable_operations.py app_tester/rube_learning/ingestion_pipeline/ingestion_stats_counter.py app_tester/rube_learning/ingestion_pipeline/cached_plan_processor.py app_tester/rube_learning/ingestion_pipeline/unmatched_cluster_processor.py app_tester/rube_learning/ingestion_pipeline/usecase_ingestion_service.py app_tester/rube_learning/ingestion_pipeline/tests/test_vector_store_writer_retries.py app_tester/rube_learning/ingestion_pipeline/tests/test_ingestion_failure_stats.py

Notes

Dry-run pipeline execution was attempted with --minutes-back 60 --dry-run --no-slack, but ClickHouse reset the TLS connection during session fetch, so it did not exercise plan processing. The run stayed dry-run/no-Slack and failed at fetch_sessions_failed.

loading diff…