feat(wbs): AEG-VS-01-05 Event/Job/Inbox - Part 4 Stage 3 Complete (Error Handling + Monitoring)
Part 4 Stage 3: Error Handling & Monitoring Infrastructure
Error Handling
- ConsumerErrorHandler: Logs to dead-letter queue on consumer failures
- Tracks retry attempts (max 3), captures error message + stacktrace
- Atomic transaction: error record + inbox status update together
- Idempotency via (message_id, attempt_number) UNIQUE constraint
- Status flow: PENDING → RETRYING (1-3 attempts) → FAILED → ARCHIVED
Dead-Letter Queue (DLQ)
- Table: building_blocks.dead_letter_message
- Columns: message_id, event_type, payload_json, error_message, attempt_number, status
- Indexes: by status, created_at, correlation_id for alerting/querying
- Used for post-mortem analysis, alerting, manual replay
Monitoring & Observability
- ConsumerMetrics: Records latency (duration_ms), success/failure per consumer
- Table: infrastructure.consumer_metrics (partitioned by month)
- Queries: P95 latency, success rate, throughput
- Alert rules per consumer: p95_latency_ms threshold, min_success_rate %
DownstreamConsumerJob Updates
- Added ConsumerErrorHandler dependency for DLQ logging
- Wraps consumer invocations with error → dead-letter path
- Graceful failure: logs to DLQ, marks inbox as FAILED, propagates exception
- Correlation ID propagated end-to-end for tracing
Database Migrations
- 0044_consumer_error_handling_and_metrics.sql
- Creates: dead_letter_message table, consumer_metrics partitioned table
- Creates: consumer_alert_rules table (alert thresholds per consumer)
- Updates: inbox schema (add status, failed_at columns)
- Inserts: default alert rules for Identity/Audit consumers
Structured Logging
- CorrelationId propagated in all log messages
- LoggerMessage for high-performance logging (compile-time safe)
- Separate log levels: DEBUG (success), ERROR (failure), CRITICAL (DLQ failure)
Architecture: Error Path
Status: 100% COMPLETE (event + endpoint + consumers + job scheduling + E2E tests + error handling + monitoring)
Build: ✅ 0 errors, 0 warnings
Remaining: Admin UI (Vue 3 identity management page), regression tests
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
kjh2064
af0f983cd7