 kjh2064
|
042db95d9b
|
Gate 5: Observability & Alerting (Metrics & Dashboard Foundation)
Implements validation gate 5: Production readiness observability infrastructure
Backend implementation:
1. IObservabilityService interface - 5 metric families
2. ObservabilityService implementation - SQL queries for metrics
3. GetObservabilityMetrics endpoint (GET /api/v1/observability/metrics)
Metric Families (Grafana/Seq integration-ready):
1. **Batch SLA Metrics**: Job completion times, queue depths, retry rates
- QueueDepth: Pending job count
- AverageCompletionTimeMs: Job execution time
- TotalJobsCompleted: Success count
- RetryCount: Retry rate tracking
2. **Data Quality Metrics**: Quarantine monitoring
- QuarantinedJobCount: Jobs marked dq (data quality)
- TopQuarantineReasons: Error pattern analysis
- AverageQuarantineAgeHours: Quarantine age tracking
3. **Duplicate Detection**: Constraint violation monitoring
- DuplicateViolationCount: Inbox dedup failures
- AffectedMessageCount: Impact analysis
- LastViolationAt: Recency tracking
4. **Reconciliation Metrics**: Audit trail completeness
- OutboxMessageCount: Total published events
- InboxProcessedCount: Processed events
- AuditTrailCompleteness %: Evidence preservation ratio
- MismatchCount: Orphaned messages
5. **Model Drift Metrics**: OOS performance tracking
- ModelsUnderMonitoring: Active model count
- AverageOosPerformance: Out-of-sample DSR
- PerformanceDegradedCount: Alert threshold
- BaselineSharpeRatio: Baseline comparison
Alert Thresholds (AGENTS.md v16.0 constraint enforcement):
- CRITICAL: Duplicate inbox messages detected
- WARNING: Audit trail completeness < 95%
- WARNING: > 10 jobs in quarantine
- WARNING: Model performance degradation detected
Test coverage (6 scenarios):
1. Batch SLA metrics structure validation
2. Data Quality quarantine monitoring
3. Duplicate detection identification
4. Reconciliation completeness calculation
5. Model drift OOS tracking
6. Alert threshold conditions
Architecture:
- Database queries (Hangfire + audit tables)
- Metrics DTOs for serialization
- REST endpoint for dashboard consumption
- Ready for Grafana/Seq/OpenTelemetry integration
AGENTS.md v16.0 compliance:
✓ Evidence-based monitoring (5 metric families)
✓ Constraint validation (alert thresholds)
✓ Audit trail traceability (correlation IDs)
✓ Complete endpoint (all gates monitored)
Build: Clean, 0 errors
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
|
2026-08-02 13:19:38 +09:00 |
|