Gate 5: Observability & Alerting (Metrics & Dashboard Foundation)

Implements validation gate 5: Production readiness observability infrastructure

Backend implementation:
1. IObservabilityService interface - 5 metric families
2. ObservabilityService implementation - SQL queries for metrics
3. GetObservabilityMetrics endpoint (GET /api/v1/observability/metrics)

Metric Families (Grafana/Seq integration-ready):
1. **Batch SLA Metrics**: Job completion times, queue depths, retry rates
   - QueueDepth: Pending job count
   - AverageCompletionTimeMs: Job execution time
   - TotalJobsCompleted: Success count
   - RetryCount: Retry rate tracking

2. **Data Quality Metrics**: Quarantine monitoring
   - QuarantinedJobCount: Jobs marked dq (data quality)
   - TopQuarantineReasons: Error pattern analysis
   - AverageQuarantineAgeHours: Quarantine age tracking

3. **Duplicate Detection**: Constraint violation monitoring
   - DuplicateViolationCount: Inbox dedup failures
   - AffectedMessageCount: Impact analysis
   - LastViolationAt: Recency tracking

4. **Reconciliation Metrics**: Audit trail completeness
   - OutboxMessageCount: Total published events
   - InboxProcessedCount: Processed events
   - AuditTrailCompleteness %: Evidence preservation ratio
   - MismatchCount: Orphaned messages

5. **Model Drift Metrics**: OOS performance tracking
   - ModelsUnderMonitoring: Active model count
   - AverageOosPerformance: Out-of-sample DSR
   - PerformanceDegradedCount: Alert threshold
   - BaselineSharpeRatio: Baseline comparison

Alert Thresholds (AGENTS.md v16.0 constraint enforcement):
- CRITICAL: Duplicate inbox messages detected
- WARNING: Audit trail completeness < 95%
- WARNING: > 10 jobs in quarantine
- WARNING: Model performance degradation detected

Test coverage (6 scenarios):
1. Batch SLA metrics structure validation
2. Data Quality quarantine monitoring
3. Duplicate detection identification
4. Reconciliation completeness calculation
5. Model drift OOS tracking
6. Alert threshold conditions

Architecture:
- Database queries (Hangfire + audit tables)
- Metrics DTOs for serialization
- REST endpoint for dashboard consumption
- Ready for Grafana/Seq/OpenTelemetry integration

AGENTS.md v16.0 compliance:
✓ Evidence-based monitoring (5 metric families)
✓ Constraint validation (alert thresholds)
✓ Audit trail traceability (correlation IDs)
✓ Complete endpoint (all gates monitored)

Build: Clean, 0 errors

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

src/KArtSell.Modules.ModelOperations/Features/GetObservabilityMetrics/Endpoint.cs

@@ -0,0 +1,34 @@
+using FastEndpoints;
+using KArtSell.BuildingBlocks.Time;
+using KArtSell.Modules.ModelOperations.Observability;
+
+namespace KArtSell.Modules.ModelOperations.Features.GetObservabilityMetrics;
+
+public sealed class Endpoint(IObservabilityService observability, IClock clock) : EndpointWithoutRequest<Response>
+{
+    public override void Configure()
+    {
+        Get("/api/v1/observability/metrics");
+        Roles("Auditor", "System", "Risk");
+        Tags("Observability");
+    }
+
+    public override async Task HandleAsync(CancellationToken ct)
+    {
+        var batchSla = await observability.GetBatchSlaMetricsAsync(ct);
+        var dataQuality = await observability.GetDataQualityMetricsAsync(ct);
+        var duplicates = await observability.GetDuplicateDetectionMetricsAsync(ct);
+        var reconciliation = await observability.GetReconciliationMetricsAsync(ct);
+        var modelDrift = await observability.GetModelDriftMetricsAsync(ct);
+
+        var response = new Response(
+            batchSla,
+            dataQuality,
+            duplicates,
+            reconciliation,
+            modelDrift,
+            clock.UtcNow.DateTime);
+
+        await Send.OkAsync(response, ct);
+    }
+}

src/KArtSell.Modules.ModelOperations/Features/GetObservabilityMetrics/Response.cs

@@ -0,0 +1,11 @@
+using KArtSell.Modules.ModelOperations.Observability;
+
+namespace KArtSell.Modules.ModelOperations.Features.GetObservabilityMetrics;
+
+public sealed record Response(
+    BatchSlaMetrics BatchSla,
+    DataQualityMetrics DataQuality,
+    DuplicateDetectionMetrics Duplicates,
+    ReconciliationMetrics Reconciliation,
+    ModelDriftMetrics ModelDrift,
+    DateTime CollectedAt);