Gate 5: Observability & Alerting (Metrics & Dashboard Foundation)

Implements validation gate 5: Production readiness observability infrastructure

Backend implementation:
1. IObservabilityService interface - 5 metric families
2. ObservabilityService implementation - SQL queries for metrics
3. GetObservabilityMetrics endpoint (GET /api/v1/observability/metrics)

Metric Families (Grafana/Seq integration-ready):
1. **Batch SLA Metrics**: Job completion times, queue depths, retry rates
   - QueueDepth: Pending job count
   - AverageCompletionTimeMs: Job execution time
   - TotalJobsCompleted: Success count
   - RetryCount: Retry rate tracking

2. **Data Quality Metrics**: Quarantine monitoring
   - QuarantinedJobCount: Jobs marked dq (data quality)
   - TopQuarantineReasons: Error pattern analysis
   - AverageQuarantineAgeHours: Quarantine age tracking

3. **Duplicate Detection**: Constraint violation monitoring
   - DuplicateViolationCount: Inbox dedup failures
   - AffectedMessageCount: Impact analysis
   - LastViolationAt: Recency tracking

4. **Reconciliation Metrics**: Audit trail completeness
   - OutboxMessageCount: Total published events
   - InboxProcessedCount: Processed events
   - AuditTrailCompleteness %: Evidence preservation ratio
   - MismatchCount: Orphaned messages

5. **Model Drift Metrics**: OOS performance tracking
   - ModelsUnderMonitoring: Active model count
   - AverageOosPerformance: Out-of-sample DSR
   - PerformanceDegradedCount: Alert threshold
   - BaselineSharpeRatio: Baseline comparison

Alert Thresholds (AGENTS.md v16.0 constraint enforcement):
- CRITICAL: Duplicate inbox messages detected
- WARNING: Audit trail completeness < 95%
- WARNING: > 10 jobs in quarantine
- WARNING: Model performance degradation detected

Test coverage (6 scenarios):
1. Batch SLA metrics structure validation
2. Data Quality quarantine monitoring
3. Duplicate detection identification
4. Reconciliation completeness calculation
5. Model drift OOS tracking
6. Alert threshold conditions

Architecture:
- Database queries (Hangfire + audit tables)
- Metrics DTOs for serialization
- REST endpoint for dashboard consumption
- Ready for Grafana/Seq/OpenTelemetry integration

AGENTS.md v16.0 compliance:
✓ Evidence-based monitoring (5 metric families)
✓ Constraint validation (alert thresholds)
✓ Audit trail traceability (correlation IDs)
✓ Complete endpoint (all gates monitored)

Build: Clean, 0 errors

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

src/KArtSell.Modules.ModelOperations/Observability/IObservabilityService.cs

@@ -0,0 +1,81 @@
+namespace KArtSell.Modules.ModelOperations.Observability;
+
+/// <summary>
+/// Observability service for production readiness metrics
+/// Covers: Batch SLA, Data Quality, Duplicates, Reconciliation, Model Drift
+/// </summary>
+public interface IObservabilityService
+{
+    /// <summary>
+    /// Batch SLA metrics: Job completion times and queue depths
+    /// </summary>
+    Task<BatchSlaMetrics> GetBatchSlaMetricsAsync(CancellationToken ct);
+
+    /// <summary>
+    /// Data Quality metrics: Jobs in quarantine (marked as `dq`)
+    /// </summary>
+    Task<DataQualityMetrics> GetDataQualityMetricsAsync(CancellationToken ct);
+
+    /// <summary>
+    /// Duplicate detection: Outbox message dedup violations
+    /// </summary>
+    Task<DuplicateDetectionMetrics> GetDuplicateDetectionMetricsAsync(CancellationToken ct);
+
+    /// <summary>
+    /// Reconciliation: Evidence vs current state mismatches
+    /// </summary>
+    Task<ReconciliationMetrics> GetReconciliationMetricsAsync(CancellationToken ct);
+
+    /// <summary>
+    /// Model Drift: Out-of-sample performance vs baseline
+    /// </summary>
+    Task<ModelDriftMetrics> GetModelDriftMetricsAsync(CancellationToken ct);
+}
+
+/// <summary>
+/// Batch SLA: Job completion times, queue depths, retry rates
+/// </summary>
+public sealed record BatchSlaMetrics(
+    int QueueDepth,
+    double AverageCompletionTimeMs,
+    int TotalJobsCompleted,
+    int RetryCount,
+    DateTime MeasuredAt);
+
+/// <summary>
+/// Data Quality: Jobs in quarantine, reasons, age
+/// </summary>
+public sealed record DataQualityMetrics(
+    int QuarantinedJobCount,
+    string[] TopQuarantineReasons,
+    double AverageQuarantineAgeHours,
+    DateTime MeasuredAt);
+
+/// <summary>
+/// Duplicate Detection: Constraint violations, affected messages
+/// </summary>
+public sealed record DuplicateDetectionMetrics(
+    int DuplicateViolationCount,
+    int AffectedMessageCount,
+    DateTime LastViolationAt,
+    DateTime MeasuredAt);
+
+/// <summary>
+/// Reconciliation: Evidence vs state mismatches, audit trail completeness
+/// </summary>
+public sealed record ReconciliationMetrics(
+    int MismatchCount,
+    double AuditTrailCompleteness,
+    int OutboxMessageCount,
+    int InboxProcessedCount,
+    DateTime MeasuredAt);
+
+/// <summary>
+/// Model Drift: OOS performance, baseline comparison, risk flags
+/// </summary>
+public sealed record ModelDriftMetrics(
+    int ModelsUnderMonitoring,
+    double AverageOosPerformance,
+    int PerformanceDegradedCount,
+    double BaselineSharpeRatio,
+    DateTime MeasuredAt);