# Downstream Event Consumers: Shadow Run Completion (AGENTS.md v16.0)

## 1. SOURCE (Requirements)

**From CLAUDE.md:**
- § "Async Coupling: Outbox/Inbox" — Use event-driven async notification
- § "SignalR (Real-Time Push)" — Live notifications (model activation events, approval notifications)

**From Shadow Run Design:**
- All validation gates (PBO ≤ 20%, DSR ≥ 95%, Cost 2x) must notify stakeholders
- Approval workflows triggered on gate passage
- Failure scenarios logged and escalated

**Business Logic:**
- Shadow run completes → Outbox event inserted (transactional)
- Hangfire Outbox Poller reads events → Inbox (idempotent delivery)
- Inbox Consumers process: SignalR notification, approval queue, audit log

---

## 2. ARCHITECTURE (Event-Driven Async)

```
ShadowRunJob (Phase 5: Persist)
  │
  ├─ INSERT shadow_run (PIT append)
  ├─ INSERT outbox {ShadowRunCompletedEvent} (same transaction)
  │
  └─ Hangfire OutboxPoller (every 30s)
     ├─ SELECT * FROM outbox WHERE processed_at IS NULL
     ├─ INSERT inbox {event_id, payload, consumer_id, status}
     ├─ UPDATE outbox SET processed_at
     │
     └─ Hangfire InboxConsumers (fanout)
        ├─ ShadowRunCompletedConsumer (SignalR push)
        │  └─ foreach user in group "model-{modelId}" → send notification
        ├─ ApprovalQueueConsumer (if AllGatesPassed)
        │  └─ INSERT approval_queue {runId, status=Pending}
        └─ AuditLogConsumer (all runs)
           └─ INSERT audit_log {runId, event_type, status}
```

---

## 3. CONTRACT (Event + Consumer)

### Event Schema

```csharp
public record ShadowRunCompletedEvent(
    Guid RunId,
    Guid ModelId,
    Guid CorrelationId,
    DateOnly WindowStartDate,
    DateOnly WindowEndDate,
    bool AllGatesPassed,
    decimal TotalReturn,
    decimal SharpeRatio,
    decimal ProbOfBacktestOverfit,
    decimal DailySharePercentile,
    string? ErrorMessage,
    DateTime CompletedAt)
{
    public string IdempotencyKey => $"{RunId}#1";  // Deduplication key
}
```

### Consumer Interface

```csharp
public interface IInboxConsumer<TEvent>
{
    Task HandleAsync(TEvent @event, CancellationToken cancellationToken);
}

// Implementations:
public sealed class ShadowRunCompletedConsumer : IInboxConsumer<ShadowRunCompletedEvent>
{
    // Push to SignalR group: model-{modelId}
    // Payload: {status, allGatesPassed, sharpe, pbo, timestamp}
}

public sealed class ApprovalQueueConsumer : IInboxConsumer<ShadowRunCompletedEvent>
{
    // If AllGatesPassed: insert approval_queue record
    // Notify: approval_queue subscribers
}

public sealed class AuditLogConsumer : IInboxConsumer<ShadowRunCompletedEvent>
{
    // Log all completions (pass/fail) for compliance
}
```

### Database

#### Outbox Table (Existing)
```sql
CREATE TABLE outbox (
    id UUID PRIMARY KEY,
    aggregate_id UUID NOT NULL,
    event_type VARCHAR(256) NOT NULL,
    payload JSONB NOT NULL,
    published_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    processed_at TIMESTAMP,
    CONSTRAINT outbox_duplicate_check
      UNIQUE (aggregate_id, event_type, payload)
);
```

#### Inbox Table (New)
```sql
CREATE TABLE inbox (
    id UUID PRIMARY KEY,
    outbox_id UUID NOT NULL,
    event_type VARCHAR(256) NOT NULL,
    payload JSONB NOT NULL,
    consumer_id VARCHAR(256) NOT NULL,
    status VARCHAR(32) NOT NULL DEFAULT 'Pending',  -- Pending, Processed, Failed
    error_message TEXT,
    attempted_at TIMESTAMP,
    processed_at TIMESTAMP,
    CONSTRAINT inbox_idempotency
      UNIQUE (outbox_id, consumer_id),
    FOREIGN KEY (outbox_id) REFERENCES outbox(id)
);
```

#### Approval Queue Table (New)
```sql
CREATE TABLE approval_queue (
    id UUID PRIMARY KEY,
    run_id UUID NOT NULL UNIQUE,
    model_id UUID NOT NULL,
    status VARCHAR(32) NOT NULL DEFAULT 'Pending',  -- Pending, Approved, Rejected
    requested_by UUID,
    approved_by UUID,
    approval_reason TEXT,
    requested_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    approved_at TIMESTAMP,
    CONSTRAINT fk_shadow_run
      FOREIGN KEY (run_id) REFERENCES model_operations.shadow_run(run_id)
);
```

### Idempotency & Replay

| Scenario | Outbox Behavior | Inbox Behavior | Consumer |
|----------|-----------------|-------------------|----------|
| First run | INSERT event | INSERT inbox (Pending) | Process → Processed |
| Duplicate event | UNIQUE constraint blocks | Inbox already has record | Skip (idempotent) |
| Consumer fails | Inbox.status = Failed | Retry on next cycle | Transient classification |
| Consumer permanent error | Inbox.error_message set | Status = Failed | Log & alert, no retry |

---

## 4. TESTS

### Unit Tests

| Test | Scenario | Expected |
|------|----------|----------|
| Event_Idempotency | Same RunId + event → IdempotencyKey identical | Deduplication works |
| Outbox_Insert | ShadowRunJob success → Event in outbox | Transactional coupling |
| Inbox_Insert | OutboxPoller reads outbox → Event in inbox | Fanout per consumer |
| Consumer_Idempotent | Handle() called twice → Same result | Safe replay |
| Consumer_SignalR | Event.AllGatesPassed=true → SignalR.Send() called | Notification sent |
| Consumer_ApprovalQueue | Event.AllGatesPassed=true → approval_queue insert | Queue populated |

### Integration Tests

| Test | Scenario | Expected |
|------|----------|----------|
| E2E_ShadowRunToSignalR | Shadow run completes → SignalR notification | End-to-end flow |
| E2E_ApprovalQueuePopulated | Gate passage → Approval queue entry | Ready for human approval |
| E2E_Idempotency | Outbox reprocessing → No duplicate inbox | Deduplication enforced |

---

## 5. OPS (Deployment + Monitoring)

### Startup
- OutboxPoller: Runs every 30 seconds (q-research queue)
- InboxConsumers: Fanout via Hangfire service resolution
- No external API calls (pure database events)

### Monitoring
- Outbox backlog: Alert if unprocessed > 100
- Inbox failures: Alert if Failed count > 10 in 1h
- Consumer latency: Track P99 time from Outbox insert → Consumer complete
- SignalR delivery: Track connection count, message drop rate

### Rollback
- Outbox: Safe to reprocess (idempotent consumers)
- Inbox: Manually mark as Processed if needed
- Consumer: Can be restarted without state loss

---

## 6. TESTS (Verification)

### Unit: Event Idempotency
```csharp
[Fact]
public void Event_IdempotencyKey_IsDeterministic()
{
    var event1 = new ShadowRunCompletedEvent(...);
    var event2 = new ShadowRunCompletedEvent(...);
    Assert.Equal(event1.IdempotencyKey, event2.IdempotencyKey);
}
```

### Integration: Outbox Insert
```csharp
[Fact]
public async Task ShadowRunJob_Success_InsertsOutbox()
{
    // Act: ShadowRunJob completes
    // Assert: SELECT * FROM outbox WHERE aggregate_id = runId
    //         → 1 row, event_type = "ShadowRunCompleted"
}
```

### Integration: Consumer Idempotency
```csharp
[Fact]
public async Task Consumer_Handle_IsSafeToRetry()
{
    // Act: consumer.HandleAsync(event) twice
    // Assert: Same result both times (no duplicate side effects)
}
```

---

## 7. OUTPUT RULE (Deliverables)

**Changed files:**
```
src/KArtSell.Modules.ModelOperations/
  ShadowRun/Events/
    ShadowRunCompletedEvent.cs       (event contract)

src/KArtSell.Host/
  Features/ShadowRun/
    DOWNSTREAM_CONSUMERS_CONTRACT.md  (this file)
  Jobs/
    OutboxPollerJob.cs               (existing, verify)
  Consumers/
    ShadowRunCompletedConsumer.cs    (SignalR push)
    ApprovalQueueConsumer.cs         (approval workflow)
    AuditLogConsumer.cs              (compliance logging)

src/KArtSell.DbMigrator/
  0009_CreateInboxTable.sql          (inbox schema)
  0010_CreateApprovalQueueTable.sql  (approval queue)

tests/KArtSell.Integration.Tests/
  DownstreamConsumersTests.cs        (integration tests)
```

**Verification:**
```bash
dotnet test --filter "DownstreamConsumers" -c Release
# Expected: All tests green
# Outbox: ✓ Event inserted
# Inbox: ✓ Consumer fanout
# Consumer: ✓ Idempotent handling
```

---

## 8. AGENTS.md v16.0 CHECKLIST

| Criterion | Status | Evidence |
|-----------|--------|----------|
| **SOLID** | ✅ Design | Consumer interface, DI per consumer type |
| **Complexity** | ✅ Design | No branching logic, idempotent predicates |
| **Audit** | ✅ Design | outbox/inbox/approval_queue fully traced |
| **Necessity** | ✅ Sourced | From CLAUDE.md Async Coupling requirement |
| **Normalization** | ✅ Design | Outbox append-only, Inbox deduplication |
| **Simplicity** | ✅ Design | Event contract, consumer pattern, no magic |
| **Pattern** | ✅ Design | Outbox/Inbox idempotent async pattern |
| **Guardrails** | ✅ Design | IdempotencyKey deduplication, error classification |
| **Traceability** | ✅ Design | CorrelationId in event, audit log all ops |
| **Safety** | ✅ Design | Transactional outbox, idempotent consumers |
| **Maturity** | ✅ Design | Contract-first, test-first sequencing |
| **Right Way** | ✅ Design | Event-driven async, no polling delays |
| **Debt** | ✅ Design | Zero new tech debt |

---

## NEXT STEPS

### Phase 1: Event & Schema
- Define ShadowRunCompletedEvent
- Create inbox and approval_queue tables (migrations)

### Phase 2: Consumers
- Implement ShadowRunCompletedConsumer (SignalR)
- Implement ApprovalQueueConsumer (approval workflow)
- Implement AuditLogConsumer (logging)

### Phase 3: Integration
- Update ShadowRunJob to emit event on success
- Wire consumer registrations in Program.cs
- Hangfire InboxProcessor jobs

### Phase 4: Testing
- Unit: Event idempotency, consumer safety
- Integration: Outbox → Inbox → Consumer fanout
- E2E: Shadow run completion → Notification

### Phase 5: Validation
- All tests green
- No AGENTS.md violations
- Commit & push

---

**Status:** `DOWNSTREAM_CONSUMERS_CONTRACT_DEFINED`
