KArtSell.Aegis/GATE_3_RESULTS_VALIDATION.md

# Gate 3 Results Validation

**Purpose:** Verify shadow run results meet all validation gates  
**Usage:** After shadow run execution completes (status = EvaluationComplete)

---

## Validation Gates Overview

| Gate | Threshold | JSON Field | Expected |
|------|-----------|-----------|----------|
| **PBO** | ≤ 20% | `pbo_under_20` | `true` |
| **DSR** | ≥ 95th | `dsr_above_95` | `true` |
| **Cost 2x** | Positive | `cost_2x_positive` | `true` |
| **All Passed** | 3/3 gates | `all_gates_passed` | `true` |

---

## Step 1: Check Overall Status

**SQL Query:**
```sql
SELECT 
  run_id,
  status,
  CAST(validation_gates_json->>'all_gates_passed' AS bool) as gates_passed,
  validation_gates_json::text as full_gates,
  published_at
FROM model_operations.shadow_run
ORDER BY published_at DESC
LIMIT 1;
```

**Expected Result:**
```
run_id       | status              | gates_passed | full_gates                              | published_at
b2c3d4e5... | EvaluationComplete  | true         | {"all_gates_passed":true, ...}         | 2026-08-02 14:30:45
```

**Interpretation:**
- ✅ **Status = EvaluationComplete**: Run finished successfully
- ✅ **gates_passed = true**: All validation gates passed
- ⚠️ **Status = Failed**: Check error_message column for failure reason
- ⚠️ **gates_passed = false**: At least one gate failed (see details below)

---

## Step 2: Validate Each Gate

### Gate 2a: PBO (Probability of Backtest Overfit) ≤ 20%

**SQL Query:**
```sql
SELECT 
  CAST(validation_gates_json->>'pbo' AS numeric) as pbo_value,
  CAST(validation_gates_json->>'pbo_under_20' AS bool) as pbo_pass
FROM model_operations.shadow_run
ORDER BY published_at DESC
LIMIT 1;
```

**Expected Result:**
```
pbo_value | pbo_pass
0.15      | true
```

**Interpretation:**
- ✅ **pbo_value ≤ 0.20**: Strategy not overfit to historical data
- ❌ **pbo_value > 0.20**: Strategy may be overfit; consider:
  - Different date range
  - Different model parameters
  - Simpler strategy

**Action if Failed:**
```
Risk Level: HIGH
Recommendation: Review strategy assumptions, try longer backtest period
Contact: Risk committee for decision on proceeding despite failed gate
```

---

### Gate 2b: DSR (Daily Sharpe Ratio) ≥ 95th Percentile

**SQL Query:**
```sql
SELECT 
  CAST(validation_gates_json->>'dsr' AS numeric) as dsr_value,
  CAST(validation_gates_json->>'dsr_above_95' AS bool) as dsr_pass,
  CAST(validation_gates_json->>'sharpe' AS numeric) as sharpe_ratio
FROM model_operations.shadow_run
ORDER BY published_at DESC
LIMIT 1;
```

**Expected Result:**
```
dsr_value | dsr_pass | sharpe_ratio
0.96      | true     | 1.45
```

**Interpretation:**
- ✅ **dsr_value ≥ 0.95**: Daily Sharpe ratio above 95th percentile (robust)
- ✅ **sharpe_ratio ≥ 1.0**: Standard Sharpe ratio is positive
- ❌ **dsr_value < 0.95**: Inconsistent daily performance
- ❌ **sharpe_ratio < 1.0**: Weak risk-adjusted returns

**Action if Failed:**
```
Risk Level: MEDIUM
Recommendation: Analyze volatility patterns, check for asymmetric risk
Contact: Quant team for robustness review
```

---

### Gate 2c: Cost 2x (Returns Survive Doubled Fees)

**SQL Query:**
```sql
SELECT 
  CAST(validation_gates_json->>'cost_2x_positive' AS bool) as cost_pass,
  CAST(validation_gates_json->>'returns' AS numeric) as total_return,
  (validation_gates_json->'cost_analysis_json'->>'doubled_fee_return') as cost_2x_return
FROM model_operations.shadow_run
ORDER BY published_at DESC
LIMIT 1;
```

**Expected Result:**
```
cost_pass | total_return | cost_2x_return
true      | 0.28         | 0.18
```

**Interpretation:**
- ✅ **cost_pass = true**: Returns remain positive even with 2x fees
- ✅ **cost_2x_return > 0**: Robust to fee increases
- ❌ **cost_pass = false**: Strategy margin eroded by fees

**Action if Failed:**
```
Risk Level: MEDIUM
Recommendation: Review trading costs, optimize execution
Contact: Trading desk for fee negotiations
```

---

## Step 3: Phase Analysis (Optional but Recommended)

**SQL Query:**
```sql
SELECT 
  (phase_analysis_json->'bull'->>'sharpe')::numeric as bull_sharpe,
  (phase_analysis_json->'bull'->>'return')::numeric as bull_return,
  (phase_analysis_json->'bear'->>'sharpe')::numeric as bear_sharpe,
  (phase_analysis_json->'bear'->>'return')::numeric as bear_return,
  (phase_analysis_json->'sideways'->>'sharpe')::numeric as sideways_sharpe,
  (phase_analysis_json->'sideways'->>'return')::numeric as sideways_return
FROM model_operations.shadow_run
ORDER BY published_at DESC
LIMIT 1;
```

**Expected Result:**
```
bull_sharpe | bull_return | bear_sharpe | bear_return | sideways_sharpe | sideways_return
1.8         | 0.35        | 0.9         | 0.15        | 1.2             | 0.22
```

**Interpretation:**
- ✅ **All non-zero**: Strategy works across market regimes
- ✅ **Bull sharpe > bear sharpe**: Better in trending markets (typical)
- ⚠️ **Bear sharpe < 1.0**: Struggles in downturns (acceptable)
- ❌ **Any = 0**: Missing data for market phase

**Insights:**
- Bull regime: +35% return (1.8 Sharpe) — strong upside capture
- Bear regime: +15% return (0.9 Sharpe) — downside protection working
- Sideways: +22% return (1.2 Sharpe) — range-bound trading effective

---

## Step 4: Audit Trail Verification

**SQL Query:**
```sql
SELECT 
  sr.run_id,
  sr.published_at,
  COUNT(DISTINCT om.correlation_id) as distinct_correlation_ids,
  COUNT(DISTINCT im.consumer_id) as consumers_processed,
  (SELECT COUNT(*) FROM model_operations.approval_queue 
   WHERE run_id = sr.run_id) as approval_records
FROM model_operations.shadow_run sr
LEFT JOIN building_blocks.outbox_message om ON sr.run_id::text = om.payload_json->>'runId'
LEFT JOIN outbox.inbox im ON om.message_id = im.outbox_id
WHERE sr.run_id = '<RUN_ID>'
GROUP BY sr.run_id, sr.published_at;
```

**Expected Result:**
```
run_id       | published_at        | distinct_correlation_ids | consumers_processed | approval_records
b2c3d4e5... | 2026-08-02 14:30:45 | 1                        | 3                   | 1
```

**Interpretation:**
- ✅ **distinct_correlation_ids = 1**: Single run traced end-to-end
- ✅ **consumers_processed ≥ 1**: Events routed to consumers
- ✅ **approval_records = 1**: Approval auto-populated
- ❌ **Any = 0**: Audit trail incomplete

---

## Summary Checklist

After execution, verify:

- [ ] Status = EvaluationComplete
- [ ] all_gates_passed = true
  - [ ] pbo_under_20 = true (PBO ≤ 20%)
  - [ ] dsr_above_95 = true (DSR ≥ 95th)
  - [ ] cost_2x_positive = true (2x fee robust)
- [ ] Phase analysis populated (bull, bear, sideways)
- [ ] Approval queue auto-populated (status = Pending)
- [ ] Correlation IDs in audit trail
- [ ] No error_message in shadow_run

---

## Decision Points

| Scenario | Action |
|----------|--------|
| All gates ✅ | Proceed to approval workflow (Gate 4) |
| PBO fails | Contact Risk committee |
| DSR fails | Contact Quant team for robustness review |
| Cost gate fails | Discuss with Trading desk |
| Audit trail incomplete | Investigate Outbox→Inbox pipeline |
| Approval not auto-populated | Check downstream consumer job logs |

---

## Next Steps (if all validated)

1. **Query Approval Queue**
   ```sql
   SELECT id, run_id, status, requested_at
   FROM model_operations.approval_queue
   WHERE run_id = '<RUN_ID>';
   ```

2. **Maker-Checker Approval**
   ```bash
   curl -X POST http://localhost:5000/api/v1/approval-queue/{id}/approve \
     -H "Authorization: Bearer <token>" \
     -d '{"approvalReason":"All gates passed, approved for activation"}'
   ```

3. **Verify Approval Updated**
   ```sql
   SELECT status, approved_by, approval_reason, approved_at
   FROM model_operations.approval_queue
   WHERE run_id = '<RUN_ID>';
   ```

---

## Troubleshooting

**Problem: Shadow run missing validation gates JSON**
```
Solution: Check error_message for execution errors. Re-run with logs enabled.
```

**Problem: Approval not auto-created**
```
Solution: Check DownstreamConsumerJob logs. Verify ShadowRunCompletedEvent was emitted.
```

**Problem: One gate failed (e.g., PBO > 20%)**
```
Solution: This is NOT a blocker for activation, but flags increased backtest risk.
Review with Risk committee before activation.
```

**Problem: Phase analysis all zeros**
```
Solution: Check date range covered all market regimes.
If short period, results are expected. Use longer window for production.
```