KArtSell.Aegis/GATE_3_RESULTS_VALIDATION.md

Gate 3 Results Validation

Purpose: Verify shadow run results meet all validation gates
Usage: After shadow run execution completes (status = EvaluationComplete)

---

Validation Gates Overview

Gate	Threshold	JSON Field	Expected
PBO	≤ 20%	pbo_under_20	true
DSR	≥ 95th	dsr_above_95	true
Cost 2x	Positive	cost_2x_positive	true
All Passed	3/3 gates	all_gates_passed	true

---

Step 1: Check Overall Status

SQL Query:

SELECT 
  run_id,
  status,
  CAST(validation_gates_json->>'all_gates_passed' AS bool) as gates_passed,
  validation_gates_json::text as full_gates,
  published_at
FROM model_operations.shadow_run
ORDER BY published_at DESC
LIMIT 1;

Expected Result:

run_id       | status              | gates_passed | full_gates                              | published_at
b2c3d4e5... | EvaluationComplete  | true         | {"all_gates_passed":true, ...}         | 2026-08-02 14:30:45

Interpretation:

• ✅ Status = EvaluationComplete: Run finished successfully
• ✅ gates_passed = true: All validation gates passed
• ⚠️ Status = Failed: Check error_message column for failure reason
• ⚠️ gates_passed = false: At least one gate failed (see details below)

---

Step 2: Validate Each Gate

Gate 2a: PBO (Probability of Backtest Overfit) ≤ 20%

SQL Query:

SELECT 
  CAST(validation_gates_json->>'pbo' AS numeric) as pbo_value,
  CAST(validation_gates_json->>'pbo_under_20' AS bool) as pbo_pass
FROM model_operations.shadow_run
ORDER BY published_at DESC
LIMIT 1;

Expected Result:

pbo_value | pbo_pass
0.15      | true

Interpretation:

• ✅ pbo_value ≤ 0.20: Strategy not overfit to historical data
• ❌ pbo_value > 0.20: Strategy may be overfit; consider:
  • Different date range
  • Different model parameters
  • Simpler strategy

Action if Failed:

Risk Level: HIGH
Recommendation: Review strategy assumptions, try longer backtest period
Contact: Risk committee for decision on proceeding despite failed gate

---

Gate 2b: DSR (Daily Sharpe Ratio) ≥ 95th Percentile

SQL Query:

SELECT 
  CAST(validation_gates_json->>'dsr' AS numeric) as dsr_value,
  CAST(validation_gates_json->>'dsr_above_95' AS bool) as dsr_pass,
  CAST(validation_gates_json->>'sharpe' AS numeric) as sharpe_ratio
FROM model_operations.shadow_run
ORDER BY published_at DESC
LIMIT 1;

Expected Result:

dsr_value | dsr_pass | sharpe_ratio
0.96      | true     | 1.45

Interpretation:

• ✅ dsr_value ≥ 0.95: Daily Sharpe ratio above 95th percentile (robust)
• ✅ sharpe_ratio ≥ 1.0: Standard Sharpe ratio is positive
• ❌ dsr_value < 0.95: Inconsistent daily performance
• ❌ sharpe_ratio < 1.0: Weak risk-adjusted returns

Action if Failed:

Risk Level: MEDIUM
Recommendation: Analyze volatility patterns, check for asymmetric risk
Contact: Quant team for robustness review

---

Gate 2c: Cost 2x (Returns Survive Doubled Fees)

SQL Query:

SELECT 
  CAST(validation_gates_json->>'cost_2x_positive' AS bool) as cost_pass,
  CAST(validation_gates_json->>'returns' AS numeric) as total_return,
  (validation_gates_json->'cost_analysis_json'->>'doubled_fee_return') as cost_2x_return
FROM model_operations.shadow_run
ORDER BY published_at DESC
LIMIT 1;

Expected Result:

cost_pass | total_return | cost_2x_return
true      | 0.28         | 0.18

Interpretation:

• ✅ cost_pass = true: Returns remain positive even with 2x fees
• ✅ cost_2x_return > 0: Robust to fee increases
• ❌ cost_pass = false: Strategy margin eroded by fees

Action if Failed:

Risk Level: MEDIUM
Recommendation: Review trading costs, optimize execution
Contact: Trading desk for fee negotiations

---

Step 3: Phase Analysis (Optional but Recommended)

SQL Query:

SELECT 
  (phase_analysis_json->'bull'->>'sharpe')::numeric as bull_sharpe,
  (phase_analysis_json->'bull'->>'return')::numeric as bull_return,
  (phase_analysis_json->'bear'->>'sharpe')::numeric as bear_sharpe,
  (phase_analysis_json->'bear'->>'return')::numeric as bear_return,
  (phase_analysis_json->'sideways'->>'sharpe')::numeric as sideways_sharpe,
  (phase_analysis_json->'sideways'->>'return')::numeric as sideways_return
FROM model_operations.shadow_run
ORDER BY published_at DESC
LIMIT 1;

Expected Result:

bull_sharpe | bull_return | bear_sharpe | bear_return | sideways_sharpe | sideways_return
1.8         | 0.35        | 0.9         | 0.15        | 1.2             | 0.22

Interpretation:

• ✅ All non-zero: Strategy works across market regimes
• ✅ Bull sharpe > bear sharpe: Better in trending markets (typical)
• ⚠️ Bear sharpe < 1.0: Struggles in downturns (acceptable)
• ❌ Any = 0: Missing data for market phase

Insights:

• Bull regime: +35% return (1.8 Sharpe) — strong upside capture
• Bear regime: +15% return (0.9 Sharpe) — downside protection working
• Sideways: +22% return (1.2 Sharpe) — range-bound trading effective

---

Step 4: Audit Trail Verification

SQL Query:

SELECT 
  sr.run_id,
  sr.published_at,
  COUNT(DISTINCT om.correlation_id) as distinct_correlation_ids,
  COUNT(DISTINCT im.consumer_id) as consumers_processed,
  (SELECT COUNT(*) FROM model_operations.approval_queue 
   WHERE run_id = sr.run_id) as approval_records
FROM model_operations.shadow_run sr
LEFT JOIN building_blocks.outbox_message om ON sr.run_id::text = om.payload_json->>'runId'
LEFT JOIN outbox.inbox im ON om.message_id = im.outbox_id
WHERE sr.run_id = '<RUN_ID>'
GROUP BY sr.run_id, sr.published_at;

Expected Result:

run_id       | published_at        | distinct_correlation_ids | consumers_processed | approval_records
b2c3d4e5... | 2026-08-02 14:30:45 | 1                        | 3                   | 1

Interpretation:

• ✅ distinct_correlation_ids = 1: Single run traced end-to-end
• ✅ consumers_processed ≥ 1: Events routed to consumers
• ✅ approval_records = 1: Approval auto-populated
• ❌ Any = 0: Audit trail incomplete

---

Summary Checklist

After execution, verify:

• Status = EvaluationComplete
• all_gates_passed = true
  • pbo_under_20 = true (PBO ≤ 20%)
  • dsr_above_95 = true (DSR ≥ 95th)
  • cost_2x_positive = true (2x fee robust)
• Phase analysis populated (bull, bear, sideways)
• Approval queue auto-populated (status = Pending)
• Correlation IDs in audit trail
• No error_message in shadow_run

---

Decision Points

Scenario	Action
All gates ✅	Proceed to approval workflow (Gate 4)
PBO fails	Contact Risk committee
DSR fails	Contact Quant team for robustness review
Cost gate fails	Discuss with Trading desk
Audit trail incomplete	Investigate Outbox→Inbox pipeline
Approval not auto-populated	Check downstream consumer job logs

---

Next Steps (if all validated)

1. Query Approval Queue

   SELECT id, run_id, status, requested_at
   FROM model_operations.approval_queue
   WHERE run_id = '<RUN_ID>';
   

2. Maker-Checker Approval

   curl -X POST http://localhost:5000/api/v1/approval-queue/{id}/approve \
     -H "Authorization: Bearer <token>" \
     -d '{"approvalReason":"All gates passed, approved for activation"}'
   

3. Verify Approval Updated

   SELECT status, approved_by, approval_reason, approved_at
   FROM model_operations.approval_queue
   WHERE run_id = '<RUN_ID>';
   

---

Troubleshooting

Problem: Shadow run missing validation gates JSON

Solution: Check error_message for execution errors. Re-run with logs enabled.

Problem: Approval not auto-created

Solution: Check DownstreamConsumerJob logs. Verify ShadowRunCompletedEvent was emitted.

Problem: One gate failed (e.g., PBO > 20%)

Solution: This is NOT a blocker for activation, but flags increased backtest risk.
Review with Risk committee before activation.

Problem: Phase analysis all zeros

Solution: Check date range covered all market regimes.
If short period, results are expected. Use longer window for production.