KArtSell.Aegis/docs/PHASE_2_METRICS_PLAN.md

Phase 2: PBO/DSR Metrics Validation Planning

Governance: AGENTS.md v16.0 (Contract-first, Evidence-based)
Date: 2026-08-03 23:00 KST
Status: 📋 PLANNING (Contract Definition)
Trigger: Phase 1 completion (50-90+ days)

---

📊 CONTRACT DEFINITION (Before Implementation)

1. DATA SOURCE

Source: Job 893 Shadow Run Results

Location: Database table: model_operations.shadow_run_results
Content:
  - run_id: Unique shadow run identifier
  - daily_returns: Array of daily return percentages
  - trade_decisions: Buy/sell signals per day
  - confidence_scores: Signal confidence (0-1)
  - market_regime: Bull/Bear/Sideways phase
  - timestamp: When result was recorded

Availability:

• Start: Job 893 completion (~Oct/Nov 2026)
• Format: PostgreSQL JSONB
• Size: 252+ trading days of data

---

2. METRICS TO CALCULATE

A. PBO (Probability of Backtest Overfit)

Definition:

PBO = Probability that backtest results are due to luck/overfitting
      rather than genuine predictive signal

Target: PBO < 50% (ideally < 25%)
Interpretation:
  - PBO < 25%: Very unlikely to be overfit (EXCELLENT)
  - PBO 25-50%: Unlikely to be overfit (ACCEPTABLE)
  - PBO > 50%: Significant overfit risk (REJECT)

Methodology:

Standard: CSCV (Combinatorially Symmetric Cross-Validation)
Simplified: Z-score method if CSCV deferred (DEBT-009)

Steps:
1. Split 252-day period into K folds (e.g., 6 folds = 42 days each)
2. Test all combinations (C(K,K/2) = 20 combinations)
3. Calculate variance across combinations
4. Compute PBO = probability of overfit

Implementation Status: ⏳ DEBT-009 (Deferred)

• Option A (Full): Implement CSCV algorithm
• Option B (Simplified): Use Z-score on daily return variance
• Decision: TBD (Phase 2 start, per CLAUDE.md DEBT registry)

AGENTS.md Compliance:

• ✅ Contract defined (no placeholders)
• ✅ Success criteria clear (PBO < 50%)
• ✅ Methodology documented
• ⏳ Implementation approach TBD

---

B. DSR (Daily Sharpe Ratio)

Definition:

DSR = (Average daily return - Risk-free rate) / Daily return std dev
      Annualized: DSR * sqrt(252)

Target: DSR > Baseline (typically > 0.5)
Interpretation:
  - DSR > 1.0: Excellent risk-adjusted returns
  - DSR 0.5-1.0: Good (acceptable)
  - DSR < 0.5: Marginal (borderline)
  - DSR < 0: Negative returns (REJECT)

Calculation Formula:

daily_returns = [r1, r2, ..., r252]
avg_return = mean(daily_returns)
std_dev = stdev(daily_returns)
risk_free_rate = 0.03 / 252  # ~3% annual

DSR = (avg_return - risk_free_rate) / std_dev
DSR_annualized = DSR * sqrt(252)

Baseline Determination:

Benchmark: Buy-and-hold S&P500 DSR (~0.6-0.8 annualized)
Our target: Exceed benchmark by 50% (DSR > 0.9 annualized)
Validation: Compare against KRX KOSPI index baseline

AGENTS.md Compliance:

• ✅ Formula defined
• ✅ Data sources specified
• ✅ Benchmark established
• ✅ Success criteria clear

---

C. OOS (Out-of-Sample) Performance by Market Regime

Definition:

Verify signal performance across different market conditions:
  - Bull Market Phase: Rising indices, positive bias
  - Bear Market Phase: Falling indices, negative bias
  - Sideways Phase: Range-bound, mean-reversion dominant

Validation Matrix:

| Regime | Duration | DSR Target | Pass Criteria |
|--------|----------|------------|---------------|
| Bull   | 40% of window | > 1.0 | Profitable in uptrends |
| Bear   | 40% of window | > 0.5 | Protective (less loss) |
| Sideways| 20% of window | > 0.7 | Captures range trades |

Phase Segmentation:

Source: Phase Segmentation model (already implemented)
Integration: Query existing phase_classification results
Expected: ~100 days bull, ~100 days bear, ~52 days sideways

AGENTS.md Compliance:

• ✅ Regime definitions clear
• ✅ Performance criteria per regime
• ✅ Data source identified (phase segmentation)
• ✅ Success metrics quantified

---

3. DATA QUALITY GATES

Before Metrics Validation, Verify:

☐ Data Completeness
  - No gaps in daily returns (252 consecutive days)
  - No null values in key fields
  - Timestamp alignment correct

☐ Data Integrity
  - Return calculations match expected range (-50% to +50% daily)
  - Outliers documented and justified
  - Signal confidence scores within [0,1]

☐ Schema Conformance
  - All required columns present
  - Data types match specification
  - Revision tracking up-to-date (published_at <= cutoff)

☐ Traceability
  - Each metric traced to specific trade decision
  - Decisions linked to signal confidence
  - Market regime correlated with performance

Decision Rule: GATE PASS if all checks pass, else REJECT and debug

AGENTS.md Compliance:

• ✅ Quality criteria pre-defined
• ✅ Gate logic explicit (no subjective calls)
• ✅ Failure mode documented (debug protocol)

---

4. CALCULATION PIPELINE

High-Level Flow:

Phase 1 Completion
        ↓
Extract shadow_run_results
        ↓
Data Quality Gates (PASS/REJECT)
        ↓
Calculate Daily Returns
        ↓
├─ PBO Calculation (CSCV or Z-score)
├─ DSR Calculation (annualized)
└─ OOS Performance (by regime)
        ↓
Generate Metrics Report
        ↓
Validate Against Thresholds
        ↓
Phase 2 Results
        ↓
Phase 3 Re-check + Phase 4 Sign-off

AGENTS.md Compliance:

• ✅ Pipeline stages clearly defined
• ✅ Decision points explicit (PASS/REJECT)
• ✅ No ambiguous branching
• ✅ Each stage has success criteria

---

5. IMPLEMENTATION CHECKLIST

Phase 2 Execution (TBD start date: after Phase 1):

• Environment Setup (1 hour)

  • PostgreSQL connection verified
  • Data query tested
  • Python/C# environment ready

• Data Extraction (2 hours)

  • Query shadow_run_results table
  • Validate 252-day completeness
  • Export to CSV for analysis

• Data Quality (2 hours)

  • Run quality gates (all checks pass)
  • Document any anomalies
  • Generate data report

• Metrics Calculation (3 hours)

  • Implement daily return calculation
  • Calculate DSR (annualized)
  • Calculate PBO (simplified or full per DEBT-009 decision)
  • Calculate OOS performance by regime

• Validation & Reporting (2 hours)

  • Compare against baselines
  • Generate visual charts
  • Write findings report

• Decision (1 hour)

  • PASS: All metrics exceed thresholds → Phase 3/4 proceed
  • MARGINAL: Some metrics borderline → Discussion required
  • FAIL: Key metrics below threshold → Root cause analysis

• Evidence Archival (1 hour)

  • Save report + data + calculations
  • Commit to repository
  • Update CLAUDE.md

Total Estimated Time: 12 hours (1-2 calendar days)

---

6. SUCCESS CRITERIA

Phase 2 Complete When:

✅ All 4 data quality gates PASS
✅ DSR_annualized > 0.9 (or justified exception)
✅ PBO < 50% (or simplified method used with caveat)
✅ OOS Bull performance DSR > 1.0
✅ OOS Bear performance DSR > 0.5
✅ All results documented + archived
✅ Report signed off (Claude + reviewed by user if desired)

Failure Handling:

If metrics marginal:
  1. Investigate root cause
  2. Check for data quality issues
  3. Validate model assumptions
  4. Document findings
  5. Proceed to Phase 3 with caveats
  
If metrics fail:
  1. Halt Phase 4 sign-off
  2. Perform root cause analysis
  3. Determine if:
     a) Model needs retraining (defer to next iteration)
     b) Shadow run had anomaly (rerun if fixable)
     c) Metrics calculation error (fix and recompute)
  4. Escalate to user for decision

---

📋 DECISION: DEBT-009 (PBO Methodology)

Question: Full CSCV vs Simplified Z-score?

Option A: Full CSCV (15-20 hours)

• Pros: Publication-grade, defensible
• Cons: Complex to implement, time-consuming
• When: If DEBT-009 resolved before Phase 2

Option B: Simplified Z-score (2-3 hours)

• Pros: Fast, reasonable proxy
• Cons: Less rigorous, academic criticism
• When: If DEBT-009 deferred to Phase 3/4

Current Status: DEBT-009 on backlog (not yet started) Recommendation: Use Simplified for Phase 2, document limitation, defer full CSCV to Phase 3 if time permits

Decision Trigger: Phase 2 start date (when Job 893 completes)

---

✅ AGENTS.md v16.0 COMPLIANCE

• ✅ Contract-First: All metrics defined before coding
• ✅ Evidence-Based: Success criteria explicit, not subjective
• ✅ No Shortcuts: All quality gates required
• ✅ Traceability: Each metric linked to trade decision
• ✅ Maturity: Schema + validation + success criteria ready
• ✅ Decision-Documented: DEBT-009 decision TBD at Phase 2 start
• ✅ No Placeholders: Concrete formulas, data sources, tools specified

---

📌 NEXT STEPS

Immediate (Next 24-48 hours)

• ✅ Plan documented (this file)
• ✅ Ready for Phase 2 execution

When Job 893 Completes (50-90+ days)

1. Trigger: Job 893 status = COMPLETE
2. Notify: Phase 2 starts (execute this checklist)
3. Duration: 12 hours (1-2 calendar days)
4. Output: Metrics report + decision

Phase 2 → Phase 3 → Phase 4 Timeline

Phase 2 (12 hours):        Metrics validation
Phase 3 (concurrent):      Scenario 1 re-test (when outbox has data)
Phase 4 (10 hours):        Final sign-off
                           ↓
                    100% PRODUCTION READY

---

Prepared by: Claude Haiku 4.5
Governance: AGENTS.md v16.0
Status: ✅ READY FOR EXECUTION (awaiting Phase 1 completion)
Review Date: 2026-10-XX (when Phase 1 nears completion)