KArtSell.Aegis/docs/PHASE3_OOS_PREPARATION.md

Phase 3 - Out-of-Sample (OOS) Testing Preparation

Status: ✅ READY FOR SETUP
Window: 2026-08-13 ~ 2027-08-13 (252+ trading days)
Requirement: AllGatesPassed = true (Phase 2)
Duration: 30-60 minutes execution

OOS Testing Strategy

Time Window Split

In-Sample (Training/Optimization):

• Start: 2025-08-12
• End: 2026-08-12
• Purpose: Model development + Phase 1 shadow run
• Status: ✅ Complete in Phase 1

Out-of-Sample (Validation):

• Start: 2026-08-13 (next trading day)
• End: 2027-08-13 (12 months forward)
• Purpose: Real-world performance validation
• Data: Actual market prices (live)
• Prevents curve-fitting bias

OOS Metrics to Calculate

1. Sharpe Ratio (OOS)

   • Formula: Expected Return / Std Dev
   • Comparison: OOS vs In-Sample
   • Target: OOS Sharpe ≥ 80% of In-Sample

2. Sortino Ratio

   • Downside volatility focus
   • Target: Positive + stable

3. Maximum Drawdown

   • Peak-to-trough decline
   • Target: < 20% of portfolio

4. Calmar Ratio

   • Return / Max Drawdown
   • Target: > 1.0

5. Information Ratio

   • Alpha over benchmark (KOSPI)
   • Target: Positive

Walk-Forward Validation

Retrain: Re-optimize model quarterly on latest data
  Q1 (Aug-Oct):    Train on 2025-08-12 ~ 2026-08-12, Test on 2026-08-13 ~ 2026-10-31
  Q2 (Nov-Jan):    Train on 2025-11-12 ~ 2026-11-12, Test on 2026-11-01 ~ 2027-01-31
  Q3 (Feb-Apr):    Train on 2025-02-12 ~ 2027-02-12, Test on 2027-02-01 ~ 2027-04-30
  Q4 (May-Jul):    Train on 2025-05-12 ~ 2027-05-12, Test on 2027-05-01 ~ 2027-08-13

Purpose: Detect model degradation over time
Action if Degrading: Reoptimize or fall back to Phase 1 model

Data Quality Checks

Pre-Phase 3 Verification:

-- Check OOS data availability
SELECT COUNT(*) as trading_days
FROM krx_data.daily_prices
WHERE ticker IN ('KOSPI', 'KOSDAQ')
  AND date BETWEEN '2026-08-13' AND '2027-08-13';

-- Expected: ~250 trading days per ticker
-- If < 200: OOS window incomplete, delay Phase 3

During Phase 3 Monitoring:

-- Monitor price gaps and anomalies
SELECT date, ticker, ABS(close - LAG(close) OVER (PARTITION BY ticker ORDER BY date)) / LAG(close) as pct_change
FROM krx_data.daily_prices
WHERE ticker = 'KOSPI'
  AND date BETWEEN '2026-08-13' AND '2027-08-13'
  AND ABS(close - LAG(close) OVER (PARTITION BY ticker ORDER BY date)) / LAG(close) > 0.05
ORDER BY date;

-- Flag unusual moves (gap days) for investigation

Auto-Execution Configuration

Trigger Condition:

IF (Phase2Result.AllGatesPassed == true)
{
    // Auto-start Phase 3
    var phase3Command = new Phase3OosTestCommand
    {
        ModelId = phase1Result.ModelId,
        InSampleEndDate = new DateOnly(2026, 8, 12),
        OosSampleStartDate = new DateOnly(2026, 8, 13),
        OosSampleEndDate = new DateOnly(2027, 8, 13),
        ValidationMetrics = new[] { "SharpeRatio", "SortinoRatio", "MaxDrawdown", "CalmarRatio", "InfoRatio" },
        RetrainingSchedule = "Quarterly"
    };
    
    // Queue for immediate execution
    BackgroundJobClient.Enqueue(() => OosTestJob.ExecuteAsync(phase3Command));
}

Expected Timeline

T+0h   Phase 2 gates judgment
  ↓
T+0.01h PASS/FAIL decision
  ↓
T+0.02h IF PASS: Phase 3 queue + start
  ↓
T+0.1h Phase 3 execution (OOS validation)
  ↓
T+1.0h Phase 3 complete (30-60 min)
  ↓
T+1.1h Phase 4 ready (manual approval)

Success Criteria for Phase 3

• ✅ OOS Sharpe >= 1.0 (positive performance)
• ✅ OOS Sharpe >= 80% of In-Sample Sharpe
• ✅ Maximum Drawdown < 20%
• ✅ Calmar Ratio > 1.0
• ✅ No curve-fitting detected (walk-forward stable)

If Phase 3 Fails

Failure Scenario 1: OOS Sharpe << In-Sample

• Indicates overfitting during Phase 1
• Action: Return to Phase 3 Unblock (model redesign)
• Timeline: 2-4 hours additional tuning

Failure Scenario 2: Large Drawdown (> 20%)

• Market regime change or model weakness
• Action: Implement stop-loss or reduce position size
• Timeline: 1-2 hours quick fix

Failure Scenario 3: Walk-Forward Degrades

• Model loses effectiveness over time
• Action: Implement quarterly retraining logic
• Timeline: 2-4 hours infrastructure change

Next Steps

1. Verify OOS data availability (2026-08-13 ~ 2027-08-13)
2. Configure walk-forward validation parameters
3. Set up monitoring dashboards for Phase 3
4. Prepare fallback strategies for failure scenarios
5. Ready for auto-trigger at Phase 2 completion