KArtSell.Aegis/docs/PHASE3_OOS_PREPARATION.md

# Phase 3 - Out-of-Sample (OOS) Testing Preparation

**Status:** ✅ READY FOR SETUP  
**Window:** 2026-08-13 ~ 2027-08-13 (252+ trading days)  
**Requirement:** AllGatesPassed = true (Phase 2)  
**Duration:** 30-60 minutes execution

## OOS Testing Strategy

### Time Window Split

**In-Sample (Training/Optimization):**
- Start: 2025-08-12
- End: 2026-08-12
- Purpose: Model development + Phase 1 shadow run
- Status: ✅ Complete in Phase 1

**Out-of-Sample (Validation):**
- Start: 2026-08-13 (next trading day)
- End: 2027-08-13 (12 months forward)
- Purpose: Real-world performance validation
- Data: Actual market prices (live)
- Prevents curve-fitting bias

### OOS Metrics to Calculate

1. **Sharpe Ratio (OOS)**
   - Formula: Expected Return / Std Dev
   - Comparison: OOS vs In-Sample
   - Target: OOS Sharpe ≥ 80% of In-Sample

2. **Sortino Ratio**
   - Downside volatility focus
   - Target: Positive + stable

3. **Maximum Drawdown**
   - Peak-to-trough decline
   - Target: < 20% of portfolio

4. **Calmar Ratio**
   - Return / Max Drawdown
   - Target: > 1.0

5. **Information Ratio**
   - Alpha over benchmark (KOSPI)
   - Target: Positive

## Walk-Forward Validation

```
Retrain: Re-optimize model quarterly on latest data
  Q1 (Aug-Oct):    Train on 2025-08-12 ~ 2026-08-12, Test on 2026-08-13 ~ 2026-10-31
  Q2 (Nov-Jan):    Train on 2025-11-12 ~ 2026-11-12, Test on 2026-11-01 ~ 2027-01-31
  Q3 (Feb-Apr):    Train on 2025-02-12 ~ 2027-02-12, Test on 2027-02-01 ~ 2027-04-30
  Q4 (May-Jul):    Train on 2025-05-12 ~ 2027-05-12, Test on 2027-05-01 ~ 2027-08-13

Purpose: Detect model degradation over time
Action if Degrading: Reoptimize or fall back to Phase 1 model
```

## Data Quality Checks

**Pre-Phase 3 Verification:**
```sql
-- Check OOS data availability
SELECT COUNT(*) as trading_days
FROM krx_data.daily_prices
WHERE ticker IN ('KOSPI', 'KOSDAQ')
  AND date BETWEEN '2026-08-13' AND '2027-08-13';

-- Expected: ~250 trading days per ticker
-- If < 200: OOS window incomplete, delay Phase 3
```

**During Phase 3 Monitoring:**
```sql
-- Monitor price gaps and anomalies
SELECT date, ticker, ABS(close - LAG(close) OVER (PARTITION BY ticker ORDER BY date)) / LAG(close) as pct_change
FROM krx_data.daily_prices
WHERE ticker = 'KOSPI'
  AND date BETWEEN '2026-08-13' AND '2027-08-13'
  AND ABS(close - LAG(close) OVER (PARTITION BY ticker ORDER BY date)) / LAG(close) > 0.05
ORDER BY date;

-- Flag unusual moves (gap days) for investigation
```

## Auto-Execution Configuration

**Trigger Condition:**
```csharp
IF (Phase2Result.AllGatesPassed == true)
{
    // Auto-start Phase 3
    var phase3Command = new Phase3OosTestCommand
    {
        ModelId = phase1Result.ModelId,
        InSampleEndDate = new DateOnly(2026, 8, 12),
        OosSampleStartDate = new DateOnly(2026, 8, 13),
        OosSampleEndDate = new DateOnly(2027, 8, 13),
        ValidationMetrics = new[] { "SharpeRatio", "SortinoRatio", "MaxDrawdown", "CalmarRatio", "InfoRatio" },
        RetrainingSchedule = "Quarterly"
    };
    
    // Queue for immediate execution
    BackgroundJobClient.Enqueue(() => OosTestJob.ExecuteAsync(phase3Command));
}
```

## Expected Timeline

```
T+0h   Phase 2 gates judgment
  ↓
T+0.01h PASS/FAIL decision
  ↓
T+0.02h IF PASS: Phase 3 queue + start
  ↓
T+0.1h Phase 3 execution (OOS validation)
  ↓
T+1.0h Phase 3 complete (30-60 min)
  ↓
T+1.1h Phase 4 ready (manual approval)
```

## Success Criteria for Phase 3

- ✅ OOS Sharpe >= 1.0 (positive performance)
- ✅ OOS Sharpe >= 80% of In-Sample Sharpe
- ✅ Maximum Drawdown < 20%
- ✅ Calmar Ratio > 1.0
- ✅ No curve-fitting detected (walk-forward stable)

## If Phase 3 Fails

**Failure Scenario 1: OOS Sharpe << In-Sample**
- Indicates overfitting during Phase 1
- Action: Return to Phase 3 Unblock (model redesign)
- Timeline: 2-4 hours additional tuning

**Failure Scenario 2: Large Drawdown (> 20%)**
- Market regime change or model weakness
- Action: Implement stop-loss or reduce position size
- Timeline: 1-2 hours quick fix

**Failure Scenario 3: Walk-Forward Degrades**
- Model loses effectiveness over time
- Action: Implement quarterly retraining logic
- Timeline: 2-4 hours infrastructure change

## Next Steps

1. Verify OOS data availability (2026-08-13 ~ 2027-08-13)
2. Configure walk-forward validation parameters
3. Set up monitoring dashboards for Phase 3
4. Prepare fallback strategies for failure scenarios
5. Ready for auto-trigger at Phase 2 completion