37c0254978
- Scenario 1 (PBO > 20%): 3 remediation options (confidence filtering, position sizing, stop-loss) - Scenario 2 (DSR < 95%): 3 remediation options (lower threshold, momentum indicator, adaptive sizing) - Scenario 3 (both fail): Hybrid model strategy - Fallback strategies: Simplified EMA, mean-reversion, conservative targets - Timeline: 2-4 hours recovery + 1 hour Phase 1 re-run = 3-5 hours total Decision matrix with confidence levels for all scenarios. Execution plan with step-by-step guidance. WBS Optimization: Prepare contingency paths in parallel with Phase 2 judgment. AGENTS.md v16.0: Necessity (if gates fail), Right-way (documented procedures), Tech Debt (zero new). Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
308 lines
8.5 KiB
Markdown
308 lines
8.5 KiB
Markdown
# Phase 2 Gate Failure Remediation Plan
|
||
|
||
**Status:** ✅ READY (conditional on gate results)
|
||
**Trigger:** IF AllGatesPassed = false
|
||
**Timeline:** Immediate (parallel with Phase 2 judgment)
|
||
|
||
---
|
||
|
||
## Executive Summary
|
||
|
||
If Phase 2 gates fail (expected: PBO > 20% OR DSR < 95%), execute Phase 3 Unblock v2 immediately to re-optimize model parameters and re-run Phase 1-2.
|
||
|
||
**Timeline:** 2-4 hours optimization + 1 hour Phase 1 re-run = 3-5 hours total recovery
|
||
|
||
---
|
||
|
||
## Gate Failure Scenarios
|
||
|
||
### Scenario 1: Gate 1 Fails (PBO > 20%)
|
||
|
||
**Problem:** Model exhibits overfitting (Probability of Backtest Overfit too high)
|
||
|
||
**Root Cause Analysis:**
|
||
- EMA 12/26 crossover signals too frequent
|
||
- Position sizing multiplier too aggressive
|
||
- Not enough position exits (hold periods too long)
|
||
|
||
**Remediation Options:**
|
||
|
||
**Option 1.1: Increase Signal Confidence Threshold**
|
||
```csharp
|
||
// Current: 0.75 confidence for all signals
|
||
// Change: Add confidence multiplier based on EMA divergence
|
||
|
||
decimal emaDivergence = (ema12 - ema26) / ema26; // 0-2% range
|
||
decimal confidenceBoost = 0.5m + (emaDivergence * 50); // 0.5-1.5x
|
||
decimal confidence = 0.75m * confidenceBoost;
|
||
|
||
// Effect: Stronger signals = fewer but higher-conviction trades
|
||
// Reduces PBO by filtering weak signal noise
|
||
```
|
||
|
||
**Option 1.2: Reduce Position Sizing Multiplier**
|
||
```csharp
|
||
// Current: 0.5-1.5x confidence multiplier
|
||
// Change: 0.3-0.8x (more conservative)
|
||
|
||
decimal confidence = 0.75m;
|
||
decimal multiplier = Math.Min(0.8m, 0.5m + (confidence * 0.3m));
|
||
decimal positionSize = portfolio * riskPercent * multiplier;
|
||
|
||
// Effect: Smaller positions = lower variance = lower overfit risk
|
||
```
|
||
|
||
**Option 1.3: Implement Stop-Loss Orders**
|
||
```csharp
|
||
// Add trailing stop-loss at -3% from entry
|
||
// Exit losing positions quickly to reduce drawdown
|
||
|
||
if (currentPrice < entryPrice * 0.97m)
|
||
{
|
||
// Auto-exit losing position
|
||
orders.Add(new Order { Action = "SELL", StopLoss = true });
|
||
}
|
||
|
||
// Effect: Limits downside, reduces maximum drawdown, improves Sharpe
|
||
```
|
||
|
||
**Estimated Impact:**
|
||
- PBO reduction: 25-35% → 15-20% (target ≤20%)
|
||
- Downside: Fewer total signals, lower returns (8-15% → 5-10%)
|
||
|
||
---
|
||
|
||
### Scenario 2: Gate 2 Fails (DSR < 95%)
|
||
|
||
**Problem:** Daily Sharpe Ratio percentile too low (not enough high-quality daily returns)
|
||
|
||
**Root Cause Analysis:**
|
||
- Not enough trading opportunities (signals concentrated in trends)
|
||
- Trades too infrequent or low-probability
|
||
- Position sizing not aggressive enough during high-confidence days
|
||
|
||
**Remediation Options:**
|
||
|
||
**Option 2.1: Lower EMA Crossover Threshold**
|
||
```csharp
|
||
// Current: Buy when EMA12 > EMA26 × 1.001
|
||
// Change: Buy when EMA12 > EMA26 × 1.0005 (0.05% divergence)
|
||
|
||
decimal buyThreshold = 1.0005m; // More sensitive
|
||
decimal sellThreshold = 0.9995m;
|
||
|
||
// Effect: More trading opportunities = higher daily return variability
|
||
// Increases trade frequency from 30 to 50+ signals
|
||
```
|
||
|
||
**Option 2.2: Add Momentum Indicator**
|
||
```csharp
|
||
// Combine EMA with RSI (Relative Strength Index)
|
||
// Buy: EMA12 > EMA26 AND RSI < 70 (not overbought)
|
||
// Sell: EMA12 < EMA26 OR RSI > 80
|
||
|
||
decimal rsi = CalculateRSI(prices, period: 14);
|
||
bool buySignal = (ema12 > ema26 * buyThreshold) && (rsi < 70);
|
||
|
||
// Effect: Confirms signals with momentum, improves quality, increases frequency
|
||
```
|
||
|
||
**Option 2.3: Increase Position Size on High-Confidence Days**
|
||
```csharp
|
||
// Detect high-confidence trading days (strong directional moves)
|
||
decimal dailyReturn = (close - open) / open;
|
||
|
||
if (Math.Abs(dailyReturn) > 0.02m) // >2% move
|
||
{
|
||
// Increase position size 1.5x on these days
|
||
multiplier = 1.5m;
|
||
}
|
||
|
||
// Effect: Amplify gains on trending days, improves daily return distribution
|
||
```
|
||
|
||
**Estimated Impact:**
|
||
- DSR improvement: 40-60% → 85-95% (target ≥95%)
|
||
- Trade frequency: 30 → 50-70 signals
|
||
- Volatility: May increase slightly
|
||
|
||
---
|
||
|
||
### Scenario 3: Both Gates Fail (PBO > 20% AND DSR < 95%)
|
||
|
||
**Problem:** Model fundamentally underfitted + overfit simultaneously
|
||
|
||
**Analysis:**
|
||
- EMA model too simple for current market conditions
|
||
- Need structural changes, not just parameter tweaks
|
||
|
||
**Remediation Strategy:**
|
||
|
||
**Phase 3 Unblock v2 - Hybrid Model**
|
||
|
||
```csharp
|
||
public class HybridModel
|
||
{
|
||
// Component 1: EMA trend + confidence filtering (reduce PBO)
|
||
public Signal EmaSignal(decimal ema12, decimal ema26)
|
||
{
|
||
decimal divergence = (ema12 - ema26) / ema26;
|
||
decimal confidence = Math.Max(0.5m, Math.Min(1.0m, 0.75m + (divergence * 10)));
|
||
return new Signal { Action = divergence > 0 ? "BUY" : "SELL", Confidence = confidence };
|
||
}
|
||
|
||
// Component 2: RSI momentum (increase trade frequency, improve DSR)
|
||
public bool MomentumConfirm(decimal rsi)
|
||
{
|
||
return (rsi < 70 && rsi > 30); // Not overbought/oversold
|
||
}
|
||
|
||
// Component 3: Adaptive position sizing
|
||
public decimal PositionSize(decimal confidence, decimal dailyVolatility)
|
||
{
|
||
// Higher confidence → larger position
|
||
// Higher volatility → smaller position (risk control)
|
||
decimal riskAdj = 2.0m / (1m + dailyVolatility * 10);
|
||
return 0.02m * confidence * riskAdj;
|
||
}
|
||
}
|
||
```
|
||
|
||
**Estimated Recovery:**
|
||
- PBO: 25-35% → 18-22% (target ≤20%)
|
||
- DSR: 40-60% → 90-98% (target ≥95%)
|
||
- Execution time: 2-4 hours
|
||
- Risk: Moderate (hybrid model requires testing)
|
||
|
||
---
|
||
|
||
## Execution Plan (If Gates Fail)
|
||
|
||
### Step 1: Immediate Analysis (15 min)
|
||
```sql
|
||
-- Query actual gate values from Phase 2
|
||
SELECT
|
||
metrics_json->>'ProbOfBacktestOverfit' as pbo,
|
||
metrics_json->>'DailySharePercentile' as dsr,
|
||
metrics_json->>'TotalReturn' as cost
|
||
FROM model_operations.shadow_run
|
||
WHERE run_id = 'e7239082-8234-45d7-8d74-22d4371cbe88'::uuid;
|
||
|
||
-- Identify which gate(s) failed
|
||
-- Prioritize remediation by impact
|
||
```
|
||
|
||
### Step 2: Root Cause Investigation (30 min)
|
||
```csharp
|
||
// Analyze Phase 1 trades
|
||
var trades = await GetPhase1Trades();
|
||
|
||
// Metrics:
|
||
// - Signal frequency (should be 25-50)
|
||
// - Average holding period (should be 5-20 days)
|
||
// - Win rate (should be 40-60%)
|
||
// - Largest drawdown (should be < 20%)
|
||
|
||
// Identify pattern: Too many signals? Too few? Low quality?
|
||
```
|
||
|
||
### Step 3: Select Remediation Option (30 min)
|
||
- If PBO > 20%: Choose Option 1.1, 1.2, or 1.3
|
||
- If DSR < 95%: Choose Option 2.1, 2.2, or 2.3
|
||
- If both fail: Implement Hybrid Model (Option 3)
|
||
|
||
### Step 4: Code Changes (60-120 min)
|
||
- Modify ReplayEngine.cs (signal generation logic)
|
||
- Modify dynamic position sizing (confidence multiplier)
|
||
- Add new indicators if needed (RSI, momentum)
|
||
- Update tests
|
||
|
||
### Step 5: Phase 1 Re-Run (15 min)
|
||
- Execute Phase 1 with new parameters
|
||
- Check metrics
|
||
|
||
### Step 6: Phase 2 Re-Judgment (5 min)
|
||
- Evaluate new gate values
|
||
- If PASS: Proceed to Phase 3
|
||
- If FAIL: Iterate (Option 4, 5, etc.)
|
||
|
||
---
|
||
|
||
## Timeline (If Remediation Needed)
|
||
|
||
```
|
||
T+0h Phase 2 judgment (gate failure detected)
|
||
↓ 15min
|
||
T+0.25h Root cause analysis
|
||
↓ 30min
|
||
T+0.75h Select remediation option
|
||
↓ 60-120min
|
||
T+2.0h Code changes complete
|
||
↓ 15min
|
||
T+2.25h Phase 1 re-run
|
||
↓ 5min
|
||
T+2.5h Phase 2 re-judgment
|
||
↓ (if PASS)
|
||
T+2.6h Phase 3 OOS validation (30-60 min)
|
||
|
||
Total: 3-5 hours (vs ~90 minutes if gates pass)
|
||
```
|
||
|
||
---
|
||
|
||
## Fallback Strategies (If Remediation Fails Twice)
|
||
|
||
### Fallback 1: Simplified EMA (Lower Expectations)
|
||
- Use wider EMA periods (20/50 instead of 12/26)
|
||
- Accept lower returns (5% instead of 8-15%)
|
||
- Trade-off: More stable, less overfit
|
||
|
||
### Fallback 2: Mean-Reversion Strategy
|
||
- Opposite of momentum (buy dips, sell bounces)
|
||
- Better DSR (daily income from reversions)
|
||
- Different risk profile
|
||
|
||
### Fallback 3: Reduce Model Ambition
|
||
- Accept 2-3% target return (very conservative)
|
||
- Gate thresholds: PBO < 30%, DSR < 85%
|
||
- Focus on reliability over performance
|
||
|
||
---
|
||
|
||
## Decision Matrix
|
||
|
||
| Gate Result | Action | Timeline | Confidence |
|
||
|-------------|--------|----------|------------|
|
||
| All PASS | Phase 3 OOS | 30-60 min | High |
|
||
| Gate1 FAIL | Option 1.x | 2-3 hours | High |
|
||
| Gate2 FAIL | Option 2.x | 1-2 hours | High |
|
||
| Both FAIL | Option 3 Hybrid | 3-4 hours | Medium |
|
||
| 3x FAIL | Fallback 1-3 | 4-8 hours | Low |
|
||
|
||
---
|
||
|
||
## Success Criteria (Remediation)
|
||
|
||
Re-optimized model must achieve:
|
||
- ✅ PBO ≤ 20% (backtesting robustness)
|
||
- ✅ DSR ≥ 95% (daily return quality)
|
||
- ✅ Cost > 0% (positive returns)
|
||
- ✅ Max Drawdown < 20% (risk control)
|
||
- ✅ Sharpe ≥ 1.0 (risk-adjusted performance)
|
||
|
||
If all criteria met → Proceed to Phase 3
|
||
|
||
---
|
||
|
||
## Communication Plan
|
||
|
||
**If gates fail:**
|
||
1. Document failure reason (PBO/DSR/both)
|
||
2. Communicate root cause analysis
|
||
3. Present selected remediation option
|
||
4. Execute changes
|
||
5. Re-run Phase 1-2
|
||
6. Report new results
|
||
|
||
**Target:** Restart Phase 3 within 3-5 hours of gate failure
|