KArtSell.Aegis/docs/PHASE4_MANUAL_ACTIVATION.md

# Phase 4 - Manual Activation & Production Deployment

**Status:** ✅ DOCUMENTATION COMPLETE  
**Timeline:** 1-2 weeks (after Phase 3)  
**Type:** Manual process with maker-checker approval  
**Requirement:** Phase 3 OOS validation complete + all criteria passed

## Pre-Activation Checklist

Before manual activation, verify all items:

```
[✓] Phase 1 Complete: 252 trading days shadow run
[✓] Phase 2 PASS: All 3 gates passed
    ├─ Gate 1: PBO ≤ 20%
    ├─ Gate 2: DSR ≥ 95%
    └─ Gate 3: Cost > 0

[✓] Phase 3 Complete: OOS validation (252+ trading days)
    ├─ OOS Sharpe ≥ 1.0
    ├─ OOS Sharpe ≥ 80% of In-Sample
    ├─ Max Drawdown < 20%
    ├─ Calmar Ratio > 1.0
    └─ Walk-forward stable (no degradation)

[✓] Model Documentation:
    ├─ Strategy description (EMA 12/26 + dynamic sizing + fees)
    ├─ Parameter tuning rationale
    ├─ Risk assessment
    ├─ Known limitations
    └─ Monitoring plan

[✓] Maker-Checker Approval:
    ├─ Researcher: Confirms model quality + OOS results
    ├─ Risk Manager: Approves risk profile + limits
    ├─ Compliance: Confirms regulatory compliance
    └─ Operations: Confirms infrastructure readiness

[✓] Infrastructure Ready:
    ├─ Production database configured
    ├─ API endpoints tested
    ├─ Monitoring dashboards built
    ├─ Alerting configured
    └─ Rollback procedure tested

[✓] Approval Sign-off:
    ├─ Trader approval: Confirms trading strategy
    ├─ CRO approval: Risk limit compliance
    └─ CEO sign-off: Final authorization
```

## Deployment Procedure

### Phase 4A: Staging Validation (Day 1-2)

**Step 1: Model Registry Update**
```sql
INSERT INTO model_registry (model_id, version, status, effective_date)
VALUES ('00000000-0000-0000-0000-000000000004', 'v1.0-prod', 'DEPLOYED', NOW());

UPDATE model_config
SET is_production = true, activation_timestamp = NOW()
WHERE model_id = '00000000-0000-0000-0000-000000000004';
```

**Step 2: Staging Deployment**
```bash
# 1. Deploy to staging environment
docker pull kartsell-model:v1.0-prod
docker run -e ENV=staging kartsell-model:v1.0-prod

# 2. Run smoke tests
./tests/staging/smoke_tests.sh

# 3. Verify API responses
curl http://staging-api:5002/api/models/predict -X POST -d '{...}'

# 4. Monitor for 24 hours
# Expected: Zero errors, normal latency
```

**Step 3: Staging Monitoring (24 hours)**
```
Metrics to watch:
  - API latency: < 500ms (p95)
  - Error rate: < 0.1%
  - Model coverage: 100% (all securities)
  - Prediction variance: Normal
  - Database query time: < 200ms

Alert if:
  - Error rate > 1%
  - Latency > 1000ms (p95)
  - Model timeout
  - Database connection issues
```

### Phase 4B: Progressive Production Rollout (Day 3-5)

**Step 4: Production Deployment (Canary)**
```bash
# 1. Deploy to production with traffic split
# Initial: 1% of traffic → model prediction
#          99% of traffic → fallback to previous model

kubectl set image deployment/model-inference \
  model=kartsell-model:v1.0-prod

# 2. Monitor canary metrics
curl http://prod-api:5002/api/metrics/canary

# Expected after 1 hour:
#   - 1% traffic serving model v1.0
#   - Zero errors in model predictions
#   - Latency within SLA
#   - Prediction quality baseline established
```

**Step 5: Incremental Traffic Shift**
```
T+1h:  1% traffic to model
  └─ Monitor: 0 errors, latency OK
  
T+6h:  10% traffic to model
  └─ Monitor: Prediction quality, business metrics
  
T+24h: 50% traffic to model
  └─ Monitor: Full 24-hour cycle, overnight behavior
  
T+48h: 100% traffic to model
  └─ FULL PRODUCTION LIVE
```

**Monitoring During Rollout:**
```sql
-- Real-time prediction quality
SELECT 
  model_version,
  COUNT(*) as predictions,
  AVG(prediction_latency_ms) as avg_latency,
  PERCENTILE(prediction_latency_ms, 0.95) as p95_latency,
  CASE WHEN error = true THEN 1 ELSE 0 END as error_count
FROM model_predictions
WHERE created_at > NOW() - INTERVAL '1 hour'
GROUP BY model_version;

-- Business impact (vs. previous model)
SELECT 
  model_version,
  AVG(portfolio_return_pct) as avg_return,
  STDDEV(portfolio_return_pct) as volatility,
  COUNT(DISTINCT trading_date) as trading_days
FROM trading_results
WHERE created_at > NOW() - INTERVAL '24 hours'
GROUP BY model_version;
```

### Phase 4C: Production Stabilization (Day 6+)

**Step 6: Full Production Monitoring**
```
Daily Metrics:
  - API P95 latency
  - Error rate
  - Prediction coverage
  - Model performance (Sharpe, return, drawdown)
  - Resource usage (CPU, memory, database)

Weekly Review:
  - Compare actual performance to historical
  - Check for anomalies or drift
  - Review logs for edge cases
  - Plan monitoring improvements

Monthly Review:
  - Full performance assessment
  - Comparison to OOS benchmarks
  - Consider retraining if drift detected
  - Update runbooks based on findings
```

## Rollback Procedure (Emergency Only)

**Trigger Conditions:**
- Error rate > 5% for > 10 minutes
- API P95 latency > 2 seconds
- Model predictions making significant losses
- Regulatory or compliance violation

**Rollback Steps (< 2 minutes):**
```bash
# 1. Immediate traffic switch back to previous model
kubectl set image deployment/model-inference \
  model=kartsell-model:v0.9-prod

# 2. Verify traffic switch
curl http://prod-api:5002/api/metrics/version
# Expected: 0% on v1.0-prod, 100% on v0.9-prod

# 3. Monitor for stability (5 minutes)
# Expected: Error rate drops below 0.1%

# 4. Post-incident review (within 24 hours)
#    - Root cause analysis
#    - Code review of v1.0-prod
#    - Fix + retest before next attempt
```

## Post-Deployment (Week 2+)

**Week 1-2:**
- Daily performance monitoring
- Collect feedback from traders
- Document edge cases
- Plan improvements

**Month 1:**
- Full quarterly retraining evaluation
- Consider model evolution (v1.1)
- Document lessons learned
- Update risk limits if needed

**Ongoing:**
- Monthly performance reviews
- Quarterly retraining cycle
- Annual comprehensive audit
- Plan Phase 2 improvements (if market conditions change)

## Success Criteria

Model is considered successfully deployed when:
- ✅ Production error rate < 0.1%
- ✅ API latency within SLA (P95 < 500ms)
- ✅ Sharpe ratio ≥ OOS baseline (within 10%)
- ✅ No regulatory violations or compliance issues
- ✅ Trading team confirms smooth operations
- ✅ One week of stable performance data collected

## Failure Exit Plan

If Phase 4 experiences critical failure:
1. Rollback to previous version (< 2 minutes)
2. Disable automated trading for this model
3. Root cause analysis (24 hours)
4. Fix issues + retest (48 hours)
5. Second activation attempt (Day 10+)

If Phase 4 fails twice: Escalate to senior leadership for decision on model retirement vs. major redesign.