KArtSell.Aegis/docs/PHASE4_MANUAL_ACTIVATION.md

Phase 4 - Manual Activation & Production Deployment

Status: ✅ DOCUMENTATION COMPLETE
Timeline: 1-2 weeks (after Phase 3)
Type: Manual process with maker-checker approval
Requirement: Phase 3 OOS validation complete + all criteria passed

Pre-Activation Checklist

Before manual activation, verify all items:

[✓] Phase 1 Complete: 252 trading days shadow run
[✓] Phase 2 PASS: All 3 gates passed
    ├─ Gate 1: PBO ≤ 20%
    ├─ Gate 2: DSR ≥ 95%
    └─ Gate 3: Cost > 0

[✓] Phase 3 Complete: OOS validation (252+ trading days)
    ├─ OOS Sharpe ≥ 1.0
    ├─ OOS Sharpe ≥ 80% of In-Sample
    ├─ Max Drawdown < 20%
    ├─ Calmar Ratio > 1.0
    └─ Walk-forward stable (no degradation)

[✓] Model Documentation:
    ├─ Strategy description (EMA 12/26 + dynamic sizing + fees)
    ├─ Parameter tuning rationale
    ├─ Risk assessment
    ├─ Known limitations
    └─ Monitoring plan

[✓] Maker-Checker Approval:
    ├─ Researcher: Confirms model quality + OOS results
    ├─ Risk Manager: Approves risk profile + limits
    ├─ Compliance: Confirms regulatory compliance
    └─ Operations: Confirms infrastructure readiness

[✓] Infrastructure Ready:
    ├─ Production database configured
    ├─ API endpoints tested
    ├─ Monitoring dashboards built
    ├─ Alerting configured
    └─ Rollback procedure tested

[✓] Approval Sign-off:
    ├─ Trader approval: Confirms trading strategy
    ├─ CRO approval: Risk limit compliance
    └─ CEO sign-off: Final authorization

Deployment Procedure

Phase 4A: Staging Validation (Day 1-2)

Step 1: Model Registry Update

INSERT INTO model_registry (model_id, version, status, effective_date)
VALUES ('00000000-0000-0000-0000-000000000004', 'v1.0-prod', 'DEPLOYED', NOW());

UPDATE model_config
SET is_production = true, activation_timestamp = NOW()
WHERE model_id = '00000000-0000-0000-0000-000000000004';

Step 2: Staging Deployment

# 1. Deploy to staging environment
docker pull kartsell-model:v1.0-prod
docker run -e ENV=staging kartsell-model:v1.0-prod

# 2. Run smoke tests
./tests/staging/smoke_tests.sh

# 3. Verify API responses
curl http://staging-api:5002/api/models/predict -X POST -d '{...}'

# 4. Monitor for 24 hours
# Expected: Zero errors, normal latency

Step 3: Staging Monitoring (24 hours)

Metrics to watch:
  - API latency: < 500ms (p95)
  - Error rate: < 0.1%
  - Model coverage: 100% (all securities)
  - Prediction variance: Normal
  - Database query time: < 200ms

Alert if:
  - Error rate > 1%
  - Latency > 1000ms (p95)
  - Model timeout
  - Database connection issues

Phase 4B: Progressive Production Rollout (Day 3-5)

Step 4: Production Deployment (Canary)

# 1. Deploy to production with traffic split
# Initial: 1% of traffic → model prediction
#          99% of traffic → fallback to previous model

kubectl set image deployment/model-inference \
  model=kartsell-model:v1.0-prod

# 2. Monitor canary metrics
curl http://prod-api:5002/api/metrics/canary

# Expected after 1 hour:
#   - 1% traffic serving model v1.0
#   - Zero errors in model predictions
#   - Latency within SLA
#   - Prediction quality baseline established

Step 5: Incremental Traffic Shift

T+1h:  1% traffic to model
  └─ Monitor: 0 errors, latency OK
  
T+6h:  10% traffic to model
  └─ Monitor: Prediction quality, business metrics
  
T+24h: 50% traffic to model
  └─ Monitor: Full 24-hour cycle, overnight behavior
  
T+48h: 100% traffic to model
  └─ FULL PRODUCTION LIVE

Monitoring During Rollout:

-- Real-time prediction quality
SELECT 
  model_version,
  COUNT(*) as predictions,
  AVG(prediction_latency_ms) as avg_latency,
  PERCENTILE(prediction_latency_ms, 0.95) as p95_latency,
  CASE WHEN error = true THEN 1 ELSE 0 END as error_count
FROM model_predictions
WHERE created_at > NOW() - INTERVAL '1 hour'
GROUP BY model_version;

-- Business impact (vs. previous model)
SELECT 
  model_version,
  AVG(portfolio_return_pct) as avg_return,
  STDDEV(portfolio_return_pct) as volatility,
  COUNT(DISTINCT trading_date) as trading_days
FROM trading_results
WHERE created_at > NOW() - INTERVAL '24 hours'
GROUP BY model_version;

Phase 4C: Production Stabilization (Day 6+)

Step 6: Full Production Monitoring

Daily Metrics:
  - API P95 latency
  - Error rate
  - Prediction coverage
  - Model performance (Sharpe, return, drawdown)
  - Resource usage (CPU, memory, database)

Weekly Review:
  - Compare actual performance to historical
  - Check for anomalies or drift
  - Review logs for edge cases
  - Plan monitoring improvements

Monthly Review:
  - Full performance assessment
  - Comparison to OOS benchmarks
  - Consider retraining if drift detected
  - Update runbooks based on findings

Rollback Procedure (Emergency Only)

Trigger Conditions:

• Error rate > 5% for > 10 minutes
• API P95 latency > 2 seconds
• Model predictions making significant losses
• Regulatory or compliance violation

Rollback Steps (< 2 minutes):

# 1. Immediate traffic switch back to previous model
kubectl set image deployment/model-inference \
  model=kartsell-model:v0.9-prod

# 2. Verify traffic switch
curl http://prod-api:5002/api/metrics/version
# Expected: 0% on v1.0-prod, 100% on v0.9-prod

# 3. Monitor for stability (5 minutes)
# Expected: Error rate drops below 0.1%

# 4. Post-incident review (within 24 hours)
#    - Root cause analysis
#    - Code review of v1.0-prod
#    - Fix + retest before next attempt

Post-Deployment (Week 2+)

Week 1-2:

• Daily performance monitoring
• Collect feedback from traders
• Document edge cases
• Plan improvements

Month 1:

• Full quarterly retraining evaluation
• Consider model evolution (v1.1)
• Document lessons learned
• Update risk limits if needed

Ongoing:

• Monthly performance reviews
• Quarterly retraining cycle
• Annual comprehensive audit
• Plan Phase 2 improvements (if market conditions change)

Success Criteria

Model is considered successfully deployed when:

• ✅ Production error rate < 0.1%
• ✅ API latency within SLA (P95 < 500ms)
• ✅ Sharpe ratio ≥ OOS baseline (within 10%)
• ✅ No regulatory violations or compliance issues
• ✅ Trading team confirms smooth operations
• ✅ One week of stable performance data collected

Failure Exit Plan

If Phase 4 experiences critical failure:

1. Rollback to previous version (< 2 minutes)
2. Disable automated trading for this model
3. Root cause analysis (24 hours)
4. Fix issues + retest (48 hours)
5. Second activation attempt (Day 10+)

If Phase 4 fails twice: Escalate to senior leadership for decision on model retirement vs. major redesign.