diff --git a/PRODUCTION_READINESS.md b/PRODUCTION_READINESS.md new file mode 100644 index 00000000..7b2c0868 --- /dev/null +++ b/PRODUCTION_READINESS.md @@ -0,0 +1,201 @@ +# Production Readiness Checklist + +**K-ArtSell Aegis v16.0** — Shadow Run Validation System + +**Status:** `IMPLEMENTATION_COMPLETE / READY_FOR_252DAY_REHEARSAL` + +--- + +## ✅ Completed (Pre-Merge) + +### Architecture & Code Quality +- [x] AGENTS.md v16.0 compliance verified (all 13 decision criteria) +- [x] Vertical Slice pattern: Complete endpoint-to-database features +- [x] Module isolation: Cross-module coupling via Outbox/Inbox pattern only +- [x] Async coupling: ShadowRunJob → IOutboxWriter → OutboxPollerJob → DownstreamConsumerJob +- [x] Zero new technical debt (all deferred work documented) +- [x] Code analysis: CA1822, CA1873 rules suppressed per CLAUDE.md + +### Testing +- [x] Unit tests: 17/17 ModelOperations ✓ +- [x] Unit tests: 18/18 SignalEngine ✓ +- [x] Architecture tests: 5/5 ✓ +- [x] Integration tests: 47/47 (including 3 E2E pipeline tests) ✓ +- [x] **Total: 87/87 tests passing (0 regressions)** + +### Database +- [x] Migrations: 0008_CreateShadowRunTable, 0009_CreateInboxTable, 0010_CreateApprovalQueueTable +- [x] Schema: JSONB payloads, PIT queries (published_at ≤ cutoff), immutability triggers +- [x] Idempotency: UNIQUE constraints (outbox_message, approval_queue), dedup by message_id +- [x] Constraints: Status transitions enforced (Pending → Processed/Failed, Approved → timestamp) + +### Features Implemented +1. **Shadow Run Validation** (252+ days) + - Phase 1: DataBackfill (OHLCV, fees, calendar) + - Phase 2: Replay (signals → orders → fills) + - Phase 3: Metrics (Sharpe, PBO, DSR, Calmar, Max DD) + - Phase 4: Phase Segmentation (Bull/Bear/Sideways/HighVolatility per-phase metrics) + - Phase 5: Persist (shadow_run table, JSONB analysis) + - Phase 6: Emit (IOutboxWriter → building_blocks.outbox_message) + +2. **Async Event Pipeline** (Real-time notifications) + - OutboxPollerJob: outbox_message → inbox_message (delivery marker) + - DownstreamConsumerJob: inbox_message → fetch payload → route to consumers + - Consumers: SignalR (push), ApprovalQueue (gate-conditional), AuditLog (compliance) + +3. **Market Data Integration** + - KRX OpenAPI: Real price data (fallback to stub for local dev) + - Retry logic: Transient (429, 503, 408) vs Permanent (400, 404) + - Cache: 24 hours per (ticker, date) + +4. **Approval Workflow** + - approval_queue table: Pending → Approved/Rejected workflow + - Constraints: approved_by, approval_reason, rejection_reason validation + - Audit: requested_at, approved_at, rejected_at timestamps + +--- + +## ⏳ Pending (Pre-Production) + +### Validation Gates (CLAUDE.md: "Not Yet Passed") + +#### 1. **PostgreSQL DbUp Fresh/Upgrade/Re-run/Failure-Recovery Tests** (REQUIRED) +- [ ] Fresh install: DbUp executes 0008, 0009, 0010 in order +- [ ] Upgrade from prior version: No data loss, schema migrations idempotent +- [ ] Re-run: Migrations safe to re-execute (checksums match) +- [ ] Failure recovery: If migration fails, retry doesn't corrupt state +- **Action:** Add migration test suite (Jest/Xunit) to CI/CD + +#### 2. **Outbox/Inbox Crash-Recovery & Audit Reconciliation** (REQUIRED) +- [ ] Outbox crash: Messages survive process restart, replay-safe +- [ ] Inbox processing: Consumer failures → retry on restart (transient vs permanent) +- [ ] Dedup: Duplicate events filtered (UNIQUE constraints hold) +- [ ] Reconciliation: Evidence of all events processed (audit log trace) +- **Action:** Add integration test for job restart scenarios + +#### 3. **252+ Trading-Day Shadow Run Execution** (REQUIRED) +- [ ] End-to-end execution with real market data (2024-2025) +- [ ] PBO (Probability of Backtest Overfit) ≤ 20% validation gate passes +- [ ] DSR (Daily Sharpe Ratio) ≥ 95th percentile validation gate passes +- [ ] Cost 2x positive: Return survives doubled fees +- [ ] Phase analysis: Bull/Bear/Sideways metrics non-zero +- [ ] Audit trail: Every event logged with CorrelationId +- **Action:** Execute shadow run via ShadowRunJob with real KRX data + +#### 4. **Manual Activation Workflow** (REQUIRED) +- [ ] Model Card review: Strategy description, risk factors, assumptions +- [ ] Maker-checker approval: Two-person sign-off before live trading +- [ ] Effective date: model_operations.model_activation.effective_at set +- [ ] Rollback plan: Documented de-activation procedure +- **Action:** Implement ModelActivationEndpoint + maker-checker UI + +#### 5. **Observability & Alerting** (REQUIRED) +- [ ] Batch SLA dashboard: Job completion times, queue depths +- [ ] Data quality quarantine: Monitor jobs marked `dq` (data quality issue) +- [ ] Duplicate detection: Alert if outbox dedup constraint violated +- [ ] Reconciliation breaks: Evidence vs current state mismatch +- [ ] Model drift: OOS performance tracking vs baseline +- **Action:** Wire Serilog → observability backend (Seq, Grafana, etc.) + +--- + +## 🚀 Pre-Production Deployment Steps + +### 1. Database Preparation +```bash +# Apply migrations (DbUp handles versioning) +dotnet run --project src/KArtSell.DbMigrator -c Release + +# Verify schema +psql -h 178.104.200.7 -U kartsell -d kartsell -c "\dt model_operations.*" +``` + +### 2. Shadow Run Rehearsal +```bash +# Via HTTP endpoint +POST /api/shadow-run/initiate +{ + "modelId": "{uuid}", + "windowStartDate": "2024-01-02", + "windowEndDate": "2024-08-31" +} + +# Monitor Hangfire dashboard +# → ShadowRunJob should complete in ~30 minutes (q-research queue) +# → Check: outbox_message, inbox_message, approval_queue populated +``` + +### 3. Validation Evidence Collection +- [ ] PBO evidence: Stored in shadow_run.validation_gates_json +- [ ] DSR evidence: Daily Sharpe percentile ≥ 0.95 +- [ ] Cost analysis: 2x fee impact documented +- [ ] Phase breakdown: Bull/Bear/Sideways metrics non-zero +- [ ] Audit log: All completions (PASS/FAIL) logged + +### 4. Approval Workflow Execution +```bash +# GET /api/approval-queue (list pending) +# POST /api/approval/{id}/approve (maker-checker sign-off) +# Verify: approved_at, approved_by populated +``` + +--- + +## 📋 Risk Mitigation + +| Risk | Mitigation | Status | +|------|-----------|--------| +| **No real data** | Use KRX OpenAPI (fallback stub available) | ✅ Code ready | +| **Migration failure** | IdUp checksums + rollback procedure | ✅ Designed | +| **Consumer crash** | Transient retry + idempotency dedup | ✅ Implemented | +| **Model drift** | OOS monitoring dashboard + alert | ⏳ Needs wiring | +| **Concurrent access** | DisableConcurrentExecution (60min max) | ✅ Configured | +| **Data loss** | JSONB immutability + audit triggers | ✅ Enforced | + +--- + +## 🎯 Success Criteria (Pre-Go-Live) + +### Functional +- [ ] Shadow run completes in < 30 minutes (with real KRX data) +- [ ] All 4 validation gates produce numeric results (no NaN, null) +- [ ] Async events flow: Outbox → Inbox → Consumer (verifiable via logs) +- [ ] Approval queue auto-populated on gate passage +- [ ] Audit log entry created for every completion (PASS/FAIL) + +### Non-Functional +- [ ] Zero test regressions (87/87 passing) +- [ ] Query response time: shadow_run SELECT < 100ms +- [ ] Job concurrency: Single execution held for 60 minutes max +- [ ] Memory usage: < 500MB per job run +- [ ] Log compression: Rotate after 10GB per day + +### Security +- [ ] No SELECT * (schema-qualified, explicit columns) +- [ ] No direct module-to-module table access (IOutboxWriter/IInboxStore only) +- [ ] No sensitive data logged (API keys, PII redacted) +- [ ] Correlation IDs present in all audit records + +--- + +## 📞 Escalation + +**If any validation gate fails:** +1. Capture evidence (logs, metrics, database state) +2. File issue with decision point (e.g., "PBO > 20%, impact assessment needed") +3. Root cause analysis: Code vs data vs external API +4. Resolution: Fix + re-run shadow run OR defer with documented exception + +**Owner:** ModelOperations team +**Stakeholders:** Risk, Trading, Compliance + +--- + +**Next Actions:** +1. Execute 252+ trading-day shadow run (this week) +2. Collect PBO/DSR evidence (evidence_table.md) +3. Activate maker-checker workflow approval +4. Go-live authorization + +**Timeline:** ≤ 2 weeks to production +**Status:** `READY_FOR_REHEARSAL`