# Gate 3 Execution Guide: 252-Day Shadow Run Validation **Purpose:** Complete end-to-end validation of model against 252+ trading-day historical window **Status:** Ready for execution (Gates 1-2-4-5 infrastructure complete) **Effort:** 30-60 minutes (depending on market data availability) **Success Criteria:** - PBO (Probability of Backtest Overfit) ≤ 20% ✓ - DSR (Daily Sharpe Ratio) ≥ 95th percentile ✓ - Cost 2x positive (returns survive doubled fees) ✓ - Phase analysis metrics (Bull/Bear/Sideways) ≠ 0 ✓ - All metrics logged with CorrelationId ✓ --- ## Prerequisites ### 1. Infrastructure Setup **SSH Port Forwarding (PostgreSQL):** ```bash ssh -L 5432:127.0.0.1:5432 kjh2064@178.104.200.7 # Keep this tunnel open during execution ``` **Environment Variables:** ```bash # PowerShell $env:KARTSELL_POSTGRES="Host=localhost;Port=5432;Database=kartsell;Username=kartsell;Password=kartsell" $env:KRX_API_KEY="" # Bash export KARTSELL_POSTGRES="Host=localhost;Port=5432;Database=kartsell;Username=kartsell;Password=kartsell" export KRX_API_KEY="" ``` **KArtSell.Host Startup:** ```bash cd D:\JobRoomz\KArtSell.Aegis dotnet run --project src/KArtSell.Host -c Release # API should be available at http://localhost:5000 ``` **Hangfire Dashboard:** - Monitor job execution at http://localhost:5000/hangfire - Queue: `q-research` (long-running shadow runs) - Max execution time: 3600 seconds (1 hour) --- ## 2. Model Setup **Option A: Use Existing Test Model** ```sql -- Query to find available models in database SELECT id, name, status FROM model_operations.model WHERE status IN ('Active', 'Validated') LIMIT 5; ``` **Option B: Create Test Model** (if none exist) ```sql INSERT INTO model_operations.model ( id, name, strategy_description, risk_factors, created_at, status ) VALUES ( 'a1b2c3d4-e5f6-7890-abcd-ef1234567890'::uuid, 'Test Model 2024', 'Simple momentum strategy for validation', 'Market regime dependency, data quality', NOW(), 'Active' ); ``` --- ## 3. Shadow Run Execution ### Initiate Shadow Run via API **Endpoint:** `POST /api/shadow-runs` **Authentication:** Bearer token (Admin or Researcher role) **Request Body:** ```json { "modelId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", "windowStart": "2024-01-02", "windowEnd": "2024-08-31", "phaseFilter": "All" } ``` **Using curl:** ```bash curl -X POST http://localhost:5000/api/shadow-runs \ -H "Content-Type: application/json" \ -H "Authorization: Bearer " \ -d '{ "modelId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", "windowStart": "2024-01-02", "windowEnd": "2024-08-31", "phaseFilter": "All" }' ``` **Expected Response (202 Accepted):** ```json { "runId": "b2c3d4e5-f6a7-8901-bcde-f12345678901", "modelId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", "status": "Queued", "jobId": "12345", "pollingUrl": "/api/shadow-runs/b2c3d4e5-f6a7-8901-bcde-f12345678901" } ``` **Save the `runId`** — You'll use this to poll results. --- ## 4. Monitor Execution ### Via Hangfire Dashboard - Go to http://localhost:5000/hangfire - Watch for `ShadowRunJob` in `q-research` queue - Stages: Enqueued → Processing → Succeeded/Failed ### Via Polling Endpoint **Endpoint:** `GET /api/shadow-runs/{runId}` ```bash curl -X GET http://localhost:5000/api/shadow-runs/b2c3d4e5-f6a7-8901-bcde-f12345678901 \ -H "Authorization: Bearer " ``` **Poll every 30 seconds** until status changes from `Pending` to `EvaluationComplete` or `Failed`. **Response while running:** ```json { "runId": "b2c3d4e5-f6a7-8901-bcde-f12345678901", "status": "Replay", "message": "Replaying model signals..." } ``` **Response when complete:** ```json { "runId": "b2c3d4e5-f6a7-8901-bcde-f12345678901", "status": "EvaluationComplete", "validationGatesJson": { "pbo": 0.15, "pbo_under_20": true, "dsr": 0.96, "dsr_above_95": true, "cost_2x_positive": true, "all_gates_passed": true, "sharpe": 1.45, "calmar": 0.82, "max_drawdown": 0.18, "returns": 0.28 }, "metricsJson": { "bull": { "sharpe": 1.8, "return": 0.35 }, "bear": { "sharpe": 0.9, "return": 0.15 }, "sideways": { "sharpe": 1.2, "return": 0.22 } }, "approvalQueueId": "c3d4e5f6-a7b8-9012-cdef-123456789012" } ``` --- ## 5. Validate Results ### Gate 5 Success Criteria | Criterion | Expected | Actual | Status | |-----------|----------|--------|--------| | **PBO ≤ 20%** | 0.20 | — | ⏳ | | **DSR ≥ 95th** | 0.95 | — | ⏳ | | **Cost 2x positive** | true | — | ⏳ | | **Phase metrics ≠ 0** | true | — | ⏳ | | **Audit logged** | CorrelationId | — | ⏳ | ### Verify in Database ```sql -- Check shadow_run results SELECT run_id, model_id, status, validation_gates_json -> 'all_gates_passed' as all_gates_passed, validation_gates_json -> 'pbo' as pbo, validation_gates_json -> 'dsr' as dsr, published_at FROM model_operations.shadow_run WHERE status = 'EvaluationComplete' ORDER BY published_at DESC LIMIT 1; -- Check approval queue auto-population SELECT id, run_id, status, requested_at FROM model_operations.approval_queue WHERE run_id = 'b2c3d4e5-f6a7-8901-bcde-f12345678901'; -- Verify outbox events SELECT COUNT(*) as event_count, COUNT(DISTINCT consumer) as consumers FROM outbox.inbox WHERE created_at >= NOW() - INTERVAL '1 hour'; ``` --- ## 6. Handle Failures ### Transient Failures (Retry) - Network timeout: Automatic retry (Hangfire) - KRX API 429 (rate limit): Exponential backoff - Database connection drop: Retry on reconnect ### Permanent Failures (Log & Alert) - Invalid model ID: Check model exists and is active - Missing market data: Verify KRX API key and data availability - Calculation error: Check logs for math domain errors (NaN, inf) **Check logs:** ```bash # Tail application logs dotnet logs KArtSell.Host | grep -i "shadow\|error" # Or in Hangfire dashboard: Failed Jobs tab ``` --- ## 7. Post-Execution ### Collect Evidence 1. **Shadow Run Metrics** — validation_gates_json (already in DB) 2. **Approval Queue** — Status = "Pending" awaiting maker-checker 3. **Audit Trail** — CorrelationId in all logs/events 4. **Outbox/Inbox** — Verify event processing completeness ### Decision Gate - ✅ **All gates passed?** → Proceed to approval workflow - ❌ **Gates failed?** → Root cause analysis, fix, re-run ### Approval Workflow (Gate 4 - Already Implemented) Once shadow run succeeds: ```bash # Get pending approval curl -X GET http://localhost:5000/api/v1/approval-queue \ -H "Authorization: Bearer " # Maker-checker approval (Risk officer) curl -X POST http://localhost:5000/api/v1/approval-queue/{id}/approve \ -H "Authorization: Bearer " \ -d '{ "approvalReason": "All validation gates passed. PBO=0.15, DSR=0.96. Approved for activation." }' ``` --- ## Timeline Expectations | Phase | Duration | Notes | |-------|----------|-------| | **DataBackfill** | 5-10 min | Fetch OHLCV, fees, calendar | | **Replay** | 10-20 min | Simulate signals & orders | | **Evaluation** | 5-10 min | Calculate metrics, gates | | **Phase Segmentation** | 2-5 min | Bull/Bear/Sideways analysis | | **Persist & Emit** | 1-2 min | Write to DB, emit events | | **Total** | 30-60 min | Depends on market data lag | --- ## Troubleshooting **Problem: Job stuck in "Processing"** - Check Hangfire logs for errors - Verify PostgreSQL connection - Restart job if stuck > 1 hour **Problem: "Model not found"** - Verify ModelId exists in database - Use query from section 2 (Model Setup) **Problem: "No market data available"** - Check KRX API credentials - Verify date range is covered by KRX - Use stub data for testing (set in KrxDataService) **Problem: "PBO > 20% or DSR < 95%"** - Model not robust in 252-day window - Consider strategy adjustments - Re-run with different date range - Log as evidence for risk review --- ## Success Confirmation **Gate 3 is PASSED when:** - ✅ Shadow run completes with status = "EvaluationComplete" - ✅ validation_gates_json.all_gates_passed = true - ✅ Approval queue auto-populated with status = "Pending" - ✅ CorrelationId present in all audit logs - ✅ Events flow through Outbox → Inbox → Consumers **Next Step:** Gate 4 (Approval Workflow) — Already implemented, awaiting results