KArtSell.Aegis/GATE_3_EXECUTION_GUIDE.md

Gate 3 Execution Guide: 252-Day Shadow Run Validation

Purpose: Complete end-to-end validation of model against 252+ trading-day historical window
Status: Ready for execution (Gates 1-2-4-5 infrastructure complete)
Effort: 30-60 minutes (depending on market data availability)
Success Criteria:

• PBO (Probability of Backtest Overfit) ≤ 20% ✓
• DSR (Daily Sharpe Ratio) ≥ 95th percentile ✓
• Cost 2x positive (returns survive doubled fees) ✓
• Phase analysis metrics (Bull/Bear/Sideways) ≠ 0 ✓
• All metrics logged with CorrelationId ✓

---

Prerequisites

1. Infrastructure Setup

SSH Port Forwarding (PostgreSQL):

ssh -L 5432:127.0.0.1:5432 kjh2064@178.104.200.7
# Keep this tunnel open during execution

Environment Variables:

# PowerShell
$env:KARTSELL_POSTGRES="Host=localhost;Port=5432;Database=kartsell;Username=kartsell;Password=kartsell"
$env:KRX_API_KEY="<real-krx-api-key-from-gitea-secrets>"

# Bash
export KARTSELL_POSTGRES="Host=localhost;Port=5432;Database=kartsell;Username=kartsell;Password=kartsell"
export KRX_API_KEY="<real-krx-api-key-from-gitea-secrets>"

KArtSell.Host Startup:

cd D:\JobRoomz\KArtSell.Aegis
dotnet run --project src/KArtSell.Host -c Release
# API should be available at http://localhost:5000

Hangfire Dashboard:

• Monitor job execution at http://localhost:5000/hangfire
• Queue: q-research (long-running shadow runs)
• Max execution time: 3600 seconds (1 hour)

---

2. Model Setup

Option A: Use Existing Test Model

-- Query to find available models in database
SELECT id, name, status FROM model_operations.model 
WHERE status IN ('Active', 'Validated')
LIMIT 5;

Option B: Create Test Model (if none exist)

INSERT INTO model_operations.model (
  id, name, strategy_description, risk_factors, 
  created_at, status
) VALUES (
  'a1b2c3d4-e5f6-7890-abcd-ef1234567890'::uuid,
  'Test Model 2024',
  'Simple momentum strategy for validation',
  'Market regime dependency, data quality',
  NOW(),
  'Active'
);

---

3. Shadow Run Execution

Initiate Shadow Run via API

Endpoint: POST /api/shadow-runs
Authentication: Bearer token (Admin or Researcher role)
Request Body:

{
  "modelId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "windowStart": "2024-01-02",
  "windowEnd": "2024-08-31",
  "phaseFilter": "All"
}

Using curl:

curl -X POST http://localhost:5000/api/shadow-runs \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <your-jwt-token>" \
  -d '{
    "modelId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "windowStart": "2024-01-02",
    "windowEnd": "2024-08-31",
    "phaseFilter": "All"
  }'

Expected Response (202 Accepted):

{
  "runId": "b2c3d4e5-f6a7-8901-bcde-f12345678901",
  "modelId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "Queued",
  "jobId": "12345",
  "pollingUrl": "/api/shadow-runs/b2c3d4e5-f6a7-8901-bcde-f12345678901"
}

Save the runId — You'll use this to poll results.

---

4. Monitor Execution

Via Hangfire Dashboard

• Go to http://localhost:5000/hangfire
• Watch for ShadowRunJob in q-research queue
• Stages: Enqueued → Processing → Succeeded/Failed

Via Polling Endpoint

Endpoint: GET /api/shadow-runs/{runId}

curl -X GET http://localhost:5000/api/shadow-runs/b2c3d4e5-f6a7-8901-bcde-f12345678901 \
  -H "Authorization: Bearer <your-jwt-token>"

Poll every 30 seconds until status changes from Pending to EvaluationComplete or Failed.

Response while running:

{
  "runId": "b2c3d4e5-f6a7-8901-bcde-f12345678901",
  "status": "Replay",
  "message": "Replaying model signals..."
}

Response when complete:

{
  "runId": "b2c3d4e5-f6a7-8901-bcde-f12345678901",
  "status": "EvaluationComplete",
  "validationGatesJson": {
    "pbo": 0.15,
    "pbo_under_20": true,
    "dsr": 0.96,
    "dsr_above_95": true,
    "cost_2x_positive": true,
    "all_gates_passed": true,
    "sharpe": 1.45,
    "calmar": 0.82,
    "max_drawdown": 0.18,
    "returns": 0.28
  },
  "metricsJson": {
    "bull": { "sharpe": 1.8, "return": 0.35 },
    "bear": { "sharpe": 0.9, "return": 0.15 },
    "sideways": { "sharpe": 1.2, "return": 0.22 }
  },
  "approvalQueueId": "c3d4e5f6-a7b8-9012-cdef-123456789012"
}

---

5. Validate Results

Gate 5 Success Criteria

Criterion	Expected	Actual	Status
PBO ≤ 20%	0.20	—	⏳
DSR ≥ 95th	0.95	—	⏳
Cost 2x positive	true	—	⏳
Phase metrics ≠ 0	true	—	⏳
Audit logged	CorrelationId	—	⏳

Verify in Database

-- Check shadow_run results
SELECT 
  run_id,
  model_id,
  status,
  validation_gates_json -> 'all_gates_passed' as all_gates_passed,
  validation_gates_json -> 'pbo' as pbo,
  validation_gates_json -> 'dsr' as dsr,
  published_at
FROM model_operations.shadow_run
WHERE status = 'EvaluationComplete'
ORDER BY published_at DESC
LIMIT 1;

-- Check approval queue auto-population
SELECT 
  id,
  run_id,
  status,
  requested_at
FROM model_operations.approval_queue
WHERE run_id = 'b2c3d4e5-f6a7-8901-bcde-f12345678901';

-- Verify outbox events
SELECT 
  COUNT(*) as event_count,
  COUNT(DISTINCT consumer) as consumers
FROM outbox.inbox
WHERE created_at >= NOW() - INTERVAL '1 hour';

---

6. Handle Failures

Transient Failures (Retry)

• Network timeout: Automatic retry (Hangfire)
• KRX API 429 (rate limit): Exponential backoff
• Database connection drop: Retry on reconnect

Permanent Failures (Log & Alert)

• Invalid model ID: Check model exists and is active
• Missing market data: Verify KRX API key and data availability
• Calculation error: Check logs for math domain errors (NaN, inf)

Check logs:

# Tail application logs
dotnet logs KArtSell.Host | grep -i "shadow\|error"

# Or in Hangfire dashboard: Failed Jobs tab

---

7. Post-Execution

Collect Evidence

1. Shadow Run Metrics — validation_gates_json (already in DB)
2. Approval Queue — Status = "Pending" awaiting maker-checker
3. Audit Trail — CorrelationId in all logs/events
4. Outbox/Inbox — Verify event processing completeness

Decision Gate

• ✅ All gates passed? → Proceed to approval workflow
• ❌ Gates failed? → Root cause analysis, fix, re-run

Approval Workflow (Gate 4 - Already Implemented)

Once shadow run succeeds:

# Get pending approval
curl -X GET http://localhost:5000/api/v1/approval-queue \
  -H "Authorization: Bearer <token>"

# Maker-checker approval (Risk officer)
curl -X POST http://localhost:5000/api/v1/approval-queue/{id}/approve \
  -H "Authorization: Bearer <risk-officer-token>" \
  -d '{
    "approvalReason": "All validation gates passed. PBO=0.15, DSR=0.96. Approved for activation."
  }'

---

Timeline Expectations

Phase	Duration	Notes
DataBackfill	5-10 min	Fetch OHLCV, fees, calendar
Replay	10-20 min	Simulate signals & orders
Evaluation	5-10 min	Calculate metrics, gates
Phase Segmentation	2-5 min	Bull/Bear/Sideways analysis
Persist & Emit	1-2 min	Write to DB, emit events
Total	30-60 min	Depends on market data lag

---

Troubleshooting

Problem: Job stuck in "Processing"

• Check Hangfire logs for errors
• Verify PostgreSQL connection
• Restart job if stuck > 1 hour

Problem: "Model not found"

• Verify ModelId exists in database
• Use query from section 2 (Model Setup)

Problem: "No market data available"

• Check KRX API credentials
• Verify date range is covered by KRX
• Use stub data for testing (set in KrxDataService)

Problem: "PBO > 20% or DSR < 95%"

• Model not robust in 252-day window
• Consider strategy adjustments
• Re-run with different date range
• Log as evidence for risk review

---

Success Confirmation

Gate 3 is PASSED when:

• ✅ Shadow run completes with status = "EvaluationComplete"
• ✅ validation_gates_json.all_gates_passed = true
• ✅ Approval queue auto-populated with status = "Pending"
• ✅ CorrelationId present in all audit logs
• ✅ Events flow through Outbox → Inbox → Consumers

Next Step: Gate 4 (Approval Workflow) — Already implemented, awaiting results