KArtSell.Aegis/GATE_3_EXECUTION_GUIDE.md

# Gate 3 Execution Guide: 252-Day Shadow Run Validation

**Purpose:** Complete end-to-end validation of model against 252+ trading-day historical window  
**Status:** Ready for execution (Gates 1-2-4-5 infrastructure complete)  
**Effort:** 30-60 minutes (depending on market data availability)  
**Success Criteria:**
- PBO (Probability of Backtest Overfit) ≤ 20% ✓
- DSR (Daily Sharpe Ratio) ≥ 95th percentile ✓
- Cost 2x positive (returns survive doubled fees) ✓
- Phase analysis metrics (Bull/Bear/Sideways) ≠ 0 ✓
- All metrics logged with CorrelationId ✓

---

## Prerequisites

### 1. Infrastructure Setup

**SSH Port Forwarding (PostgreSQL):**
```bash
ssh -L 5432:127.0.0.1:5432 kjh2064@178.104.200.7
# Keep this tunnel open during execution
```

**Environment Variables:**
```bash
# PowerShell
$env:KARTSELL_POSTGRES="Host=localhost;Port=5432;Database=kartsell;Username=kartsell;Password=kartsell"
$env:KRX_API_KEY="<real-krx-api-key-from-gitea-secrets>"

# Bash
export KARTSELL_POSTGRES="Host=localhost;Port=5432;Database=kartsell;Username=kartsell;Password=kartsell"
export KRX_API_KEY="<real-krx-api-key-from-gitea-secrets>"
```

**KArtSell.Host Startup:**
```bash
cd D:\JobRoomz\KArtSell.Aegis
dotnet run --project src/KArtSell.Host -c Release
# API should be available at http://localhost:5000
```

**Hangfire Dashboard:**
- Monitor job execution at http://localhost:5000/hangfire
- Queue: `q-research` (long-running shadow runs)
- Max execution time: 3600 seconds (1 hour)

---

## 2. Model Setup

**Option A: Use Existing Test Model**
```sql
-- Query to find available models in database
SELECT id, name, status FROM model_operations.model 
WHERE status IN ('Active', 'Validated')
LIMIT 5;
```

**Option B: Create Test Model** (if none exist)
```sql
INSERT INTO model_operations.model (
  id, name, strategy_description, risk_factors, 
  created_at, status
) VALUES (
  'a1b2c3d4-e5f6-7890-abcd-ef1234567890'::uuid,
  'Test Model 2024',
  'Simple momentum strategy for validation',
  'Market regime dependency, data quality',
  NOW(),
  'Active'
);
```

---

## 3. Shadow Run Execution

### Initiate Shadow Run via API

**Endpoint:** `POST /api/shadow-runs`  
**Authentication:** Bearer token (Admin or Researcher role)  
**Request Body:**

```json
{
  "modelId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "windowStart": "2024-01-02",
  "windowEnd": "2024-08-31",
  "phaseFilter": "All"
}
```

**Using curl:**
```bash
curl -X POST http://localhost:5000/api/shadow-runs \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <your-jwt-token>" \
  -d '{
    "modelId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "windowStart": "2024-01-02",
    "windowEnd": "2024-08-31",
    "phaseFilter": "All"
  }'
```

**Expected Response (202 Accepted):**
```json
{
  "runId": "b2c3d4e5-f6a7-8901-bcde-f12345678901",
  "modelId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "Queued",
  "jobId": "12345",
  "pollingUrl": "/api/shadow-runs/b2c3d4e5-f6a7-8901-bcde-f12345678901"
}
```

**Save the `runId`** — You'll use this to poll results.

---

## 4. Monitor Execution

### Via Hangfire Dashboard
- Go to http://localhost:5000/hangfire
- Watch for `ShadowRunJob` in `q-research` queue
- Stages: Enqueued → Processing → Succeeded/Failed

### Via Polling Endpoint

**Endpoint:** `GET /api/shadow-runs/{runId}`

```bash
curl -X GET http://localhost:5000/api/shadow-runs/b2c3d4e5-f6a7-8901-bcde-f12345678901 \
  -H "Authorization: Bearer <your-jwt-token>"
```

**Poll every 30 seconds** until status changes from `Pending` to `EvaluationComplete` or `Failed`.

**Response while running:**
```json
{
  "runId": "b2c3d4e5-f6a7-8901-bcde-f12345678901",
  "status": "Replay",
  "message": "Replaying model signals..."
}
```

**Response when complete:**
```json
{
  "runId": "b2c3d4e5-f6a7-8901-bcde-f12345678901",
  "status": "EvaluationComplete",
  "validationGatesJson": {
    "pbo": 0.15,
    "pbo_under_20": true,
    "dsr": 0.96,
    "dsr_above_95": true,
    "cost_2x_positive": true,
    "all_gates_passed": true,
    "sharpe": 1.45,
    "calmar": 0.82,
    "max_drawdown": 0.18,
    "returns": 0.28
  },
  "metricsJson": {
    "bull": { "sharpe": 1.8, "return": 0.35 },
    "bear": { "sharpe": 0.9, "return": 0.15 },
    "sideways": { "sharpe": 1.2, "return": 0.22 }
  },
  "approvalQueueId": "c3d4e5f6-a7b8-9012-cdef-123456789012"
}
```

---

## 5. Validate Results

### Gate 5 Success Criteria

| Criterion | Expected | Actual | Status |
|-----------|----------|--------|--------|
| **PBO ≤ 20%** | 0.20 | — | ⏳ |
| **DSR ≥ 95th** | 0.95 | — | ⏳ |
| **Cost 2x positive** | true | — | ⏳ |
| **Phase metrics ≠ 0** | true | — | ⏳ |
| **Audit logged** | CorrelationId | — | ⏳ |

### Verify in Database

```sql
-- Check shadow_run results
SELECT 
  run_id,
  model_id,
  status,
  validation_gates_json -> 'all_gates_passed' as all_gates_passed,
  validation_gates_json -> 'pbo' as pbo,
  validation_gates_json -> 'dsr' as dsr,
  published_at
FROM model_operations.shadow_run
WHERE status = 'EvaluationComplete'
ORDER BY published_at DESC
LIMIT 1;

-- Check approval queue auto-population
SELECT 
  id,
  run_id,
  status,
  requested_at
FROM model_operations.approval_queue
WHERE run_id = 'b2c3d4e5-f6a7-8901-bcde-f12345678901';

-- Verify outbox events
SELECT 
  COUNT(*) as event_count,
  COUNT(DISTINCT consumer) as consumers
FROM outbox.inbox
WHERE created_at >= NOW() - INTERVAL '1 hour';
```

---

## 6. Handle Failures

### Transient Failures (Retry)
- Network timeout: Automatic retry (Hangfire)
- KRX API 429 (rate limit): Exponential backoff
- Database connection drop: Retry on reconnect

### Permanent Failures (Log & Alert)
- Invalid model ID: Check model exists and is active
- Missing market data: Verify KRX API key and data availability
- Calculation error: Check logs for math domain errors (NaN, inf)

**Check logs:**
```bash
# Tail application logs
dotnet logs KArtSell.Host | grep -i "shadow\|error"

# Or in Hangfire dashboard: Failed Jobs tab
```

---

## 7. Post-Execution

### Collect Evidence
1. **Shadow Run Metrics** — validation_gates_json (already in DB)
2. **Approval Queue** — Status = "Pending" awaiting maker-checker
3. **Audit Trail** — CorrelationId in all logs/events
4. **Outbox/Inbox** — Verify event processing completeness

### Decision Gate
- ✅ **All gates passed?** → Proceed to approval workflow
- ❌ **Gates failed?** → Root cause analysis, fix, re-run

### Approval Workflow (Gate 4 - Already Implemented)

Once shadow run succeeds:

```bash
# Get pending approval
curl -X GET http://localhost:5000/api/v1/approval-queue \
  -H "Authorization: Bearer <token>"

# Maker-checker approval (Risk officer)
curl -X POST http://localhost:5000/api/v1/approval-queue/{id}/approve \
  -H "Authorization: Bearer <risk-officer-token>" \
  -d '{
    "approvalReason": "All validation gates passed. PBO=0.15, DSR=0.96. Approved for activation."
  }'
```

---

## Timeline Expectations

| Phase | Duration | Notes |
|-------|----------|-------|
| **DataBackfill** | 5-10 min | Fetch OHLCV, fees, calendar |
| **Replay** | 10-20 min | Simulate signals & orders |
| **Evaluation** | 5-10 min | Calculate metrics, gates |
| **Phase Segmentation** | 2-5 min | Bull/Bear/Sideways analysis |
| **Persist & Emit** | 1-2 min | Write to DB, emit events |
| **Total** | 30-60 min | Depends on market data lag |

---

## Troubleshooting

**Problem: Job stuck in "Processing"**
- Check Hangfire logs for errors
- Verify PostgreSQL connection
- Restart job if stuck > 1 hour

**Problem: "Model not found"**
- Verify ModelId exists in database
- Use query from section 2 (Model Setup)

**Problem: "No market data available"**
- Check KRX API credentials
- Verify date range is covered by KRX
- Use stub data for testing (set in KrxDataService)

**Problem: "PBO > 20% or DSR < 95%"**
- Model not robust in 252-day window
- Consider strategy adjustments
- Re-run with different date range
- Log as evidence for risk review

---

## Success Confirmation

**Gate 3 is PASSED when:**
- ✅ Shadow run completes with status = "EvaluationComplete"
- ✅ validation_gates_json.all_gates_passed = true
- ✅ Approval queue auto-populated with status = "Pending"
- ✅ CorrelationId present in all audit logs
- ✅ Events flow through Outbox → Inbox → Consumers

**Next Step:** Gate 4 (Approval Workflow) — Already implemented, awaiting results