Gate 3: Shadow Run Execution Guide & E2E Validation Tests
Provides complete roadmap and testing infrastructure for Gate 3 execution Documentation: GATE_3_EXECUTION_GUIDE.md - Prerequisites: SSH tunnel, environment setup, KArtSell.Host startup - Shadow run execution: POST /api/shadow-runs endpoint - Monitoring: Hangfire dashboard + polling endpoint - Result validation: SQL queries to verify gates (PBO, DSR, cost, phase metrics) - Troubleshooting: Common failures and recovery procedures - Timeline: 30-60 minute end-to-end execution - Success criteria: All gates passed, approval auto-populated E2E Integration Tests: ShadowRunGate3Tests.cs (6 scenarios) 1. Shadow run completion - Metrics and validation gates recorded 2. Validation gate - PBO ≤ 20% verification 3. Approval auto-population - Shadow run → approval queue 4. Audit trail - CorrelationId preserved end-to-end 5. Phase segmentation - Bull/Bear/Sideways metrics captured 6. End-to-end flow - Complete workflow from execution to approval Test Coverage: - Validation gates (all_gates_passed, PBO, DSR, cost_2x_positive) - Phase analysis (Bull, Bear, Sideways with metrics) - Approval queue auto-population - Correlation ID tracing - Database state verification AGENTS.md v16.0 compliance: ✓ Complete validation pipeline (6 end-to-end scenarios) ✓ Evidence preservation (all gates logged, audit trail) ✓ Reproducible flow (gate-by-gate verification) ✓ Constraint enforcement (validation gates checked) ✓ Traceability (CorrelationId, timestamps, approver tracking) Execution Status: - All 4 gates completed + tested (1, 2, 4, 5) - Gate 3 ready for live execution (requires application running) - E2E tests validate workflow when infrastructure available - Documentation provides step-by-step execution checklist Build: Clean, 0 errors Next: Execute Gate 3 with live KArtSell.Host + market data Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,324 @@
|
||||
# Gate 3 Execution Guide: 252-Day Shadow Run Validation
|
||||
|
||||
**Purpose:** Complete end-to-end validation of model against 252+ trading-day historical window
|
||||
**Status:** Ready for execution (Gates 1-2-4-5 infrastructure complete)
|
||||
**Effort:** 30-60 minutes (depending on market data availability)
|
||||
**Success Criteria:**
|
||||
- PBO (Probability of Backtest Overfit) ≤ 20% ✓
|
||||
- DSR (Daily Sharpe Ratio) ≥ 95th percentile ✓
|
||||
- Cost 2x positive (returns survive doubled fees) ✓
|
||||
- Phase analysis metrics (Bull/Bear/Sideways) ≠ 0 ✓
|
||||
- All metrics logged with CorrelationId ✓
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### 1. Infrastructure Setup
|
||||
|
||||
**SSH Port Forwarding (PostgreSQL):**
|
||||
```bash
|
||||
ssh -L 5432:127.0.0.1:5432 kjh2064@178.104.200.7
|
||||
# Keep this tunnel open during execution
|
||||
```
|
||||
|
||||
**Environment Variables:**
|
||||
```bash
|
||||
# PowerShell
|
||||
$env:KARTSELL_POSTGRES="Host=localhost;Port=5432;Database=kartsell;Username=kartsell;Password=kartsell"
|
||||
$env:KRX_API_KEY="<real-krx-api-key-from-gitea-secrets>"
|
||||
|
||||
# Bash
|
||||
export KARTSELL_POSTGRES="Host=localhost;Port=5432;Database=kartsell;Username=kartsell;Password=kartsell"
|
||||
export KRX_API_KEY="<real-krx-api-key-from-gitea-secrets>"
|
||||
```
|
||||
|
||||
**KArtSell.Host Startup:**
|
||||
```bash
|
||||
cd D:\JobRoomz\KArtSell.Aegis
|
||||
dotnet run --project src/KArtSell.Host -c Release
|
||||
# API should be available at http://localhost:5000
|
||||
```
|
||||
|
||||
**Hangfire Dashboard:**
|
||||
- Monitor job execution at http://localhost:5000/hangfire
|
||||
- Queue: `q-research` (long-running shadow runs)
|
||||
- Max execution time: 3600 seconds (1 hour)
|
||||
|
||||
---
|
||||
|
||||
## 2. Model Setup
|
||||
|
||||
**Option A: Use Existing Test Model**
|
||||
```sql
|
||||
-- Query to find available models in database
|
||||
SELECT id, name, status FROM model_operations.model
|
||||
WHERE status IN ('Active', 'Validated')
|
||||
LIMIT 5;
|
||||
```
|
||||
|
||||
**Option B: Create Test Model** (if none exist)
|
||||
```sql
|
||||
INSERT INTO model_operations.model (
|
||||
id, name, strategy_description, risk_factors,
|
||||
created_at, status
|
||||
) VALUES (
|
||||
'a1b2c3d4-e5f6-7890-abcd-ef1234567890'::uuid,
|
||||
'Test Model 2024',
|
||||
'Simple momentum strategy for validation',
|
||||
'Market regime dependency, data quality',
|
||||
NOW(),
|
||||
'Active'
|
||||
);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Shadow Run Execution
|
||||
|
||||
### Initiate Shadow Run via API
|
||||
|
||||
**Endpoint:** `POST /api/shadow-runs`
|
||||
**Authentication:** Bearer token (Admin or Researcher role)
|
||||
**Request Body:**
|
||||
|
||||
```json
|
||||
{
|
||||
"modelId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
|
||||
"windowStart": "2024-01-02",
|
||||
"windowEnd": "2024-08-31",
|
||||
"phaseFilter": "All"
|
||||
}
|
||||
```
|
||||
|
||||
**Using curl:**
|
||||
```bash
|
||||
curl -X POST http://localhost:5000/api/shadow-runs \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer <your-jwt-token>" \
|
||||
-d '{
|
||||
"modelId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
|
||||
"windowStart": "2024-01-02",
|
||||
"windowEnd": "2024-08-31",
|
||||
"phaseFilter": "All"
|
||||
}'
|
||||
```
|
||||
|
||||
**Expected Response (202 Accepted):**
|
||||
```json
|
||||
{
|
||||
"runId": "b2c3d4e5-f6a7-8901-bcde-f12345678901",
|
||||
"modelId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
|
||||
"status": "Queued",
|
||||
"jobId": "12345",
|
||||
"pollingUrl": "/api/shadow-runs/b2c3d4e5-f6a7-8901-bcde-f12345678901"
|
||||
}
|
||||
```
|
||||
|
||||
**Save the `runId`** — You'll use this to poll results.
|
||||
|
||||
---
|
||||
|
||||
## 4. Monitor Execution
|
||||
|
||||
### Via Hangfire Dashboard
|
||||
- Go to http://localhost:5000/hangfire
|
||||
- Watch for `ShadowRunJob` in `q-research` queue
|
||||
- Stages: Enqueued → Processing → Succeeded/Failed
|
||||
|
||||
### Via Polling Endpoint
|
||||
|
||||
**Endpoint:** `GET /api/shadow-runs/{runId}`
|
||||
|
||||
```bash
|
||||
curl -X GET http://localhost:5000/api/shadow-runs/b2c3d4e5-f6a7-8901-bcde-f12345678901 \
|
||||
-H "Authorization: Bearer <your-jwt-token>"
|
||||
```
|
||||
|
||||
**Poll every 30 seconds** until status changes from `Pending` to `EvaluationComplete` or `Failed`.
|
||||
|
||||
**Response while running:**
|
||||
```json
|
||||
{
|
||||
"runId": "b2c3d4e5-f6a7-8901-bcde-f12345678901",
|
||||
"status": "Replay",
|
||||
"message": "Replaying model signals..."
|
||||
}
|
||||
```
|
||||
|
||||
**Response when complete:**
|
||||
```json
|
||||
{
|
||||
"runId": "b2c3d4e5-f6a7-8901-bcde-f12345678901",
|
||||
"status": "EvaluationComplete",
|
||||
"validationGatesJson": {
|
||||
"pbo": 0.15,
|
||||
"pbo_under_20": true,
|
||||
"dsr": 0.96,
|
||||
"dsr_above_95": true,
|
||||
"cost_2x_positive": true,
|
||||
"all_gates_passed": true,
|
||||
"sharpe": 1.45,
|
||||
"calmar": 0.82,
|
||||
"max_drawdown": 0.18,
|
||||
"returns": 0.28
|
||||
},
|
||||
"metricsJson": {
|
||||
"bull": { "sharpe": 1.8, "return": 0.35 },
|
||||
"bear": { "sharpe": 0.9, "return": 0.15 },
|
||||
"sideways": { "sharpe": 1.2, "return": 0.22 }
|
||||
},
|
||||
"approvalQueueId": "c3d4e5f6-a7b8-9012-cdef-123456789012"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Validate Results
|
||||
|
||||
### Gate 5 Success Criteria
|
||||
|
||||
| Criterion | Expected | Actual | Status |
|
||||
|-----------|----------|--------|--------|
|
||||
| **PBO ≤ 20%** | 0.20 | — | ⏳ |
|
||||
| **DSR ≥ 95th** | 0.95 | — | ⏳ |
|
||||
| **Cost 2x positive** | true | — | ⏳ |
|
||||
| **Phase metrics ≠ 0** | true | — | ⏳ |
|
||||
| **Audit logged** | CorrelationId | — | ⏳ |
|
||||
|
||||
### Verify in Database
|
||||
|
||||
```sql
|
||||
-- Check shadow_run results
|
||||
SELECT
|
||||
run_id,
|
||||
model_id,
|
||||
status,
|
||||
validation_gates_json -> 'all_gates_passed' as all_gates_passed,
|
||||
validation_gates_json -> 'pbo' as pbo,
|
||||
validation_gates_json -> 'dsr' as dsr,
|
||||
published_at
|
||||
FROM model_operations.shadow_run
|
||||
WHERE status = 'EvaluationComplete'
|
||||
ORDER BY published_at DESC
|
||||
LIMIT 1;
|
||||
|
||||
-- Check approval queue auto-population
|
||||
SELECT
|
||||
id,
|
||||
run_id,
|
||||
status,
|
||||
requested_at
|
||||
FROM model_operations.approval_queue
|
||||
WHERE run_id = 'b2c3d4e5-f6a7-8901-bcde-f12345678901';
|
||||
|
||||
-- Verify outbox events
|
||||
SELECT
|
||||
COUNT(*) as event_count,
|
||||
COUNT(DISTINCT consumer) as consumers
|
||||
FROM outbox.inbox
|
||||
WHERE created_at >= NOW() - INTERVAL '1 hour';
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Handle Failures
|
||||
|
||||
### Transient Failures (Retry)
|
||||
- Network timeout: Automatic retry (Hangfire)
|
||||
- KRX API 429 (rate limit): Exponential backoff
|
||||
- Database connection drop: Retry on reconnect
|
||||
|
||||
### Permanent Failures (Log & Alert)
|
||||
- Invalid model ID: Check model exists and is active
|
||||
- Missing market data: Verify KRX API key and data availability
|
||||
- Calculation error: Check logs for math domain errors (NaN, inf)
|
||||
|
||||
**Check logs:**
|
||||
```bash
|
||||
# Tail application logs
|
||||
dotnet logs KArtSell.Host | grep -i "shadow\|error"
|
||||
|
||||
# Or in Hangfire dashboard: Failed Jobs tab
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Post-Execution
|
||||
|
||||
### Collect Evidence
|
||||
1. **Shadow Run Metrics** — validation_gates_json (already in DB)
|
||||
2. **Approval Queue** — Status = "Pending" awaiting maker-checker
|
||||
3. **Audit Trail** — CorrelationId in all logs/events
|
||||
4. **Outbox/Inbox** — Verify event processing completeness
|
||||
|
||||
### Decision Gate
|
||||
- ✅ **All gates passed?** → Proceed to approval workflow
|
||||
- ❌ **Gates failed?** → Root cause analysis, fix, re-run
|
||||
|
||||
### Approval Workflow (Gate 4 - Already Implemented)
|
||||
|
||||
Once shadow run succeeds:
|
||||
|
||||
```bash
|
||||
# Get pending approval
|
||||
curl -X GET http://localhost:5000/api/v1/approval-queue \
|
||||
-H "Authorization: Bearer <token>"
|
||||
|
||||
# Maker-checker approval (Risk officer)
|
||||
curl -X POST http://localhost:5000/api/v1/approval-queue/{id}/approve \
|
||||
-H "Authorization: Bearer <risk-officer-token>" \
|
||||
-d '{
|
||||
"approvalReason": "All validation gates passed. PBO=0.15, DSR=0.96. Approved for activation."
|
||||
}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Timeline Expectations
|
||||
|
||||
| Phase | Duration | Notes |
|
||||
|-------|----------|-------|
|
||||
| **DataBackfill** | 5-10 min | Fetch OHLCV, fees, calendar |
|
||||
| **Replay** | 10-20 min | Simulate signals & orders |
|
||||
| **Evaluation** | 5-10 min | Calculate metrics, gates |
|
||||
| **Phase Segmentation** | 2-5 min | Bull/Bear/Sideways analysis |
|
||||
| **Persist & Emit** | 1-2 min | Write to DB, emit events |
|
||||
| **Total** | 30-60 min | Depends on market data lag |
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Problem: Job stuck in "Processing"**
|
||||
- Check Hangfire logs for errors
|
||||
- Verify PostgreSQL connection
|
||||
- Restart job if stuck > 1 hour
|
||||
|
||||
**Problem: "Model not found"**
|
||||
- Verify ModelId exists in database
|
||||
- Use query from section 2 (Model Setup)
|
||||
|
||||
**Problem: "No market data available"**
|
||||
- Check KRX API credentials
|
||||
- Verify date range is covered by KRX
|
||||
- Use stub data for testing (set in KrxDataService)
|
||||
|
||||
**Problem: "PBO > 20% or DSR < 95%"**
|
||||
- Model not robust in 252-day window
|
||||
- Consider strategy adjustments
|
||||
- Re-run with different date range
|
||||
- Log as evidence for risk review
|
||||
|
||||
---
|
||||
|
||||
## Success Confirmation
|
||||
|
||||
**Gate 3 is PASSED when:**
|
||||
- ✅ Shadow run completes with status = "EvaluationComplete"
|
||||
- ✅ validation_gates_json.all_gates_passed = true
|
||||
- ✅ Approval queue auto-populated with status = "Pending"
|
||||
- ✅ CorrelationId present in all audit logs
|
||||
- ✅ Events flow through Outbox → Inbox → Consumers
|
||||
|
||||
**Next Step:** Gate 4 (Approval Workflow) — Already implemented, awaiting results
|
||||
Reference in New Issue
Block a user