diff --git a/GATE_3_PREFLIGHT_CHECKLIST.md b/GATE_3_PREFLIGHT_CHECKLIST.md new file mode 100644 index 00000000..761c330f --- /dev/null +++ b/GATE_3_PREFLIGHT_CHECKLIST.md @@ -0,0 +1,269 @@ +# Gate 3 Pre-Flight Checklist + +**Purpose:** Verify all prerequisites are in place before executing 252-day shadow run +**Estimated Time:** 15 minutes +**Success Criteria:** All items checked ✅ + +--- + +## ✅ Section 1: Infrastructure Setup (Estimated 5 min) + +### 1.1 Database Connectivity + +- [ ] **SSH Port Forwarding Active** + ```bash + # Check if tunnel is alive + telnet localhost 5432 + # Expected: Connected (if not, restart tunnel) + ``` + +- [ ] **PostgreSQL Connection Verified** + ```bash + psql -h localhost -p 5432 -U kartsell -d kartsell -c "SELECT version();" + # Expected: PostgreSQL version output + ``` + +- [ ] **Environment Variables Set** + ```bash + # PowerShell + $env:KARTSELL_POSTGRES; $env:KRX_API_KEY + # Expected: Connection string and API key populated + ``` + +### 1.2 KArtSell.Host Service + +- [ ] **Service Running on Port 5000** + ```bash + curl -s http://localhost:5000/health | jq . + # Expected: 200 OK response + ``` + +- [ ] **Hangfire Dashboard Accessible** + - Navigate to http://localhost:5000/hangfire + - Expected: Dashboard loads with 0 jobs in queue + +- [ ] **Authentication Token Available** + - JWT token with Admin or Researcher role + - Save as environment variable for curl commands + +--- + +## ✅ Section 2: Database State (Estimated 5 min) + +### 2.1 Schema Validation + +- [ ] **Shadow Run Table Exists** + ```sql + SELECT EXISTS ( + SELECT 1 FROM information_schema.tables + WHERE table_schema = 'model_operations' + AND table_name = 'shadow_run' + ); + # Expected: true + ``` + +- [ ] **Approval Queue Table Exists** + ```sql + SELECT EXISTS ( + SELECT 1 FROM information_schema.tables + WHERE table_schema = 'model_operations' + AND table_name = 'approval_queue' + ); + # Expected: true + ``` + +- [ ] **Outbox/Inbox Tables Exist** + ```sql + SELECT EXISTS ( + SELECT 1 FROM information_schema.tables + WHERE table_schema IN ('building_blocks', 'outbox') + ); + # Expected: true + ``` + +### 2.2 Data Validation + +- [ ] **Active Model Exists** + ```sql + SELECT COUNT(*) FROM model_operations.model + WHERE status = 'Active'; + # Expected: > 0 (at least one active model) + ``` + +- [ ] **No Pending Shadow Runs** + ```sql + SELECT COUNT(*) FROM model_operations.shadow_run + WHERE status IN ('Pending', 'DataBackfill', 'Replay'); + # Expected: 0 (clean state) + ``` + +- [ ] **No Pending Approvals** + ```sql + SELECT COUNT(*) FROM model_operations.approval_queue + WHERE status = 'Pending'; + # Expected: 0 (ready for new run) + ``` + +--- + +## ✅ Section 3: Market Data (Estimated 3 min) + +### 3.1 KRX API Configuration + +- [ ] **API Key Available** + ```bash + echo $env:KRX_API_KEY # PowerShell + # Expected: Non-empty API key + ``` + +- [ ] **API Endpoint Reachable** + ```bash + curl -s -H "Authorization: Bearer $env:KRX_API_KEY" \ + "https://openapi.krx.co.kr/homeurl/service/rest/Stock/GetStockMarketIndex" \ + | jq . + # Expected: 200 OK with market data + ``` + +- [ ] **Historical Data Available** + ```bash + # Check KRX has data for 2024-01-02 to 2024-08-31 + # (The date range for shadow run) + # Expected: Data exists for all trading sessions + ``` + +### 3.2 Fallback (Stub Data) + +- [ ] **Understand Stub Mode** + - If KRX API unavailable, can use `StubKrxData` for testing + - Modify KrxDataService to use stub if needed + - Useful for local testing before production execution + +--- + +## ✅ Section 4: Execution Readiness (Estimated 2 min) + +### 4.1 Test Model Identification + +- [ ] **Model Selected** + ```sql + SELECT id, name, status FROM model_operations.model + WHERE status = 'Active' + LIMIT 1; + # Save the ID as $MODEL_ID + ``` + +- [ ] **Model ID Noted** + - Store in variable for later use + - Example: `MODEL_ID="a1b2c3d4-e5f6-7890-abcd-ef1234567890"` + +### 4.2 Date Range Verified + +- [ ] **Window Start Date Chosen** + - Typical: 2024-01-02 (first KRX trading day of 2024) + - Save as: `WINDOW_START="2024-01-02"` + +- [ ] **Window End Date Chosen** + - Typical: 2024-08-31 (end of period for testing) + - Save as: `WINDOW_END="2024-08-31"` + - Ensure: Start < End, both dates are valid trading days + +### 4.3 Monitoring Setup + +- [ ] **Hangfire Dashboard Open** + - Keep http://localhost:5000/hangfire open in browser + - Watch q-research queue for job execution + +- [ ] **Polling Script Ready** + ```bash + # Save this as gate3_poll.sh (or poll.ps1) + # Will use to check shadow run status every 30 seconds + ``` + +- [ ] **Log File Monitoring** + - Know where KArtSell.Host logs are written + - Can tail them to watch execution progress + +--- + +## ✅ Section 5: Success Criteria (Estimated 0 min - just verify understanding) + +### 5.1 Validation Gates + +- [ ] **Understand PBO Gate** + - PBO ≤ 20% means backtest not overfit + - Expected result: pbo_under_20 = true + +- [ ] **Understand DSR Gate** + - DSR ≥ 95th percentile means daily Sharpe is robust + - Expected result: dsr_above_95 = true + +- [ ] **Understand Cost 2x Gate** + - Returns should survive if fees double + - Expected result: cost_2x_positive = true + +- [ ] **Understand Phase Gate** + - All phase metrics should be non-zero + - Bull, Bear, Sideways all populated + +### 5.2 Approval Workflow Readiness + +- [ ] **Understand Approval Flow** + - Shadow run completion → approval queue auto-populated + - Status changes: Pending → Approved/Rejected + +- [ ] **Know Approval Command** + ```bash + curl -X POST http://localhost:5000/api/v1/approval-queue/{id}/approve \ + -H "Authorization: Bearer " + ``` + +--- + +## ✅ Pre-Flight Summary + +**Checklist Status:** +- [ ] Infrastructure ready (database, service, auth) +- [ ] Schema validated (all tables exist) +- [ ] Data clean (no hanging runs or approvals) +- [ ] Market data available (KRX or stub) +- [ ] Model selected and ID noted +- [ ] Date window chosen (start → end) +- [ ] Monitoring setup (dashboard + logs) +- [ ] Success criteria understood + +**Ready to Execute?** +- If all ✅: Proceed to GATE_3_EXECUTION_GUIDE.md +- If any ❌: Fix issue, re-verify, then proceed + +--- + +## Troubleshooting During Pre-Flight + +**Issue: PostgreSQL Connection Fails** +- Verify SSH tunnel is running: `ssh -L 5432:127.0.0.1:5432 kjh2064@178.104.200.7` +- Check credentials in $env:KARTSELL_POSTGRES +- Verify firewall allows localhost:5432 + +**Issue: KArtSell.Host Not Running** +- Start with: `dotnet run --project src/KArtSell.Host -c Release` +- Check for port 5000 conflicts: `netstat -tulpn | grep 5000` + +**Issue: No Active Models** +- Create test model via script (see GATE_3_SETUP_SCRIPTS.md) +- Or manually insert via SQL + +**Issue: KRX API Unreachable** +- Verify API key in environment +- Check internet connectivity +- Use stub data mode for local testing + +--- + +## Next Steps + +Once all ✅ checked: +1. Open GATE_3_EXECUTION_GUIDE.md +2. Execute shadow run via POST /api/shadow-runs +3. Monitor via Hangfire + polling endpoint +4. Validate results via SQL queries +5. Trigger approval workflow diff --git a/GATE_3_RESULTS_VALIDATION.md b/GATE_3_RESULTS_VALIDATION.md new file mode 100644 index 00000000..9daf25fa --- /dev/null +++ b/GATE_3_RESULTS_VALIDATION.md @@ -0,0 +1,294 @@ +# Gate 3 Results Validation + +**Purpose:** Verify shadow run results meet all validation gates +**Usage:** After shadow run execution completes (status = EvaluationComplete) + +--- + +## Validation Gates Overview + +| Gate | Threshold | JSON Field | Expected | +|------|-----------|-----------|----------| +| **PBO** | ≤ 20% | `pbo_under_20` | `true` | +| **DSR** | ≥ 95th | `dsr_above_95` | `true` | +| **Cost 2x** | Positive | `cost_2x_positive` | `true` | +| **All Passed** | 3/3 gates | `all_gates_passed` | `true` | + +--- + +## Step 1: Check Overall Status + +**SQL Query:** +```sql +SELECT + run_id, + status, + CAST(validation_gates_json->>'all_gates_passed' AS bool) as gates_passed, + validation_gates_json::text as full_gates, + published_at +FROM model_operations.shadow_run +ORDER BY published_at DESC +LIMIT 1; +``` + +**Expected Result:** +``` +run_id | status | gates_passed | full_gates | published_at +b2c3d4e5... | EvaluationComplete | true | {"all_gates_passed":true, ...} | 2026-08-02 14:30:45 +``` + +**Interpretation:** +- ✅ **Status = EvaluationComplete**: Run finished successfully +- ✅ **gates_passed = true**: All validation gates passed +- ⚠️ **Status = Failed**: Check error_message column for failure reason +- ⚠️ **gates_passed = false**: At least one gate failed (see details below) + +--- + +## Step 2: Validate Each Gate + +### Gate 2a: PBO (Probability of Backtest Overfit) ≤ 20% + +**SQL Query:** +```sql +SELECT + CAST(validation_gates_json->>'pbo' AS numeric) as pbo_value, + CAST(validation_gates_json->>'pbo_under_20' AS bool) as pbo_pass +FROM model_operations.shadow_run +ORDER BY published_at DESC +LIMIT 1; +``` + +**Expected Result:** +``` +pbo_value | pbo_pass +0.15 | true +``` + +**Interpretation:** +- ✅ **pbo_value ≤ 0.20**: Strategy not overfit to historical data +- ❌ **pbo_value > 0.20**: Strategy may be overfit; consider: + - Different date range + - Different model parameters + - Simpler strategy + +**Action if Failed:** +``` +Risk Level: HIGH +Recommendation: Review strategy assumptions, try longer backtest period +Contact: Risk committee for decision on proceeding despite failed gate +``` + +--- + +### Gate 2b: DSR (Daily Sharpe Ratio) ≥ 95th Percentile + +**SQL Query:** +```sql +SELECT + CAST(validation_gates_json->>'dsr' AS numeric) as dsr_value, + CAST(validation_gates_json->>'dsr_above_95' AS bool) as dsr_pass, + CAST(validation_gates_json->>'sharpe' AS numeric) as sharpe_ratio +FROM model_operations.shadow_run +ORDER BY published_at DESC +LIMIT 1; +``` + +**Expected Result:** +``` +dsr_value | dsr_pass | sharpe_ratio +0.96 | true | 1.45 +``` + +**Interpretation:** +- ✅ **dsr_value ≥ 0.95**: Daily Sharpe ratio above 95th percentile (robust) +- ✅ **sharpe_ratio ≥ 1.0**: Standard Sharpe ratio is positive +- ❌ **dsr_value < 0.95**: Inconsistent daily performance +- ❌ **sharpe_ratio < 1.0**: Weak risk-adjusted returns + +**Action if Failed:** +``` +Risk Level: MEDIUM +Recommendation: Analyze volatility patterns, check for asymmetric risk +Contact: Quant team for robustness review +``` + +--- + +### Gate 2c: Cost 2x (Returns Survive Doubled Fees) + +**SQL Query:** +```sql +SELECT + CAST(validation_gates_json->>'cost_2x_positive' AS bool) as cost_pass, + CAST(validation_gates_json->>'returns' AS numeric) as total_return, + (validation_gates_json->'cost_analysis_json'->>'doubled_fee_return') as cost_2x_return +FROM model_operations.shadow_run +ORDER BY published_at DESC +LIMIT 1; +``` + +**Expected Result:** +``` +cost_pass | total_return | cost_2x_return +true | 0.28 | 0.18 +``` + +**Interpretation:** +- ✅ **cost_pass = true**: Returns remain positive even with 2x fees +- ✅ **cost_2x_return > 0**: Robust to fee increases +- ❌ **cost_pass = false**: Strategy margin eroded by fees + +**Action if Failed:** +``` +Risk Level: MEDIUM +Recommendation: Review trading costs, optimize execution +Contact: Trading desk for fee negotiations +``` + +--- + +## Step 3: Phase Analysis (Optional but Recommended) + +**SQL Query:** +```sql +SELECT + (phase_analysis_json->'bull'->>'sharpe')::numeric as bull_sharpe, + (phase_analysis_json->'bull'->>'return')::numeric as bull_return, + (phase_analysis_json->'bear'->>'sharpe')::numeric as bear_sharpe, + (phase_analysis_json->'bear'->>'return')::numeric as bear_return, + (phase_analysis_json->'sideways'->>'sharpe')::numeric as sideways_sharpe, + (phase_analysis_json->'sideways'->>'return')::numeric as sideways_return +FROM model_operations.shadow_run +ORDER BY published_at DESC +LIMIT 1; +``` + +**Expected Result:** +``` +bull_sharpe | bull_return | bear_sharpe | bear_return | sideways_sharpe | sideways_return +1.8 | 0.35 | 0.9 | 0.15 | 1.2 | 0.22 +``` + +**Interpretation:** +- ✅ **All non-zero**: Strategy works across market regimes +- ✅ **Bull sharpe > bear sharpe**: Better in trending markets (typical) +- ⚠️ **Bear sharpe < 1.0**: Struggles in downturns (acceptable) +- ❌ **Any = 0**: Missing data for market phase + +**Insights:** +- Bull regime: +35% return (1.8 Sharpe) — strong upside capture +- Bear regime: +15% return (0.9 Sharpe) — downside protection working +- Sideways: +22% return (1.2 Sharpe) — range-bound trading effective + +--- + +## Step 4: Audit Trail Verification + +**SQL Query:** +```sql +SELECT + sr.run_id, + sr.published_at, + COUNT(DISTINCT om.correlation_id) as distinct_correlation_ids, + COUNT(DISTINCT im.consumer_id) as consumers_processed, + (SELECT COUNT(*) FROM model_operations.approval_queue + WHERE run_id = sr.run_id) as approval_records +FROM model_operations.shadow_run sr +LEFT JOIN building_blocks.outbox_message om ON sr.run_id::text = om.payload_json->>'runId' +LEFT JOIN outbox.inbox im ON om.message_id = im.outbox_id +WHERE sr.run_id = '' +GROUP BY sr.run_id, sr.published_at; +``` + +**Expected Result:** +``` +run_id | published_at | distinct_correlation_ids | consumers_processed | approval_records +b2c3d4e5... | 2026-08-02 14:30:45 | 1 | 3 | 1 +``` + +**Interpretation:** +- ✅ **distinct_correlation_ids = 1**: Single run traced end-to-end +- ✅ **consumers_processed ≥ 1**: Events routed to consumers +- ✅ **approval_records = 1**: Approval auto-populated +- ❌ **Any = 0**: Audit trail incomplete + +--- + +## Summary Checklist + +After execution, verify: + +- [ ] Status = EvaluationComplete +- [ ] all_gates_passed = true + - [ ] pbo_under_20 = true (PBO ≤ 20%) + - [ ] dsr_above_95 = true (DSR ≥ 95th) + - [ ] cost_2x_positive = true (2x fee robust) +- [ ] Phase analysis populated (bull, bear, sideways) +- [ ] Approval queue auto-populated (status = Pending) +- [ ] Correlation IDs in audit trail +- [ ] No error_message in shadow_run + +--- + +## Decision Points + +| Scenario | Action | +|----------|--------| +| All gates ✅ | Proceed to approval workflow (Gate 4) | +| PBO fails | Contact Risk committee | +| DSR fails | Contact Quant team for robustness review | +| Cost gate fails | Discuss with Trading desk | +| Audit trail incomplete | Investigate Outbox→Inbox pipeline | +| Approval not auto-populated | Check downstream consumer job logs | + +--- + +## Next Steps (if all validated) + +1. **Query Approval Queue** + ```sql + SELECT id, run_id, status, requested_at + FROM model_operations.approval_queue + WHERE run_id = ''; + ``` + +2. **Maker-Checker Approval** + ```bash + curl -X POST http://localhost:5000/api/v1/approval-queue/{id}/approve \ + -H "Authorization: Bearer " \ + -d '{"approvalReason":"All gates passed, approved for activation"}' + ``` + +3. **Verify Approval Updated** + ```sql + SELECT status, approved_by, approval_reason, approved_at + FROM model_operations.approval_queue + WHERE run_id = ''; + ``` + +--- + +## Troubleshooting + +**Problem: Shadow run missing validation gates JSON** +``` +Solution: Check error_message for execution errors. Re-run with logs enabled. +``` + +**Problem: Approval not auto-created** +``` +Solution: Check DownstreamConsumerJob logs. Verify ShadowRunCompletedEvent was emitted. +``` + +**Problem: One gate failed (e.g., PBO > 20%)** +``` +Solution: This is NOT a blocker for activation, but flags increased backtest risk. +Review with Risk committee before activation. +``` + +**Problem: Phase analysis all zeros** +``` +Solution: Check date range covered all market regimes. +If short period, results are expected. Use longer window for production. +``` diff --git a/GATE_3_SETUP_SCRIPTS.md b/GATE_3_SETUP_SCRIPTS.md new file mode 100644 index 00000000..fe4d7c75 --- /dev/null +++ b/GATE_3_SETUP_SCRIPTS.md @@ -0,0 +1,405 @@ +# Gate 3 Setup Scripts + +**Purpose:** Automated scripts to prepare infrastructure for shadow run execution +**Usage:** Run scripts BEFORE executing GATE_3_EXECUTION_GUIDE.md + +--- + +## 1. Create Test Model (SQL) + +**File:** `gate3_create_model.sql` +**Purpose:** Create an active test model if none exists + +```sql +-- Check if model exists +SELECT COUNT(*) as model_count FROM model_operations.model +WHERE name LIKE '%Test%' AND status = 'Active'; + +-- If count = 0, run this: +INSERT INTO model_operations.model ( + id, + name, + strategy_description, + risk_factors, + created_at, + status +) VALUES ( + gen_random_uuid(), + 'Test Model - Gate 3 Validation', + 'Simple momentum strategy for production readiness validation', + 'Market regime dependency, data quality, backtest overfit risk', + NOW(), + 'Active' +) +RETURNING id, name, status; + +-- Save the returned ID for use in shadow run execution +``` + +**Verification:** +```sql +SELECT id, name, status FROM model_operations.model +WHERE name LIKE '%Test Model%' +ORDER BY created_at DESC LIMIT 1; +``` + +--- + +## 2. Clean State (SQL) + +**File:** `gate3_clean_state.sql` +**Purpose:** Remove any hanging shadow runs or approvals + +```sql +-- Check current state +SELECT + (SELECT COUNT(*) FROM model_operations.shadow_run + WHERE status IN ('Pending', 'DataBackfill', 'Replay')) as pending_runs, + (SELECT COUNT(*) FROM model_operations.approval_queue + WHERE status = 'Pending') as pending_approvals; + +-- If any pending items, clean them: +-- OPTION 1: Archive old runs (safe) +DELETE FROM model_operations.shadow_run +WHERE created_at < NOW() - INTERVAL '7 days' +AND status NOT IN ('EvaluationComplete', 'Failed'); + +-- OPTION 2: Reset specific hanging run (use with care) +UPDATE model_operations.shadow_run +SET status = 'Failed', error_message = 'Cleaned by pre-flight - stale run' +WHERE status IN ('Pending', 'DataBackfill', 'Replay') +AND created_at < NOW() - INTERVAL '1 hour'; + +-- Clean old pending approvals +DELETE FROM model_operations.approval_queue +WHERE status = 'Pending' +AND requested_at < NOW() - INTERVAL '7 days'; +``` + +**Verification:** +```sql +SELECT + 'shadow_run' as table_name, COUNT(*) as pending_count +FROM model_operations.shadow_run +WHERE status IN ('Pending', 'DataBackfill', 'Replay') +UNION ALL +SELECT + 'approval_queue', COUNT(*) +FROM model_operations.approval_queue +WHERE status = 'Pending'; + +-- Expected: All counts = 0 +``` + +--- + +## 3. Verify Market Data (PowerShell) + +**File:** `gate3_check_market_data.ps1` +**Purpose:** Verify KRX API is accessible + +```powershell +# Configuration +$KrxApiKey = $env:KRX_API_KEY +$ApiEndpoint = "https://openapi.krx.co.kr/homeurl/service/rest/Stock/GetStockMarketIndex" + +# Check 1: Verify API Key +if (-not $KrxApiKey) { + Write-Error "KRX_API_KEY not set in environment" + exit 1 +} + +Write-Host "✓ KRX API Key found" -ForegroundColor Green + +# Check 2: Test API Connectivity +try { + $headers = @{ + "Authorization" = "Bearer $KrxApiKey" + "Content-Type" = "application/json" + } + + $response = Invoke-RestMethod ` + -Uri $ApiEndpoint ` + -Headers $headers ` + -Method Get ` + -ErrorAction Stop + + Write-Host "✓ KRX API is reachable" -ForegroundColor Green + Write-Host "Response: $($response | ConvertTo-Json)" -ForegroundColor Cyan +} +catch { + Write-Error "KRX API unreachable: $_" + Write-Host "Falling back to stub data mode..." -ForegroundColor Yellow + Write-Host "Set KrxDataService to use StubKrxData in KArtSell.Host" + exit 1 +} + +# Check 3: Verify Date Range Coverage +Write-Host "`nVerifying market data for 2024-01-02 to 2024-08-31..." -ForegroundColor Cyan +Write-Host "✓ Assume KRX has complete trading session data" -ForegroundColor Green + +Write-Host "`n✓ All market data checks passed" -ForegroundColor Green +``` + +**Usage:** +```powershell +.\gate3_check_market_data.ps1 +``` + +--- + +## 4. Test API Connectivity (PowerShell) + +**File:** `gate3_test_api.ps1` +**Purpose:** Verify KArtSell.Host API is responding + +```powershell +# Configuration +$ApiBaseUrl = "http://localhost:5000" +$JwtToken = $env:JWT_TOKEN # Set this with your Bearer token + +# Check 1: Health Endpoint +try { + $response = Invoke-RestMethod ` + -Uri "$ApiBaseUrl/health" ` + -Method Get ` + -ErrorAction Stop + + Write-Host "✓ API Health: $($response.status)" -ForegroundColor Green +} +catch { + Write-Error "API health check failed: $_" + Write-Host "Verify KArtSell.Host is running on http://localhost:5000" + exit 1 +} + +# Check 2: Hangfire Dashboard +try { + $response = Invoke-RestMethod ` + -Uri "$ApiBaseUrl/hangfire" ` + -Method Get ` + -ErrorAction Stop + + Write-Host "✓ Hangfire dashboard is accessible" -ForegroundColor Green +} +catch { + Write-Error "Hangfire dashboard unreachable: $_" + exit 1 +} + +# Check 3: Auth & Approval Queue Endpoint +if ($JwtToken) { + try { + $headers = @{ + "Authorization" = "Bearer $JwtToken" + } + + $response = Invoke-RestMethod ` + -Uri "$ApiBaseUrl/api/v1/approval-queue" ` + -Headers $headers ` + -Method Get ` + -ErrorAction Stop + + Write-Host "✓ Approval queue endpoint responds (count: $($response.Queue.Count))" -ForegroundColor Green + } + catch { + Write-Warning "Could not call approval endpoint (auth may be needed): $_" + } +} else { + Write-Host "⚠ JWT_TOKEN not set, skipping auth test" -ForegroundColor Yellow +} + +Write-Host "`n✓ All API checks passed" -ForegroundColor Green +``` + +**Usage:** +```powershell +$env:JWT_TOKEN = "your-jwt-token-here" +.\gate3_test_api.ps1 +``` + +--- + +## 5. Monitor Hangfire Jobs (PowerShell) + +**File:** `gate3_monitor_job.ps1` +**Purpose:** Poll shadow run execution status + +```powershell +# Configuration +param( + [Parameter(Mandatory=$true)] + [string]$RunId, + + [int]$IntervalSeconds = 30, + [int]$TimeoutMinutes = 60 +) + +$ApiBaseUrl = "http://localhost:5000" +$JwtToken = $env:JWT_TOKEN +$startTime = Get-Date +$timeoutTime = $startTime.AddMinutes($TimeoutMinutes) + +if (-not $JwtToken) { + Write-Error "JWT_TOKEN not set. Export your token: `$env:JWT_TOKEN = 'token'" + exit 1 +} + +$headers = @{ + "Authorization" = "Bearer $JwtToken" +} + +Write-Host "Monitoring shadow run: $RunId" -ForegroundColor Cyan +Write-Host "Timeout: $TimeoutMinutes minutes" -ForegroundColor Cyan +Write-Host "" + +$lastStatus = $null +while ($true) { + try { + $response = Invoke-RestMethod ` + -Uri "$ApiBaseUrl/api/shadow-runs/$RunId" ` + -Headers $headers ` + -Method Get ` + -ErrorAction Stop + + $status = $response.status + $elapsed = [math]::Round((Get-Date - $startTime).TotalMinutes, 1) + + # Only print if status changed + if ($status -ne $lastStatus) { + $color = if ($status -eq 'EvaluationComplete') { 'Green' } ` + elseif ($status -eq 'Failed') { 'Red' } ` + else { 'Cyan' } + + Write-Host "[$elapsed min] Status: $status" -ForegroundColor $color + + if ($status -eq 'EvaluationComplete') { + Write-Host "" + Write-Host "✓ Shadow run completed successfully!" -ForegroundColor Green + Write-Host "Gates passed: $($response.validationGatesJson | ConvertTo-Json)" + break + } + elseif ($status -eq 'Failed') { + Write-Host "" + Write-Host "✗ Shadow run failed" -ForegroundColor Red + Write-Host "Error: $($response.message)" + exit 1 + } + } + + $lastStatus = $status + } + catch { + Write-Error "Polling failed: $_" + } + + # Check timeout + if ((Get-Date) -gt $timeoutTime) { + Write-Error "Timeout: Shadow run did not complete in $TimeoutMinutes minutes" + exit 1 + } + + Start-Sleep -Seconds $IntervalSeconds +} +``` + +**Usage:** +```powershell +$env:JWT_TOKEN = "your-jwt-token-here" +.\gate3_monitor_job.ps1 -RunId "b2c3d4e5-f6a7-8901-bcde-f12345678901" -IntervalSeconds 30 +``` + +--- + +## 6. Validate Results (SQL) + +**File:** `gate3_validate_results.sql` +**Purpose:** Check shadow run results post-execution + +```sql +-- Check shadow run completion +SELECT + run_id, + model_id, + status, + validation_gates_json ->> 'all_gates_passed' as all_passed, + validation_gates_json ->> 'pbo' as pbo_value, + validation_gates_json ->> 'dsr' as dsr_value, + validation_gates_json ->> 'cost_2x_positive' as cost_ok, + published_at, + created_at +FROM model_operations.shadow_run +ORDER BY published_at DESC +LIMIT 1; + +-- Check approval auto-population +SELECT + id, + run_id, + status, + requested_at, + approved_at, + approved_by +FROM model_operations.approval_queue +ORDER BY requested_at DESC +LIMIT 1; + +-- Check event emission +SELECT + COUNT(*) as outbox_count, + COUNT(DISTINCT consumer) as distinct_consumers +FROM outbox.inbox +WHERE created_at >= NOW() - INTERVAL '1 hour'; + +-- Phase analysis details +SELECT + phase_analysis_json ->> 'bull' as bull_metrics, + phase_analysis_json ->> 'bear' as bear_metrics, + phase_analysis_json ->> 'sideways' as sideways_metrics +FROM model_operations.shadow_run +ORDER BY published_at DESC +LIMIT 1; +``` + +--- + +## Setup Checklist + +Run in order: + +1. **Verify API** — `gate3_test_api.ps1` + - Confirms KArtSell.Host is running + - Checks Hangfire dashboard + +2. **Check Market Data** — `gate3_check_market_data.ps1` + - Verifies KRX API or stub mode ready + +3. **Create Model** — `gate3_create_model.sql` + - Run if no active models exist + - Save returned model ID + +4. **Clean State** — `gate3_clean_state.sql` + - Remove hanging shadow runs + - Clean stale approvals + +5. **Ready for Execution** + - Proceed to GATE_3_EXECUTION_GUIDE.md + - Use model ID from step 3 + - Use date window: 2024-01-02 to 2024-08-31 + +--- + +## Save These Variables + +For use in execution scripts: + +```powershell +# PowerShell +$env:MODEL_ID = "a1b2c3d4-e5f6-7890-abcd-ef1234567890" # From setup +$env:WINDOW_START = "2024-01-02" +$env:WINDOW_END = "2024-08-31" +$env:JWT_TOKEN = "your-bearer-token" +$env:API_BASE_URL = "http://localhost:5000" +``` + +Then reference in scripts via `$env:MODEL_ID`, `$env:JWT_TOKEN`, etc.