KArtSell.Aegis/GATE_3_TROUBLESHOOTING.md

# Gate 3 Troubleshooting Guide

**Purpose:** Resolve common issues during shadow run execution  
**Usage:** Reference when execution encounters errors or unexpected behavior

---

## Pre-Execution Issues

### Issue: "Connection refused" when connecting to PostgreSQL

**Symptoms:**
```
psql: could not translate host name "localhost" to address: Unknown host
or
could not connect to server: Connection refused
```

**Root Cause:**
- SSH tunnel not running
- Wrong connection string
- PostgreSQL port already in use

**Fix:**

1. **Verify SSH tunnel:**
   ```bash
   ssh -L 5432:127.0.0.1:5432 kjh2064@178.104.200.7
   # Keep this running in separate terminal
   ```

2. **Check if port 5432 is listening:**
   ```bash
   # PowerShell
   Get-NetTcpConnection -LocalPort 5432
   # Expected: State = Listen
   ```

3. **Verify connection string:**
   ```bash
   $env:KARTSELL_POSTGRES
   # Should be: Host=localhost;Port=5432;Database=kartsell;Username=kartsell;Password=kartsell
   ```

---

### Issue: "KArtSell.Host not responding" on port 5000

**Symptoms:**
```
curl: (7) Failed to connect to localhost port 5000
or
HTTP Error: Connection refused
```

**Root Cause:**
- Service not started
- Port 5000 already in use
- Service crashed

**Fix:**

1. **Check if service is running:**
   ```bash
   curl -s http://localhost:5000/health
   # Expected: 200 OK
   ```

2. **Start service if not running:**
   ```bash
   dotnet run --project src/KArtSell.Host -c Release
   # Wait for: "Application started" message
   ```

3. **Check if port is in use:**
   ```bash
   # PowerShell
   Get-NetTcpConnection -LocalPort 5000
   # If shows STATE = Listen, restart service
   # Stop-Process -Name dotnet
   # Re-run: dotnet run --project src/KArtSell.Host
   ```

4. **Check service logs:**
   ```bash
   # Look for error messages in console output
   # Common: "Address already in use" → change port or kill process
   ```

---

### Issue: "KRX_API_KEY not set" or API returns 401 Unauthorized

**Symptoms:**
```
401 Unauthorized from KRX API
or
error: "authentication failed"
```

**Root Cause:**
- Missing API key environment variable
- Expired or invalid API key
- KRX API credentials not in Gitea Secrets

**Fix:**

1. **Set API key:**
   ```bash
   # PowerShell
   $env:KRX_API_KEY = "your-krx-api-key"
   
   # Bash
   export KRX_API_KEY="your-krx-api-key"
   ```

2. **Verify it's set:**
   ```bash
   echo $env:KRX_API_KEY  # PowerShell
   echo $KRX_API_KEY      # Bash
   ```

3. **Get fresh key from Gitea:**
   - Go to: https://gitea.taxbaik.com/kjh2064/KArtSell.Aegis/settings/actions/secrets
   - Copy `KRX_API_KEY` value
   - Set in your local environment

4. **Test API connectivity:**
   ```bash
   # PowerShell
   .\gate3_check_market_data.ps1
   # Should show: "✓ KRX API is reachable"
   ```

---

## Execution Issues

### Issue: Shadow run stuck in "Pending" or "DataBackfill" status

**Symptoms:**
```
Hangfire dashboard shows job in "Processing" for > 10 minutes
or
GET /api/shadow-runs/{runId} always returns "Pending"
```

**Root Cause:**
- Job exception or hang
- Market data not available
- Database connection lost
- Job timeout (max 1 hour)

**Fix:**

1. **Check Hangfire dashboard:**
   - Go to http://localhost:5000/hangfire
   - Click "Failed Jobs" tab
   - Look for ShadowRunJob with error message

2. **Check application logs:**
   ```bash
   # If you still have console output from KArtSell.Host:
   # Look for ERROR or WARN messages
   # Copy full error stack trace
   ```

3. **Check database state:**
   ```sql
   SELECT run_id, status, error_message, created_at
   FROM model_operations.shadow_run
   WHERE status IN ('Pending', 'DataBackfill', 'Replay')
   ORDER BY created_at DESC LIMIT 1;
   ```

4. **If > 1 hour stuck: Manual intervention**
   ```sql
   -- Mark as failed (if certain it won't complete)
   UPDATE model_operations.shadow_run
   SET status = 'Failed', error_message = 'Timeout: Job stuck > 1 hour'
   WHERE run_id = '<RUN_ID>' AND status IN ('Pending', 'DataBackfill', 'Replay');
   
   -- Then re-run shadow run
   ```

---

### Issue: "No market data available" error during DataBackfill phase

**Symptoms:**
```
Status: DataBackfill
Error: "No OHLCV data for KOSPI on 2024-01-02"
or
"KRX API rate limit exceeded"
```

**Root Cause:**
- KRX API down or no data for date range
- Rate limit hit (too many requests)
- Network timeout

**Fix:**

1. **Check KRX API status:**
   ```bash
   # Test API connectivity
   .\gate3_check_market_data.ps1
   
   # If fails, KRX may be down
   # Option A: Wait and retry in 1 hour
   # Option B: Use stub data (local testing)
   ```

2. **Switch to stub data (testing mode):**
   - Edit: `src/KArtSell.Host/Services/KrxDataService.cs`
   - Change: Use `StubKrxData` instead of real API
   - Rebuild: `dotnet build -c Release`
   - Restart: `dotnet run --project src/KArtSell.Host`

3. **Handle rate limiting:**
   - Add delay between API calls
   - Check KRX documentation for rate limits
   - Use cache if available

4. **Verify date range is valid:**
   ```sql
   -- Check if dates are trading days
   SELECT trading_day FROM model_operations.market_calendar
   WHERE trading_day BETWEEN '2024-01-02' AND '2024-08-31'
   LIMIT 1;
   -- Expected: At least one row (if market_calendar populated)
   ```

---

### Issue: Shadow run completes but validation_gates_json is empty

**Symptoms:**
```sql
SELECT validation_gates_json
FROM model_operations.shadow_run
WHERE run_id = '<RUN_ID>';
-- Result: null or {}
```

**Root Cause:**
- Metrics calculation skipped
- JSON serialization error
- Incomplete phase execution

**Fix:**

1. **Check error_message:**
   ```sql
   SELECT error_message
   FROM model_operations.shadow_run
   WHERE run_id = '<RUN_ID>';
   ```

2. **Common calculation errors:**
   - Division by zero (volatility = 0)
   - NaN in Sharpe calculation
   - Missing phase data

3. **Re-run with diagnostics:**
   - Enable DEBUG logging in KArtSell.Host
   - Re-execute shadow run
   - Check logs for "Metrics calculation" debug output

---

## Post-Execution Issues

### Issue: Shadow run completed but all_gates_passed = false

**Symptoms:**
```
status = "EvaluationComplete"
all_gates_passed = false
```

**Root Cause:**
- PBO > 20% (backtest overfit)
- DSR < 95th percentile (inconsistent daily performance)
- Cost 2x < 0 (returns eroded by fees)
- Model not robust for production

**Fix:**

1. **Identify failed gate:**
   ```sql
   SELECT 
     pbo_under_20,
     dsr_above_95,
     cost_2x_positive
   FROM (
     SELECT 
       CAST(validation_gates_json->>'pbo_under_20' AS bool) as pbo_under_20,
       CAST(validation_gates_json->>'dsr_above_95' AS bool) as dsr_above_95,
       CAST(validation_gates_json->>'cost_2x_positive' AS bool) as cost_2x_positive
     FROM model_operations.shadow_run
     WHERE run_id = '<RUN_ID>'
   );
   ```

2. **If PBO fails (backtest overfit):**
   - Try different model parameters
   - Use longer historical period (if available)
   - Simplify strategy to reduce overfitting
   - Contact: Risk committee for approval decision

3. **If DSR fails (inconsistent daily performance):**
   - Analyze daily returns: Are there extreme outliers?
   - Check for concentrated risk on specific days
   - Verify market regime coverage (did run include downturns?)
   - Contact: Quant team for robustness review

4. **If Cost 2x fails (fees erode profits):**
   - Trading costs too high relative to alpha
   - Optimize execution to reduce costs
   - Widen trading bands to reduce frequency
   - Contact: Trading desk for cost negotiation

---

### Issue: Approval queue not auto-populated

**Symptoms:**
```sql
SELECT COUNT(*) FROM model_operations.approval_queue
WHERE run_id = '<RUN_ID>';
-- Result: 0
```

**Root Cause:**
- Downstream consumer job didn't run
- Event not emitted to Outbox
- Job failed silently

**Fix:**

1. **Check if event was emitted:**
   ```sql
   SELECT COUNT(*) FROM building_blocks.outbox_message
   WHERE payload_json->>'runId' = '<RUN_ID>'
   AND event_type = 'ShadowRunCompleted';
   -- Expected: 1
   ```

2. **Check Outbox → Inbox flow:**
   ```sql
   SELECT COUNT(*) FROM outbox.inbox
   WHERE outbox_id IN (
     SELECT id FROM building_blocks.outbox_message
     WHERE payload_json->>'runId' = '<RUN_ID>'
   );
   -- Expected: >= 1 (one per consumer)
   ```

3. **Check DownstreamConsumerJob logs:**
   - Look for errors in KArtSell.Host logs
   - Check Hangfire dashboard for failed jobs

4. **Manual approval creation (if needed):**
   ```sql
   INSERT INTO model_operations.approval_queue (run_id, model_id, status)
   VALUES ('<RUN_ID>', '<MODEL_ID>', 'Pending');
   ```

---

### Issue: "No model found" error during initialization

**Symptoms:**
```
POST /api/shadow-runs returns 400
Error: "Model not found: <MODEL_ID>"
```

**Root Cause:**
- Model ID doesn't exist
- Model status not 'Active'
- Wrong model ID copied

**Fix:**

1. **Verify model exists:**
   ```sql
   SELECT id, name, status FROM model_operations.model
   WHERE id = '<MODEL_ID>';
   -- Expected: 1 row with status = 'Active'
   ```

2. **If not found, get correct ID:**
   ```sql
   SELECT id, name, status FROM model_operations.model
   WHERE status = 'Active'
   ORDER BY created_at DESC LIMIT 5;
   ```

3. **If no active models:**
   - Create test model using script from GATE_3_SETUP_SCRIPTS.md
   - Or use this SQL:
   ```sql
   INSERT INTO model_operations.model (id, name, status)
   VALUES (
     gen_random_uuid(),
     'Test Model',
     'Active'
   )
   RETURNING id;
   ```

---

## Common Patterns & Quick Fixes

| Error | Quick Fix |
|-------|-----------|
| Connection refused | Restart SSH tunnel |
| 401 Unauthorized | Set `$env:KRX_API_KEY` |
| Port 5000 in use | Kill dotnet process, restart host |
| Job timeout | Increase timeout, check logs |
| Gates failed | Expected — contact Risk/Quant |
| Approval not created | Manually insert via SQL |
| Market data missing | Use stub data for testing |

---

## Escalation Paths

**Issue Category → Contact**

| Category | Contact | Slack Channel |
|----------|---------|---------------|
| API Connectivity | DevOps / Infrastructure | #infrastructure |
| Market Data | Trading / Data Engineering | #trading-ops |
| Gate Failures (Risk) | Risk Committee | #risk-governance |
| Gate Failures (Quant) | Quant Team | #research |
| Database | DB Admin | #database-ops |
| Approval Workflow | Compliance | #compliance |

---

## Prevention Checklist

Before executing shadow run, verify:

- [ ] SSH tunnel running: `telnet localhost 5432`
- [ ] PostgreSQL responding: `psql ... -c "SELECT 1"`
- [ ] KArtSell.Host running: `curl http://localhost:5000/health`
- [ ] KRX API key set: `echo $env:KRX_API_KEY`
- [ ] Model exists & active: Query model table
- [ ] No hanging jobs: `SELECT COUNT(*) WHERE status IN ('Pending', 'DataBackfill')`
- [ ] Hangfire dashboard accessible: Navigate to `/hangfire`
- [ ] JWT token available: For approval endpoints

---

## Recovery Procedure (if execution fails)

1. **Stop KArtSell.Host** — `Ctrl+C` in terminal
2. **Check PostgreSQL** — Verify tunnel & connection
3. **Review logs** — Look for error messages
4. **Fix root cause** — Use troubleshooting guide above
5. **Restart KArtSell.Host** — `dotnet run --project src/KArtSell.Host`
6. **Clean failed run** — Mark as Failed in DB if stale
7. **Re-execute** — POST /api/shadow-runs with same parameters

---

## Still Stuck?

If none of the above resolve the issue:

1. **Collect evidence:**
   - Screenshot of error message
   - Full application log output
   - Database state (shadow_run + approval_queue rows)
   - Hangfire dashboard status

2. **Escalate to team lead with:**
   - What you were trying to do
   - What error you got
   - What you already tried
   - All evidence collected above

3. **Reference this guide** — Quote the section number for context