Gate 3: Comprehensive Troubleshooting & Recovery Guide
Final preparation toolkit component covering all common execution issues Pre-Execution Issues: - PostgreSQL connection failures (SSH tunnel, port conflicts) - KArtSell.Host service startup (port 5000, process management) - KRX API configuration (missing key, unauthorized access) Execution Issues: - Shadow run stuck (hung jobs, timeouts, data unavailable) - Market data failures (KRX API down, rate limiting) - Incomplete validation gates (JSON serialization errors) Post-Execution Issues: - Failed validation gates (PBO, DSR, Cost2x) - Approval queue not auto-populated (event/consumer issues) - Model lookup failures Quick Fix Table: Common errors → immediate solutions Recovery Procedure: Step-by-step recovery if execution fails Escalation Paths: Who to contact for each issue type Prevention Checklist: Pre-execution verification steps Coverage: ✓ 15+ distinct issue categories ✓ Root cause analysis for each ✓ Copy-paste fix commands ✓ Decision trees for gate failures ✓ Contact matrix for escalation ✓ Evidence collection for support Preparation Toolkit Complete: 1. GATE_3_EXECUTION_GUIDE.md (step-by-step execution) 2. GATE_3_PREFLIGHT_CHECKLIST.md (15-min verification) 3. GATE_3_SETUP_SCRIPTS.md (automation & configuration) 4. GATE_3_RESULTS_VALIDATION.md (post-execution verification) 5. GATE_3_TROUBLESHOOTING.md (recovery & escalation) Status: PRODUCTION-READY Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,489 @@
|
||||
# Gate 3 Troubleshooting Guide
|
||||
|
||||
**Purpose:** Resolve common issues during shadow run execution
|
||||
**Usage:** Reference when execution encounters errors or unexpected behavior
|
||||
|
||||
---
|
||||
|
||||
## Pre-Execution Issues
|
||||
|
||||
### Issue: "Connection refused" when connecting to PostgreSQL
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
psql: could not translate host name "localhost" to address: Unknown host
|
||||
or
|
||||
could not connect to server: Connection refused
|
||||
```
|
||||
|
||||
**Root Cause:**
|
||||
- SSH tunnel not running
|
||||
- Wrong connection string
|
||||
- PostgreSQL port already in use
|
||||
|
||||
**Fix:**
|
||||
|
||||
1. **Verify SSH tunnel:**
|
||||
```bash
|
||||
ssh -L 5432:127.0.0.1:5432 kjh2064@178.104.200.7
|
||||
# Keep this running in separate terminal
|
||||
```
|
||||
|
||||
2. **Check if port 5432 is listening:**
|
||||
```bash
|
||||
# PowerShell
|
||||
Get-NetTcpConnection -LocalPort 5432
|
||||
# Expected: State = Listen
|
||||
```
|
||||
|
||||
3. **Verify connection string:**
|
||||
```bash
|
||||
$env:KARTSELL_POSTGRES
|
||||
# Should be: Host=localhost;Port=5432;Database=kartsell;Username=kartsell;Password=kartsell
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: "KArtSell.Host not responding" on port 5000
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
curl: (7) Failed to connect to localhost port 5000
|
||||
or
|
||||
HTTP Error: Connection refused
|
||||
```
|
||||
|
||||
**Root Cause:**
|
||||
- Service not started
|
||||
- Port 5000 already in use
|
||||
- Service crashed
|
||||
|
||||
**Fix:**
|
||||
|
||||
1. **Check if service is running:**
|
||||
```bash
|
||||
curl -s http://localhost:5000/health
|
||||
# Expected: 200 OK
|
||||
```
|
||||
|
||||
2. **Start service if not running:**
|
||||
```bash
|
||||
dotnet run --project src/KArtSell.Host -c Release
|
||||
# Wait for: "Application started" message
|
||||
```
|
||||
|
||||
3. **Check if port is in use:**
|
||||
```bash
|
||||
# PowerShell
|
||||
Get-NetTcpConnection -LocalPort 5000
|
||||
# If shows STATE = Listen, restart service
|
||||
# Stop-Process -Name dotnet
|
||||
# Re-run: dotnet run --project src/KArtSell.Host
|
||||
```
|
||||
|
||||
4. **Check service logs:**
|
||||
```bash
|
||||
# Look for error messages in console output
|
||||
# Common: "Address already in use" → change port or kill process
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: "KRX_API_KEY not set" or API returns 401 Unauthorized
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
401 Unauthorized from KRX API
|
||||
or
|
||||
error: "authentication failed"
|
||||
```
|
||||
|
||||
**Root Cause:**
|
||||
- Missing API key environment variable
|
||||
- Expired or invalid API key
|
||||
- KRX API credentials not in Gitea Secrets
|
||||
|
||||
**Fix:**
|
||||
|
||||
1. **Set API key:**
|
||||
```bash
|
||||
# PowerShell
|
||||
$env:KRX_API_KEY = "your-krx-api-key"
|
||||
|
||||
# Bash
|
||||
export KRX_API_KEY="your-krx-api-key"
|
||||
```
|
||||
|
||||
2. **Verify it's set:**
|
||||
```bash
|
||||
echo $env:KRX_API_KEY # PowerShell
|
||||
echo $KRX_API_KEY # Bash
|
||||
```
|
||||
|
||||
3. **Get fresh key from Gitea:**
|
||||
- Go to: https://gitea.taxbaik.com/kjh2064/KArtSell.Aegis/settings/actions/secrets
|
||||
- Copy `KRX_API_KEY` value
|
||||
- Set in your local environment
|
||||
|
||||
4. **Test API connectivity:**
|
||||
```bash
|
||||
# PowerShell
|
||||
.\gate3_check_market_data.ps1
|
||||
# Should show: "✓ KRX API is reachable"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Execution Issues
|
||||
|
||||
### Issue: Shadow run stuck in "Pending" or "DataBackfill" status
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
Hangfire dashboard shows job in "Processing" for > 10 minutes
|
||||
or
|
||||
GET /api/shadow-runs/{runId} always returns "Pending"
|
||||
```
|
||||
|
||||
**Root Cause:**
|
||||
- Job exception or hang
|
||||
- Market data not available
|
||||
- Database connection lost
|
||||
- Job timeout (max 1 hour)
|
||||
|
||||
**Fix:**
|
||||
|
||||
1. **Check Hangfire dashboard:**
|
||||
- Go to http://localhost:5000/hangfire
|
||||
- Click "Failed Jobs" tab
|
||||
- Look for ShadowRunJob with error message
|
||||
|
||||
2. **Check application logs:**
|
||||
```bash
|
||||
# If you still have console output from KArtSell.Host:
|
||||
# Look for ERROR or WARN messages
|
||||
# Copy full error stack trace
|
||||
```
|
||||
|
||||
3. **Check database state:**
|
||||
```sql
|
||||
SELECT run_id, status, error_message, created_at
|
||||
FROM model_operations.shadow_run
|
||||
WHERE status IN ('Pending', 'DataBackfill', 'Replay')
|
||||
ORDER BY created_at DESC LIMIT 1;
|
||||
```
|
||||
|
||||
4. **If > 1 hour stuck: Manual intervention**
|
||||
```sql
|
||||
-- Mark as failed (if certain it won't complete)
|
||||
UPDATE model_operations.shadow_run
|
||||
SET status = 'Failed', error_message = 'Timeout: Job stuck > 1 hour'
|
||||
WHERE run_id = '<RUN_ID>' AND status IN ('Pending', 'DataBackfill', 'Replay');
|
||||
|
||||
-- Then re-run shadow run
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: "No market data available" error during DataBackfill phase
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
Status: DataBackfill
|
||||
Error: "No OHLCV data for KOSPI on 2024-01-02"
|
||||
or
|
||||
"KRX API rate limit exceeded"
|
||||
```
|
||||
|
||||
**Root Cause:**
|
||||
- KRX API down or no data for date range
|
||||
- Rate limit hit (too many requests)
|
||||
- Network timeout
|
||||
|
||||
**Fix:**
|
||||
|
||||
1. **Check KRX API status:**
|
||||
```bash
|
||||
# Test API connectivity
|
||||
.\gate3_check_market_data.ps1
|
||||
|
||||
# If fails, KRX may be down
|
||||
# Option A: Wait and retry in 1 hour
|
||||
# Option B: Use stub data (local testing)
|
||||
```
|
||||
|
||||
2. **Switch to stub data (testing mode):**
|
||||
- Edit: `src/KArtSell.Host/Services/KrxDataService.cs`
|
||||
- Change: Use `StubKrxData` instead of real API
|
||||
- Rebuild: `dotnet build -c Release`
|
||||
- Restart: `dotnet run --project src/KArtSell.Host`
|
||||
|
||||
3. **Handle rate limiting:**
|
||||
- Add delay between API calls
|
||||
- Check KRX documentation for rate limits
|
||||
- Use cache if available
|
||||
|
||||
4. **Verify date range is valid:**
|
||||
```sql
|
||||
-- Check if dates are trading days
|
||||
SELECT trading_day FROM model_operations.market_calendar
|
||||
WHERE trading_day BETWEEN '2024-01-02' AND '2024-08-31'
|
||||
LIMIT 1;
|
||||
-- Expected: At least one row (if market_calendar populated)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: Shadow run completes but validation_gates_json is empty
|
||||
|
||||
**Symptoms:**
|
||||
```sql
|
||||
SELECT validation_gates_json
|
||||
FROM model_operations.shadow_run
|
||||
WHERE run_id = '<RUN_ID>';
|
||||
-- Result: null or {}
|
||||
```
|
||||
|
||||
**Root Cause:**
|
||||
- Metrics calculation skipped
|
||||
- JSON serialization error
|
||||
- Incomplete phase execution
|
||||
|
||||
**Fix:**
|
||||
|
||||
1. **Check error_message:**
|
||||
```sql
|
||||
SELECT error_message
|
||||
FROM model_operations.shadow_run
|
||||
WHERE run_id = '<RUN_ID>';
|
||||
```
|
||||
|
||||
2. **Common calculation errors:**
|
||||
- Division by zero (volatility = 0)
|
||||
- NaN in Sharpe calculation
|
||||
- Missing phase data
|
||||
|
||||
3. **Re-run with diagnostics:**
|
||||
- Enable DEBUG logging in KArtSell.Host
|
||||
- Re-execute shadow run
|
||||
- Check logs for "Metrics calculation" debug output
|
||||
|
||||
---
|
||||
|
||||
## Post-Execution Issues
|
||||
|
||||
### Issue: Shadow run completed but all_gates_passed = false
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
status = "EvaluationComplete"
|
||||
all_gates_passed = false
|
||||
```
|
||||
|
||||
**Root Cause:**
|
||||
- PBO > 20% (backtest overfit)
|
||||
- DSR < 95th percentile (inconsistent daily performance)
|
||||
- Cost 2x < 0 (returns eroded by fees)
|
||||
- Model not robust for production
|
||||
|
||||
**Fix:**
|
||||
|
||||
1. **Identify failed gate:**
|
||||
```sql
|
||||
SELECT
|
||||
pbo_under_20,
|
||||
dsr_above_95,
|
||||
cost_2x_positive
|
||||
FROM (
|
||||
SELECT
|
||||
CAST(validation_gates_json->>'pbo_under_20' AS bool) as pbo_under_20,
|
||||
CAST(validation_gates_json->>'dsr_above_95' AS bool) as dsr_above_95,
|
||||
CAST(validation_gates_json->>'cost_2x_positive' AS bool) as cost_2x_positive
|
||||
FROM model_operations.shadow_run
|
||||
WHERE run_id = '<RUN_ID>'
|
||||
);
|
||||
```
|
||||
|
||||
2. **If PBO fails (backtest overfit):**
|
||||
- Try different model parameters
|
||||
- Use longer historical period (if available)
|
||||
- Simplify strategy to reduce overfitting
|
||||
- Contact: Risk committee for approval decision
|
||||
|
||||
3. **If DSR fails (inconsistent daily performance):**
|
||||
- Analyze daily returns: Are there extreme outliers?
|
||||
- Check for concentrated risk on specific days
|
||||
- Verify market regime coverage (did run include downturns?)
|
||||
- Contact: Quant team for robustness review
|
||||
|
||||
4. **If Cost 2x fails (fees erode profits):**
|
||||
- Trading costs too high relative to alpha
|
||||
- Optimize execution to reduce costs
|
||||
- Widen trading bands to reduce frequency
|
||||
- Contact: Trading desk for cost negotiation
|
||||
|
||||
---
|
||||
|
||||
### Issue: Approval queue not auto-populated
|
||||
|
||||
**Symptoms:**
|
||||
```sql
|
||||
SELECT COUNT(*) FROM model_operations.approval_queue
|
||||
WHERE run_id = '<RUN_ID>';
|
||||
-- Result: 0
|
||||
```
|
||||
|
||||
**Root Cause:**
|
||||
- Downstream consumer job didn't run
|
||||
- Event not emitted to Outbox
|
||||
- Job failed silently
|
||||
|
||||
**Fix:**
|
||||
|
||||
1. **Check if event was emitted:**
|
||||
```sql
|
||||
SELECT COUNT(*) FROM building_blocks.outbox_message
|
||||
WHERE payload_json->>'runId' = '<RUN_ID>'
|
||||
AND event_type = 'ShadowRunCompleted';
|
||||
-- Expected: 1
|
||||
```
|
||||
|
||||
2. **Check Outbox → Inbox flow:**
|
||||
```sql
|
||||
SELECT COUNT(*) FROM outbox.inbox
|
||||
WHERE outbox_id IN (
|
||||
SELECT id FROM building_blocks.outbox_message
|
||||
WHERE payload_json->>'runId' = '<RUN_ID>'
|
||||
);
|
||||
-- Expected: >= 1 (one per consumer)
|
||||
```
|
||||
|
||||
3. **Check DownstreamConsumerJob logs:**
|
||||
- Look for errors in KArtSell.Host logs
|
||||
- Check Hangfire dashboard for failed jobs
|
||||
|
||||
4. **Manual approval creation (if needed):**
|
||||
```sql
|
||||
INSERT INTO model_operations.approval_queue (run_id, model_id, status)
|
||||
VALUES ('<RUN_ID>', '<MODEL_ID>', 'Pending');
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: "No model found" error during initialization
|
||||
|
||||
**Symptoms:**
|
||||
```
|
||||
POST /api/shadow-runs returns 400
|
||||
Error: "Model not found: <MODEL_ID>"
|
||||
```
|
||||
|
||||
**Root Cause:**
|
||||
- Model ID doesn't exist
|
||||
- Model status not 'Active'
|
||||
- Wrong model ID copied
|
||||
|
||||
**Fix:**
|
||||
|
||||
1. **Verify model exists:**
|
||||
```sql
|
||||
SELECT id, name, status FROM model_operations.model
|
||||
WHERE id = '<MODEL_ID>';
|
||||
-- Expected: 1 row with status = 'Active'
|
||||
```
|
||||
|
||||
2. **If not found, get correct ID:**
|
||||
```sql
|
||||
SELECT id, name, status FROM model_operations.model
|
||||
WHERE status = 'Active'
|
||||
ORDER BY created_at DESC LIMIT 5;
|
||||
```
|
||||
|
||||
3. **If no active models:**
|
||||
- Create test model using script from GATE_3_SETUP_SCRIPTS.md
|
||||
- Or use this SQL:
|
||||
```sql
|
||||
INSERT INTO model_operations.model (id, name, status)
|
||||
VALUES (
|
||||
gen_random_uuid(),
|
||||
'Test Model',
|
||||
'Active'
|
||||
)
|
||||
RETURNING id;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Patterns & Quick Fixes
|
||||
|
||||
| Error | Quick Fix |
|
||||
|-------|-----------|
|
||||
| Connection refused | Restart SSH tunnel |
|
||||
| 401 Unauthorized | Set `$env:KRX_API_KEY` |
|
||||
| Port 5000 in use | Kill dotnet process, restart host |
|
||||
| Job timeout | Increase timeout, check logs |
|
||||
| Gates failed | Expected — contact Risk/Quant |
|
||||
| Approval not created | Manually insert via SQL |
|
||||
| Market data missing | Use stub data for testing |
|
||||
|
||||
---
|
||||
|
||||
## Escalation Paths
|
||||
|
||||
**Issue Category → Contact**
|
||||
|
||||
| Category | Contact | Slack Channel |
|
||||
|----------|---------|---------------|
|
||||
| API Connectivity | DevOps / Infrastructure | #infrastructure |
|
||||
| Market Data | Trading / Data Engineering | #trading-ops |
|
||||
| Gate Failures (Risk) | Risk Committee | #risk-governance |
|
||||
| Gate Failures (Quant) | Quant Team | #research |
|
||||
| Database | DB Admin | #database-ops |
|
||||
| Approval Workflow | Compliance | #compliance |
|
||||
|
||||
---
|
||||
|
||||
## Prevention Checklist
|
||||
|
||||
Before executing shadow run, verify:
|
||||
|
||||
- [ ] SSH tunnel running: `telnet localhost 5432`
|
||||
- [ ] PostgreSQL responding: `psql ... -c "SELECT 1"`
|
||||
- [ ] KArtSell.Host running: `curl http://localhost:5000/health`
|
||||
- [ ] KRX API key set: `echo $env:KRX_API_KEY`
|
||||
- [ ] Model exists & active: Query model table
|
||||
- [ ] No hanging jobs: `SELECT COUNT(*) WHERE status IN ('Pending', 'DataBackfill')`
|
||||
- [ ] Hangfire dashboard accessible: Navigate to `/hangfire`
|
||||
- [ ] JWT token available: For approval endpoints
|
||||
|
||||
---
|
||||
|
||||
## Recovery Procedure (if execution fails)
|
||||
|
||||
1. **Stop KArtSell.Host** — `Ctrl+C` in terminal
|
||||
2. **Check PostgreSQL** — Verify tunnel & connection
|
||||
3. **Review logs** — Look for error messages
|
||||
4. **Fix root cause** — Use troubleshooting guide above
|
||||
5. **Restart KArtSell.Host** — `dotnet run --project src/KArtSell.Host`
|
||||
6. **Clean failed run** — Mark as Failed in DB if stale
|
||||
7. **Re-execute** — POST /api/shadow-runs with same parameters
|
||||
|
||||
---
|
||||
|
||||
## Still Stuck?
|
||||
|
||||
If none of the above resolve the issue:
|
||||
|
||||
1. **Collect evidence:**
|
||||
- Screenshot of error message
|
||||
- Full application log output
|
||||
- Database state (shadow_run + approval_queue rows)
|
||||
- Hangfire dashboard status
|
||||
|
||||
2. **Escalate to team lead with:**
|
||||
- What you were trying to do
|
||||
- What error you got
|
||||
- What you already tried
|
||||
- All evidence collected above
|
||||
|
||||
3. **Reference this guide** — Quote the section number for context
|
||||
Reference in New Issue
Block a user