KArtSell.Aegis/GATE_3_TROUBLESHOOTING.md

Gate 3 Troubleshooting Guide

Purpose: Resolve common issues during shadow run execution
Usage: Reference when execution encounters errors or unexpected behavior

---

Pre-Execution Issues

Issue: "Connection refused" when connecting to PostgreSQL

Symptoms:

psql: could not translate host name "localhost" to address: Unknown host
or
could not connect to server: Connection refused

Root Cause:

• SSH tunnel not running
• Wrong connection string
• PostgreSQL port already in use

Fix:

1. Verify SSH tunnel:

   ssh -L 5432:127.0.0.1:5432 kjh2064@178.104.200.7
   # Keep this running in separate terminal
   

2. Check if port 5432 is listening:

   # PowerShell
   Get-NetTcpConnection -LocalPort 5432
   # Expected: State = Listen
   

3. Verify connection string:

   $env:KARTSELL_POSTGRES
   # Should be: Host=localhost;Port=5432;Database=kartsell;Username=kartsell;Password=kartsell
   

---

Issue: "KArtSell.Host not responding" on port 5000

Symptoms:

curl: (7) Failed to connect to localhost port 5000
or
HTTP Error: Connection refused

Root Cause:

• Service not started
• Port 5000 already in use
• Service crashed

Fix:

1. Check if service is running:

   curl -s http://localhost:5000/health
   # Expected: 200 OK
   

2. Start service if not running:

   dotnet run --project src/KArtSell.Host -c Release
   # Wait for: "Application started" message
   

3. Check if port is in use:

   # PowerShell
   Get-NetTcpConnection -LocalPort 5000
   # If shows STATE = Listen, restart service
   # Stop-Process -Name dotnet
   # Re-run: dotnet run --project src/KArtSell.Host
   

4. Check service logs:

   # Look for error messages in console output
   # Common: "Address already in use" → change port or kill process
   

---

Issue: "KRX_API_KEY not set" or API returns 401 Unauthorized

Symptoms:

401 Unauthorized from KRX API
or
error: "authentication failed"

Root Cause:

• Missing API key environment variable
• Expired or invalid API key
• KRX API credentials not in Gitea Secrets

Fix:

1. Set API key:

   # PowerShell
   $env:KRX_API_KEY = "your-krx-api-key"
   
   # Bash
   export KRX_API_KEY="your-krx-api-key"
   

2. Verify it's set:

   echo $env:KRX_API_KEY  # PowerShell
   echo $KRX_API_KEY      # Bash
   

3. Get fresh key from Gitea:

   • Go to: https://gitea.taxbaik.com/kjh2064/KArtSell.Aegis/settings/actions/secrets
   • Copy KRX_API_KEY value
   • Set in your local environment

4. Test API connectivity:

   # PowerShell
   .\gate3_check_market_data.ps1
   # Should show: "✓ KRX API is reachable"
   

---

Execution Issues

Issue: Shadow run stuck in "Pending" or "DataBackfill" status

Symptoms:

Hangfire dashboard shows job in "Processing" for > 10 minutes
or
GET /api/shadow-runs/{runId} always returns "Pending"

Root Cause:

• Job exception or hang
• Market data not available
• Database connection lost
• Job timeout (max 1 hour)

Fix:

1. Check Hangfire dashboard:

   • Go to http://localhost:5000/hangfire
   • Click "Failed Jobs" tab
   • Look for ShadowRunJob with error message

2. Check application logs:

   # If you still have console output from KArtSell.Host:
   # Look for ERROR or WARN messages
   # Copy full error stack trace
   

3. Check database state:

   SELECT run_id, status, error_message, created_at
   FROM model_operations.shadow_run
   WHERE status IN ('Pending', 'DataBackfill', 'Replay')
   ORDER BY created_at DESC LIMIT 1;
   

4. If > 1 hour stuck: Manual intervention

   -- Mark as failed (if certain it won't complete)
   UPDATE model_operations.shadow_run
   SET status = 'Failed', error_message = 'Timeout: Job stuck > 1 hour'
   WHERE run_id = '<RUN_ID>' AND status IN ('Pending', 'DataBackfill', 'Replay');
   
   -- Then re-run shadow run
   

---

Issue: "No market data available" error during DataBackfill phase

Symptoms:

Status: DataBackfill
Error: "No OHLCV data for KOSPI on 2024-01-02"
or
"KRX API rate limit exceeded"

Root Cause:

• KRX API down or no data for date range
• Rate limit hit (too many requests)
• Network timeout

Fix:

1. Check KRX API status:

   # Test API connectivity
   .\gate3_check_market_data.ps1
   
   # If fails, KRX may be down
   # Option A: Wait and retry in 1 hour
   # Option B: Use stub data (local testing)
   

2. Switch to stub data (testing mode):

   • Edit: src/KArtSell.Host/Services/KrxDataService.cs
   • Change: Use StubKrxData instead of real API
   • Rebuild: dotnet build -c Release
   • Restart: dotnet run --project src/KArtSell.Host

3. Handle rate limiting:

   • Add delay between API calls
   • Check KRX documentation for rate limits
   • Use cache if available

4. Verify date range is valid:

   -- Check if dates are trading days
   SELECT trading_day FROM model_operations.market_calendar
   WHERE trading_day BETWEEN '2024-01-02' AND '2024-08-31'
   LIMIT 1;
   -- Expected: At least one row (if market_calendar populated)
   

---

Issue: Shadow run completes but validation_gates_json is empty

Symptoms:

SELECT validation_gates_json
FROM model_operations.shadow_run
WHERE run_id = '<RUN_ID>';
-- Result: null or {}

Root Cause:

• Metrics calculation skipped
• JSON serialization error
• Incomplete phase execution

Fix:

1. Check error_message:

   SELECT error_message
   FROM model_operations.shadow_run
   WHERE run_id = '<RUN_ID>';
   

2. Common calculation errors:

   • Division by zero (volatility = 0)
   • NaN in Sharpe calculation
   • Missing phase data

3. Re-run with diagnostics:

   • Enable DEBUG logging in KArtSell.Host
   • Re-execute shadow run
   • Check logs for "Metrics calculation" debug output

---

Post-Execution Issues

Issue: Shadow run completed but all_gates_passed = false

Symptoms:

status = "EvaluationComplete"
all_gates_passed = false

Root Cause:

• PBO > 20% (backtest overfit)
• DSR < 95th percentile (inconsistent daily performance)
• Cost 2x < 0 (returns eroded by fees)
• Model not robust for production

Fix:

1. Identify failed gate:

   SELECT 
     pbo_under_20,
     dsr_above_95,
     cost_2x_positive
   FROM (
     SELECT 
       CAST(validation_gates_json->>'pbo_under_20' AS bool) as pbo_under_20,
       CAST(validation_gates_json->>'dsr_above_95' AS bool) as dsr_above_95,
       CAST(validation_gates_json->>'cost_2x_positive' AS bool) as cost_2x_positive
     FROM model_operations.shadow_run
     WHERE run_id = '<RUN_ID>'
   );
   

2. If PBO fails (backtest overfit):

   • Try different model parameters
   • Use longer historical period (if available)
   • Simplify strategy to reduce overfitting
   • Contact: Risk committee for approval decision

3. If DSR fails (inconsistent daily performance):

   • Analyze daily returns: Are there extreme outliers?
   • Check for concentrated risk on specific days
   • Verify market regime coverage (did run include downturns?)
   • Contact: Quant team for robustness review

4. If Cost 2x fails (fees erode profits):

   • Trading costs too high relative to alpha
   • Optimize execution to reduce costs
   • Widen trading bands to reduce frequency
   • Contact: Trading desk for cost negotiation

---

Issue: Approval queue not auto-populated

Symptoms:

SELECT COUNT(*) FROM model_operations.approval_queue
WHERE run_id = '<RUN_ID>';
-- Result: 0

Root Cause:

• Downstream consumer job didn't run
• Event not emitted to Outbox
• Job failed silently

Fix:

1. Check if event was emitted:

   SELECT COUNT(*) FROM building_blocks.outbox_message
   WHERE payload_json->>'runId' = '<RUN_ID>'
   AND event_type = 'ShadowRunCompleted';
   -- Expected: 1
   

2. Check Outbox → Inbox flow:

   SELECT COUNT(*) FROM outbox.inbox
   WHERE outbox_id IN (
     SELECT id FROM building_blocks.outbox_message
     WHERE payload_json->>'runId' = '<RUN_ID>'
   );
   -- Expected: >= 1 (one per consumer)
   

3. Check DownstreamConsumerJob logs:

   • Look for errors in KArtSell.Host logs
   • Check Hangfire dashboard for failed jobs

4. Manual approval creation (if needed):

   INSERT INTO model_operations.approval_queue (run_id, model_id, status)
   VALUES ('<RUN_ID>', '<MODEL_ID>', 'Pending');
   

---

Issue: "No model found" error during initialization

Symptoms:

POST /api/shadow-runs returns 400
Error: "Model not found: <MODEL_ID>"

Root Cause:

• Model ID doesn't exist
• Model status not 'Active'
• Wrong model ID copied

Fix:

1. Verify model exists:

   SELECT id, name, status FROM model_operations.model
   WHERE id = '<MODEL_ID>';
   -- Expected: 1 row with status = 'Active'
   

2. If not found, get correct ID:

   SELECT id, name, status FROM model_operations.model
   WHERE status = 'Active'
   ORDER BY created_at DESC LIMIT 5;
   

3. If no active models:

   • Create test model using script from GATE_3_SETUP_SCRIPTS.md
   • Or use this SQL:

   INSERT INTO model_operations.model (id, name, status)
   VALUES (
     gen_random_uuid(),
     'Test Model',
     'Active'
   )
   RETURNING id;
   

---

Common Patterns & Quick Fixes

Error	Quick Fix
Connection refused	Restart SSH tunnel
401 Unauthorized	Set $env:KRX_API_KEY
Port 5000 in use	Kill dotnet process, restart host
Job timeout	Increase timeout, check logs
Gates failed	Expected — contact Risk/Quant
Approval not created	Manually insert via SQL
Market data missing	Use stub data for testing

---

Escalation Paths

Issue Category → Contact

Category	Contact	Slack Channel
API Connectivity	DevOps / Infrastructure	#infrastructure
Market Data	Trading / Data Engineering	#trading-ops
Gate Failures (Risk)	Risk Committee	#risk-governance
Gate Failures (Quant)	Quant Team	#research
Database	DB Admin	#database-ops
Approval Workflow	Compliance	#compliance

---

Prevention Checklist

Before executing shadow run, verify:

• SSH tunnel running: telnet localhost 5432
• PostgreSQL responding: psql ... -c "SELECT 1"
• KArtSell.Host running: curl http://localhost:5000/health
• KRX API key set: echo $env:KRX_API_KEY
• Model exists & active: Query model table
• No hanging jobs: SELECT COUNT(*) WHERE status IN ('Pending', 'DataBackfill')
• Hangfire dashboard accessible: Navigate to /hangfire
• JWT token available: For approval endpoints

---

Recovery Procedure (if execution fails)

1. Stop KArtSell.Host — Ctrl+C in terminal
2. Check PostgreSQL — Verify tunnel & connection
3. Review logs — Look for error messages
4. Fix root cause — Use troubleshooting guide above
5. Restart KArtSell.Host — dotnet run --project src/KArtSell.Host
6. Clean failed run — Mark as Failed in DB if stale
7. Re-execute — POST /api/shadow-runs with same parameters

---

Still Stuck?

If none of the above resolve the issue:

1. Collect evidence:

   • Screenshot of error message
   • Full application log output
   • Database state (shadow_run + approval_queue rows)
   • Hangfire dashboard status

2. Escalate to team lead with:

   • What you were trying to do
   • What error you got
   • What you already tried
   • All evidence collected above

3. Reference this guide — Quote the section number for context