KArtSell.Aegis/docs/operational-runbook.md

# K-ArtSell Aegis v16.0 Operational Runbook

**Purpose:** Decision tree + resolution steps for common incidents  
**Governance:** AGENTS.md v16.0 "Safety & Reliability" (Criterion #10)  
**Last Updated:** 2026-08-04

---

## Table of Contents

1. [Quick Reference](#quick-reference)
2. [Incident Classification](#incident-classification)
3. [Common Scenarios & Resolutions](#common-scenarios--resolutions)
4. [Escalation Path](#escalation-path)
5. [Post-Incident Review](#post-incident-review)

---

## Quick Reference

| Symptom | Root Cause | Resolution | Time |
|---------|-----------|-----------|------|
| High API latency (> 2s p99) | DB query backlog | Scale connections or optimize slow queries | 5-15 min |
| All requests return 403 | Auth provider misconfigured | Check ASPNETCORE_ENVIRONMENT, redeploy | 10 min |
| Hangfire jobs stuck | Distributed lock timeout | Delete stale locks from DB | 3 min |
| Outbox/Inbox deadlock | Concurrent writes collision | Trigger manual OutboxPollerJob | 5 min |
| Memory leak (usage > 1GB) | Unfreed objects in graph | Graceful restart + drain queue | 20 min |
| DB connection pool exhausted | Max connections reached | Increase pool size or kill idle connections | 10 min |

---

## Incident Classification

### By Severity

**🔴 CRITICAL (Page on-call immediately)**
- All users cannot access system (Host down, Auth failed)
- Data corruption or loss
- Security breach (credentials exposed, unauthorized access)
- Revenue-impacting transactions failing

**🟠 HIGH (Start work within 15 minutes)**
- Subset of users affected (single queue stuck)
- Degraded performance (p99 > 5s)
- Data quality issue (DQ jobs accumulating)
- Non-critical feature unavailable

**🟡 MEDIUM (Start work within 1 hour)**
- Single job failing repeatedly
- Increased error rate (but < 1%)
- Observability gap (dashboard not updating)
- Non-critical background task delayed

**🟢 LOW (Schedule in next sprint)**
- Code improvements (tech debt)
- Documentation updates
- Performance optimization (non-critical path)

---

## Common Scenarios & Resolutions

### Scenario 1: High API Response Time (CRITICAL/HIGH)

**Detection:**
- Monitoring alert: `p99_latency > 2s`
- User complaint: "System is slow"
- Hangfire queue depth > 1000 jobs

**Decision Tree:**

```
Is Host running?
├─ NO  → Restart Host (Scenario 7)
├─ YES → Is DB reachable?
    ├─ NO  → SSH tunnel issue (Scenario 5)
    ├─ YES → Check queue depth
        ├─ Depth > 1000 → Scale Hangfire workers or analyze slowest queries
        ├─ Depth < 100  → Analyze application memory/CPU
```

**Resolution Steps:**

1. **Quick Health Check (1 min)**
   ```bash
   curl http://127.0.0.1:5002/health
   psql -U kartsell -d kartsell -c "SELECT now()" # DB latency
   ```

2. **Check Queue Depth (1 min)**
   ```sql
   SELECT queue, COUNT(*) FROM hangfire.job WHERE state_name='Enqueued' GROUP BY queue;
   ```

3. **Identify Slow Queries (3 min)**
   ```sql
   SELECT query, calls, mean_time FROM pg_stat_statements 
   WHERE mean_time > 100 ORDER BY mean_time DESC LIMIT 10;
   ```

4. **Scale Hangfire Workers (5 min)**
   - Edit `appsettings.Production.json`: `"WorkerCount": 16` (from 8)
   - Restart Host
   - Monitor: Should process queue faster

5. **Optimize Slow Query (10-30 min)**
   - Run `EXPLAIN ANALYZE` on slowest query
   - Check for missing indexes: `SELECT * FROM pg_indexes WHERE tablename='...'`
   - Add index if needed: `CREATE INDEX idx_... ON table(...)`
   - Test performance: `SELECT ... EXPLAIN ANALYZE`

**Success Criteria:** p99_latency < 2s, queue depth < 100

---

### Scenario 2: Authentication Failures (CRITICAL)

**Detection:**
- HTTP 403/404 responses on valid endpoints
- Error log: "FailClosedAuthenticationHandler denies request"
- All users affected

**Root Cause Analysis:**

```
Is ASPNETCORE_ENVIRONMENT correct?
├─ Release mode but missing auth config  → Add FailClosedAuthenticationHandler config
├─ Development mode (wrong for prod)     → Redeploy with Release
├─ API key format incorrect              → Check Gitea Secrets vs. code
```

**Resolution Steps:**

1. **Verify Environment (1 min)**
   ```powershell
   # Check running process
   Get-Process -Name dotnet | Select-Object CommandLine
   # Should show: --configuration Release
   ```

2. **Check Auth Configuration (2 min)**
   ```bash
   cat src/KArtSell.Host/appsettings.Production.json | grep -A 10 "Authentication"
   ```

3. **Verify API Key Format (2 min)**
   - Check Gitea Secrets: https://gitea.taxbaik.com/kjh2064/KArtSell.Aegis/settings/actions/secrets
   - Expected: `KRX_OPENAPI=<actual-key>` (not `stub-key-for-testing`)

4. **Temporary Workaround (1 min)**
   ```powershell
   # If stuck: Start in Development mode temporarily
   $env:ASPNETCORE_ENVIRONMENT = "Development"
   dotnet run --project src/KArtSell.Host --configuration Debug
   # This uses DevelopmentHeaderAuthenticationHandler (accepts X-KArtSell-User header)
   ```

5. **Permanent Fix (5 min)**
   - Update `appsettings.Production.json` with correct auth provider
   - Redeploy with Release configuration

**Success Criteria:** GET /api/health returns 200

---

### Scenario 3: Hangfire Job Stuck in "Scheduled" State (HIGH)

**Detection:**
- Monitoring: Jobs in "Scheduled" state > 5 minutes
- Hangfire dashboard: Red warning on recurring job
- Log: "Recurring job registration timeout"

**Root Cause:** Distributed lock held too long (network latency, DB contention)

**Decision Tree:**

```
Is the Hangfire server running?
├─ NO  → Start Host
├─ YES → Is there a distributed lock?
    ├─ NO  → Job definition error (check code)
    ├─ YES → Is lock stale?
        ├─ YES (> 10 min)  → Delete lock (Scenario 3 Resolution)
        ├─ NO (< 5 min)    → Wait or increase timeout (DEBT-015)
```

**Resolution Steps:**

1. **Verify Hangfire Server (1 min)**
   ```sql
   SELECT name, last_heartbeat, worker_count FROM hangfire.server;
   ```
   - If empty: Host not running (Scenario 7)
   - If stale: Server crashed, restart Host

2. **Check Distributed Lock (1 min)**
   ```sql
   SELECT * FROM hangfire.lock WHERE Key LIKE 'Recurring:%' ORDER BY TimeOut DESC;
   ```

3. **Identify Stale Lock (1 min)**
   - If `TimeOut > CURRENT_TIMESTAMP` by > 10 minutes → lock is stale
   - This prevents job from dequeuing

4. **Delete Stale Lock (1 min)**
   ```sql
   DELETE FROM hangfire.lock WHERE Key = 'Recurring:JobId' AND TimeOut < CURRENT_TIMESTAMP - INTERVAL '5 minutes';
   ```

5. **Monitor Next Run (2 min)**
   - Job should dequeue within 15 seconds
   - Check Hangfire dashboard: Job should move to "Processing"

**Prevention:** DEBT-015 already applied (consistent timeout handling)

**Success Criteria:** Job processes immediately after lock removal

---

### Scenario 4: Outbox/Inbox Deadlock (HIGH)

**Detection:**
- Event processing stalled
- `SELECT COUNT(*) FROM outbox.outbox WHERE published_at IS NULL` > 100
- Inbox consumers not progressing (check logs)

**Root Cause:** Concurrent writes to inbox, or published event not being consumed

**Resolution Steps:**

1. **Assess Situation (2 min)**
   ```sql
   SELECT COUNT(*) as unpublished FROM outbox.outbox WHERE published_at IS NULL;
   SELECT COUNT(*) as unprocessed FROM inbox.inbox WHERE processed_at IS NULL;
   ```

2. **Check Outbox Poller Logs (3 min)**
   ```bash
   grep -i "OutboxPollerJob" host.log | tail -20
   # Look for errors: "Duplicate event", "Database timeout", "Constraint violation"
   ```

3. **Option A: Trigger Manual Poll (2 min)**
   ```bash
   # If queue is small (< 1000), manually trigger:
   curl -X POST http://127.0.0.1:5002/internal/outbox-poll \
     -H "X-KArtSell-User: operator" -H "X-KArtSell-Role: Admin"
   ```

4. **Option B: Drain Stuck Events (5 min)**
   ```sql
   -- Mark old unpublished events as published (if safe)
   UPDATE outbox.outbox 
   SET published_at = now() 
   WHERE published_at IS NULL AND created_at < now() - INTERVAL '1 hour';
   ```

5. **Monitor Recovery (5 min)**
   - Inbox consumer should resume
   - Check: `SELECT COUNT(*) FROM inbox.inbox WHERE processed_at IS NULL`
   - Should decrease over time

**Success Criteria:** All outbox events published, inbox processing resumes

---

### Scenario 5: SSH Tunnel Disconnected (CRITICAL)

**Detection:**
- Connection timeout on DB queries
- Error: "Connection refused: localhost:5432"
- Hangfire jobs failing with DB connection errors

**Resolution Steps:**

1. **Verify Tunnel Status (1 min)**
   ```bash
   # Check if SSH tunnel is running
   netstat -an | grep 5432  # Should show LISTENING
   ps aux | grep ssh        # Should show "-L 5432:..."
   ```

2. **Reconnect SSH Tunnel (2 min)**
   ```bash
   ssh -L 5432:127.0.0.1:5432 kjh2064@178.104.200.7
   # Should show "Permission granted" or prompt for password
   ```

3. **Verify Tunnel Works (1 min)**
   ```bash
   psql -h localhost -p 5432 -U kartsell -d kartsell -c "SELECT 1"
   # Should return: 1
   ```

4. **Keep Tunnel Open (Ongoing)**
   - Do not close this terminal/session
   - If tunnel dies, reconnect immediately

**Success Criteria:** `psql` command succeeds, Host can reach DB

---

### Scenario 6: Memory Leak (Application Usage > 1GB)

**Detection:**
- Monitoring: Application memory > 1GB (baseline ~500MB)
- Host CPU spike + memory growth
- Response time degradation

**Root Cause:** Unfreed cached data, event accumulation, or circular references

**Resolution Steps (Graceful):**

1. **Verify Memory Usage (1 min)**
   ```powershell
   Get-Process | Where-Object { $_.Name -like "*dotnet*" } | Select-Object Name, @{N="MemMB";E={$_.WorkingSet/1MB}}
   ```

2. **Check Outbox Size (2 min)**
   ```sql
   SELECT pg_size_pretty(pg_total_relation_size('outbox.outbox')) as size;
   ```
   - If > 500MB: Truncate old published events

3. **Drain Hangfire Queue (5 min)**
   - Wait for all jobs to complete
   - Stop accepting new jobs
   - Monitor queue depth → 0

4. **Graceful Restart (10 min)**
   ```powershell
   # Stop Host
   Stop-Process -Name dotnet -Force
   Start-Sleep -Seconds 5
   
   # Restart
   $env:ASPNETCORE_ENVIRONMENT = "Production"
   dotnet run --project src/KArtSell.Host --configuration Release
   ```

5. **Verify Recovery (3 min)**
   ```powershell
   # Check memory is back to baseline
   Get-Process | Where-Object { $_.Name -like "*dotnet*" } | Select-Object Name, @{N="MemMB";E={$_.WorkingSet/1MB}}
   # Should be ~500MB
   ```

**Success Criteria:** Memory < 500MB, all services resume

---

### Scenario 7: Host Crashed / Not Running

**Detection:**
- HTTP connection refused: localhost:5002
- netstat shows no listener on 5002
- Hangfire jobs accumulating (no processing)

**Resolution Steps:**

1. **Verify Host is Down (1 min)**
   ```bash
   curl http://127.0.0.1:5002/health 2>&1 | grep -i "refused"
   # If connection refused: Host is down
   ```

2. **Check Logs (3 min)**
   ```bash
   tail -100 host.log | grep -i "error\|crash\|exception"
   # Look for root cause
   ```

3. **Verify Prerequisites (3 min)**
   ```bash
   # SSH tunnel
   netstat -an | grep 5432 | grep LISTENING
   
   # Database
   psql -h localhost -U kartsell -d kartsell -c "SELECT 1"
   
   # .NET SDK
   dotnet --version
   ```

4. **Start Host (1 min)**
   ```bash
   cd D:\JobRoomz\KArtSell.Aegis
   .\scripts\gate-4-startup.ps1 -Environment Debug  # or Release for production
   ```

5. **Verify Startup (2 min)**
   ```bash
   # Wait for "Now listening on: http://127.0.0.1:5002"
   curl http://127.0.0.1:5002/health
   # Should return {"status":"healthy"}
   ```

**Success Criteria:** Health check passes, Hangfire resumes processing

---

## Escalation Path

| Scenario | On-Call | Manager | CTO | Action Time |
|----------|---------|---------|-----|-------------|
| Auth failure | ✅ Page immediately | ✅ Notify | ✅ If > 15 min | < 15 min |
| Data loss | ✅ Page immediately | ✅ Notify | ✅ Page | < 5 min |
| Host crash | ✅ Try self-heal | ✅ Notify if > 10 min | ✅ If still down | < 20 min |
| Slow performance | ✅ Analyze | ✅ Notify if > 1 hour | ⏸️ Info only | < 60 min |
| DB connection issue | ✅ Check SSH tunnel | ✅ Notify | ⏸️ Info only | < 10 min |

---

## Post-Incident Review

After resolving any CRITICAL or HIGH incident:

1. **Log Incident (15 min)**
   - Incident ID: [Auto-generated timestamp]
   - Severity: [Critical/High/Medium]
   - Detection time: [When first alerted]
   - Resolution time: [When service restored]
   - Root cause: [Brief summary]
   - Steps taken: [What worked, what didn't]

2. **Document Root Cause (30 min)**
   - Why did this happen?
   - Is it a known issue or new?
   - Is there a tech debt item to track?

3. **Implement Prevention (1-4 weeks)**
   - Can we detect this earlier?
   - Can we automate the fix?
   - Should we add monitoring or alerts?

4. **Update This Runbook (15 min)**
   - Did any steps not work as documented?
   - Add new scenarios if different from existing

5. **Team Debrief (30 min)**
   - Share findings in team Slack/meeting
   - Celebrate quick resolution
   - Commit to follow-up actions

---

## Contact Information

| Role | Name | Slack | Email | On-Call |
|------|------|-------|-------|---------|
| Engineering Lead | [TBD] | @lead | lead@company.com | Schedule |
| DevOps Lead | [TBD] | @devops | devops@company.com | Schedule |
| DBA | [TBD] | @dba | dba@company.com | Schedule |

---

**Last Updated:** 2026-08-04  
**Next Review:** Upon critical incident or quarterly  
**Maintained by:** Engineering Team