KArtSell.Aegis/docs/operational-runbook.md

K-ArtSell Aegis v16.0 Operational Runbook

Purpose: Decision tree + resolution steps for common incidents
Governance: AGENTS.md v16.0 "Safety & Reliability" (Criterion #10)
Last Updated: 2026-08-04

---

Table of Contents

1. Quick Reference
2. Incident Classification
3. Common Scenarios & Resolutions
4. Escalation Path
5. Post-Incident Review

---

Quick Reference

Symptom	Root Cause	Resolution	Time
High API latency (> 2s p99)	DB query backlog	Scale connections or optimize slow queries	5-15 min
All requests return 403	Auth provider misconfigured	Check ASPNETCORE_ENVIRONMENT, redeploy	10 min
Hangfire jobs stuck	Distributed lock timeout	Delete stale locks from DB	3 min
Outbox/Inbox deadlock	Concurrent writes collision	Trigger manual OutboxPollerJob	5 min
Memory leak (usage > 1GB)	Unfreed objects in graph	Graceful restart + drain queue	20 min
DB connection pool exhausted	Max connections reached	Increase pool size or kill idle connections	10 min

---

Incident Classification

By Severity

🔴 CRITICAL (Page on-call immediately)

• All users cannot access system (Host down, Auth failed)
• Data corruption or loss
• Security breach (credentials exposed, unauthorized access)
• Revenue-impacting transactions failing

🟠 HIGH (Start work within 15 minutes)

• Subset of users affected (single queue stuck)
• Degraded performance (p99 > 5s)
• Data quality issue (DQ jobs accumulating)
• Non-critical feature unavailable

🟡 MEDIUM (Start work within 1 hour)

• Single job failing repeatedly
• Increased error rate (but < 1%)
• Observability gap (dashboard not updating)
• Non-critical background task delayed

🟢 LOW (Schedule in next sprint)

• Code improvements (tech debt)
• Documentation updates
• Performance optimization (non-critical path)

---

Common Scenarios & Resolutions

Scenario 1: High API Response Time (CRITICAL/HIGH)

Detection:

• Monitoring alert: p99_latency > 2s
• User complaint: "System is slow"
• Hangfire queue depth > 1000 jobs

Decision Tree:

Is Host running?
├─ NO  → Restart Host (Scenario 7)
├─ YES → Is DB reachable?
    ├─ NO  → SSH tunnel issue (Scenario 5)
    ├─ YES → Check queue depth
        ├─ Depth > 1000 → Scale Hangfire workers or analyze slowest queries
        ├─ Depth < 100  → Analyze application memory/CPU

Resolution Steps:

1. Quick Health Check (1 min)

   curl http://127.0.0.1:5002/health
   psql -U kartsell -d kartsell -c "SELECT now()" # DB latency
   

2. Check Queue Depth (1 min)

   SELECT queue, COUNT(*) FROM hangfire.job WHERE state_name='Enqueued' GROUP BY queue;
   

3. Identify Slow Queries (3 min)

   SELECT query, calls, mean_time FROM pg_stat_statements 
   WHERE mean_time > 100 ORDER BY mean_time DESC LIMIT 10;
   

4. Scale Hangfire Workers (5 min)

   • Edit appsettings.Production.json: "WorkerCount": 16 (from 8)
   • Restart Host
   • Monitor: Should process queue faster

5. Optimize Slow Query (10-30 min)

   • Run EXPLAIN ANALYZE on slowest query
   • Check for missing indexes: SELECT * FROM pg_indexes WHERE tablename='...'
   • Add index if needed: CREATE INDEX idx_... ON table(...)
   • Test performance: SELECT ... EXPLAIN ANALYZE

Success Criteria: p99_latency < 2s, queue depth < 100

---

Scenario 2: Authentication Failures (CRITICAL)

Detection:

• HTTP 403/404 responses on valid endpoints
• Error log: "FailClosedAuthenticationHandler denies request"
• All users affected

Root Cause Analysis:

Is ASPNETCORE_ENVIRONMENT correct?
├─ Release mode but missing auth config  → Add FailClosedAuthenticationHandler config
├─ Development mode (wrong for prod)     → Redeploy with Release
├─ API key format incorrect              → Check Gitea Secrets vs. code

Resolution Steps:

1. Verify Environment (1 min)

   # Check running process
   Get-Process -Name dotnet | Select-Object CommandLine
   # Should show: --configuration Release
   

2. Check Auth Configuration (2 min)

   cat src/KArtSell.Host/appsettings.Production.json | grep -A 10 "Authentication"
   

3. Verify API Key Format (2 min)

   • Check Gitea Secrets: https://gitea.taxbaik.com/kjh2064/KArtSell.Aegis/settings/actions/secrets
   • Expected: KRX_OPENAPI=<actual-key> (not stub-key-for-testing)

4. Temporary Workaround (1 min)

   # If stuck: Start in Development mode temporarily
   $env:ASPNETCORE_ENVIRONMENT = "Development"
   dotnet run --project src/KArtSell.Host --configuration Debug
   # This uses DevelopmentHeaderAuthenticationHandler (accepts X-KArtSell-User header)
   

5. Permanent Fix (5 min)

   • Update appsettings.Production.json with correct auth provider
   • Redeploy with Release configuration

Success Criteria: GET /api/health returns 200

---

Scenario 3: Hangfire Job Stuck in "Scheduled" State (HIGH)

Detection:

• Monitoring: Jobs in "Scheduled" state > 5 minutes
• Hangfire dashboard: Red warning on recurring job
• Log: "Recurring job registration timeout"

Root Cause: Distributed lock held too long (network latency, DB contention)

Decision Tree:

Is the Hangfire server running?
├─ NO  → Start Host
├─ YES → Is there a distributed lock?
    ├─ NO  → Job definition error (check code)
    ├─ YES → Is lock stale?
        ├─ YES (> 10 min)  → Delete lock (Scenario 3 Resolution)
        ├─ NO (< 5 min)    → Wait or increase timeout (DEBT-015)

Resolution Steps:

1. Verify Hangfire Server (1 min)

   SELECT name, last_heartbeat, worker_count FROM hangfire.server;
   

   • If empty: Host not running (Scenario 7)
   • If stale: Server crashed, restart Host

2. Check Distributed Lock (1 min)

   SELECT * FROM hangfire.lock WHERE Key LIKE 'Recurring:%' ORDER BY TimeOut DESC;
   

3. Identify Stale Lock (1 min)

   • If TimeOut > CURRENT_TIMESTAMP by > 10 minutes → lock is stale
   • This prevents job from dequeuing

4. Delete Stale Lock (1 min)

   DELETE FROM hangfire.lock WHERE Key = 'Recurring:JobId' AND TimeOut < CURRENT_TIMESTAMP - INTERVAL '5 minutes';
   

5. Monitor Next Run (2 min)

   • Job should dequeue within 15 seconds
   • Check Hangfire dashboard: Job should move to "Processing"

Prevention: DEBT-015 already applied (consistent timeout handling)

Success Criteria: Job processes immediately after lock removal

---

Scenario 4: Outbox/Inbox Deadlock (HIGH)

Detection:

• Event processing stalled
• SELECT COUNT(*) FROM outbox.outbox WHERE published_at IS NULL > 100
• Inbox consumers not progressing (check logs)

Root Cause: Concurrent writes to inbox, or published event not being consumed

Resolution Steps:

1. Assess Situation (2 min)

   SELECT COUNT(*) as unpublished FROM outbox.outbox WHERE published_at IS NULL;
   SELECT COUNT(*) as unprocessed FROM inbox.inbox WHERE processed_at IS NULL;
   

2. Check Outbox Poller Logs (3 min)

   grep -i "OutboxPollerJob" host.log | tail -20
   # Look for errors: "Duplicate event", "Database timeout", "Constraint violation"
   

3. Option A: Trigger Manual Poll (2 min)

   # If queue is small (< 1000), manually trigger:
   curl -X POST http://127.0.0.1:5002/internal/outbox-poll \
     -H "X-KArtSell-User: operator" -H "X-KArtSell-Role: Admin"
   

4. Option B: Drain Stuck Events (5 min)

   -- Mark old unpublished events as published (if safe)
   UPDATE outbox.outbox 
   SET published_at = now() 
   WHERE published_at IS NULL AND created_at < now() - INTERVAL '1 hour';
   

5. Monitor Recovery (5 min)

   • Inbox consumer should resume
   • Check: SELECT COUNT(*) FROM inbox.inbox WHERE processed_at IS NULL
   • Should decrease over time

Success Criteria: All outbox events published, inbox processing resumes

---

Scenario 5: SSH Tunnel Disconnected (CRITICAL)

Detection:

• Connection timeout on DB queries
• Error: "Connection refused: localhost:5432"
• Hangfire jobs failing with DB connection errors

Resolution Steps:

1. Verify Tunnel Status (1 min)

   # Check if SSH tunnel is running
   netstat -an | grep 5432  # Should show LISTENING
   ps aux | grep ssh        # Should show "-L 5432:..."
   

2. Reconnect SSH Tunnel (2 min)

   ssh -L 5432:127.0.0.1:5432 kjh2064@178.104.200.7
   # Should show "Permission granted" or prompt for password
   

3. Verify Tunnel Works (1 min)

   psql -h localhost -p 5432 -U kartsell -d kartsell -c "SELECT 1"
   # Should return: 1
   

4. Keep Tunnel Open (Ongoing)

   • Do not close this terminal/session
   • If tunnel dies, reconnect immediately

Success Criteria: psql command succeeds, Host can reach DB

---

Scenario 6: Memory Leak (Application Usage > 1GB)

Detection:

• Monitoring: Application memory > 1GB (baseline ~500MB)
• Host CPU spike + memory growth
• Response time degradation

Root Cause: Unfreed cached data, event accumulation, or circular references

Resolution Steps (Graceful):

1. Verify Memory Usage (1 min)

   Get-Process | Where-Object { $_.Name -like "*dotnet*" } | Select-Object Name, @{N="MemMB";E={$_.WorkingSet/1MB}}
   

2. Check Outbox Size (2 min)

   SELECT pg_size_pretty(pg_total_relation_size('outbox.outbox')) as size;
   

   • If > 500MB: Truncate old published events

3. Drain Hangfire Queue (5 min)

   • Wait for all jobs to complete
   • Stop accepting new jobs
   • Monitor queue depth → 0

4. Graceful Restart (10 min)

   # Stop Host
   Stop-Process -Name dotnet -Force
   Start-Sleep -Seconds 5
   
   # Restart
   $env:ASPNETCORE_ENVIRONMENT = "Production"
   dotnet run --project src/KArtSell.Host --configuration Release
   

5. Verify Recovery (3 min)

   # Check memory is back to baseline
   Get-Process | Where-Object { $_.Name -like "*dotnet*" } | Select-Object Name, @{N="MemMB";E={$_.WorkingSet/1MB}}
   # Should be ~500MB
   

Success Criteria: Memory < 500MB, all services resume

---

Scenario 7: Host Crashed / Not Running

Detection:

• HTTP connection refused: localhost:5002
• netstat shows no listener on 5002
• Hangfire jobs accumulating (no processing)

Resolution Steps:

1. Verify Host is Down (1 min)

   curl http://127.0.0.1:5002/health 2>&1 | grep -i "refused"
   # If connection refused: Host is down
   

2. Check Logs (3 min)

   tail -100 host.log | grep -i "error\|crash\|exception"
   # Look for root cause
   

3. Verify Prerequisites (3 min)

   # SSH tunnel
   netstat -an | grep 5432 | grep LISTENING
   
   # Database
   psql -h localhost -U kartsell -d kartsell -c "SELECT 1"
   
   # .NET SDK
   dotnet --version
   

4. Start Host (1 min)

   cd D:\JobRoomz\KArtSell.Aegis
   .\scripts\gate-4-startup.ps1 -Environment Debug  # or Release for production
   

5. Verify Startup (2 min)

   # Wait for "Now listening on: http://127.0.0.1:5002"
   curl http://127.0.0.1:5002/health
   # Should return {"status":"healthy"}
   

Success Criteria: Health check passes, Hangfire resumes processing

---

Escalation Path

Scenario	On-Call	Manager	CTO	Action Time
Auth failure	✅ Page immediately	✅ Notify	✅ If > 15 min	< 15 min
Data loss	✅ Page immediately	✅ Notify	✅ Page	< 5 min
Host crash	✅ Try self-heal	✅ Notify if > 10 min	✅ If still down	< 20 min
Slow performance	✅ Analyze	✅ Notify if > 1 hour	⏸️ Info only	< 60 min
DB connection issue	✅ Check SSH tunnel	✅ Notify	⏸️ Info only	< 10 min

---

Post-Incident Review

After resolving any CRITICAL or HIGH incident:

1. Log Incident (15 min)

   • Incident ID: [Auto-generated timestamp]
   • Severity: [Critical/High/Medium]
   • Detection time: [When first alerted]
   • Resolution time: [When service restored]
   • Root cause: [Brief summary]
   • Steps taken: [What worked, what didn't]

2. Document Root Cause (30 min)

   • Why did this happen?
   • Is it a known issue or new?
   • Is there a tech debt item to track?

3. Implement Prevention (1-4 weeks)

   • Can we detect this earlier?
   • Can we automate the fix?
   • Should we add monitoring or alerts?

4. Update This Runbook (15 min)

   • Did any steps not work as documented?
   • Add new scenarios if different from existing

5. Team Debrief (30 min)

   • Share findings in team Slack/meeting
   • Celebrate quick resolution
   • Commit to follow-up actions

---

Contact Information

Role	Name	Slack	Email	On-Call
Engineering Lead	[TBD]	@lead	lead@company.com	Schedule
DevOps Lead	[TBD]	@devops	devops@company.com	Schedule
DBA	[TBD]	@dba	dba@company.com	Schedule

---

Last Updated: 2026-08-04
Next Review: Upon critical incident or quarterly
Maintained by: Engineering Team