KArtSell.Aegis/tests/PHASE_3_TEST_PROCEDURES.md

Phase 3: Crash Recovery Test Procedures

Document: Detailed procedures for 4 crash recovery scenarios
Version: 2026-08-03
Status: 🚀 READY TO EXECUTE

---

🧪 Scenario 1: Outbox Message Loss Recovery

Procedure

1. Identify Current State
   □ Connect to PostgreSQL (via SSH tunnel)
   □ Query: SELECT id, run_id FROM outbox.outbox LIMIT 1
   □ Note the message ID

2. Simulate Loss
   □ DELETE FROM outbox.outbox WHERE id = <noted-id>
   □ Verify deletion: SELECT COUNT(*) FROM outbox.outbox

3. Trigger Recovery
   □ Watch ShadowRunCompletedConsumer logs in Host
   □ Monitor log pattern: "outbox message.*not found"
   □ Should auto-recover or retry

4. Verify Results
   □ Check if message re-processed
   □ No errors in Host logs
   □ Outbox state consistent

Evidence Capture

- Before: SELECT id, run_id FROM outbox.outbox WHERE id = X
- After: SELECT COUNT(*) FROM outbox.outbox
- Logs: grep "outbox" host.log | tail -50

Pass Criteria

✅ PASS: Message loss detected, recovery completed, no corruption
❌ FAIL: Unrecovered state divergence or data corruption

---

🧪 Scenario 2: PostgreSQL Connection Drop

Procedure

1. Baseline State
   □ Verify Host can execute queries
   □ Check connection pool status
   □ Note current connection count

2. Simulate Connection Drop
   □ Terminate SSH tunnel (close terminal or Ctrl+C)
   □ Database becomes unreachable
   □ Connection pool timeout triggered

3. Monitor Recovery
   □ Watch Host logs for connection error
   □ Wait for reconnection attempt
   □ SSH tunnel comes back online
   □ Connection re-established

4. Verify Resumption
   □ Host queries successful again
   □ No hanging requests
   □ Transaction consistency maintained

Evidence Capture

- Connection logs: grep -i "connection\|timeout\|reconnect" host.log
- Query execution: Monitor query response times
- Timeline: Record start→drop→recovery time

Pass Criteria

✅ PASS: Connection recovered, queries resumed, no data loss
❌ FAIL: Hanging requests, connection pool exhausted, duplicate processing

---

🧪 Scenario 3: Hangfire Distributed Lock Timeout

Procedure

1. Monitor Hangfire State
   □ Check Hangfire dashboard (if available) or logs
   □ Note active recurring jobs
   □ Verify worker threads

2. Trigger Lock Contention
   □ Multiple Hangfire workers attempt same lock
   □ Simulate timeout (>30s acquisition attempt)
   □ DEBT-015 fallback mechanism engages

3. Monitor Behavior
   □ Watch Host logs for lock timeout trace
   □ Verify: "lock timeout.*fallback" pattern
   □ Check that worker continues (no deadlock)
   □ Other workers unaffected

4. Verify Resolution
   □ Next job attempt succeeds
   □ No stuck locks in DB
   □ Logs show recovery

Evidence Capture

- Lock logs: grep -i "lock\|timeout\|distributed" host.log
- Job status: SELECT * FROM hangfire.job WHERE StateName IN ('Processing', 'Succeeded')
- Duration: Time from timeout to recovery

Pass Criteria

✅ PASS: Lock timeout detected, DEBT-015 fallback activated, job continues
❌ FAIL: Deadlock, stuck lock, worker hang

---

🧪 Scenario 4: Inbox Message Processing Failure

Procedure

1. Identify Consumer
   □ ApprovalQueueConsumer or AuditLogConsumer
   □ Monitor active processing

2. Inject Malformed Message
   □ Insert test message with invalid JSON:
     INSERT INTO inbox.inbox (msg_type, payload, created_at, processed_at)
     VALUES ('approval', '{"invalid": json}', NOW(), NULL)
   □ Trigger consumer to process

3. Monitor Error Handling
   □ Watch Host logs for deserialization error
   □ Verify error caught (no unhandled exception crash)
   □ Check if moved to DLQ

4. Verify Impact
   □ Consumer continues processing next message
   □ No cascade failure
   □ Error logged with context

Evidence Capture

- Error logs: grep -i "deserialization\|inbox\|error" host.log
- DLQ check: SELECT COUNT(*) FROM inbox.dead_letter_queue
- Message state: SELECT * FROM inbox.inbox WHERE processed_at IS NULL

Pass Criteria

✅ PASS: Error caught, message isolated, consumer continues
❌ FAIL: Cascade failure, consumer crash, message loss

---

⚙️ Execution Sequence (Parallel)

Option 1: Manual Sequential

Time | Scenario 1 | Scenario 2 | Scenario 3 | Scenario 4
-----|-----------|-----------|-----------|----------
+0m  | Setup     |           |           |
+5m  | Execute   | Setup     |           |
+10m | Verify    | Execute   | Setup     |
+15m |           | Verify    | Execute   | Setup
+20m |           |           | Verify    | Execute
+25m |           |           |           | Verify
+30m | Done      | Done      | Done      | Done

Option 2: Parallel (Recommended)

All 4 scenarios execute in parallel
Estimated duration: 15-20 minutes total
Each scenario: 5-7 minutes

---

📋 Required Prerequisites

Before executing tests:

✅ Host is running (verified)
✅ SSH tunnel is open (verified)
✅ PostgreSQL is accessible (need to verify)
✅ Job 893 is running (verified)
✅ All consumer services are deployed (need to verify)

Verification Checklist

□ Host logs accessible and monitored
□ PostgreSQL connection working
□ Hangfire workers active
□ Consumer message handlers ready
□ DLQ (Dead Letter Queue) exists
□ Outbox/Inbox tables accessible

---

📊 Evidence Collection Plan

After each scenario:

1. Capture logs (PHASE_3_EXECUTION_LOG.md)
2. Record query results
3. Document timeline
4. Note any issues
5. Mark PASS/FAIL

Final deliverable:

tests/crash_recovery_evidence.md
├─ Scenario 1: [PASS/FAIL with evidence]
├─ Scenario 2: [PASS/FAIL with evidence]
├─ Scenario 3: [PASS/FAIL with evidence]
└─ Scenario 4: [PASS/FAIL with evidence]

---

🚀 Status

Current: ✅ Procedures documented, environment verified
Next: Execute Scenarios 1-4
Estimated Duration: 15-20 minutes (parallel) or 30 minutes (sequential)
Target: Complete Phase 3 testing before Phase 1 (Job 893) finishes