# Phase 3: Crash Recovery Test Procedures

**Document:** Detailed procedures for 4 crash recovery scenarios  
**Version:** 2026-08-03  
**Status:** 🚀 READY TO EXECUTE

---

## 🧪 **Scenario 1: Outbox Message Loss Recovery**

### Procedure
```
1. Identify Current State
   □ Connect to PostgreSQL (via SSH tunnel)
   □ Query: SELECT id, run_id FROM outbox.outbox LIMIT 1
   □ Note the message ID

2. Simulate Loss
   □ DELETE FROM outbox.outbox WHERE id = <noted-id>
   □ Verify deletion: SELECT COUNT(*) FROM outbox.outbox

3. Trigger Recovery
   □ Watch ShadowRunCompletedConsumer logs in Host
   □ Monitor log pattern: "outbox message.*not found"
   □ Should auto-recover or retry

4. Verify Results
   □ Check if message re-processed
   □ No errors in Host logs
   □ Outbox state consistent
```

### Evidence Capture
```
- Before: SELECT id, run_id FROM outbox.outbox WHERE id = X
- After: SELECT COUNT(*) FROM outbox.outbox
- Logs: grep "outbox" host.log | tail -50
```

### Pass Criteria
```
✅ PASS: Message loss detected, recovery completed, no corruption
❌ FAIL: Unrecovered state divergence or data corruption
```

---

## 🧪 **Scenario 2: PostgreSQL Connection Drop**

### Procedure
```
1. Baseline State
   □ Verify Host can execute queries
   □ Check connection pool status
   □ Note current connection count

2. Simulate Connection Drop
   □ Terminate SSH tunnel (close terminal or Ctrl+C)
   □ Database becomes unreachable
   □ Connection pool timeout triggered

3. Monitor Recovery
   □ Watch Host logs for connection error
   □ Wait for reconnection attempt
   □ SSH tunnel comes back online
   □ Connection re-established

4. Verify Resumption
   □ Host queries successful again
   □ No hanging requests
   □ Transaction consistency maintained
```

### Evidence Capture
```
- Connection logs: grep -i "connection\|timeout\|reconnect" host.log
- Query execution: Monitor query response times
- Timeline: Record start→drop→recovery time
```

### Pass Criteria
```
✅ PASS: Connection recovered, queries resumed, no data loss
❌ FAIL: Hanging requests, connection pool exhausted, duplicate processing
```

---

## 🧪 **Scenario 3: Hangfire Distributed Lock Timeout**

### Procedure
```
1. Monitor Hangfire State
   □ Check Hangfire dashboard (if available) or logs
   □ Note active recurring jobs
   □ Verify worker threads

2. Trigger Lock Contention
   □ Multiple Hangfire workers attempt same lock
   □ Simulate timeout (>30s acquisition attempt)
   □ DEBT-015 fallback mechanism engages

3. Monitor Behavior
   □ Watch Host logs for lock timeout trace
   □ Verify: "lock timeout.*fallback" pattern
   □ Check that worker continues (no deadlock)
   □ Other workers unaffected

4. Verify Resolution
   □ Next job attempt succeeds
   □ No stuck locks in DB
   □ Logs show recovery
```

### Evidence Capture
```
- Lock logs: grep -i "lock\|timeout\|distributed" host.log
- Job status: SELECT * FROM hangfire.job WHERE StateName IN ('Processing', 'Succeeded')
- Duration: Time from timeout to recovery
```

### Pass Criteria
```
✅ PASS: Lock timeout detected, DEBT-015 fallback activated, job continues
❌ FAIL: Deadlock, stuck lock, worker hang
```

---

## 🧪 **Scenario 4: Inbox Message Processing Failure**

### Procedure
```
1. Identify Consumer
   □ ApprovalQueueConsumer or AuditLogConsumer
   □ Monitor active processing

2. Inject Malformed Message
   □ Insert test message with invalid JSON:
     INSERT INTO inbox.inbox (msg_type, payload, created_at, processed_at)
     VALUES ('approval', '{"invalid": json}', NOW(), NULL)
   □ Trigger consumer to process

3. Monitor Error Handling
   □ Watch Host logs for deserialization error
   □ Verify error caught (no unhandled exception crash)
   □ Check if moved to DLQ

4. Verify Impact
   □ Consumer continues processing next message
   □ No cascade failure
   □ Error logged with context
```

### Evidence Capture
```
- Error logs: grep -i "deserialization\|inbox\|error" host.log
- DLQ check: SELECT COUNT(*) FROM inbox.dead_letter_queue
- Message state: SELECT * FROM inbox.inbox WHERE processed_at IS NULL
```

### Pass Criteria
```
✅ PASS: Error caught, message isolated, consumer continues
❌ FAIL: Cascade failure, consumer crash, message loss
```

---

## ⚙️ **Execution Sequence (Parallel)**

### Option 1: Manual Sequential
```
Time | Scenario 1 | Scenario 2 | Scenario 3 | Scenario 4
-----|-----------|-----------|-----------|----------
+0m  | Setup     |           |           |
+5m  | Execute   | Setup     |           |
+10m | Verify    | Execute   | Setup     |
+15m |           | Verify    | Execute   | Setup
+20m |           |           | Verify    | Execute
+25m |           |           |           | Verify
+30m | Done      | Done      | Done      | Done
```

### Option 2: Parallel (Recommended)
```
All 4 scenarios execute in parallel
Estimated duration: 15-20 minutes total
Each scenario: 5-7 minutes
```

---

## 📋 **Required Prerequisites**

Before executing tests:

```
✅ Host is running (verified)
✅ SSH tunnel is open (verified)
✅ PostgreSQL is accessible (need to verify)
✅ Job 893 is running (verified)
✅ All consumer services are deployed (need to verify)
```

### Verification Checklist
```
□ Host logs accessible and monitored
□ PostgreSQL connection working
□ Hangfire workers active
□ Consumer message handlers ready
□ DLQ (Dead Letter Queue) exists
□ Outbox/Inbox tables accessible
```

---

## 📊 **Evidence Collection Plan**

After each scenario:
```
1. Capture logs (PHASE_3_EXECUTION_LOG.md)
2. Record query results
3. Document timeline
4. Note any issues
5. Mark PASS/FAIL
```

Final deliverable:
```
tests/crash_recovery_evidence.md
├─ Scenario 1: [PASS/FAIL with evidence]
├─ Scenario 2: [PASS/FAIL with evidence]
├─ Scenario 3: [PASS/FAIL with evidence]
└─ Scenario 4: [PASS/FAIL with evidence]
```

---

## 🚀 **Status**

**Current:** ✅ Procedures documented, environment verified  
**Next:** Execute Scenarios 1-4  
**Estimated Duration:** 15-20 minutes (parallel) or 30 minutes (sequential)  
**Target:** Complete Phase 3 testing before Phase 1 (Job 893) finishes