# Gate 5: Crash Recovery Rehearsal Report

**Status:** ⏳ IN PREPARATION  
**Phase:** 3 (Parallel to Phase 2)  
**Timeline:** TBD (5 days during Phase 2)  
**Template Version:** 2026-08-03

---

## 🧪 **Test Scenarios**

### Scenario 1: Outbox Message Loss
```
Setup:
  □ Start Job 893 processing
  □ Simulate message loss in Outbox
  
Execution:
  □ ShadowRunCompletedConsumer detects missing message
  □ Retry mechanism activates
  □ Message re-queued
  □ Processing resumes
  
Validation:
  □ No state corruption
  □ Message eventually processed
  □ Logs contain recovery trace
  
Result: [ ] PASS  [ ] FAIL
Notes: ___________
```

### Scenario 2: PostgreSQL Connection Drop
```
Setup:
  □ Establish normal operation
  □ Simulate sudden connection drop
  
Execution:
  □ Connection pool detects failure
  □ Retry logic activates
  □ Connection re-established
  □ Processing resumes from checkpoint
  
Validation:
  □ No data loss
  □ No duplicate processing
  □ Transaction consistency maintained
  □ Connection restored within timeout
  
Result: [ ] PASS  [ ] FAIL
Notes: ___________
```

### Scenario 3: Hangfire Distributed Lock Timeout
```
Setup:
  □ Multiple Hangfire workers active
  □ Simulate lock contention
  □ Trigger timeout condition
  
Execution:
  □ Lock acquisition times out
  □ Fallback mechanism activates (DEBT-015)
  □ Job continues without blocking
  □ Other workers unaffected
  
Validation:
  □ No deadlock observed
  □ Graceful degradation
  □ Logs contain timeout trace
  □ Recovery automatic
  
Result: [ ] PASS  [ ] FAIL
Notes: ___________
```

### Scenario 4: Inbox Message Processing Failure
```
Setup:
  □ ApprovalQueueConsumer / AuditLogConsumer processing
  □ Simulate message deserialization failure
  
Execution:
  □ Error caught by consumer
  □ Message moved to DLQ (Dead Letter Queue)
  □ Alert/notification sent
  □ Processing continues
  
Validation:
  □ No data loss
  □ Failure logged with context
  □ Manual intervention possible
  □ Main pipeline unaffected
  
Result: [ ] PASS  [ ] FAIL
Notes: ___________
```

---

## ✅ **Recovery Procedures**

### Procedure 1: State Reconciliation
```
When state divergence detected:
  □ Identify divergence scope
  □ Verify data integrity
  □ Re-sync from source of truth
  □ Validate reconciliation
  □ Log reconciliation action
  
Verification: [ ] PASS  [ ] FAIL
```

### Procedure 2: Message Replay
```
When messages need replay:
  □ Extract failed messages from logs
  □ Create replay batch
  □ Re-queue with idempotency check
  □ Monitor replay execution
  □ Verify all messages processed
  
Verification: [ ] PASS  [ ] FAIL
```

### Procedure 3: Lock Recovery
```
When Hangfire lock stuck:
  □ Identify hung lock
  □ Check lock timeout (should auto-recover)
  □ Verify fallback activated (DEBT-015)
  □ Resume processing
  □ Monitor for re-occurrence
  
Verification: [ ] PASS  [ ] FAIL
```

---

## 📊 **Test Results Summary**

| Scenario | Status | Duration | Notes |
|----------|--------|----------|-------|
| Outbox Message Loss | [ ] PASS  [ ] FAIL | TBD | ⏳ Pending |
| PostgreSQL Drop | [ ] PASS  [ ] FAIL | TBD | ⏳ Pending |
| Hangfire Lock Timeout | [ ] PASS  [ ] FAIL | TBD | ⏳ Pending |
| Inbox Failure | [ ] PASS  [ ] FAIL | TBD | ⏳ Pending |

**Overall Result:** ⏳ Pending

---

## ✅ **Gate 5 Phase 3 Completion**

- [ ] All scenarios tested
- [ ] All procedures verified
- [ ] No unrecoverable failures
- [ ] Recovery mechanisms work
- [ ] Evidence archived
- **Status:** ✅ **PASS** or ❌ **FAIL** (TBD)

---

**Next:** Phase 4 (Gate 5 Sign-Off)