KArtSell.Aegis/tests/crash_recovery_validation.md

Gate 5: Crash Recovery Rehearsal Report

Status: ⏳ IN PREPARATION
Phase: 3 (Parallel to Phase 2)
Timeline: TBD (5 days during Phase 2)
Template Version: 2026-08-03

---

🧪 Test Scenarios

Scenario 1: Outbox Message Loss

Setup:
  □ Start Job 893 processing
  □ Simulate message loss in Outbox
  
Execution:
  □ ShadowRunCompletedConsumer detects missing message
  □ Retry mechanism activates
  □ Message re-queued
  □ Processing resumes
  
Validation:
  □ No state corruption
  □ Message eventually processed
  □ Logs contain recovery trace
  
Result: [ ] PASS  [ ] FAIL
Notes: ___________

Scenario 2: PostgreSQL Connection Drop

Setup:
  □ Establish normal operation
  □ Simulate sudden connection drop
  
Execution:
  □ Connection pool detects failure
  □ Retry logic activates
  □ Connection re-established
  □ Processing resumes from checkpoint
  
Validation:
  □ No data loss
  □ No duplicate processing
  □ Transaction consistency maintained
  □ Connection restored within timeout
  
Result: [ ] PASS  [ ] FAIL
Notes: ___________

Scenario 3: Hangfire Distributed Lock Timeout

Setup:
  □ Multiple Hangfire workers active
  □ Simulate lock contention
  □ Trigger timeout condition
  
Execution:
  □ Lock acquisition times out
  □ Fallback mechanism activates (DEBT-015)
  □ Job continues without blocking
  □ Other workers unaffected
  
Validation:
  □ No deadlock observed
  □ Graceful degradation
  □ Logs contain timeout trace
  □ Recovery automatic
  
Result: [ ] PASS  [ ] FAIL
Notes: ___________

Scenario 4: Inbox Message Processing Failure

Setup:
  □ ApprovalQueueConsumer / AuditLogConsumer processing
  □ Simulate message deserialization failure
  
Execution:
  □ Error caught by consumer
  □ Message moved to DLQ (Dead Letter Queue)
  □ Alert/notification sent
  □ Processing continues
  
Validation:
  □ No data loss
  □ Failure logged with context
  □ Manual intervention possible
  □ Main pipeline unaffected
  
Result: [ ] PASS  [ ] FAIL
Notes: ___________

---

✅ Recovery Procedures

Procedure 1: State Reconciliation

When state divergence detected:
  □ Identify divergence scope
  □ Verify data integrity
  □ Re-sync from source of truth
  □ Validate reconciliation
  □ Log reconciliation action
  
Verification: [ ] PASS  [ ] FAIL

Procedure 2: Message Replay

When messages need replay:
  □ Extract failed messages from logs
  □ Create replay batch
  □ Re-queue with idempotency check
  □ Monitor replay execution
  □ Verify all messages processed
  
Verification: [ ] PASS  [ ] FAIL

Procedure 3: Lock Recovery

When Hangfire lock stuck:
  □ Identify hung lock
  □ Check lock timeout (should auto-recover)
  □ Verify fallback activated (DEBT-015)
  □ Resume processing
  □ Monitor for re-occurrence
  
Verification: [ ] PASS  [ ] FAIL

---

📊 Test Results Summary

Scenario	Status	Duration	Notes
Outbox Message Loss	[ ] PASS [ ] FAIL	TBD	⏳ Pending
PostgreSQL Drop	[ ] PASS [ ] FAIL	TBD	⏳ Pending
Hangfire Lock Timeout	[ ] PASS [ ] FAIL	TBD	⏳ Pending
Inbox Failure	[ ] PASS [ ] FAIL	TBD	⏳ Pending

Overall Result: ⏳ Pending

---

✅ Gate 5 Phase 3 Completion

• All scenarios tested
• All procedures verified
• No unrecoverable failures
• Recovery mechanisms work
• Evidence archived
• Status: ✅ PASS or ❌ FAIL (TBD)

---

Next: Phase 4 (Gate 5 Sign-Off)