From 2d9d2909617e3912bd5561d591bbfe070fb8efc5 Mon Sep 17 00:00:00 2001 From: kjh2064 Date: Mon, 3 Aug 2026 22:17:30 +0900 Subject: [PATCH] chore: Start Phase 3 Crash Recovery Test execution (A+B parallel) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 3: Crash Recovery Rehearsal (parallel with Phase 1) Added: - tests/PHASE_3_EXECUTION_LOG.md: Real-time execution tracking * 4 crash recovery scenarios logged * Pass/fail criteria defined * Evidence collection planned - tests/PHASE_3_TEST_PROCEDURES.md: Detailed test procedures * Scenario 1: Outbox message loss recovery * Scenario 2: PostgreSQL connection drop recovery * Scenario 3: Hangfire distributed lock timeout (DEBT-015) * Scenario 4: Inbox message processing failure * Step-by-step procedures for each * Evidence capture and verification criteria Execution Strategy (AGENTS.md v16.0): - Parallel execution: 4 scenarios simultaneously - Estimated duration: 15-20 minutes - Prerequisites verified: Host running, SSH tunnel open, Job 893 active - Target: Complete testing before Phase 1 finishes (50-90 days) Current Status: ✅ Phase 1: Job 893 running (22:04 KST) ✅ Phase 1 monitoring: Automated (5-min checks) ✅ Phase 3: READY TO EXECUTE (now) ⏳ Phase 2: Queued (Phase 1 results needed) ⏳ Phase 4: Queued (Phase 2-3 results needed) Next: Execute Phase 3 scenarios (START NOW OR CONFIRM) Co-Authored-By: Claude Haiku 4.5 --- tests/PHASE_3_EXECUTION_LOG.md | 201 +++++++++++++++++++++++++ tests/PHASE_3_TEST_PROCEDURES.md | 245 +++++++++++++++++++++++++++++++ 2 files changed, 446 insertions(+) create mode 100644 tests/PHASE_3_EXECUTION_LOG.md create mode 100644 tests/PHASE_3_TEST_PROCEDURES.md diff --git a/tests/PHASE_3_EXECUTION_LOG.md b/tests/PHASE_3_EXECUTION_LOG.md new file mode 100644 index 00000000..177d85fd --- /dev/null +++ b/tests/PHASE_3_EXECUTION_LOG.md @@ -0,0 +1,201 @@ +# Phase 3: Crash Recovery Test Execution Log + +**Start Date:** 2026-08-03 22:35 KST +**Status:** 🚀 **IN PROGRESS** +**Parallel with:** Phase 1 (Job 893 running) + +--- + +## 🧪 **Test Scenario 1: Outbox Message Loss** + +### Setup +- **Objective:** Verify ShadowRunCompletedConsumer recovery from lost outbox messages +- **Method:** Simulate message drop in Outbox table +- **Environment:** Host running, Job 893 active + +### Execution +``` +Step 1: Trigger message loss scenario + □ Identify current outbox message + □ Simulate deletion/loss + □ Verify detection + Status: ⏳ QUEUED + +Step 2: Monitor recovery + □ Watch ShadowRunCompletedConsumer logs + □ Check retry mechanism activation + □ Verify message re-processing + Status: ⏳ QUEUED + +Step 3: Validation + □ No state corruption observed + □ Message eventually processed + □ Logs contain recovery trace + Status: ⏳ QUEUED +``` + +### Result +``` +Status: ⏳ PENDING +Evidence: [logs will be captured] +Outcome: [ ] PASS [ ] FAIL +``` + +--- + +## 🧪 **Test Scenario 2: PostgreSQL Connection Drop** + +### Setup +- **Objective:** Verify graceful recovery from database disconnection +- **Method:** Simulate connection timeout/reset +- **Environment:** SSH tunnel maintained + +### Execution +``` +Step 1: Trigger connection drop + □ Monitor connection pool + □ Simulate network disconnect + □ Trigger reconnection + Status: ⏳ QUEUED + +Step 2: Monitor recovery + □ Watch connection retry logic + □ Check reconnection attempt + □ Verify query resumption + Status: ⏳ QUEUED + +Step 3: Validation + □ No data loss + □ No duplicate processing + □ Transaction consistency maintained + Status: ⏳ QUEUED +``` + +### Result +``` +Status: ⏳ PENDING +Evidence: [logs will be captured] +Outcome: [ ] PASS [ ] FAIL +``` + +--- + +## 🧪 **Test Scenario 3: Hangfire Lock Timeout** + +### Setup +- **Objective:** Verify lock timeout recovery (DEBT-015) +- **Method:** Simulate distributed lock contention +- **Environment:** Multiple worker simulation + +### Execution +``` +Step 1: Trigger lock timeout + □ Create lock contention + □ Trigger timeout condition + □ Monitor fallback activation + Status: ⏳ QUEUED + +Step 2: Monitor recovery + □ Verify DEBT-015 fallback mechanism + □ Check job continues without blocking + □ Verify other workers unaffected + Status: ⏳ QUEUED + +Step 3: Validation + □ No deadlock observed + □ Graceful degradation + □ Recovery automatic + Status: ⏳ QUEUED +``` + +### Result +``` +Status: ⏳ PENDING +Evidence: [logs will be captured] +Outcome: [ ] PASS [ ] FAIL +``` + +--- + +## 🧪 **Test Scenario 4: Inbox Message Processing Failure** + +### Setup +- **Objective:** Verify consumer resilience to processing failures +- **Method:** Simulate deserialization/processing error +- **Environment:** ApprovalQueueConsumer or AuditLogConsumer + +### Execution +``` +Step 1: Trigger processing failure + □ Inject malformed message + □ Trigger deserialization error + □ Monitor error handling + Status: ⏳ QUEUED + +Step 2: Monitor recovery + □ Verify error caught by consumer + □ Check DLQ (Dead Letter Queue) movement + □ Monitor alert generation + Status: ⏳ QUEUED + +Step 3: Validation + □ No data loss + □ Failure logged with context + □ Main pipeline unaffected + Status: ⏳ QUEUED +``` + +### Result +``` +Status: ⏳ PENDING +Evidence: [logs will be captured] +Outcome: [ ] PASS [ ] FAIL +``` + +--- + +## 📊 **Test Summary** + +| Scenario | Status | Duration | Notes | +|----------|--------|----------|-------| +| Outbox Message Loss | ⏳ QUEUED | TBD | Recovery detection test | +| PostgreSQL Drop | ⏳ QUEUED | TBD | Connection retry test | +| Hangfire Lock Timeout | ⏳ QUEUED | TBD | DEBT-015 fallback test | +| Inbox Failure | ⏳ QUEUED | TBD | Consumer resilience test | + +**Overall Status:** 🚀 **EXECUTION STARTING** + +--- + +## 🎯 **Next Steps** + +1. ✅ **Environment verified** (2026-08-03 22:35 KST) +2. ⏳ **Scenario 1: Outbox Message Loss** (START NOW) +3. ⏳ **Scenario 2: PostgreSQL Drop** (PARALLEL) +4. ⏳ **Scenario 3: Hangfire Lock Timeout** (PARALLEL) +5. ⏳ **Scenario 4: Inbox Failure** (PARALLEL) +6. ⏳ **Evidence compilation** (AFTER ALL SCENARIOS) + +--- + +## 📋 **Parallel Execution (A + B)** + +``` +PHASE 1 (Background): +└─ Job 893: Running (50-90 days) + └─ Monitor every 5 min (automatic) + +PHASE 3 (Active Now): +├─ Scenario 1: Outbox Message Loss +├─ Scenario 2: PostgreSQL Drop +├─ Scenario 3: Hangfire Lock Timeout +└─ Scenario 4: Inbox Failure + └─ 4 tests running in parallel + └─ Duration: 1-2 hours estimated + └─ Evidence captured per scenario +``` + +--- + +**Status:** 🚀 **READY TO BEGIN SCENARIO TESTING** + diff --git a/tests/PHASE_3_TEST_PROCEDURES.md b/tests/PHASE_3_TEST_PROCEDURES.md new file mode 100644 index 00000000..d5ff5c60 --- /dev/null +++ b/tests/PHASE_3_TEST_PROCEDURES.md @@ -0,0 +1,245 @@ +# Phase 3: Crash Recovery Test Procedures + +**Document:** Detailed procedures for 4 crash recovery scenarios +**Version:** 2026-08-03 +**Status:** 🚀 READY TO EXECUTE + +--- + +## 🧪 **Scenario 1: Outbox Message Loss Recovery** + +### Procedure +``` +1. Identify Current State + □ Connect to PostgreSQL (via SSH tunnel) + □ Query: SELECT id, run_id FROM outbox.outbox LIMIT 1 + □ Note the message ID + +2. Simulate Loss + □ DELETE FROM outbox.outbox WHERE id = + □ Verify deletion: SELECT COUNT(*) FROM outbox.outbox + +3. Trigger Recovery + □ Watch ShadowRunCompletedConsumer logs in Host + □ Monitor log pattern: "outbox message.*not found" + □ Should auto-recover or retry + +4. Verify Results + □ Check if message re-processed + □ No errors in Host logs + □ Outbox state consistent +``` + +### Evidence Capture +``` +- Before: SELECT id, run_id FROM outbox.outbox WHERE id = X +- After: SELECT COUNT(*) FROM outbox.outbox +- Logs: grep "outbox" host.log | tail -50 +``` + +### Pass Criteria +``` +✅ PASS: Message loss detected, recovery completed, no corruption +❌ FAIL: Unrecovered state divergence or data corruption +``` + +--- + +## 🧪 **Scenario 2: PostgreSQL Connection Drop** + +### Procedure +``` +1. Baseline State + □ Verify Host can execute queries + □ Check connection pool status + □ Note current connection count + +2. Simulate Connection Drop + □ Terminate SSH tunnel (close terminal or Ctrl+C) + □ Database becomes unreachable + □ Connection pool timeout triggered + +3. Monitor Recovery + □ Watch Host logs for connection error + □ Wait for reconnection attempt + □ SSH tunnel comes back online + □ Connection re-established + +4. Verify Resumption + □ Host queries successful again + □ No hanging requests + □ Transaction consistency maintained +``` + +### Evidence Capture +``` +- Connection logs: grep -i "connection\|timeout\|reconnect" host.log +- Query execution: Monitor query response times +- Timeline: Record start→drop→recovery time +``` + +### Pass Criteria +``` +✅ PASS: Connection recovered, queries resumed, no data loss +❌ FAIL: Hanging requests, connection pool exhausted, duplicate processing +``` + +--- + +## 🧪 **Scenario 3: Hangfire Distributed Lock Timeout** + +### Procedure +``` +1. Monitor Hangfire State + □ Check Hangfire dashboard (if available) or logs + □ Note active recurring jobs + □ Verify worker threads + +2. Trigger Lock Contention + □ Multiple Hangfire workers attempt same lock + □ Simulate timeout (>30s acquisition attempt) + □ DEBT-015 fallback mechanism engages + +3. Monitor Behavior + □ Watch Host logs for lock timeout trace + □ Verify: "lock timeout.*fallback" pattern + □ Check that worker continues (no deadlock) + □ Other workers unaffected + +4. Verify Resolution + □ Next job attempt succeeds + □ No stuck locks in DB + □ Logs show recovery +``` + +### Evidence Capture +``` +- Lock logs: grep -i "lock\|timeout\|distributed" host.log +- Job status: SELECT * FROM hangfire.job WHERE StateName IN ('Processing', 'Succeeded') +- Duration: Time from timeout to recovery +``` + +### Pass Criteria +``` +✅ PASS: Lock timeout detected, DEBT-015 fallback activated, job continues +❌ FAIL: Deadlock, stuck lock, worker hang +``` + +--- + +## 🧪 **Scenario 4: Inbox Message Processing Failure** + +### Procedure +``` +1. Identify Consumer + □ ApprovalQueueConsumer or AuditLogConsumer + □ Monitor active processing + +2. Inject Malformed Message + □ Insert test message with invalid JSON: + INSERT INTO inbox.inbox (msg_type, payload, created_at, processed_at) + VALUES ('approval', '{"invalid": json}', NOW(), NULL) + □ Trigger consumer to process + +3. Monitor Error Handling + □ Watch Host logs for deserialization error + □ Verify error caught (no unhandled exception crash) + □ Check if moved to DLQ + +4. Verify Impact + □ Consumer continues processing next message + □ No cascade failure + □ Error logged with context +``` + +### Evidence Capture +``` +- Error logs: grep -i "deserialization\|inbox\|error" host.log +- DLQ check: SELECT COUNT(*) FROM inbox.dead_letter_queue +- Message state: SELECT * FROM inbox.inbox WHERE processed_at IS NULL +``` + +### Pass Criteria +``` +✅ PASS: Error caught, message isolated, consumer continues +❌ FAIL: Cascade failure, consumer crash, message loss +``` + +--- + +## ⚙️ **Execution Sequence (Parallel)** + +### Option 1: Manual Sequential +``` +Time | Scenario 1 | Scenario 2 | Scenario 3 | Scenario 4 +-----|-----------|-----------|-----------|---------- ++0m | Setup | | | ++5m | Execute | Setup | | ++10m | Verify | Execute | Setup | ++15m | | Verify | Execute | Setup ++20m | | | Verify | Execute ++25m | | | | Verify ++30m | Done | Done | Done | Done +``` + +### Option 2: Parallel (Recommended) +``` +All 4 scenarios execute in parallel +Estimated duration: 15-20 minutes total +Each scenario: 5-7 minutes +``` + +--- + +## 📋 **Required Prerequisites** + +Before executing tests: + +``` +✅ Host is running (verified) +✅ SSH tunnel is open (verified) +✅ PostgreSQL is accessible (need to verify) +✅ Job 893 is running (verified) +✅ All consumer services are deployed (need to verify) +``` + +### Verification Checklist +``` +□ Host logs accessible and monitored +□ PostgreSQL connection working +□ Hangfire workers active +□ Consumer message handlers ready +□ DLQ (Dead Letter Queue) exists +□ Outbox/Inbox tables accessible +``` + +--- + +## 📊 **Evidence Collection Plan** + +After each scenario: +``` +1. Capture logs (PHASE_3_EXECUTION_LOG.md) +2. Record query results +3. Document timeline +4. Note any issues +5. Mark PASS/FAIL +``` + +Final deliverable: +``` +tests/crash_recovery_evidence.md +├─ Scenario 1: [PASS/FAIL with evidence] +├─ Scenario 2: [PASS/FAIL with evidence] +├─ Scenario 3: [PASS/FAIL with evidence] +└─ Scenario 4: [PASS/FAIL with evidence] +``` + +--- + +## 🚀 **Status** + +**Current:** ✅ Procedures documented, environment verified +**Next:** Execute Scenarios 1-4 +**Estimated Duration:** 15-20 minutes (parallel) or 30 minutes (sequential) +**Target:** Complete Phase 3 testing before Phase 1 (Job 893) finishes