docs+fix: Harness the 2026-07-12 DB password incident into pipeline

- CLAUDE.md: Add "DB Secret Management" section documenting the
  incident, the root cause (stale password baked into
  appsettings.Production.json, real password only ever lived in
  /home/kjh2064/.config/quantengine.env, never wired into systemd),
  and the permanent fix (EnvironmentFile= drop-in, applied by hand
  on 2026-07-12 with 'sudo systemctl restart quantengine' verified
  active and journalctl clean).
- CLAUDE.md: Refresh the stale "Gitea Actions Workflows" section
  (was still describing an on:push deploy-prod.yml with a single
  Build stage; now lists prepare-release.yml + deploy-prod.yml
  correctly as workflow_dispatch-only, 6-point health check).
- deploy-prod.yml: Add Check 6 (DB authentication) to the health
  check step. The existing checks only hit GET /Account/Login, which
  returns HTTP 200 even when ConnectionStrings is broken -- that's
  exactly why tonight's outage passed every prior health check. The
  new check greps journalctl for '28P01'/'password authentication
  failed' in the minute after restart and fails the deployment if
  found, so a broken DB connection string can no longer masquerade
  as a successful deploy.
This commit is contained in:
2026-07-12 00:43:31 +09:00
parent e49922e188
commit 6e9a9aa41b
2 changed files with 68 additions and 7 deletions
+21
View File
@@ -309,6 +309,27 @@ jobs:
# Check 5: Release verified
echo "✓ [5/5] Deployment release: ${{ needs.fetch-release.outputs.release-tag }} (commit: ${{ needs.fetch-release.outputs.commit-hash }})"
# Check 6: DB connectivity (GET /Account/Login returns 200 even when
# the DB password is stale -- the page itself has no DB dependency.
# Only an actual login POST, or the app logs, reveal a broken
# connection string. See CLAUDE.md "DB Secret Management" incident
# 2026-07-12: this check would have caught it, the HTTP check alone
# did not.)
sleep 2
DB_ERRORS=$(ssh -i ~/.ssh/deploy_key \
-p 22 \
-o StrictHostKeyChecking=accept-new \
kjh2064@$DEPLOY_HOST \
"journalctl -u quantengine --since '1 minute ago' --no-pager 2>/dev/null | grep -c '28P01\|password authentication failed'" || echo "0")
if [ "$DB_ERRORS" = "0" ]; then
echo "✓ [6/6] No DB authentication errors in recent logs"
else
echo "❌ [6/6] DB authentication errors found in logs ($DB_ERRORS occurrences)"
echo ""
echo "❌ FAILED: Deployment reachable over HTTP but DB connection is broken"
exit 1
fi
echo ""
echo "✅ All health checks passed!"
exit 0