docs+fix: Harness the 2026-07-12 DB password incident into pipeline
- CLAUDE.md: Add "DB Secret Management" section documenting the incident, the root cause (stale password baked into appsettings.Production.json, real password only ever lived in /home/kjh2064/.config/quantengine.env, never wired into systemd), and the permanent fix (EnvironmentFile= drop-in, applied by hand on 2026-07-12 with 'sudo systemctl restart quantengine' verified active and journalctl clean). - CLAUDE.md: Refresh the stale "Gitea Actions Workflows" section (was still describing an on:push deploy-prod.yml with a single Build stage; now lists prepare-release.yml + deploy-prod.yml correctly as workflow_dispatch-only, 6-point health check). - deploy-prod.yml: Add Check 6 (DB authentication) to the health check step. The existing checks only hit GET /Account/Login, which returns HTTP 200 even when ConnectionStrings is broken -- that's exactly why tonight's outage passed every prior health check. The new check greps journalctl for '28P01'/'password authentication failed' in the minute after restart and fails the deployment if found, so a broken DB connection string can no longer masquerade as a successful deploy.
This commit is contained in:
@@ -309,6 +309,27 @@ jobs:
|
||||
# Check 5: Release verified
|
||||
echo "✓ [5/5] Deployment release: ${{ needs.fetch-release.outputs.release-tag }} (commit: ${{ needs.fetch-release.outputs.commit-hash }})"
|
||||
|
||||
# Check 6: DB connectivity (GET /Account/Login returns 200 even when
|
||||
# the DB password is stale -- the page itself has no DB dependency.
|
||||
# Only an actual login POST, or the app logs, reveal a broken
|
||||
# connection string. See CLAUDE.md "DB Secret Management" incident
|
||||
# 2026-07-12: this check would have caught it, the HTTP check alone
|
||||
# did not.)
|
||||
sleep 2
|
||||
DB_ERRORS=$(ssh -i ~/.ssh/deploy_key \
|
||||
-p 22 \
|
||||
-o StrictHostKeyChecking=accept-new \
|
||||
kjh2064@$DEPLOY_HOST \
|
||||
"journalctl -u quantengine --since '1 minute ago' --no-pager 2>/dev/null | grep -c '28P01\|password authentication failed'" || echo "0")
|
||||
if [ "$DB_ERRORS" = "0" ]; then
|
||||
echo "✓ [6/6] No DB authentication errors in recent logs"
|
||||
else
|
||||
echo "❌ [6/6] DB authentication errors found in logs ($DB_ERRORS occurrences)"
|
||||
echo ""
|
||||
echo "❌ FAILED: Deployment reachable over HTTP but DB connection is broken"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "✅ All health checks passed!"
|
||||
exit 0
|
||||
|
||||
Reference in New Issue
Block a user