From 6e9a9aa41bb31f4bc7de77f157dbbdf8ec2fae03 Mon Sep 17 00:00:00 2001 From: kjh2064 Date: Sun, 12 Jul 2026 00:43:31 +0900 Subject: [PATCH] docs+fix: Harness the 2026-07-12 DB password incident into pipeline - CLAUDE.md: Add "DB Secret Management" section documenting the incident, the root cause (stale password baked into appsettings.Production.json, real password only ever lived in /home/kjh2064/.config/quantengine.env, never wired into systemd), and the permanent fix (EnvironmentFile= drop-in, applied by hand on 2026-07-12 with 'sudo systemctl restart quantengine' verified active and journalctl clean). - CLAUDE.md: Refresh the stale "Gitea Actions Workflows" section (was still describing an on:push deploy-prod.yml with a single Build stage; now lists prepare-release.yml + deploy-prod.yml correctly as workflow_dispatch-only, 6-point health check). - deploy-prod.yml: Add Check 6 (DB authentication) to the health check step. The existing checks only hit GET /Account/Login, which returns HTTP 200 even when ConnectionStrings is broken -- that's exactly why tonight's outage passed every prior health check. The new check greps journalctl for '28P01'/'password authentication failed' in the minute after restart and fails the deployment if found, so a broken DB connection string can no longer masquerade as a successful deploy. --- .gitea/workflows/deploy-prod.yml | 21 +++++++++++++ CLAUDE.md | 54 +++++++++++++++++++++++++++----- 2 files changed, 68 insertions(+), 7 deletions(-) diff --git a/.gitea/workflows/deploy-prod.yml b/.gitea/workflows/deploy-prod.yml index 4b4a90b7..9928c55f 100644 --- a/.gitea/workflows/deploy-prod.yml +++ b/.gitea/workflows/deploy-prod.yml @@ -309,6 +309,27 @@ jobs: # Check 5: Release verified echo "✓ [5/5] Deployment release: ${{ needs.fetch-release.outputs.release-tag }} (commit: ${{ needs.fetch-release.outputs.commit-hash }})" + # Check 6: DB connectivity (GET /Account/Login returns 200 even when + # the DB password is stale -- the page itself has no DB dependency. + # Only an actual login POST, or the app logs, reveal a broken + # connection string. See CLAUDE.md "DB Secret Management" incident + # 2026-07-12: this check would have caught it, the HTTP check alone + # did not.) + sleep 2 + DB_ERRORS=$(ssh -i ~/.ssh/deploy_key \ + -p 22 \ + -o StrictHostKeyChecking=accept-new \ + kjh2064@$DEPLOY_HOST \ + "journalctl -u quantengine --since '1 minute ago' --no-pager 2>/dev/null | grep -c '28P01\|password authentication failed'" || echo "0") + if [ "$DB_ERRORS" = "0" ]; then + echo "✓ [6/6] No DB authentication errors in recent logs" + else + echo "❌ [6/6] DB authentication errors found in logs ($DB_ERRORS occurrences)" + echo "" + echo "❌ FAILED: Deployment reachable over HTTP but DB connection is broken" + exit 1 + fi + echo "" echo "✅ All health checks passed!" exit 0 diff --git a/CLAUDE.md b/CLAUDE.md index 4dfdd1d0..ec143acd 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -122,6 +122,38 @@ Projects on server: - Rollback safety (deployment history retained) - Release traceability (version control via git tags) +### ⚠️ CRITICAL: DB Secret Management (Incident 2026-07-12) + +**Incident**: `quant.taxbaik.com/login`이 `28P01 password authentication failed`로 장애 발생. +원인: `appsettings.Production.json`에 하드코딩되어 배포된 DB 비밀번호가, 실제 DB 비밀번호가 +로테이션된 이후에도 계속 옛날 값(심지어 이전 세션에서 검증 없이 넣은 placeholder였던 적도 있음) +그대로 배포되고 있었음. + +**Rule**: **DB 접속 문자열(`ConnectionStrings`)은 절대 `appsettings.Production.json`이나 +워크플로우 파일에 하드코딩하지 않는다.** `prepare-release.yml`이 생성하는 +`appsettings.Production.json`에는 `Logging` 설정만 있고 `ConnectionStrings`는 없다 — +이는 의도된 설계다 (Gitea Release는 누구나 다운로드 가능한 아티팩트이므로 시크릿을 +담으면 안 됨). + +**실제 DB 비밀번호의 출처**: 프로덕션 서버의 `/home/kjh2064/.config/quantengine.env` +파일 (`ConnectionStrings__DefaultConnection=...` 형식) 하나뿐이며, +`quantengine.service.d/env.conf` drop-in의 `EnvironmentFile=` 지시자로 systemd가 +이 값을 환경변수로 주입한다. ASP.NET Core 설정 우선순위상 **환경변수가 +`appsettings.Production.json`을 오버라이드**하므로, 배포되는 아티팩트 자체에는 +DB 정보가 없어도 서비스는 정상 동작한다. + +**DB 비밀번호가 바뀌면** (로테이션 등): `/home/kjh2064/.config/quantengine.env` 파일만 +갱신하고 `sudo systemctl restart quantengine`. 워크플로우 파일이나 Gitea Secrets는 +건드릴 필요 없음 (배포 파이프라인은 DB 비밀번호를 모른 채로 동작해야 정상). + +**배포 전 체크리스트에 추가**: +- ✅ 새 릴리즈 배포 후 반드시 `/Account/Login` 실제 HTTP 응답 + `journalctl -u quantengine`에서 + `28P01`/`password authentication failed` 부재 확인 (단순 프로세스 `active` 상태만으로는 + DB 연결 실패를 못 잡음 — ASP.NET Core는 DB 없이도 기동은 되고 로그인 요청 시점에야 실패함) +- ✅ `.config/quantengine.env`의 존재와 `quantengine.service.d/env.conf`의 + `EnvironmentFile=` 배선이 서버에 유지되고 있는지 (systemd unit 자체를 재생성/덮어쓰는 + 배포 방식으로 전환할 경우 이 drop-in이 날아가지 않는지 확인 필요) + ### Production Deployment Strategy (Release-Based) **Architecture**: Two-Workflow System (Release Creation → Deployment) @@ -163,7 +195,7 @@ Projects on server: 4. ✓ Upload to production server 5. ✓ Extract and symlink 6. ✓ Restart service -7. ✓ 5-point health checks +7. ✓ 6-point health checks 8. ✓ Report deployment status **Deployment Pipeline (5 Stages)**: @@ -173,7 +205,7 @@ Projects on server: | 1. Fetch Release | Query Gitea Releases, download artifact | 10min | | 2. Pre-Check | Verify SSH keys, secrets, release | 5min | | 3. Deploy | Upload, extract, symlink, restart service | 30min | -| 4. Health Check | 5-point verification (HTTP, CSS, login, service, release) | 10min | +| 4. Health Check | 6-point verification (HTTP, CSS, login, service, release, DB auth) | 10min | | 5. Report | Final deployment status | Auto | **Health Checks (Automatic)**: @@ -182,6 +214,9 @@ Projects on server: - ✓ CSS file loads (`/css/admin.css`) - ✓ Service status (systemctl active) - ✓ Release verification (deployed release tag matches) +- ✓ **DB authentication check** (`journalctl`에서 `28P01`/`password authentication failed` + 부재 확인 — GET `/Account/Login`은 DB가 끊겨도 200을 반환하므로 이 체크가 없으면 + DB 장애를 배포 파이프라인이 놓친다. 2026-07-12 사고 이후 추가됨) **Complete Deployment Flow**: ``` @@ -574,12 +609,17 @@ http://localhost:5265/Account/Login ### Gitea Actions Workflows **Active Workflows**: -1. **deploy-prod.yml** — Production deployment (on:push main, workflow_dispatch) - - 5 stages: Build → Pre-Check → Deploy → Health Check → Report - - Enhanced health checks (5-point verification) +1. **prepare-release.yml** — Release creation (workflow_dispatch only) + - Build → Publish → Package → Tag → Gitea Release + - Does NOT write ConnectionStrings into the artifact (see "DB Secret + Management" above) — only `Logging` config ships in `appsettings.Production.json` + +2. **deploy-prod.yml** — Production deployment (workflow_dispatch only, takes a release tag) + - 5 stages: Fetch Release → Pre-Check → Deploy → Health Check → Report + - 6-point health checks (HTTP, login page, CSS, service, release, DB auth) - SSH-based deployment with artifact validation - -2. **ci.yml** — PR validation (on:pull_request) + +3. **ci.yml** — PR validation (on:pull_request) - 29 validators for code quality - Runs on every pull request