docs+fix: Harness the 2026-07-12 DB password incident into pipeline
- CLAUDE.md: Add "DB Secret Management" section documenting the incident, the root cause (stale password baked into appsettings.Production.json, real password only ever lived in /home/kjh2064/.config/quantengine.env, never wired into systemd), and the permanent fix (EnvironmentFile= drop-in, applied by hand on 2026-07-12 with 'sudo systemctl restart quantengine' verified active and journalctl clean). - CLAUDE.md: Refresh the stale "Gitea Actions Workflows" section (was still describing an on:push deploy-prod.yml with a single Build stage; now lists prepare-release.yml + deploy-prod.yml correctly as workflow_dispatch-only, 6-point health check). - deploy-prod.yml: Add Check 6 (DB authentication) to the health check step. The existing checks only hit GET /Account/Login, which returns HTTP 200 even when ConnectionStrings is broken -- that's exactly why tonight's outage passed every prior health check. The new check greps journalctl for '28P01'/'password authentication failed' in the minute after restart and fails the deployment if found, so a broken DB connection string can no longer masquerade as a successful deploy.
This commit is contained in:
@@ -309,6 +309,27 @@ jobs:
|
||||
# Check 5: Release verified
|
||||
echo "✓ [5/5] Deployment release: ${{ needs.fetch-release.outputs.release-tag }} (commit: ${{ needs.fetch-release.outputs.commit-hash }})"
|
||||
|
||||
# Check 6: DB connectivity (GET /Account/Login returns 200 even when
|
||||
# the DB password is stale -- the page itself has no DB dependency.
|
||||
# Only an actual login POST, or the app logs, reveal a broken
|
||||
# connection string. See CLAUDE.md "DB Secret Management" incident
|
||||
# 2026-07-12: this check would have caught it, the HTTP check alone
|
||||
# did not.)
|
||||
sleep 2
|
||||
DB_ERRORS=$(ssh -i ~/.ssh/deploy_key \
|
||||
-p 22 \
|
||||
-o StrictHostKeyChecking=accept-new \
|
||||
kjh2064@$DEPLOY_HOST \
|
||||
"journalctl -u quantengine --since '1 minute ago' --no-pager 2>/dev/null | grep -c '28P01\|password authentication failed'" || echo "0")
|
||||
if [ "$DB_ERRORS" = "0" ]; then
|
||||
echo "✓ [6/6] No DB authentication errors in recent logs"
|
||||
else
|
||||
echo "❌ [6/6] DB authentication errors found in logs ($DB_ERRORS occurrences)"
|
||||
echo ""
|
||||
echo "❌ FAILED: Deployment reachable over HTTP but DB connection is broken"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "✅ All health checks passed!"
|
||||
exit 0
|
||||
|
||||
@@ -122,6 +122,38 @@ Projects on server:
|
||||
- Rollback safety (deployment history retained)
|
||||
- Release traceability (version control via git tags)
|
||||
|
||||
### ⚠️ CRITICAL: DB Secret Management (Incident 2026-07-12)
|
||||
|
||||
**Incident**: `quant.taxbaik.com/login`이 `28P01 password authentication failed`로 장애 발생.
|
||||
원인: `appsettings.Production.json`에 하드코딩되어 배포된 DB 비밀번호가, 실제 DB 비밀번호가
|
||||
로테이션된 이후에도 계속 옛날 값(심지어 이전 세션에서 검증 없이 넣은 placeholder였던 적도 있음)
|
||||
그대로 배포되고 있었음.
|
||||
|
||||
**Rule**: **DB 접속 문자열(`ConnectionStrings`)은 절대 `appsettings.Production.json`이나
|
||||
워크플로우 파일에 하드코딩하지 않는다.** `prepare-release.yml`이 생성하는
|
||||
`appsettings.Production.json`에는 `Logging` 설정만 있고 `ConnectionStrings`는 없다 —
|
||||
이는 의도된 설계다 (Gitea Release는 누구나 다운로드 가능한 아티팩트이므로 시크릿을
|
||||
담으면 안 됨).
|
||||
|
||||
**실제 DB 비밀번호의 출처**: 프로덕션 서버의 `/home/kjh2064/.config/quantengine.env`
|
||||
파일 (`ConnectionStrings__DefaultConnection=...` 형식) 하나뿐이며,
|
||||
`quantengine.service.d/env.conf` drop-in의 `EnvironmentFile=` 지시자로 systemd가
|
||||
이 값을 환경변수로 주입한다. ASP.NET Core 설정 우선순위상 **환경변수가
|
||||
`appsettings.Production.json`을 오버라이드**하므로, 배포되는 아티팩트 자체에는
|
||||
DB 정보가 없어도 서비스는 정상 동작한다.
|
||||
|
||||
**DB 비밀번호가 바뀌면** (로테이션 등): `/home/kjh2064/.config/quantengine.env` 파일만
|
||||
갱신하고 `sudo systemctl restart quantengine`. 워크플로우 파일이나 Gitea Secrets는
|
||||
건드릴 필요 없음 (배포 파이프라인은 DB 비밀번호를 모른 채로 동작해야 정상).
|
||||
|
||||
**배포 전 체크리스트에 추가**:
|
||||
- ✅ 새 릴리즈 배포 후 반드시 `/Account/Login` 실제 HTTP 응답 + `journalctl -u quantengine`에서
|
||||
`28P01`/`password authentication failed` 부재 확인 (단순 프로세스 `active` 상태만으로는
|
||||
DB 연결 실패를 못 잡음 — ASP.NET Core는 DB 없이도 기동은 되고 로그인 요청 시점에야 실패함)
|
||||
- ✅ `.config/quantengine.env`의 존재와 `quantengine.service.d/env.conf`의
|
||||
`EnvironmentFile=` 배선이 서버에 유지되고 있는지 (systemd unit 자체를 재생성/덮어쓰는
|
||||
배포 방식으로 전환할 경우 이 drop-in이 날아가지 않는지 확인 필요)
|
||||
|
||||
### Production Deployment Strategy (Release-Based)
|
||||
|
||||
**Architecture**: Two-Workflow System (Release Creation → Deployment)
|
||||
@@ -163,7 +195,7 @@ Projects on server:
|
||||
4. ✓ Upload to production server
|
||||
5. ✓ Extract and symlink
|
||||
6. ✓ Restart service
|
||||
7. ✓ 5-point health checks
|
||||
7. ✓ 6-point health checks
|
||||
8. ✓ Report deployment status
|
||||
|
||||
**Deployment Pipeline (5 Stages)**:
|
||||
@@ -173,7 +205,7 @@ Projects on server:
|
||||
| 1. Fetch Release | Query Gitea Releases, download artifact | 10min |
|
||||
| 2. Pre-Check | Verify SSH keys, secrets, release | 5min |
|
||||
| 3. Deploy | Upload, extract, symlink, restart service | 30min |
|
||||
| 4. Health Check | 5-point verification (HTTP, CSS, login, service, release) | 10min |
|
||||
| 4. Health Check | 6-point verification (HTTP, CSS, login, service, release, DB auth) | 10min |
|
||||
| 5. Report | Final deployment status | Auto |
|
||||
|
||||
**Health Checks (Automatic)**:
|
||||
@@ -182,6 +214,9 @@ Projects on server:
|
||||
- ✓ CSS file loads (`/css/admin.css`)
|
||||
- ✓ Service status (systemctl active)
|
||||
- ✓ Release verification (deployed release tag matches)
|
||||
- ✓ **DB authentication check** (`journalctl`에서 `28P01`/`password authentication failed`
|
||||
부재 확인 — GET `/Account/Login`은 DB가 끊겨도 200을 반환하므로 이 체크가 없으면
|
||||
DB 장애를 배포 파이프라인이 놓친다. 2026-07-12 사고 이후 추가됨)
|
||||
|
||||
**Complete Deployment Flow**:
|
||||
```
|
||||
@@ -574,12 +609,17 @@ http://localhost:5265/Account/Login
|
||||
### Gitea Actions Workflows
|
||||
|
||||
**Active Workflows**:
|
||||
1. **deploy-prod.yml** — Production deployment (on:push main, workflow_dispatch)
|
||||
- 5 stages: Build → Pre-Check → Deploy → Health Check → Report
|
||||
- Enhanced health checks (5-point verification)
|
||||
1. **prepare-release.yml** — Release creation (workflow_dispatch only)
|
||||
- Build → Publish → Package → Tag → Gitea Release
|
||||
- Does NOT write ConnectionStrings into the artifact (see "DB Secret
|
||||
Management" above) — only `Logging` config ships in `appsettings.Production.json`
|
||||
|
||||
2. **deploy-prod.yml** — Production deployment (workflow_dispatch only, takes a release tag)
|
||||
- 5 stages: Fetch Release → Pre-Check → Deploy → Health Check → Report
|
||||
- 6-point health checks (HTTP, login page, CSS, service, release, DB auth)
|
||||
- SSH-based deployment with artifact validation
|
||||
|
||||
2. **ci.yml** — PR validation (on:pull_request)
|
||||
|
||||
3. **ci.yml** — PR validation (on:pull_request)
|
||||
- 29 validators for code quality
|
||||
- Runs on every pull request
|
||||
|
||||
|
||||
Reference in New Issue
Block a user