QuantEngineByItz/docs/DEPLOYMENT_RUNBOOK.md

QuantEngine Deployment Runbook

Full deployment procedure, extracted from CLAUDE.md (2026-07-30) to keep the main file within the character budget. CLAUDE.md keeps the CRITICAL rules (CI/CD-only mandate, DB secret management); this file has the complete step-by-step runbook.

Production Server: Hetzner Cloud 178.104.200.7 (kjh2064@178.104.200.7)

Projects on server:

1. TaxBaik (홈페이지) — Nginx location /taxbaik
2. QuantEngine (데이터 수집/분석) — Nginx location /quantengine

⚠️ CRITICAL: CI/CD-Only Deployment Mandate

Rule: ALL production deployments MUST go through Gitea Actions CI/CD. Manual SSH deployments are FORBIDDEN.

Why:

• Automatic validation (build, health checks, version verification)
• Audit trail (all deployments logged in Gitea Actions)
• Consistent process (no manual errors)
• Rollback safety (deployment history retained)
• Release traceability (version control via git tags)

⚠️ CRITICAL: DB Secret Management (Incident 2026-07-12)

Incident: quant.taxbaik.com/login이 28P01 password authentication failed로 장애 발생. 원인: appsettings.Production.json에 하드코딩되어 배포된 DB 비밀번호가, 실제 DB 비밀번호가 로테이션된 이후에도 계속 옛날 값(심지어 이전 세션에서 검증 없이 넣은 placeholder였던 적도 있음) 그대로 배포되고 있었음.

Rule: DB 접속 문자열(ConnectionStrings)은 절대 appsettings.Production.json이나 워크플로우 파일에 하드코딩하지 않는다. prepare-release.yml이 생성하는 appsettings.Production.json에는 Logging 설정만 있고 ConnectionStrings는 없다 — 이는 의도된 설계다 (Gitea Release는 누구나 다운로드 가능한 아티팩트이므로 시크릿을 담으면 안 됨).

실제 DB 비밀번호의 출처: 프로덕션 서버의 /home/kjh2064/.config/quantengine.env 파일 (ConnectionStrings__DefaultConnection=... 형식) 하나뿐이며, quantengine.service.d/env.conf drop-in의 EnvironmentFile= 지시자로 systemd가 이 값을 환경변수로 주입한다. ASP.NET Core 설정 우선순위상 환경변수가 appsettings.Production.json을 오버라이드하므로, 배포되는 아티팩트 자체에는 DB 정보가 없어도 서비스는 정상 동작한다.

DB 비밀번호가 바뀌면 (로테이션 등): /home/kjh2064/.config/quantengine.env 파일만 갱신하고 sudo systemctl restart quantengine. 워크플로우 파일이나 Gitea Secrets는 건드릴 필요 없음 (배포 파이프라인은 DB 비밀번호를 모른 채로 동작해야 정상).

배포 전 체크리스트에 추가:

• ✅ 새 릴리즈 배포 후 반드시 /Account/Login 실제 HTTP 응답 + journalctl -u quantengine에서 28P01/password authentication failed 부재 확인 (단순 프로세스 active 상태만으로는 DB 연결 실패를 못 잡음 — ASP.NET Core는 DB 없이도 기동은 되고 로그인 요청 시점에야 실패함)
• ✅ .config/quantengine.env의 존재와 quantengine.service.d/env.conf의 EnvironmentFile= 배선이 서버에 유지되고 있는지 (systemd unit 자체를 재생성/덮어쓰는 배포 방식으로 전환할 경우 이 drop-in이 날아가지 않는지 확인 필요)

Production Deployment Strategy (Release-Based)

Architecture: Two-Workflow System (Release Creation → Deployment)

Workflow 1: prepare-release.yml (Release Creation)

Purpose: Create a release with built artifact

Trigger: Manual (workflow_dispatch)

# Visit Gitea Actions and select prepare-release.yml
# Input version: v0.1.20260711 (or any semantic version)

What it does:

1. ✓ Build (restore, build, publish)
2. ✓ Generate appsettings.Production.json
3. ✓ Package artifact: .tar.gz
4. ✓ Create git tag: v0.1.20260711
5. ✓ Create Gitea Release with artifact attached
6. ✓ Notify: Release ready for deployment

Output: Gitea Release with downloadable artifact

Workflow 2: deploy-prod.yml (Deployment)

Purpose: Deploy a release to production

Trigger: Manual (workflow_dispatch)

# Visit Gitea Actions and select deploy-prod.yml
# Input release: v0.1.20260711 (optional — uses latest if empty)

What it does:

1. ✓ Fetch Release (from Gitea Releases)
2. ✓ Download artifact
3. ✓ Verify SSH credentials
4. ✓ Upload to production server
5. ✓ Extract and symlink
6. ✓ Restart service
7. ✓ 6-point health checks
8. ✓ Report deployment status

Deployment Pipeline (5 Stages):

Stage	Purpose	Timeout
1. Fetch Release	Query Gitea Releases, download artifact	10min
2. Pre-Check	Verify SSH keys, secrets, release	5min
3. Deploy	Upload, extract, symlink, restart service	30min
4. Health Check	6-point verification (HTTP, CSS, login, service, release, DB auth)	10min
5. Report	Final deployment status	Auto

Health Checks (Automatic):

• ✓ HTTP 200 on /Account/Login
• ✓ Login page content verification
• ✓ CSS file loads (/css/admin.css)
• ✓ Service status (systemctl active)
• ✓ Release verification (deployed release tag matches)
• ✓ DB authentication check (journalctl에서 28P01/password authentication failed 부재 확인 — GET /Account/Login은 DB가 끊겨도 200을 반환하므로 이 체크가 없으면 DB 장애를 배포 파이프라인이 놓친다. 2026-07-12 사고 이후 추가됨)

Complete Deployment Flow:

1. Code committed to main branch
2. Create release: prepare-release.yml workflow_dispatch (manual)
   → Builds code
   → Creates Gitea Release with artifact
   → Tags repository
3. Deploy release: deploy-prod.yml workflow_dispatch (manual)
   → Selects release version
   → Downloads artifact from Gitea Release
   → Deploys to production server
   → Runs health checks
   → Reports status

Pre-Deployment Checklist

Before creating a release, verify:

1. ✅ Local build: dotnet build src/dotnet/QuantEngine.Web/QuantEngine.Web.csproj -c Release (0 errors, 0 warnings)
2. ✅ E2E tests pass: npx playwright test
3. ✅ Admin pages verified (200 status, no 500 errors)
4. ✅ All changes committed and pushed to main branch
5. ✅ No uncommitted changes: git status

Release & Deployment Workflow

Step 1: Create Release (prepare-release.yml)

# Visit Gitea Actions
# https://gitea.taxbaik.com/kjh2064/QuantEngineByItz/actions

# Run prepare-release.yml workflow
# Input: version = v0.1.20260711

# Workflow will:
# - Build and publish
# - Package artifact
# - Create git tag
# - Create Gitea Release
# - Attach artifact

Step 2: Deploy Release (deploy-prod.yml)

# Visit Gitea Actions (same page)
# Run deploy-prod.yml workflow
# Input: release = v0.1.20260711 (leave empty for latest)

# Workflow will:
# - Download artifact from release
# - Deploy to production server
# - Run health checks
# - Report status

SSH Key Configuration (Required)

Setup (One-time):

1. Generate ED25519 key locally (or reuse existing):

   ssh-keygen -t ed25519 -f ~/.ssh/quantengine_deploy -C "QuantEngine CI/CD"
   

2. Add public key to production server:

   ssh-copy-id -i ~/.ssh/quantengine_deploy.pub kjh2064@178.104.200.7
   

3. Get private key in base64 format:

   # macOS/Linux
   base64 -w 0 ~/.ssh/quantengine_deploy > /tmp/key_b64.txt
   cat /tmp/key_b64.txt | pbcopy
   
   # Or Windows PowerShell
   $key = Get-Content ~/.ssh/quantengine_deploy -Raw
   [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($key)) | Set-Clipboard
   

4. Configure in Gitea:

   • URL: https://gitea.taxbaik.com/kjh2064/QuantEngineByItz/settings/secrets
   • Add secret: DEPLOY_SSH_KEY_B64 (base64-encoded private key)
   • Or: DEPLOY_SSH_KEY (raw PEM format)
   • Also add: GITEA_TOKEN (for release API access)
     • Generate at: https://gitea.taxbaik.com/user/settings/applications
     • Required permissions: repo + read:actions

Deployment Monitoring

During Deployment:

• Watch live in Gitea Actions UI
• Jobs complete in order: Build → Pre-Check → Deploy → Health Check → Report

After Deployment:

# SSH into server
ssh kjh2064@178.104.200.7

# Check active deployment
readlink ~/quantengine_active

# View service status
systemctl status quantengine

# Tail live logs
journalctl -u quantengine -f

# Health check
curl -I http://127.0.0.1:5000/Account/Login

Automatic Rollback (if health check fails)

If health check fails, deployment stops automatically:

1. Service restart may fail
2. Symlink update reverts to previous deployment
3. Gitea Actions marks deployment as FAILED
4. Logs include failure details

Manual rollback (if needed):

# List deployments
ls -lht ~/deployments/quantengine_*

# Revert symlink to previous version
ln -sfn /home/kjh2064/deployments/quantengine_YYYYMMDD_HHMMSS_COMMIT ~/quantengine_active

# Restart service
sudo systemctl restart quantengine

# Verify
curl http://127.0.0.1:5000/Account/Login

Troubleshooting Deployment Failures

Issue: Build fails

• Check: dotnet build locally first
• Ensure: No compilation errors, 0 warnings

Issue: Health check timeout

• Check: Service logs: journalctl -u quantengine -n 50
• Check: Port 5000 listening: ss -tlnp | grep 5000
• Check: DB connectivity in appsettings.Production.json

Issue: SSH key error

• Verify: DEPLOY_SSH_KEY_B64 or DEPLOY_SSH_KEY in Gitea Secrets
• Check: Public key added to ~/.ssh/authorized_keys on server
• Test: ssh -i ~/.ssh/key_file kjh2064@178.104.200.7 echo OK

Git Repository

Gitea Server (동일 호스트):

• HTTP: https://gitea.taxbaik.com/kjh2064/QuantEngineByItz.git
• SSH: ssh://git@gitea.taxbaik.com:2222/kjh2064/QuantEngineByItz.git

Active Gitea Workflows (summary)

1. prepare-release.yml — Release creation (workflow_dispatch only)
   • Build → Publish → Package → Tag → Gitea Release
   • Does NOT write ConnectionStrings into the artifact (see DB Secret Management above) — only Logging config ships in appsettings.Production.json
2. deploy-prod.yml — Production deployment (workflow_dispatch only, takes a release tag)
   • 5 stages: Fetch Release → Pre-Check → Deploy → Health Check → Report
   • 6-point health checks (HTTP, login page, CSS, service, release, DB auth)
   • SSH-based deployment with artifact validation
3. ci.yml — PR validation (on:pull_request), 29 validators, runs on every pull request

Accessing Gitea Actions:

• Web UI: https://gitea.taxbaik.com/kjh2064/QuantEngineByItz/actions
• Runs API: https://gitea.taxbaik.com/api/v1/repos/kjh2064/QuantEngineByItz/actions/runs

API Monitoring (CLI)

Monitor deployment status from command line:

# Setup (one-time)
$env:GITEA_TOKEN_TAXBAIK = "your_gitea_personal_token"

# List recent deployment runs
$token = $env:GITEA_TOKEN_TAXBAIK
$response = Invoke-WebRequest `
    -Uri "https://gitea.taxbaik.com/api/v1/repos/kjh2064/QuantEngineByItz/actions/runs?limit=5" `
    -Headers @{ "Authorization" = "token $token" }
($response.Content | ConvertFrom-Json).workflow_runs | ForEach-Object {
    Write-Host "Run #$($_.id): $($_.display_title) [$($_.conclusion)]"
}

# Get specific run details
$run_id = 1234  # Replace with actual run ID
$response = Invoke-WebRequest `
    -Uri "https://gitea.taxbaik.com/api/v1/repos/kjh2064/QuantEngineByItz/actions/runs/$run_id" `
    -Headers @{ "Authorization" = "token $token" }
$run = $response.Content | ConvertFrom-Json
Write-Host "Commit: $($run.head_sha)"
Write-Host "Status: $($run.status) / $($run.conclusion)"

See docs/GITEA_ACTIONS_API_GUIDE.md for the complete API reference.

Deployment Secrets Configuration

Required Secrets (Gitea Repository Settings → Secrets):

Secret	Type	Purpose
DEPLOY_SSH_KEY_B64	Base64 (recommended)	ED25519 private key for SSH
DEPLOY_SSH_KEY	PEM (alternative)	Raw private key format
DEPLOY_HOST	Text	Production server IP (178.104.200.7)
DEPLOY_USER	Text	SSH username (kjh2064)

How to add secrets:

1. Go to: https://gitea.taxbaik.com/kjh2064/QuantEngineByItz/settings/secrets
2. Click "Add Secret"
3. Name: DEPLOY_SSH_KEY_B64
4. Value: base64 -w 0 ~/.ssh/deploy_key | pbcopy (macOS) or certutil -encode deploy_key deploy_key.b64 (Windows)
5. Save