QuantEngineByItz/docs/DEPLOYMENT_RUNBOOK.md

# QuantEngine Deployment Runbook

Full deployment procedure, extracted from CLAUDE.md (2026-07-30) to keep the main file within
the character budget. CLAUDE.md keeps the CRITICAL rules (CI/CD-only mandate, DB secret
management); this file has the complete step-by-step runbook.

**Production Server**: Hetzner Cloud `178.104.200.7` (kjh2064@178.104.200.7)

Projects on server:
1. **TaxBaik** (홈페이지) — Nginx location `/taxbaik`
2. **QuantEngine** (데이터 수집/분석) — Nginx location `/quantengine`

## ⚠️ CRITICAL: CI/CD-Only Deployment Mandate

**Rule**: ALL production deployments MUST go through Gitea Actions CI/CD. Manual SSH deployments are **FORBIDDEN**.

**Why**:
- Automatic validation (build, health checks, version verification)
- Audit trail (all deployments logged in Gitea Actions)
- Consistent process (no manual errors)
- Rollback safety (deployment history retained)
- Release traceability (version control via git tags)

## ⚠️ CRITICAL: DB Secret Management (Incident 2026-07-12)

**Incident**: `quant.taxbaik.com/login`이 `28P01 password authentication failed`로 장애 발생.
원인: `appsettings.Production.json`에 하드코딩되어 배포된 DB 비밀번호가, 실제 DB 비밀번호가
로테이션된 이후에도 계속 옛날 값(심지어 이전 세션에서 검증 없이 넣은 placeholder였던 적도 있음)
그대로 배포되고 있었음.

**Rule**: **DB 접속 문자열(`ConnectionStrings`)은 절대 `appsettings.Production.json`이나
워크플로우 파일에 하드코딩하지 않는다.** `prepare-release.yml`이 생성하는
`appsettings.Production.json`에는 `Logging` 설정만 있고 `ConnectionStrings`는 없다 —
이는 의도된 설계다 (Gitea Release는 누구나 다운로드 가능한 아티팩트이므로 시크릿을
담으면 안 됨).

**실제 DB 비밀번호의 출처**: 프로덕션 서버의 `/home/kjh2064/.config/quantengine.env`
파일 (`ConnectionStrings__DefaultConnection=...` 형식) 하나뿐이며,
`quantengine.service.d/env.conf` drop-in의 `EnvironmentFile=` 지시자로 systemd가
이 값을 환경변수로 주입한다. ASP.NET Core 설정 우선순위상 **환경변수가
`appsettings.Production.json`을 오버라이드**하므로, 배포되는 아티팩트 자체에는
DB 정보가 없어도 서비스는 정상 동작한다.

**DB 비밀번호가 바뀌면** (로테이션 등): `/home/kjh2064/.config/quantengine.env` 파일만
갱신하고 `sudo systemctl restart quantengine`. 워크플로우 파일이나 Gitea Secrets는
건드릴 필요 없음 (배포 파이프라인은 DB 비밀번호를 모른 채로 동작해야 정상).

**배포 전 체크리스트에 추가**:
- ✅ 새 릴리즈 배포 후 반드시 `/Account/Login` 실제 HTTP 응답 + `journalctl -u quantengine`에서
  `28P01`/`password authentication failed` 부재 확인 (단순 프로세스 `active` 상태만으로는
  DB 연결 실패를 못 잡음 — ASP.NET Core는 DB 없이도 기동은 되고 로그인 요청 시점에야 실패함)
- ✅ `.config/quantengine.env`의 존재와 `quantengine.service.d/env.conf`의
  `EnvironmentFile=` 배선이 서버에 유지되고 있는지 (systemd unit 자체를 재생성/덮어쓰는
  배포 방식으로 전환할 경우 이 drop-in이 날아가지 않는지 확인 필요)

## Production Deployment Strategy (Release-Based)

**Architecture**: Two-Workflow System (Release Creation → Deployment)

### Workflow 1: prepare-release.yml (Release Creation)

**Purpose**: Create a release with built artifact

**Trigger**: Manual (`workflow_dispatch`)
```bash
# Visit Gitea Actions and select prepare-release.yml
# Input version: v0.1.20260711 (or any semantic version)
```

**What it does**:
1. ✓ Build (restore, build, publish)
2. ✓ Generate `appsettings.Production.json`
3. ✓ Package artifact: `.tar.gz`
4. ✓ Create git tag: `v0.1.20260711`
5. ✓ Create Gitea Release with artifact attached
6. ✓ Notify: Release ready for deployment

**Output**: Gitea Release with downloadable artifact

### Workflow 2: deploy-prod.yml (Deployment)

**Purpose**: Deploy a release to production

**Trigger**: Manual (`workflow_dispatch`)
```bash
# Visit Gitea Actions and select deploy-prod.yml
# Input release: v0.1.20260711 (optional — uses latest if empty)
```

**What it does**:
1. ✓ Fetch Release (from Gitea Releases)
2. ✓ Download artifact
3. ✓ Verify SSH credentials
4. ✓ Upload to production server
5. ✓ Extract and symlink
6. ✓ Restart service
7. ✓ 6-point health checks
8. ✓ Report deployment status

**Deployment Pipeline (5 Stages)**:

| Stage | Purpose | Timeout |
|-------|---------|---------|
| 1. Fetch Release | Query Gitea Releases, download artifact | 10min |
| 2. Pre-Check | Verify SSH keys, secrets, release | 5min |
| 3. Deploy | Upload, extract, symlink, restart service | 30min |
| 4. Health Check | 6-point verification (HTTP, CSS, login, service, release, DB auth) | 10min |
| 5. Report | Final deployment status | Auto |

**Health Checks (Automatic)**:
- ✓ HTTP 200 on `/Account/Login`
- ✓ Login page content verification
- ✓ CSS file loads (`/css/admin.css`)
- ✓ Service status (systemctl active)
- ✓ Release verification (deployed release tag matches)
- ✓ **DB authentication check** (`journalctl`에서 `28P01`/`password authentication failed`
  부재 확인 — GET `/Account/Login`은 DB가 끊겨도 200을 반환하므로 이 체크가 없으면
  DB 장애를 배포 파이프라인이 놓친다. 2026-07-12 사고 이후 추가됨)

**Complete Deployment Flow**:
```
1. Code committed to main branch
2. Create release: prepare-release.yml workflow_dispatch (manual)
   → Builds code
   → Creates Gitea Release with artifact
   → Tags repository
3. Deploy release: deploy-prod.yml workflow_dispatch (manual)
   → Selects release version
   → Downloads artifact from Gitea Release
   → Deploys to production server
   → Runs health checks
   → Reports status
```

## Pre-Deployment Checklist

**Before creating a release**, verify:
1. ✅ Local build: `dotnet build src/dotnet/QuantEngine.Web/QuantEngine.Web.csproj -c Release` (0 errors, 0 warnings)
2. ✅ E2E tests pass: `npx playwright test`
3. ✅ Admin pages verified (200 status, no 500 errors)
4. ✅ All changes committed and pushed to main branch
5. ✅ No uncommitted changes: `git status`

## Release & Deployment Workflow

**Step 1: Create Release (prepare-release.yml)**
```bash
# Visit Gitea Actions
# https://gitea.taxbaik.com/kjh2064/QuantEngineByItz/actions

# Run prepare-release.yml workflow
# Input: version = v0.1.20260711

# Workflow will:
# - Build and publish
# - Package artifact
# - Create git tag
# - Create Gitea Release
# - Attach artifact
```

**Step 2: Deploy Release (deploy-prod.yml)**
```bash
# Visit Gitea Actions (same page)
# Run deploy-prod.yml workflow
# Input: release = v0.1.20260711 (leave empty for latest)

# Workflow will:
# - Download artifact from release
# - Deploy to production server
# - Run health checks
# - Report status
```

## SSH Key Configuration (Required)

**Setup (One-time)**:
1. Generate ED25519 key locally (or reuse existing):
   ```bash
   ssh-keygen -t ed25519 -f ~/.ssh/quantengine_deploy -C "QuantEngine CI/CD"
   ```

2. Add public key to production server:
   ```bash
   ssh-copy-id -i ~/.ssh/quantengine_deploy.pub kjh2064@178.104.200.7
   ```

3. Get private key in base64 format:
   ```bash
   # macOS/Linux
   base64 -w 0 ~/.ssh/quantengine_deploy > /tmp/key_b64.txt
   cat /tmp/key_b64.txt | pbcopy

   # Or Windows PowerShell
   $key = Get-Content ~/.ssh/quantengine_deploy -Raw
   [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($key)) | Set-Clipboard
   ```

4. Configure in Gitea:
   - URL: https://gitea.taxbaik.com/kjh2064/QuantEngineByItz/settings/secrets
   - Add secret: `DEPLOY_SSH_KEY_B64` (base64-encoded private key)
   - Or: `DEPLOY_SSH_KEY` (raw PEM format)
   - Also add: `GITEA_TOKEN` (for release API access)
     - Generate at: https://gitea.taxbaik.com/user/settings/applications
     - Required permissions: `repo` + `read:actions`

## Deployment Monitoring

**During Deployment**:
- Watch live in Gitea Actions UI
- Jobs complete in order: Build → Pre-Check → Deploy → Health Check → Report

**After Deployment**:
```bash
# SSH into server
ssh kjh2064@178.104.200.7

# Check active deployment
readlink ~/quantengine_active

# View service status
systemctl status quantengine

# Tail live logs
journalctl -u quantengine -f

# Health check
curl -I http://127.0.0.1:5000/Account/Login
```

## Automatic Rollback (if health check fails)

If health check fails, deployment stops automatically:
1. Service restart may fail
2. Symlink update reverts to previous deployment
3. Gitea Actions marks deployment as FAILED
4. Logs include failure details

Manual rollback (if needed):
```bash
# List deployments
ls -lht ~/deployments/quantengine_*

# Revert symlink to previous version
ln -sfn /home/kjh2064/deployments/quantengine_YYYYMMDD_HHMMSS_COMMIT ~/quantengine_active

# Restart service
sudo systemctl restart quantengine

# Verify
curl http://127.0.0.1:5000/Account/Login
```

## Troubleshooting Deployment Failures

**Issue**: Build fails
- Check: `dotnet build` locally first
- Ensure: No compilation errors, 0 warnings

**Issue**: Health check timeout
- Check: Service logs: `journalctl -u quantengine -n 50`
- Check: Port 5000 listening: `ss -tlnp | grep 5000`
- Check: DB connectivity in appsettings.Production.json

**Issue**: SSH key error
- Verify: `DEPLOY_SSH_KEY_B64` or `DEPLOY_SSH_KEY` in Gitea Secrets
- Check: Public key added to `~/.ssh/authorized_keys` on server
- Test: `ssh -i ~/.ssh/key_file kjh2064@178.104.200.7 echo OK`

## Git Repository

**Gitea Server** (동일 호스트):
- **HTTP**: `https://gitea.taxbaik.com/kjh2064/QuantEngineByItz.git`
- **SSH**: `ssh://git@gitea.taxbaik.com:2222/kjh2064/QuantEngineByItz.git`

## Active Gitea Workflows (summary)

1. **prepare-release.yml** — Release creation (workflow_dispatch only)
   - Build → Publish → Package → Tag → Gitea Release
   - Does NOT write ConnectionStrings into the artifact (see DB Secret Management above) —
     only `Logging` config ships in `appsettings.Production.json`
2. **deploy-prod.yml** — Production deployment (workflow_dispatch only, takes a release tag)
   - 5 stages: Fetch Release → Pre-Check → Deploy → Health Check → Report
   - 6-point health checks (HTTP, login page, CSS, service, release, DB auth)
   - SSH-based deployment with artifact validation
3. **ci.yml** — PR validation (on:pull_request), 29 validators, runs on every pull request

**Accessing Gitea Actions**:
- Web UI: https://gitea.taxbaik.com/kjh2064/QuantEngineByItz/actions
- Runs API: https://gitea.taxbaik.com/api/v1/repos/kjh2064/QuantEngineByItz/actions/runs

## API Monitoring (CLI)

Monitor deployment status from command line:

```powershell
# Setup (one-time)
$env:GITEA_TOKEN_TAXBAIK = "your_gitea_personal_token"

# List recent deployment runs
$token = $env:GITEA_TOKEN_TAXBAIK
$response = Invoke-WebRequest `
    -Uri "https://gitea.taxbaik.com/api/v1/repos/kjh2064/QuantEngineByItz/actions/runs?limit=5" `
    -Headers @{ "Authorization" = "token $token" }
($response.Content | ConvertFrom-Json).workflow_runs | ForEach-Object {
    Write-Host "Run #$($_.id): $($_.display_title) [$($_.conclusion)]"
}

# Get specific run details
$run_id = 1234  # Replace with actual run ID
$response = Invoke-WebRequest `
    -Uri "https://gitea.taxbaik.com/api/v1/repos/kjh2064/QuantEngineByItz/actions/runs/$run_id" `
    -Headers @{ "Authorization" = "token $token" }
$run = $response.Content | ConvertFrom-Json
Write-Host "Commit: $($run.head_sha)"
Write-Host "Status: $($run.status) / $($run.conclusion)"
```

See `docs/GITEA_ACTIONS_API_GUIDE.md` for the complete API reference.

## Deployment Secrets Configuration

**Required Secrets** (Gitea Repository Settings → Secrets):

| Secret | Type | Purpose |
|--------|------|---------|
| `DEPLOY_SSH_KEY_B64` | Base64 (recommended) | ED25519 private key for SSH |
| `DEPLOY_SSH_KEY` | PEM (alternative) | Raw private key format |
| `DEPLOY_HOST` | Text | Production server IP (178.104.200.7) |
| `DEPLOY_USER` | Text | SSH username (kjh2064) |

**How to add secrets**:
1. Go to: https://gitea.taxbaik.com/kjh2064/QuantEngineByItz/settings/secrets
2. Click "Add Secret"
3. Name: `DEPLOY_SSH_KEY_B64`
4. Value: `base64 -w 0 ~/.ssh/deploy_key | pbcopy` (macOS) or `certutil -encode deploy_key deploy_key.b64` (Windows)
5. Save