User caught this directly: it's already 2026-07-12 in Korea, but
Run #2008's release was tagged quant_20260711.3.7150737 -- the wrong
date.
Confirmed: UTC was still 2026-07-11 16:2x when KST was already
2026-07-12 01:2x (9-hour offset). prepare-release.yml computed
TODAY via `TZ=UTC date +%Y%m%d`, which is only "correct" if the team
operates on UTC -- but this project's production server logs,
deployment cadence, and team are all Korea Standard Time. Any release
cut between midnight and 9am KST would silently tag itself with
yesterday's date.
Fixed by using `TZ=Asia/Seoul date +%Y%m%d` instead.
Per spec: the sequence number is a per-day counter that resets on
date change and starts at 0, not 1. The first release of a day is
quant_YYYYMMDD.0.hash, the second .1, etc.
Previous commit fixed *counting* today's releases via the Gitea API
(instead of the always-empty local git tags from a shallow checkout),
but still added +1 on top, which would have produced 1, 2, 3... for
the first, second, third releases of a day instead of 0, 1, 2.
DEPLOY_COUNT is now just RELEASES_TODAY directly.
User asked why every release tonight had the same "sequence number"
(quant_20260711.1.*) despite creating three of them. Confirmed via
API: tags b7591fb, 6ab270f, and e49922e all exist for 2026-07-11, all
claiming to be deploy #1.
Root cause: `actions/checkout@v4` (no fetch-depth/fetch-tags options)
does a shallow, tags-less clone by default. Each prepare-release.yml
run happens in a brand-new container, so `git tag -l "quant_${TODAY}.*"`
always sees zero local tags regardless of how many releases actually
exist -- DEPLOY_COUNT was permanently stuck at 0+1=1.
Fix: query GET /repos/{repo}/tags via the Gitea API (same token/curl
pattern already used elsewhere in this workflow) to count today's
actual tags, instead of relying on the job's local, incomplete git
state.
Run #2006 proved the deploy itself is fully working now: checks 1-5
all passed (HTTP 200, login content, CSS, service active, release
verified) -- only check 6 failed, and the log shows exactly why:
❌ [6/6] DB authentication errors found in logs (0
0 occurrences)
`grep -c PATTERN` exits with status 1 whenever the count is 0, even
though it still correctly prints "0" to stdout. The old
`grep -c ... || echo "0"` therefore printed grep's own "0" AND (because
grep's nonzero exit triggered the `||`) a second "0" from the fallback
-- a two-line "0\n0" that can never equal the string "0" in the
subsequent `[ "$DB_ERRORS" = "0" ]` check. So the *healthy* case (zero
DB errors) was the one that always failed this check.
Fixed by using `|| true` instead of `|| echo "0"`: it neutralizes
grep's exit code (needed to avoid an instant abort under `set -e
-o pipefail`, same class of bug as the earlier `git config user.name`
incident) without adding any extra output.
Run #2005's Health Check job hung for 18+ minutes (well past its own
timeout-minutes: 10) instead of failing within seconds. Killed the
zombie container manually via 'docker kill' on the runner host.
Root cause: the pre-fix curl calls to the unreachable
$DEPLOY_HOST:5000 had no --connect-timeout/--max-time, so each of the
20 retry attempts could hang on the OS's default TCP timeout instead
of failing fast; the job-level timeout-minutes didn't reliably cut it
off either (act_runner enforcement gap, not something we control from
the workflow file).
This is now largely moot after the previous commit (health checks run
against 127.0.0.1 on the server itself, where curl returns
near-instantly), but added explicit timeouts everywhere as a second
line of defense against the same failure mode recurring:
- Gitea API curl calls (release fetch, artifact download):
--connect-timeout 10 --max-time 30/120
- Local 127.0.0.1 health-check curls: --connect-timeout 5 --max-time 10
- All ssh/scp invocations: -o ConnectTimeout=10
No single curl or ssh call in this workflow should now be able to
hang indefinitely.
Root cause confirmed by direct test:
curl --connect-timeout 5 http://178.104.200.7:5000/Account/Login -> 000
quantengine.service sets ASPNETCORE_URLS=http://127.0.0.1:5000 (loopback
only, by design -- Nginx is the only public entry point, proxying
quant.taxbaik.com to it). The Gitea Actions runner is not the
production host, so its direct curl to $DEPLOY_HOST:5000 was always
going to hit a closed port. Run #2005 is direct proof: "Deploy to
Production" succeeded, the site was reachable over HTTPS the whole
time, and journalctl was clean -- yet "Health Check & Verification"
burned through all 20 retries (60s) because it was polling the wrong
address entirely. This check has likely never once passed on this
service's actual network layout.
Fix: wrap the HTTP-200 / login-content / CSS retry loop in a single
SSH session that runs curl against 127.0.0.1:5000 on the production
server itself -- consistent with how the service-status and DB-error
checks already correctly run remotely. Removed the redundant
per-attempt SSH round trips for service status (now a plain local
command inside the same remote script) and dropped the separate
"Setup SSH (for service check)" step's curl usage entirely.
Found via SSH log analysis (Run #2004, task 2336): the deploy script's
own echo output revealed the bug directly --
Deploy Dir: /home/kjh2064/deployments/quantengine_$RELEASE_TAG_$COMMIT
tar (child): /tmp/$ARTIFACT: Cannot open: No such file or directory
$ARTIFACT, $RELEASE_TAG, $COMMIT were printed as LITERAL TEXT instead
of their values. Root cause: the heredoc used a quoted delimiter
(<< 'REMOTE'), which correctly prevents the local runner shell from
expanding anything inside it -- but the script still relied on that
expansion happening for these three variables. They were never
actually being passed to the remote bash process at all; this path
had likely never worked.
Fix: pass ARTIFACT/RELEASE_TAG/COMMIT/SERVICE_NAME as env-var
prefixes on the remote `bash -s` invocation (`"VAR='...' bash -s"`),
which the LOCAL shell does expand (since it's a normal double-quoted
string, not part of the quoted heredoc). The heredoc body itself
stays fully remote-evaluated (DEPLOY_HOME=$HOME correctly resolves
to the remote user's home, not the runner's).
Also fixed: COMMIT was being read from the release's
`target_commitish` field, which is the branch name the tag points to
("main"), not a commit SHA -- confirmed by the same log ("Commit:
$COMMIT" would have printed "main" once the heredoc bug was fixed).
Since our tags are always "quant_YYYYMMDD.count.hash"
(prepare-release.yml), the hash is now parsed directly out of the
tag name instead.
Found via SSH log analysis (Run #2003, task 2334): the "Verify SSH
Key and Secrets" step failed immediately with
"DEPLOY_SSH_KEY_B64 or DEPLOY_SSH_KEY not configured" -- both were
empty. Queried GET /repos/{repo}/actions/secrets directly and found
the actually-registered secrets are named SSH_PRIVATE_KEY and
QUANTENGINE_DB_PASSWORD; DEPLOY_SSH_KEY_B64/DEPLOY_SSH_KEY were never
created, despite CLAUDE.md claiming "SSH credentials: SSH_KEY
registered in Gitea Secrets".
Every past deploy-prod.yml run that reached the SSH step (e.g. Run
#1991's Pre-Deployment Verification) failed here for the same reason
-- this was never a working path, just never diagnosed down to the
secret name before now.
Fix: check secrets.SSH_PRIVATE_KEY first (with the same PEM-vs-base64
auto-detection used for the legacy names), falling back to
DEPLOY_SSH_KEY_B64 / DEPLOY_SSH_KEY in case those get added later.
Applied to all three places that build ~/.ssh/deploy_key (deploy job
verify + setup, and post-deploy-check's setup).
Found via SSH log analysis (actions_log/.../2332.log, Run #2002):
1. This Gitea Actions instance's runner explicitly rejects the
actions/upload-artifact@v4 / download-artifact@v4 protocol:
"GHESNotSupportedError: @actions/artifact v2.0.0+,
upload-artifact@v4+ and download-artifact@v4+ are not
currently supported on GHES."
The old 3-job split (fetch-release -> pre-deploy-check -> deploy)
relied on upload-artifact/download-artifact to hand the .tar.gz
from the fetch job to the deploy job, so it could never succeed
on this server regardless of any other fix.
2. Independently, the guessed download URL pattern
/releases/download/{tag}/{filename} doesn't exist on this Gitea
instance -- it silently downloaded a 19-byte "404 page not found"
body as if it were the artifact (curl exited 0, file "existed").
Fixes:
- Merge fetch-release + pre-deploy-check + deploy into a single
`deploy` job so the downloaded artifact never needs to cross a
job boundary -- it's downloaded and scp'd from the same runner
filesystem in one shot.
- Fetch the real `browser_download_url` from the release JSON
instead of constructing the URL by convention.
- Add a `file "$ARTIFACT" | grep -q "gzip compressed"` guard right
after download so a wrong-URL / error-page download fails loudly
instead of silently proceeding with garbage bytes.
- Update post-deploy-check / post-deploy-report to read from
`needs.deploy.outputs.*` now that fetch-release no longer exists
as a separate job.
- CLAUDE.md: Add "DB Secret Management" section documenting the
incident, the root cause (stale password baked into
appsettings.Production.json, real password only ever lived in
/home/kjh2064/.config/quantengine.env, never wired into systemd),
and the permanent fix (EnvironmentFile= drop-in, applied by hand
on 2026-07-12 with 'sudo systemctl restart quantengine' verified
active and journalctl clean).
- CLAUDE.md: Refresh the stale "Gitea Actions Workflows" section
(was still describing an on:push deploy-prod.yml with a single
Build stage; now lists prepare-release.yml + deploy-prod.yml
correctly as workflow_dispatch-only, 6-point health check).
- deploy-prod.yml: Add Check 6 (DB authentication) to the health
check step. The existing checks only hit GET /Account/Login, which
returns HTTP 200 even when ConnectionStrings is broken -- that's
exactly why tonight's outage passed every prior health check. The
new check greps journalctl for '28P01'/'password authentication
failed' in the minute after restart and fails the deployment if
found, so a broken DB connection string can no longer masquerade
as a successful deploy.
Production incident: quant.taxbaik.com/login threw 28P01 (password
authentication failed) after the July 7 deployment's
appsettings.Production.json carried a stale DB password. Root cause
chain:
1. The DB password for quantengine_app had been rotated at some
point; the new password was saved to
/home/kjh2064/.config/quantengine.env on the server, but that
file was never wired into the quantengine.service systemd unit
(no EnvironmentFile= directive), so it was silently unused.
2. Every appsettings.Production.json we've generated in CI
(including tonight's prepare-release.yml) baked in a PLACEHOLDER
password ("quantengine_app") that was never the real credential
to begin with -- copied forward from an earlier debugging session
without ever being verified against the live DB.
Immediate production fix (out of band, via SSH): patched the active
deployment's appsettings.Production.json with the current working
password (verified via direct psql connection) and restarted the
service. Login confirmed HTTP 200 with a clean journalctl afterward.
This commit fixes the root cause in the pipeline: prepare-release.yml
no longer writes a ConnectionStrings block into the artifact at all.
Baking any DB password (even a correct one) into a build artifact
that ships as a downloadable Gitea Release asset is unsafe and goes
stale on every credential rotation. The correct fix is for
quantengine.service to load ConnectionStrings__DefaultConnection from
/home/kjh2064/.config/quantengine.env via systemd's EnvironmentFile=,
which overrides appsettings.Production.json at runtime per standard
ASP.NET Core configuration precedence. That unit-file edit requires
interactive sudo and must be applied by hand on the server (tracked
separately, not part of this commit).
IMPORTANT: the release quant_20260711.1.6ab270f already published
tonight was built before this fix and still lacks any DB config --
do not deploy it via deploy-prod.yml until the systemd
EnvironmentFile wiring is confirmed on the server, or the login
outage will recur.
Root cause found via SSH log analysis (actions_log/.../2326.log):
'gh release create' failed with exit code 127 (command not found).
The act_runner Docker image used for jobs does not ship the
GitHub CLI (gh), so any step relying on it fails immediately.
Fix: Replace gh CLI calls with direct Gitea REST API calls using
curl, which is available in the base image:
1. POST /repos/{repo}/releases -- create release, parse id via python3
2. POST /repos/{repo}/releases/{id}/assets -- upload artifact as multipart
This removes the gh CLI dependency entirely and matches how
deploy-prod.yml already talks to Gitea (curl + REST API).
Root cause found via SSH log analysis (actions_log/.../2324.log):
'git config user.name' returned exit code 1 (no global identity set
in the Gitea Actions runner container), and since the step uses
'bash -e -o pipefail', the script aborted immediately at that line
before ever reaching 'git tag'.
Fix: explicitly set git user.name/user.email before tagging, and
remove the fragile bare 'git config user.name' debug calls.
Also removed the '|| echo ...continuing' fallback on git push so
push failures are now visible as real failures instead of swallowed.
- Add git config output for debugging tag creation
- Add artifact existence check
- Add gh CLI version check
- Add explicit --repo parameter for gh release create
- Make tag push non-fatal to continue workflow
- Auto-generate version format: quant_YYYYMMDD.count.hash
- Count existing tags for today to determine deploy count
- Add job outputs for version and commit
- Simplify release notes format to fix YAML parsing error
- Make version input optional (auto-generated if empty)
Missing configuration file step caused DB authentication failure.
Added Python config generator (taxbaik pattern) to create
appsettings.Production.json with DB connection string before packaging.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
【 개선사항 】
1. Build 단계 분리: metadata 생성, artifact 관리
2. Pre-deployment 검증: SSH, secrets, artifact, connectivity
3. 실제 배포: SSH를 통한 원격 배포, symlink 관리
4. 헬스 체크: 10회 재시도, 상세 검증
5. 배포 후 검증: 실제 서비스 상태 확인
6. 완벽한 에러 처리: 각 단계별 fail-fast
7. 배포 결과 리포팅: 성공/실패 알림
【 구조 】
- Build: .NET 빌드 + 아티팩트 생성
- Pre-deploy-check: SSH/Secrets/Artifact/Connectivity 검증
- Deploy: 실제 배포 + 헬스 체크
- Post-deploy: 배포 결과 리포팅
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
kjh2064