User reported: "배포가 되면 인증이 풀린다" (auth resets after every
deployment).
Root cause: Program.cs had no explicit Data Protection configuration.
Without SetApplicationName, ASP.NET Core derives the key-ring
discriminator from the app's physical content root path. Every
deployment lands in a brand-new directory
(~/deployments/quantengine_{tag}_{hash}/), so the discriminator
changed on every single release. The cookie authentication ticket is
encrypted/signed via this key ring, so once the discriminator
changed, every previously-issued auth cookie became undecryptable --
forcing all logged-in users to authenticate again after each deploy,
even well inside their 12-hour ExpireTimeSpan.
Fix: explicit .SetApplicationName("QuantEngine") pins a stable
discriminator across deployments, and .PersistKeysToFileSystem points
at %LOCALAPPDATA%/quantengine-keys (Linux: ~/.local/share/quantengine-keys
via User=kjh2064 in the systemd unit) -- a location outside the
versioned deployment directories, so the actual key material also
survives every redeploy and service restart instead of only the
discriminator being stable.
Discovered while verifying the new Operations page against a local
instance (SSH-tunneled to prod Postgres): every startup logged
'Hangfire setup failed: Cannot resolve scoped service
QuantEngine.Web.Services.SchedulerService from root provider' and
silently skipped InitializeSchedules() entirely.
SchedulerService is registered AddScoped, but UseHangfireSetup()
resolved it directly from app.Services (the root/singleton-level
provider), which cannot construct scoped services without an active
scope. This has apparently been broken for a while -- the 4 recurring
jobs (daily-collection, hourly-price-update, weekly-report,
monthly-optimization) only kept showing up because Hangfire persists
recurring job definitions in PostgreSQL from whatever earlier
deployment last managed to register them; any code change to those
schedules would silently never take effect on redeploy.
Fixed by creating an explicit scope (serviceProvider.CreateScope())
before resolving SchedulerService. Verified locally: the warning is
gone and the log now shows "Hangfire schedules initialized
successfully" followed by the dispatchers starting.
Root cause: user asked why logout was missing. Playwright audit against
production found logout works fine, but surfaced two real defects and
led to a wider audit that found extensive fabricated data across the
admin pages -- none of it backed by the database despite CLAUDE.md's
policy that all data must come from DB records.
Layout (_AdminLayout.cshtml):
- Full rewrite using Tabler's actual navbar-vertical/page-wrapper/footer
component structure instead of ad-hoc inline CSS. The old layout had
no footer element at all, and its mobile breakpoint CSS hid the
sidebar off-screen (left: -260px) with no hamburger button to bring
it back -- verified via Playwright screenshot at 375px width that
the entire nav menu was inaccessible on mobile, leaving only Logout
reachable. Tabler's navbar-toggler + Bootstrap collapse (bundled in
tabler.min.js) now restores it; verified the toggle actually opens
the menu via Playwright.
- Active nav-link highlighting moved from client-side JS string
matching to a server-side Razor helper against Context.Request.Path.
Fake/hardcoded data removed or replaced with real DB/Hangfire state:
- Dashboard: deleted the "최근 시스템 이벤트" table (3 rows hardcoded
from DateTime.Now with fake descriptions like "시스템 초기화" /
"데이터베이스 백업" -- no backing table exists). Removed hardcoded
"정상"/"연결됨" status badges and "버전: v0.1.0"/"업타임: 정상";
replaced with a real IsDatabaseConnected flag (true only if the
page's actual DB queries succeeded) and the real
IWebHostEnvironment.EnvironmentName.
- Monitoring: removed hardcoded "API 서버: 운영 중" (no real signal
backs it) and wired "데이터베이스: 연결 정상/끊김" to the same
real success/failure state as the page's own DB calls.
- Operations: this page was entirely fabricated -- ScheduledJobs,
RecentExecutions, IsJobProcessorRunning, PendingJobsCount, and
StatusMessage were all static values with zero connection to
Hangfire, despite Hangfire actually running in production
(confirmed via journalctl: ServerWatchdog, RecurringJobScheduler
dispatchers active) with 4 real recurring jobs registered in
SchedulerService (daily-collection, hourly-price-update,
weekly-report, monthly-optimization). Rewrote to query
JobStorage.Current.GetConnection().GetRecurringJobs() and
GetMonitoringApi() directly: real scheduled jobs, real succeeded/
failed executions, real server count, real enqueued count. Verified
locally (SSH-tunneled to prod DB) that this now returns the actual
4 registered jobs with correct next-run times and one real
RunDailyCollectionAsync execution.
Also fixed the page-title duplication on Monitoring/Operations
(ViewData["Title"] included "- QuantEngine" AND the layout appended
it again -> "모니터링 - QuantEngine - QuantEngine" in the browser tab).
Separately discovered (not fixed in this commit, flagging for
follow-up): Hangfire's SchedulerService.InitializeSchedules() fails
every startup with "Cannot resolve scoped service 'SchedulerService'
from root provider" -- the 4 recurring jobs above still show up
because they persist from an earlier successful registration, but
re-registration is silently broken on every current boot.
Verified end-to-end with Playwright against a local instance (SSH
tunnel to production Postgres): login, all 5 admin pages render
without errors, mobile hamburger opens the sidebar, and Operations
shows genuine Hangfire data.
User caught this directly: it's already 2026-07-12 in Korea, but
Run #2008's release was tagged quant_20260711.3.7150737 -- the wrong
date.
Confirmed: UTC was still 2026-07-11 16:2x when KST was already
2026-07-12 01:2x (9-hour offset). prepare-release.yml computed
TODAY via `TZ=UTC date +%Y%m%d`, which is only "correct" if the team
operates on UTC -- but this project's production server logs,
deployment cadence, and team are all Korea Standard Time. Any release
cut between midnight and 9am KST would silently tag itself with
yesterday's date.
Fixed by using `TZ=Asia/Seoul date +%Y%m%d` instead.
Per spec: the sequence number is a per-day counter that resets on
date change and starts at 0, not 1. The first release of a day is
quant_YYYYMMDD.0.hash, the second .1, etc.
Previous commit fixed *counting* today's releases via the Gitea API
(instead of the always-empty local git tags from a shallow checkout),
but still added +1 on top, which would have produced 1, 2, 3... for
the first, second, third releases of a day instead of 0, 1, 2.
DEPLOY_COUNT is now just RELEASES_TODAY directly.
User asked why every release tonight had the same "sequence number"
(quant_20260711.1.*) despite creating three of them. Confirmed via
API: tags b7591fb, 6ab270f, and e49922e all exist for 2026-07-11, all
claiming to be deploy #1.
Root cause: `actions/checkout@v4` (no fetch-depth/fetch-tags options)
does a shallow, tags-less clone by default. Each prepare-release.yml
run happens in a brand-new container, so `git tag -l "quant_${TODAY}.*"`
always sees zero local tags regardless of how many releases actually
exist -- DEPLOY_COUNT was permanently stuck at 0+1=1.
Fix: query GET /repos/{repo}/tags via the Gitea API (same token/curl
pattern already used elsewhere in this workflow) to count today's
actual tags, instead of relying on the job's local, incomplete git
state.
Run #2006 proved the deploy itself is fully working now: checks 1-5
all passed (HTTP 200, login content, CSS, service active, release
verified) -- only check 6 failed, and the log shows exactly why:
❌ [6/6] DB authentication errors found in logs (0
0 occurrences)
`grep -c PATTERN` exits with status 1 whenever the count is 0, even
though it still correctly prints "0" to stdout. The old
`grep -c ... || echo "0"` therefore printed grep's own "0" AND (because
grep's nonzero exit triggered the `||`) a second "0" from the fallback
-- a two-line "0\n0" that can never equal the string "0" in the
subsequent `[ "$DB_ERRORS" = "0" ]` check. So the *healthy* case (zero
DB errors) was the one that always failed this check.
Fixed by using `|| true` instead of `|| echo "0"`: it neutralizes
grep's exit code (needed to avoid an instant abort under `set -e
-o pipefail`, same class of bug as the earlier `git config user.name`
incident) without adding any extra output.
Run #2005's Health Check job hung for 18+ minutes (well past its own
timeout-minutes: 10) instead of failing within seconds. Killed the
zombie container manually via 'docker kill' on the runner host.
Root cause: the pre-fix curl calls to the unreachable
$DEPLOY_HOST:5000 had no --connect-timeout/--max-time, so each of the
20 retry attempts could hang on the OS's default TCP timeout instead
of failing fast; the job-level timeout-minutes didn't reliably cut it
off either (act_runner enforcement gap, not something we control from
the workflow file).
This is now largely moot after the previous commit (health checks run
against 127.0.0.1 on the server itself, where curl returns
near-instantly), but added explicit timeouts everywhere as a second
line of defense against the same failure mode recurring:
- Gitea API curl calls (release fetch, artifact download):
--connect-timeout 10 --max-time 30/120
- Local 127.0.0.1 health-check curls: --connect-timeout 5 --max-time 10
- All ssh/scp invocations: -o ConnectTimeout=10
No single curl or ssh call in this workflow should now be able to
hang indefinitely.
Root cause confirmed by direct test:
curl --connect-timeout 5 http://178.104.200.7:5000/Account/Login -> 000
quantengine.service sets ASPNETCORE_URLS=http://127.0.0.1:5000 (loopback
only, by design -- Nginx is the only public entry point, proxying
quant.taxbaik.com to it). The Gitea Actions runner is not the
production host, so its direct curl to $DEPLOY_HOST:5000 was always
going to hit a closed port. Run #2005 is direct proof: "Deploy to
Production" succeeded, the site was reachable over HTTPS the whole
time, and journalctl was clean -- yet "Health Check & Verification"
burned through all 20 retries (60s) because it was polling the wrong
address entirely. This check has likely never once passed on this
service's actual network layout.
Fix: wrap the HTTP-200 / login-content / CSS retry loop in a single
SSH session that runs curl against 127.0.0.1:5000 on the production
server itself -- consistent with how the service-status and DB-error
checks already correctly run remotely. Removed the redundant
per-attempt SSH round trips for service status (now a plain local
command inside the same remote script) and dropped the separate
"Setup SSH (for service check)" step's curl usage entirely.
Found via SSH log analysis (Run #2004, task 2336): the deploy script's
own echo output revealed the bug directly --
Deploy Dir: /home/kjh2064/deployments/quantengine_$RELEASE_TAG_$COMMIT
tar (child): /tmp/$ARTIFACT: Cannot open: No such file or directory
$ARTIFACT, $RELEASE_TAG, $COMMIT were printed as LITERAL TEXT instead
of their values. Root cause: the heredoc used a quoted delimiter
(<< 'REMOTE'), which correctly prevents the local runner shell from
expanding anything inside it -- but the script still relied on that
expansion happening for these three variables. They were never
actually being passed to the remote bash process at all; this path
had likely never worked.
Fix: pass ARTIFACT/RELEASE_TAG/COMMIT/SERVICE_NAME as env-var
prefixes on the remote `bash -s` invocation (`"VAR='...' bash -s"`),
which the LOCAL shell does expand (since it's a normal double-quoted
string, not part of the quoted heredoc). The heredoc body itself
stays fully remote-evaluated (DEPLOY_HOME=$HOME correctly resolves
to the remote user's home, not the runner's).
Also fixed: COMMIT was being read from the release's
`target_commitish` field, which is the branch name the tag points to
("main"), not a commit SHA -- confirmed by the same log ("Commit:
$COMMIT" would have printed "main" once the heredoc bug was fixed).
Since our tags are always "quant_YYYYMMDD.count.hash"
(prepare-release.yml), the hash is now parsed directly out of the
tag name instead.
Found via SSH log analysis (Run #2003, task 2334): the "Verify SSH
Key and Secrets" step failed immediately with
"DEPLOY_SSH_KEY_B64 or DEPLOY_SSH_KEY not configured" -- both were
empty. Queried GET /repos/{repo}/actions/secrets directly and found
the actually-registered secrets are named SSH_PRIVATE_KEY and
QUANTENGINE_DB_PASSWORD; DEPLOY_SSH_KEY_B64/DEPLOY_SSH_KEY were never
created, despite CLAUDE.md claiming "SSH credentials: SSH_KEY
registered in Gitea Secrets".
Every past deploy-prod.yml run that reached the SSH step (e.g. Run
#1991's Pre-Deployment Verification) failed here for the same reason
-- this was never a working path, just never diagnosed down to the
secret name before now.
Fix: check secrets.SSH_PRIVATE_KEY first (with the same PEM-vs-base64
auto-detection used for the legacy names), falling back to
DEPLOY_SSH_KEY_B64 / DEPLOY_SSH_KEY in case those get added later.
Applied to all three places that build ~/.ssh/deploy_key (deploy job
verify + setup, and post-deploy-check's setup).
Found via SSH log analysis (actions_log/.../2332.log, Run #2002):
1. This Gitea Actions instance's runner explicitly rejects the
actions/upload-artifact@v4 / download-artifact@v4 protocol:
"GHESNotSupportedError: @actions/artifact v2.0.0+,
upload-artifact@v4+ and download-artifact@v4+ are not
currently supported on GHES."
The old 3-job split (fetch-release -> pre-deploy-check -> deploy)
relied on upload-artifact/download-artifact to hand the .tar.gz
from the fetch job to the deploy job, so it could never succeed
on this server regardless of any other fix.
2. Independently, the guessed download URL pattern
/releases/download/{tag}/{filename} doesn't exist on this Gitea
instance -- it silently downloaded a 19-byte "404 page not found"
body as if it were the artifact (curl exited 0, file "existed").
Fixes:
- Merge fetch-release + pre-deploy-check + deploy into a single
`deploy` job so the downloaded artifact never needs to cross a
job boundary -- it's downloaded and scp'd from the same runner
filesystem in one shot.
- Fetch the real `browser_download_url` from the release JSON
instead of constructing the URL by convention.
- Add a `file "$ARTIFACT" | grep -q "gzip compressed"` guard right
after download so a wrong-URL / error-page download fails loudly
instead of silently proceeding with garbage bytes.
- Update post-deploy-check / post-deploy-report to read from
`needs.deploy.outputs.*` now that fetch-release no longer exists
as a separate job.
- CLAUDE.md: Add "DB Secret Management" section documenting the
incident, the root cause (stale password baked into
appsettings.Production.json, real password only ever lived in
/home/kjh2064/.config/quantengine.env, never wired into systemd),
and the permanent fix (EnvironmentFile= drop-in, applied by hand
on 2026-07-12 with 'sudo systemctl restart quantengine' verified
active and journalctl clean).
- CLAUDE.md: Refresh the stale "Gitea Actions Workflows" section
(was still describing an on:push deploy-prod.yml with a single
Build stage; now lists prepare-release.yml + deploy-prod.yml
correctly as workflow_dispatch-only, 6-point health check).
- deploy-prod.yml: Add Check 6 (DB authentication) to the health
check step. The existing checks only hit GET /Account/Login, which
returns HTTP 200 even when ConnectionStrings is broken -- that's
exactly why tonight's outage passed every prior health check. The
new check greps journalctl for '28P01'/'password authentication
failed' in the minute after restart and fails the deployment if
found, so a broken DB connection string can no longer masquerade
as a successful deploy.
Production incident: quant.taxbaik.com/login threw 28P01 (password
authentication failed) after the July 7 deployment's
appsettings.Production.json carried a stale DB password. Root cause
chain:
1. The DB password for quantengine_app had been rotated at some
point; the new password was saved to
/home/kjh2064/.config/quantengine.env on the server, but that
file was never wired into the quantengine.service systemd unit
(no EnvironmentFile= directive), so it was silently unused.
2. Every appsettings.Production.json we've generated in CI
(including tonight's prepare-release.yml) baked in a PLACEHOLDER
password ("quantengine_app") that was never the real credential
to begin with -- copied forward from an earlier debugging session
without ever being verified against the live DB.
Immediate production fix (out of band, via SSH): patched the active
deployment's appsettings.Production.json with the current working
password (verified via direct psql connection) and restarted the
service. Login confirmed HTTP 200 with a clean journalctl afterward.
This commit fixes the root cause in the pipeline: prepare-release.yml
no longer writes a ConnectionStrings block into the artifact at all.
Baking any DB password (even a correct one) into a build artifact
that ships as a downloadable Gitea Release asset is unsafe and goes
stale on every credential rotation. The correct fix is for
quantengine.service to load ConnectionStrings__DefaultConnection from
/home/kjh2064/.config/quantengine.env via systemd's EnvironmentFile=,
which overrides appsettings.Production.json at runtime per standard
ASP.NET Core configuration precedence. That unit-file edit requires
interactive sudo and must be applied by hand on the server (tracked
separately, not part of this commit).
IMPORTANT: the release quant_20260711.1.6ab270f already published
tonight was built before this fix and still lacks any DB config --
do not deploy it via deploy-prod.yml until the systemd
EnvironmentFile wiring is confirmed on the server, or the login
outage will recur.
Document the two-stage debugging pattern discovered while fixing
prepare-release.yml (Run #1996-2000):
1. PowerShell harness for workflow_dispatch trigger + poll-to-completion
- Working pattern for POST .../dispatches (204 = success)
- Known PowerShell/HttpClient limitation: cannot read error response
body via GetResponseStream() in PS7
2. SSH log-reading harness for when the Gitea API has no working
/logs endpoint (404 on job logs):
- Match runner container logs (task ID) to the triggered run
- Locate actions_log/{owner}/{repo}/{shard}/{taskId}.log.zst
- Stream-decompress with 'zstd -dc' and grep for 'Failure'/'exitcode'
3. Network debugging commands for dispatch 500s / stuck runners
(docker network inspect, restart timing, exec connectivity test)
4. Table of real failure patterns hit and their fixes (YAML multiline
notes, unset git identity, missing gh CLI in runner image)
Root cause found via SSH log analysis (actions_log/.../2326.log):
'gh release create' failed with exit code 127 (command not found).
The act_runner Docker image used for jobs does not ship the
GitHub CLI (gh), so any step relying on it fails immediately.
Fix: Replace gh CLI calls with direct Gitea REST API calls using
curl, which is available in the base image:
1. POST /repos/{repo}/releases -- create release, parse id via python3
2. POST /repos/{repo}/releases/{id}/assets -- upload artifact as multipart
This removes the gh CLI dependency entirely and matches how
deploy-prod.yml already talks to Gitea (curl + REST API).
Root cause found via SSH log analysis (actions_log/.../2324.log):
'git config user.name' returned exit code 1 (no global identity set
in the Gitea Actions runner container), and since the step uses
'bash -e -o pipefail', the script aborted immediately at that line
before ever reaching 'git tag'.
Fix: explicitly set git user.name/user.email before tagging, and
remove the fragile bare 'git config user.name' debug calls.
Also removed the '|| echo ...continuing' fallback on git push so
push failures are now visible as real failures instead of swallowed.
- Add git config output for debugging tag creation
- Add artifact existence check
- Add gh CLI version check
- Add explicit --repo parameter for gh release create
- Make tag push non-fatal to continue workflow
- Auto-generate version format: quant_YYYYMMDD.count.hash
- Count existing tags for today to determine deploy count
- Add job outputs for version and commit
- Simplify release notes format to fix YAML parsing error
- Make version input optional (auto-generated if empty)
Fixed deploy-prod.yml now includes Python config generation step
to create appsettings.Production.json with DB connection string
before packaging artifact.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
kjh2064