Compare commits

...

10 Commits

Author SHA1 Message Date
kjh2064 e0dd400371 feat(phase1): add parameterized activation tooling + runbook
- scripts/freeze-versionset.ps1: Parameterized tool to freeze model/dataset VersionSet
  * Inserts FROZEN records into governance.model_version_registry + evaluation.dataset_manifest
  * NO default values — all 5 parameters REQUIRED (-ModelId, -DatasetId, -ApprovedBy, -ConfigVersion, -CodeSha)
  * Fails immediately if any parameter missing (enforces discipline)
  * Pre-flight: Verifies migration 0032 deployed before proceeding
  * Rationale: Prevents accidental partial freezes; seed data only with explicit values

- scripts/generate-shadow-run-identifiers.ps1: Reusable UUID generation script
  * Generates cryptographic UUIDs for Phase 1: RunId, JobId, JobRunId, CorrelationId, IdempotencyKey
  * Outputs JSON (versionset.json) for use in STEP 3
  * Rationale: Decoupled from freeze; can re-run if identifiers lost

- docs/CURRENT/PHASE-1_ACTIVATION_RUNBOOK.md: Executable runbook
  * 3-step activation: (1) FREEZE VersionSet, (2) GENERATE identifiers, (3) ENQUEUE Job 893
  * Pre-flight checklist: Migration, Host (Debug mode), SSH tunnel, Hangfire, scripts
  * Detailed expected outputs + troubleshooting matrix
  * Monitoring: Live logs, Grafana metrics, evidence artifacts
  * Emergency rollback procedure
  * ~15 min setup; 50-90 day autonomous execution

AGENTS.md: Necessity (real tooling gap: no VersionSet freeze script), Maturity (runbook before execution),
Right-Way (parameterized + validation vs ad-hoc SQL).

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
2026-08-07 14:06:56 +09:00
kjh2064 67274cbdb6 Merge pull request '배포: 날짜·일련번호 기반 프런트엔드 버전 계약 적용' (#15) from fix/deploy-build-frontend-artifact into main
deploy / deploy (push) Successful in 2m15s
deploy / notify (push) Successful in 1s
Reviewed-on: #15
2026-08-06 16:28:38 +09:00
kjh2064 9e4346efa9 DEPLOY: generate date sequence semantic frontend version 2026-08-06 16:23:47 +09:00
kjh2064 07b6fc6bb3 Merge pull request 'DEPLOY: rebuild frontend before publishing host artifact' (#14) from fix/deploy-build-frontend-artifact into main
ci / static (push) Successful in 10s
ci / frontend (push) Has been cancelled
ci / publish (push) Has been cancelled
ci / backend (push) Has been cancelled
Build & Test with Secrets / frontend (push) Has been cancelled
Build & Test with Secrets / security-scan (push) Has been cancelled
Build & Test with Secrets / notification (push) Has been cancelled
Build & Test with Secrets / build (push) Has been cancelled
deploy / deploy (push) Successful in 1m53s
deploy / notify (push) Successful in 0s
Reviewed-on: #14
2026-08-06 16:13:37 +09:00
kjh2064 366978ce0f DEPLOY: rebuild frontend before publishing host artifact
ci / static (push) Has been cancelled
ci / publish (push) Has been cancelled
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
Build & Test with Secrets / build (pull_request) Has been cancelled
Build & Test with Secrets / frontend (pull_request) Has been cancelled
Build & Test with Secrets / security-scan (pull_request) Has been cancelled
Build & Test with Secrets / notification (pull_request) Has been cancelled
ci / static (pull_request) Successful in 7s
ci / publish (pull_request) Has been cancelled
ci / frontend (pull_request) Has been cancelled
ci / backend (pull_request) Has been cancelled
2026-08-06 16:13:01 +09:00
kjh2064 c0b49959d4 Merge pull request 'DEPLOY: one-time sudo delegation for kartsell restart' (#13) from fix/deploy-kartsell-nopasswd into main
ci / static (push) Has been cancelled
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / publish (push) Has been cancelled
Build & Test with Secrets / build (push) Has been cancelled
Build & Test with Secrets / frontend (push) Has been cancelled
Build & Test with Secrets / security-scan (push) Has been cancelled
Build & Test with Secrets / notification (push) Has been cancelled
deploy / deploy (push) Successful in 2m35s
deploy / notify (push) Successful in 1s
Reviewed-on: #13
2026-08-06 16:05:28 +09:00
kjh2064 1c685e2285 DEPLOY: delegate kartsell restart without interactive sudo
ci / static (push) Successful in 11s
ci / publish (push) Has been cancelled
ci / backend (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / static (pull_request) Successful in 9s
Build & Test with Secrets / build (pull_request) Failing after 2s
Build & Test with Secrets / security-scan (pull_request) Has been cancelled
Build & Test with Secrets / notification (pull_request) Has been cancelled
Build & Test with Secrets / frontend (pull_request) Has been cancelled
ci / backend (pull_request) Has been cancelled
ci / frontend (pull_request) Has been cancelled
ci / publish (pull_request) Has been cancelled
2026-08-06 15:59:16 +09:00
kjh2064 aee4a4d624 Merge pull request 'Deploy: fail closed on migration and restart errors' (#12) from fix/deploy-fail-closed into main
Build & Test with Secrets / build (push) Has been cancelled
Build & Test with Secrets / frontend (push) Has been cancelled
Build & Test with Secrets / security-scan (push) Has been cancelled
Build & Test with Secrets / notification (push) Has been cancelled
ci / static (push) Successful in 13s
ci / publish (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / backend (push) Has been cancelled
deploy / deploy (push) Failing after 1m41s
deploy / notify (push) Successful in 0s
Reviewed-on: #12
2026-08-06 15:32:26 +09:00
kjh2064 36479307e9 Deploy: fail closed when migration or restart fails
ci / static (push) Successful in 14s
ci / static (pull_request) Successful in 12s
ci / frontend (pull_request) Has been cancelled
ci / publish (pull_request) Has been cancelled
ci / backend (pull_request) Has been cancelled
Build & Test with Secrets / build (pull_request) Has been cancelled
Build & Test with Secrets / frontend (pull_request) Has been cancelled
Build & Test with Secrets / security-scan (pull_request) Has been cancelled
Build & Test with Secrets / notification (pull_request) Has been cancelled
ci / publish (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / backend (push) Has been cancelled
2026-08-06 15:30:08 +09:00
kjh2064 74b50465fe Merge pull request 'AEG-X-004: deploy DbUp migrations with release artifact' (#11) from fix/deploy-db-migrator-migrations into main
ci / static (push) Successful in 11s
ci / backend (push) Successful in 4m1s
Build & Test with Secrets / build (push) Failing after 2s
deploy / deploy (push) Successful in 3m54s
ci / frontend (push) Successful in 5m15s
Build & Test with Secrets / security-scan (push) Failing after 9s
deploy / notify (push) Successful in 2s
ci / publish (push) Failing after 1m53s
Build & Test with Secrets / frontend (push) Successful in 4m17s
Build & Test with Secrets / notification (push) Failing after 1s
2026-08-06 15:21:21 +09:00
6 changed files with 748 additions and 5 deletions
+41 -5
View File
@@ -22,6 +22,33 @@ jobs:
with:
dotnet-version: '10.0.x'
- uses: pnpm/action-setup@v4
with:
version: 10
- uses: actions/setup-node@v4
with:
node-version: 22
cache: pnpm
cache-dependency-path: frontend/pnpm-lock.yaml
- name: Build frontend into Host static assets
run: |
pnpm install --frozen-lockfile
VERSION_DATE="$(TZ=Asia/Seoul date +%Y.%m.%d)"
RELEASE_COUNT="$(git ls-remote --tags origin "refs/tags/v${VERSION_DATE}.*" | wc -l | tr -d ' ')"
VERSION_SEQUENCE="$((RELEASE_COUNT + 1))"
APP_VERSION="${VERSION_DATE}.${VERSION_SEQUENCE}.${GITHUB_SHA::10}"
echo "VITE_APP_VERSION=${APP_VERSION}" >> "$GITHUB_ENV"
echo "release_version=${APP_VERSION}"
VITE_APP_VERSION="${APP_VERSION}" pnpm build
grep -R -q 'app-version' dist
grep -R -q 'UI contract 4.0' dist
grep -R -q "${APP_VERSION}" dist
find ../src/KArtSell.Host/wwwroot -mindepth 1 -delete
cp -R dist/. ../src/KArtSell.Host/wwwroot/
working-directory: frontend
- run: dotnet restore KArtSell.sln
- run: dotnet build KArtSell.sln --no-restore -c Release
@@ -57,14 +84,23 @@ jobs:
echo "✅ File transferred"
echo ""
ssh -i /tmp/deploy_key.pem -o StrictHostKeyChecking=no kjh2064@178.104.200.7 \
"export KARTSELL_POSTGRES='${{ secrets.KARTSELL_POSTGRES }}'; \
sudo mkdir -p /app/kartsell/current; \
sudo unzip -oq /tmp/kartsell-release.zip -d /app/kartsell/current; \
"set -euo pipefail; \
sudo -n -l | grep -Fq '/usr/bin/systemctl restart kartsell' || { \
echo 'Deployment blocked: one-time sudoers delegation is missing for kartsell.' >&2; \
echo 'Expected: kjh2064 ALL=(root) NOPASSWD: /usr/bin/systemctl restart kartsell' >&2; \
exit 77; \
}; \
export KARTSELL_POSTGRES='${{ secrets.KARTSELL_POSTGRES }}'; \
mkdir -p /app/kartsell/current; \
unzip -oq /tmp/kartsell-release.zip -d /app/kartsell/current; \
cd /app/kartsell/current; \
test -f KArtSell.DbMigrator.dll; \
test -f 0032_shadow_run_queued_status_contract.sql; \
dotnet KArtSell.DbMigrator.dll; \
sudo systemctl restart kartsell; \
sudo -n systemctl restart kartsell; \
sleep 3; \
systemctl is-active --quiet kartsell"
systemctl is-active --quiet kartsell; \
echo 'deployment_verified=true'"
echo "✅ Artifact deployed, DbMigrator executed, and kartsell restarted"
@@ -0,0 +1,22 @@
# 배포 frontend artifact 계약
## Source
- 운영 배포 Run 3357 로그: `dotnet publish` 전 frontend build 단계 없음
- 운영 bundle에 `app-version``UI contract 4.0` marker 없음
- `frontend`의 재현 가능한 `pnpm-lock.yaml` 및 기존 CI frontend job
## Decision
배포 workflow는 Host publish 전에 다음 규칙으로 버전을 계산하고 frontend를 재생성한다.
```text
YYYY.MM.DD.<당일 release 순번>.<commit SHA 10자리>
```
당일 순번은 `vYYYY.MM.DD.*` release tag 개수에 1을 더해 계산한다. 예: `2026.08.06.1.acaa731b3f`. 생성된 `frontend/dist`를 Host `wwwroot`에 복사하고, `app-version`, `UI contract 4.0`, 계산된 전체 버전 marker가 없으면 배포를 중단한다.
## Evidence / Unknown
- Source 변경과 운영 artifact를 분리하지 않고, 매 배포 시 동일 commit에서 재생성한다.
- 실제 운영 반영 증거는 이 Slice의 CI 및 deploy run 완료 후 보존한다.
@@ -0,0 +1,32 @@
# KArtSell 배포 재기동 권한 계약
## Source
- 운영 호스트 `hz-prod-01`의 실제 sudo 정책 조회 결과
- 기존 `quantengine``taxbaik` 서비스의 특정 `systemctl restart` `NOPASSWD` 위임 패턴
- `.gitea/workflows/deploy.yml`
## Assumption
- 배포 SSH 계정은 `kjh2064`로 유지한다.
- 운영 서비스는 `/etc/systemd/system/kartsell.service`로 유지한다.
- DbMigrator와 artifact 복사는 현재처럼 `kjh2064` 권한으로 수행한다.
## Decision
`kjh2064`에 전체 sudo 권한을 부여하지 않고, 운영자가 한 번만 다음 단일 명령을 `/etc/sudoers.d/kartsell-deploy`에 등록한다.
```sudoers
kjh2064 ALL=(root) NOPASSWD: /usr/bin/systemctl restart kartsell
```
파일 권한은 `0440`이어야 하며 `visudo -cf /etc/sudoers.d/kartsell-deploy` 검증 후 적용한다. 이후 CI는 비대화형 `sudo -n systemctl restart kartsell`만 사용하므로 배포마다 비밀번호 입력이나 sudo 등록이 필요 없다.
## Deployment guard
워크플로우는 artifact 복사와 DbMigrator 실행 전에 `sudo -n -l`로 위임 존재 여부를 검사한다. 위임이 없으면 운영 DB를 변경하지 않고 exit 77로 종료한다.
## Unknown / Decision Required
- 이 파일을 운영 호스트에 설치할 권한은 root 운영자에게만 있다.
- 설치 후 필요한 증거: `visudo -cf` 결과, `sudo -n -l` 결과, 다음 deploy run의 성공 로그, 서비스 active 상태.
+331
View File
@@ -0,0 +1,331 @@
# Phase 1 Activation Runbook
**Date:** 2026-08-07
**Purpose:** Step-by-step activation of Phase 1 shadow run (252+ trading days)
**Owner:** Platform SRE
**Status:** READY FOR EXECUTION (All tools prepared)
---
## 🎯 Objective
Launch **Job 893 (Shadow Run)** with frozen model/dataset VersionSet, generating 252+ trading days of market simulation with auditable evidence trail.
**Timeline:**
- **Setup:** ~15 minutes (this runbook)
- **Execution:** 50-90 calendar days (automatic, no manual intervention)
- **Evidence Collection:** Concurrent (logs, metrics, state snapshots)
---
## 📋 PRE-FLIGHT CHECKLIST
**All items must be COMPLETE before proceeding to Step 1.**
- [ ] **1. Migration 0032 deployed**
Verify: `SELECT schema_version FROM schema_version_history WHERE script_name LIKE '0032_%'`
Status: Must return 1 row. If missing, run `dotnet run --project src/KArtSell.DbMigrator`
- [ ] **2. Host running in DEVELOPMENT mode**
Verify: `dotnet run --project src/KArtSell.Host -c Debug --no-build`
Expected: "Now listening on: http://127.0.0.1:5002"
**Why Debug mode?** `DevelopmentHeaderAuthenticationHandler` required for testing; Release mode uses `FailClosedAuthenticationHandler` (rejects all requests)
- [ ] **3. PostgreSQL accessible via SSH tunnel**
Verify: `ssh -L 5432:127.0.0.1:5432 kjh2064@178.104.200.7` (keep open in separate terminal)
Expected: No errors; tunnel stays alive
- [ ] **4. Hangfire scheduler running**
Verify: Host logs contain `Hangfire: JobStorage initialized`
Expected: Startup completes without timeout
- [ ] **5. Scripts available in ./scripts/**
Verify: `ls scripts/freeze-versionset.ps1 scripts/generate-shadow-run-identifiers.ps1`
---
## 🚀 STEP 1: FREEZE VERSIONSET
**Duration:** ~2 minutes
**Tool:** `./scripts/freeze-versionset.ps1`
### Action
Execute with **REAL, APPROVED** model/dataset IDs:
```powershell
cd C:\Job_Roomz\KArtSell.Aegis
$env:KARTSELL_POSTGRES = "Host=localhost;Port=5432;Database=kartsell;Username=kartsell;Password=kartsell"
.\scripts\freeze-versionset.ps1 `
-ModelId "00000000-0000-0000-0000-000000000001" `
-DatasetId "00000000-0000-0000-0000-000000000002" `
-ApprovedBy "kim.jae.hyun@example.com" `
-ConfigVersion "v1.0.0" `
-CodeSha "acaa731b3f"
```
### Expected Output
```
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Phase 1: Freeze VersionSet
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[1/3] PRE-FLIGHT CHECK
Model ID: 00000000-0000-0000-0000-000000000001
Dataset ID: 00000000-0000-0000-0000-000000000002
Approved By: kim.jae.hyun@example.com
Config Version: v1.0.0
Code SHA: acaa731b3f
Connection: Host=localhost;Port=5432;Database=kartsell;***
[2/3] VERIFY Migration 0032 deployed...
✅ Migration 0032 deployed (schema_version: 32)
[3/3] FREEZE VersionSet...
✅ Inserted governance.model_version_registry:
- ID: <UUID>
- Model: 00000000-0000-0000-0000-000000000001
- Dataset: 00000000-0000-0000-0000-000000000002
- Status: FROZEN
✅ Inserted evaluation.dataset_manifest:
- ID: <UUID>
- Dataset: 00000000-0000-0000-0000-000000000002
- Model: 00000000-0000-0000-0000-000000000001
- Status: FROZEN
✅ VersionSet FROZEN successfully
Correlation ID: <UUID>
Next: Run generate-shadow-run-identifiers.ps1 to create RunId/JobId
```
### Troubleshooting
| Error | Cause | Fix |
|-------|-------|-----|
| "Migration 0032 NOT FOUND" | DbMigrator hasn't run yet | Run: `dotnet run --project src/KArtSell.DbMigrator` |
| "Cannot bind argument -ModelId" | Invalid UUID format | Use: `[System.Guid]::NewGuid() \| % { $_.ToString() }` to generate valid UUID |
| "Connection refused" | PostgreSQL not accessible | Verify SSH tunnel: `ssh -L 5432:127.0.0.1:5432 kjh2064@178.104.200.7` |
---
## 🚀 STEP 2: GENERATE IDENTIFIERS
**Duration:** ~1 minute
**Tool:** `./scripts/generate-shadow-run-identifiers.ps1`
### Action
```powershell
.\scripts\generate-shadow-run-identifiers.ps1 -OutputPath ./phase1-versionset.json
```
### Expected Output
```
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Phase 1: Generate Shadow Run Identifiers
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[1/3] Generating cryptographic UUIDs...
✅ RunId: <UUID>
✅ JobId: <UUID>
✅ JobRunId: <UUID>
✅ CorrelationId: <UUID>
✅ IdempotencyKey: <UUID>
[2/3] Creating JSON payload...
✅ JSON payload generated
[3/3] Writing to file: ./phase1-versionset.json
✅ File saved: C:\Job_Roomz\KArtSell.Aegis\phase1-versionset.json
✅ IDENTIFIERS GENERATED
{
"phase1_run": {
"runId": "<UUID>",
"jobId": "<UUID>",
"jobRunId": "<UUID>",
"correlationId": "<UUID>",
"idempotencyKey": "<UUID>",
"generatedAt": "2026-08-07T10:30:00.000Z",
...
}
}
Next Steps:
1. Copy the identifiers from above or read from ./phase1-versionset.json
2. Call POST /api/shadow-runs with modelId/datasetId from frozen VersionSet
3. Hangfire will enqueue Job 893 with these correlation IDs
4. Monitor logs: grep 'CorrelationId: <UUID>' app.log
```
### Save for Reference
Copy output to clipboard or save in a secure file. You'll need these IDs in STEP 3.
---
## 🚀 STEP 3: ENQUEUE SHADOW RUN JOB
**Duration:** ~1 minute
**Method:** PowerShell HTTP request
### Prerequisites
- [ ] Host running on `http://127.0.0.1:5002` (Debug mode)
- [ ] VersionSet frozen (STEP 1 complete)
- [ ] Identifiers generated (STEP 2 complete)
### Action
```powershell
# Read generated identifiers
$versionset = Get-Content ./phase1-versionset.json | ConvertFrom-Json
$correlationId = $versionset.phase1_run.correlationId
$runId = $versionset.phase1_run.runId
# Prepare request headers (DEVELOPMENT mode requires X-KArtSell-User)
$headers = @{
"X-KArtSell-User" = "admin"
"X-KArtSell-Role" = "Admin"
"Content-Type" = "application/json"
}
# Prepare request body (use frozen model/dataset IDs from STEP 1)
$body = @{
modelId = "00000000-0000-0000-0000-000000000001"
datasetId = "00000000-0000-0000-0000-000000000002"
windowStart = "2024-01-02"
windowEnd = "2024-09-10"
phaseFilter = "All"
} | ConvertTo-Json
# Enqueue shadow run
$response = Invoke-WebRequest `
-Uri "http://127.0.0.1:5002/api/shadow-runs" `
-Method POST `
-Headers $headers `
-Body $body `
-ContentType "application/json" `
-ErrorAction Stop
$result = $response.Content | ConvertFrom-Json
Write-Host "✅ Shadow run enqueued!"
Write-Host " Job ID: $($result.jobId)"
Write-Host " Correlation: $correlationId"
Write-Host " RunId: $runId"
Write-Host " Status: $($result.status)"
```
### Expected Output (HTTP 202 Accepted)
```
✅ Shadow run enqueued!
Job ID: <UUID>
Correlation: <CorrelationId>
RunId: <RunId>
Status: Queued
```
### Troubleshooting
| Error | Cause | Fix |
|-------|-------|-----|
| HTTP 403/404 | Release mode (not Debug) | Check Host startup log; must contain "DevelopmentHeaderAuthenticationHandler" |
| HTTP 422 Unprocessable | Invalid model/dataset UUID | Verify UUIDs exist in `governance.model_version_registry` via SQL: `SELECT * FROM governance.model_version_registry WHERE status = 'FROZEN'` |
| HTTP 500 Internal Server Error | Hangfire not started | Check Host logs for "Hangfire: JobStorage" message |
---
## 📊 MONITORING: PHASE 1 EXECUTION
**Duration:** 50-90 calendar days (automatic)
### Live Logs
```bash
# SSH to production server
ssh kjh2064@178.104.200.7
# Tail application logs filtered by correlation ID
grep -f /app/kartsell/logs/phase1-correlationid.txt /app/kartsell/logs/app.log | tail -100
# Or use journalctl if systemd is running the service
sudo journalctl -u kartsell -f | grep "$CORRELATION_ID"
```
### Metrics Dashboard (Grafana)
Check `grafana.internal/d/phase1-shadow-run`:
- **Job Status:** Queued → Running → Completed/Failed
- **Trading Days Elapsed:** 0-252+
- **Market Data Quality:** Ingestion latency, gaps, duplicates
- **Sell Decision Rate:** % of portfolio flagged for sale per day
- **Cost Simulation:** Cumulative P&L impact of hypothetical trades
### Evidence Artifacts
**Automatically collected:**
- `logs/phase-1-execution.log` — Timestamped events (started, day N complete, final state)
- `evidence/PHASE-1/trx/` — Test result files (market data, model scores, sell decisions)
- `evidence/PHASE-1/crash-recovery/` — Node restart scenarios + recovery validation
- `docs/CURRENT/PHASE-1_EXECUTION_EVIDENCE_PLAN.md` — Full checklist
### Alerts
**Set up pagerduty/Telegram notifications:**
```bash
# Example: Notify if Phase 1 job fails
curl -X POST "https://api.telegram.org/bot$TELEGRAM_TOKEN/sendMessage" \
-d "chat_id=$TELEGRAM_CHAT_ID" \
-d "text=⚠️ Phase 1 Job $JOB_ID failed: $ERROR_MESSAGE"
```
---
## ✅ COMPLETION: PHASE 1 EXECUTION COMPLETE
**When:**
- Job 893 reaches 252+ trading days
- All sell decisions generated + cost impact simulated
- No gaps or anomalies in market data
**What to do:**
1. Download `logs/phase-1-execution.log` (evidence of completion)
2. Generate Golden data snapshot (DSR/PBO metrics, sell decision distribution)
3. Unlock Gates 2-5 (downstream slices depend on this data)
4. Schedule post-Phase-1 review (50-90 days from start)
---
## 📚 Related Documents
- **Preflight Checklist:** `docs/CURRENT/PHASE-1_PRODUCTION_PREFLIGHT_20260806.md`
- **Architecture Decision:** `docs/DECISIONS/ADR-SEC-001.md`
- **Hangfire Jobs:** `src/KArtSell.Host/Jobs/ShadowRunJob.cs`
- **Evidence Plan:** `docs/CURRENT/PHASE-1_EXECUTION_EVIDENCE_PLAN.md`
---
## 🆘 Emergency Rollback
**If Phase 1 must be stopped:**
1. SSH to production
2. `sudo systemctl stop kartsell`
3. Kill Job 893 in Hangfire Dashboard (Admin UI)
4. Archive logs: `cp /app/kartsell/logs/phase-1-execution.log evidence/PHASE-1/rollback-$(date +%s).log`
5. Notify team (Telegram/Email)
6. Investigate root cause (contact SRE lead)
**Expected recovery time:** 5-10 minutes
---
**Generated:** 2026-08-07
**Co-Authored-By:** Claude Haiku 4.5 <noreply@anthropic.com>
+232
View File
@@ -0,0 +1,232 @@
#!/usr/bin/env pwsh
<#
.SYNOPSIS
Freeze an approved model/dataset VersionSet for Phase 1 shadow run.
.DESCRIPTION
Parameterized tool to INSERT approved model_id + dataset_id into:
- governance.model_version_registry (FROZEN status)
- evaluation.dataset_manifest (FROZEN status)
NO default values; all parameters REQUIRED. Fails immediately if any parameter is missing.
.PARAMETER ModelId
UUID of the approved model (e.g., "00000000-0000-0000-0000-000000000001")
Required. No default.
.PARAMETER DatasetId
UUID of the approved dataset (e.g., "00000000-0000-0000-0000-000000000002")
Required. No default.
.PARAMETER ApprovedBy
Email/ID of the approver (e.g., "kjh2064@gmail.com")
Required. No default.
.PARAMETER ConfigVersion
Configuration version string (e.g., "v1.0.0")
Required. No default.
.PARAMETER CodeSha
Git commit SHA (e.g., "acaa731b3f")
Required. No default.
.PARAMETER ConnectionString
PostgreSQL connection string.
Default: $env:KARTSELL_POSTGRES
.EXAMPLE
# Freeze a versionset (all parameters required)
.\freeze-versionset.ps1 `
-ModelId "00000000-0000-0000-0000-000000000001" `
-DatasetId "00000000-0000-0000-0000-000000000002" `
-ApprovedBy "kjh2064@gmail.com" `
-ConfigVersion "v1.0.0" `
-CodeSha "acaa731b3f"
.EXAMPLE
# Will fail: missing -ConfigVersion
.\freeze-versionset.ps1 `
-ModelId "00000000-0000-0000-0000-000000000001" `
-DatasetId "00000000-0000-0000-0000-000000000002" `
-ApprovedBy "kjh2064@gmail.com" `
-CodeSha "acaa731b3f"
# Error: Cannot bind argument to parameter 'ConfigVersion' because it is an empty string.
#>
[CmdletBinding()]
param(
[Parameter(Mandatory, HelpMessage = "Model UUID (e.g., 00000000-0000-0000-0000-000000000001)")]
[ValidateScript({ $_ -match '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$' })]
[string]$ModelId,
[Parameter(Mandatory, HelpMessage = "Dataset UUID")]
[ValidateScript({ $_ -match '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$' })]
[string]$DatasetId,
[Parameter(Mandatory, HelpMessage = "Approver email/ID (e.g., kjh2064@gmail.com)")]
[ValidateScript({ $_ -match '^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$' })]
[string]$ApprovedBy,
[Parameter(Mandatory, HelpMessage = "Config version (e.g., v1.0.0)")]
[ValidateScript({ $_ -match '^v[0-9]+\.[0-9]+\.[0-9]+' })]
[string]$ConfigVersion,
[Parameter(Mandatory, HelpMessage = "Git commit SHA (at least 10 chars)")]
[ValidateScript({ $_.Length -ge 10 })]
[string]$CodeSha,
[string]$ConnectionString = $env:KARTSELL_POSTGRES
)
$ErrorActionPreference = 'Stop'
Write-Host "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" -ForegroundColor Cyan
Write-Host "Phase 1: Freeze VersionSet" -ForegroundColor Cyan
Write-Host "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" -ForegroundColor Cyan
# Validate connection string
if (-not $ConnectionString) {
Write-Error "ConnectionString not provided and `$env:KARTSELL_POSTGRES not set. Aborting."
exit 1
}
Write-Host "`n[1/3] PRE-FLIGHT CHECK"
Write-Host " Model ID: $ModelId"
Write-Host " Dataset ID: $DatasetId"
Write-Host " Approved By: $ApprovedBy"
Write-Host " Config Version: $ConfigVersion"
Write-Host " Code SHA: $CodeSha"
Write-Host " Connection: $(($ConnectionString -split 'Password=')[0])***"
# Verify 0032 migration is deployed
Write-Host "`n[2/3] VERIFY Migration 0032 deployed..."
try {
$conn = New-Object System.Data.NpgsqlClient.NpgsqlConnection($ConnectionString)
$conn.Open()
$cmd = $conn.CreateCommand()
$cmd.CommandText = @"
SELECT schema_version FROM schema_version_history
WHERE script_name = '0032_shadow_run_queued_status_contract.sql'
LIMIT 1
"@
$result = $cmd.ExecuteScalar()
if ($null -eq $result) {
throw "Migration 0032 NOT FOUND. Run DbMigrator first."
}
Write-Host " ✅ Migration 0032 deployed (schema_version: $result)"
$conn.Close()
}
catch {
Write-Error " ❌ Pre-flight failed: $_`n`nCorrective: Run DbMigrator to deploy 0032_*.sql before freezing."
exit 1
}
# Insert into governance.model_version_registry
Write-Host "`n[3/3] FREEZE VersionSet..."
try {
$conn = New-Object System.Data.NpgsqlClient.NpgsqlConnection($ConnectionString)
$conn.Open()
$correlationId = [System.Guid]::NewGuid()
$now = [System.DateTime]::UtcNow
$cmd = $conn.CreateCommand()
$cmd.CommandText = @"
INSERT INTO governance.model_version_registry (
id, model_id, dataset_id, status, approved_by, config_version, code_sha,
effective_at, published_at, revision, correlation_id
) VALUES (
@id, @model_id, @dataset_id, 'FROZEN', @approved_by, @config_version, @code_sha,
@effective_at, @published_at, 1, @correlation_id
)
ON CONFLICT (model_id, dataset_id) DO UPDATE SET
status = 'FROZEN',
approved_by = EXCLUDED.approved_by,
config_version = EXCLUDED.config_version,
code_sha = EXCLUDED.code_sha,
effective_at = EXCLUDED.effective_at,
revision = governance.model_version_registry.revision + 1,
published_at = EXCLUDED.published_at
RETURNING id, model_id, dataset_id, status, effective_at
"@
$cmd.Parameters.AddWithValue("@id", [System.Guid]::NewGuid()) | Out-Null
$cmd.Parameters.AddWithValue("@model_id", [System.Guid]$ModelId) | Out-Null
$cmd.Parameters.AddWithValue("@dataset_id", [System.Guid]$DatasetId) | Out-Null
$cmd.Parameters.AddWithValue("@approved_by", $ApprovedBy) | Out-Null
$cmd.Parameters.AddWithValue("@config_version", $ConfigVersion) | Out-Null
$cmd.Parameters.AddWithValue("@code_sha", $CodeSha) | Out-Null
$cmd.Parameters.AddWithValue("@effective_at", $now) | Out-Null
$cmd.Parameters.AddWithValue("@published_at", $now) | Out-Null
$cmd.Parameters.AddWithValue("@correlation_id", $correlationId) | Out-Null
$reader = $cmd.ExecuteReader()
if ($reader.Read()) {
$insertedId = $reader['id']
$insertedModelId = $reader['model_id']
$insertedDatasetId = $reader['dataset_id']
$insertedStatus = $reader['status']
Write-Host " ✅ Inserted governance.model_version_registry:"
Write-Host " - ID: $insertedId"
Write-Host " - Model: $insertedModelId"
Write-Host " - Dataset: $insertedDatasetId"
Write-Host " - Status: $insertedStatus"
}
$reader.Close()
# Update evaluation.dataset_manifest
$cmd2 = $conn.CreateCommand()
$cmd2.CommandText = @"
INSERT INTO evaluation.dataset_manifest (
id, dataset_id, model_id, status, freeze_reason,
published_at, revision, correlation_id
) VALUES (
@id, @dataset_id, @model_id, 'FROZEN', 'Phase 1 VersionSet freeze',
@published_at, 1, @correlation_id
)
ON CONFLICT (dataset_id, model_id) DO UPDATE SET
status = 'FROZEN',
freeze_reason = 'Phase 1 VersionSet freeze',
revision = evaluation.dataset_manifest.revision + 1,
published_at = EXCLUDED.published_at
RETURNING id, dataset_id, model_id, status
"@
$cmd2.Parameters.AddWithValue("@id", [System.Guid]::NewGuid()) | Out-Null
$cmd2.Parameters.AddWithValue("@dataset_id", [System.Guid]$DatasetId) | Out-Null
$cmd2.Parameters.AddWithValue("@model_id", [System.Guid]$ModelId) | Out-Null
$cmd2.Parameters.AddWithValue("@published_at", $now) | Out-Null
$cmd2.Parameters.AddWithValue("@correlation_id", $correlationId) | Out-Null
$reader2 = $cmd2.ExecuteReader()
if ($reader2.Read()) {
$mId = $reader2['id']
$mDatasetId = $reader2['dataset_id']
$mModelId = $reader2['model_id']
$mStatus = $reader2['status']
Write-Host " ✅ Inserted evaluation.dataset_manifest:"
Write-Host " - ID: $mId"
Write-Host " - Dataset: $mDatasetId"
Write-Host " - Model: $mModelId"
Write-Host " - Status: $mStatus"
}
$reader2.Close()
$conn.Close()
Write-Host "`n✅ VersionSet FROZEN successfully"
Write-Host " Correlation ID: $correlationId"
Write-Host " Next: Run generate-shadow-run-identifiers.ps1 to create RunId/JobId"
}
catch {
Write-Error " ❌ Failed to freeze VersionSet: $_"
exit 1
}
Write-Host "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" -ForegroundColor Cyan
@@ -0,0 +1,90 @@
#!/usr/bin/env pwsh
<#
.SYNOPSIS
Generate Phase 1 shadow run identifiers (RunId, JobId, JobRunId, CorrelationId, Idempotency-Key).
.DESCRIPTION
Produces a JSON-formatted versionset.json file with all identifiers needed to enqueue Phase 1.
Uses CRYPTOGRAPHIC random UUIDs and correlation for full traceability.
.PARAMETER OutputPath
Path to save versionset.json (default: ./versionset.json in current directory)
.EXAMPLE
.\generate-shadow-run-identifiers.ps1 -OutputPath ./phase1-versionset.json
.OUTPUTS
JSON file with structure:
{
"phase1_run": {
"runId": "UUID",
"jobId": "UUID",
"jobRunId": "UUID",
"correlationId": "UUID",
"idempotencyKey": "UUID",
"generatedAt": "ISO8601 timestamp",
"usage": "Use these IDs to enqueue Job 893 in Hangfire..."
}
}
#>
[CmdletBinding()]
param(
[string]$OutputPath = "./versionset.json"
)
$ErrorActionPreference = 'Stop'
Write-Host "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" -ForegroundColor Cyan
Write-Host "Phase 1: Generate Shadow Run Identifiers" -ForegroundColor Cyan
Write-Host "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" -ForegroundColor Cyan
Write-Host "`n[1/3] Generating cryptographic UUIDs..."
$runId = [System.Guid]::NewGuid()
$jobId = [System.Guid]::NewGuid()
$jobRunId = [System.Guid]::NewGuid()
$correlationId = [System.Guid]::NewGuid()
$idempotencyKey = [System.Guid]::NewGuid()
Write-Host " ✅ RunId: $runId"
Write-Host " ✅ JobId: $jobId"
Write-Host " ✅ JobRunId: $jobRunId"
Write-Host " ✅ CorrelationId: $correlationId"
Write-Host " ✅ IdempotencyKey: $idempotencyKey"
Write-Host "`n[2/3] Creating JSON payload..."
$payload = @{
phase1_run = @{
runId = $runId.ToString()
jobId = $jobId.ToString()
jobRunId = $jobRunId.ToString()
correlationId = $correlationId.ToString()
idempotencyKey = $idempotencyKey.ToString()
generatedAt = [System.DateTime]::UtcNow.ToString("o")
windowStart = "2024-01-02"
windowEnd = "2024-09-10"
usage = "Use these IDs to enqueue Job 893 (Phase 1 shadow run) in Hangfire. Command: `n Invoke-WebRequest -Uri 'http://127.0.0.1:5002/api/shadow-runs' -Method POST -Headers @{ 'X-KArtSell-User'='admin'; 'X-KArtSell-Role'='Admin'; 'Content-Type'='application/json' } -Body (ConvertTo-Json @{ modelId='<modelId>'; datasetId='<datasetId>'; windowStart='2024-01-02'; windowEnd='2024-09-10'; phaseFilter='All' })"
}
}
Write-Host " ✅ JSON payload generated"
Write-Host "`n[3/3] Writing to file: $OutputPath"
$json = $payload | ConvertTo-Json -Depth 10
$json | Out-File -FilePath $OutputPath -Encoding UTF8
Write-Host " ✅ File saved: $(Resolve-Path $OutputPath)"
Write-Host "`n✅ IDENTIFIERS GENERATED`n"
Write-Host $json -ForegroundColor Green
Write-Host "`nNext Steps:`n"
Write-Host " 1. Copy the identifiers from above or read from $OutputPath"
Write-Host " 2. Call POST /api/shadow-runs with modelId/datasetId from frozen VersionSet"
Write-Host " 3. Hangfire will enqueue Job 893 with these correlation IDs"
Write-Host " 4. Monitor logs: grep 'CorrelationId: $correlationId' app.log"
Write-Host "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" -ForegroundColor Cyan