kjh2064
d731800954
Co-authored-by: Claude Code <kjh2064@gmail.com> Co-committed-by: Claude Code <kjh2064@gmail.com>
332 lines
10 KiB
Markdown
332 lines
10 KiB
Markdown
# Phase 1 Activation Runbook
|
|
|
|
**Date:** 2026-08-07
|
|
**Purpose:** Step-by-step activation of Phase 1 shadow run (252+ trading days)
|
|
**Owner:** Platform SRE
|
|
**Status:** READY FOR EXECUTION (All tools prepared)
|
|
|
|
---
|
|
|
|
## 🎯 Objective
|
|
|
|
Launch **Job 893 (Shadow Run)** with frozen model/dataset VersionSet, generating 252+ trading days of market simulation with auditable evidence trail.
|
|
|
|
**Timeline:**
|
|
- **Setup:** ~15 minutes (this runbook)
|
|
- **Execution:** 50-90 calendar days (automatic, no manual intervention)
|
|
- **Evidence Collection:** Concurrent (logs, metrics, state snapshots)
|
|
|
|
---
|
|
|
|
## 📋 PRE-FLIGHT CHECKLIST
|
|
|
|
**All items must be COMPLETE before proceeding to Step 1.**
|
|
|
|
- [ ] **1. Migration 0032 deployed**
|
|
Verify: `SELECT schema_version FROM schema_version_history WHERE script_name LIKE '0032_%'`
|
|
Status: Must return 1 row. If missing, run `dotnet run --project src/KArtSell.DbMigrator`
|
|
|
|
- [ ] **2. Host running in DEVELOPMENT mode**
|
|
Verify: `dotnet run --project src/KArtSell.Host -c Debug --no-build`
|
|
Expected: "Now listening on: http://127.0.0.1:5002"
|
|
**Why Debug mode?** `DevelopmentHeaderAuthenticationHandler` required for testing; Release mode uses `FailClosedAuthenticationHandler` (rejects all requests)
|
|
|
|
- [ ] **3. PostgreSQL accessible via SSH tunnel**
|
|
Verify: `ssh -L 5432:127.0.0.1:5432 kjh2064@178.104.200.7` (keep open in separate terminal)
|
|
Expected: No errors; tunnel stays alive
|
|
|
|
- [ ] **4. Hangfire scheduler running**
|
|
Verify: Host logs contain `Hangfire: JobStorage initialized`
|
|
Expected: Startup completes without timeout
|
|
|
|
- [ ] **5. Scripts available in ./scripts/**
|
|
Verify: `ls scripts/freeze-versionset.ps1 scripts/generate-shadow-run-identifiers.ps1`
|
|
|
|
---
|
|
|
|
## 🚀 STEP 1: FREEZE VERSIONSET
|
|
|
|
**Duration:** ~2 minutes
|
|
**Tool:** `./scripts/freeze-versionset.ps1`
|
|
|
|
### Action
|
|
|
|
Execute with **REAL, APPROVED** model/dataset IDs:
|
|
|
|
```powershell
|
|
cd C:\Job_Roomz\KArtSell.Aegis
|
|
|
|
$env:KARTSELL_POSTGRES = "Host=localhost;Port=5432;Database=kartsell;Username=kartsell;Password=kartsell"
|
|
|
|
.\scripts\freeze-versionset.ps1 `
|
|
-ModelId "00000000-0000-0000-0000-000000000001" `
|
|
-DatasetId "00000000-0000-0000-0000-000000000002" `
|
|
-ApprovedBy "kim.jae.hyun@example.com" `
|
|
-ConfigVersion "v1.0.0" `
|
|
-CodeSha "acaa731b3f"
|
|
```
|
|
|
|
### Expected Output
|
|
|
|
```
|
|
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
Phase 1: Freeze VersionSet
|
|
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
[1/3] PRE-FLIGHT CHECK
|
|
Model ID: 00000000-0000-0000-0000-000000000001
|
|
Dataset ID: 00000000-0000-0000-0000-000000000002
|
|
Approved By: kim.jae.hyun@example.com
|
|
Config Version: v1.0.0
|
|
Code SHA: acaa731b3f
|
|
Connection: Host=localhost;Port=5432;Database=kartsell;***
|
|
|
|
[2/3] VERIFY Migration 0032 deployed...
|
|
✅ Migration 0032 deployed (schema_version: 32)
|
|
|
|
[3/3] FREEZE VersionSet...
|
|
✅ Inserted governance.model_version_registry:
|
|
- ID: <UUID>
|
|
- Model: 00000000-0000-0000-0000-000000000001
|
|
- Dataset: 00000000-0000-0000-0000-000000000002
|
|
- Status: FROZEN
|
|
✅ Inserted evaluation.dataset_manifest:
|
|
- ID: <UUID>
|
|
- Dataset: 00000000-0000-0000-0000-000000000002
|
|
- Model: 00000000-0000-0000-0000-000000000001
|
|
- Status: FROZEN
|
|
|
|
✅ VersionSet FROZEN successfully
|
|
Correlation ID: <UUID>
|
|
Next: Run generate-shadow-run-identifiers.ps1 to create RunId/JobId
|
|
```
|
|
|
|
### Troubleshooting
|
|
|
|
| Error | Cause | Fix |
|
|
|-------|-------|-----|
|
|
| "Migration 0032 NOT FOUND" | DbMigrator hasn't run yet | Run: `dotnet run --project src/KArtSell.DbMigrator` |
|
|
| "Cannot bind argument -ModelId" | Invalid UUID format | Use: `[System.Guid]::NewGuid() \| % { $_.ToString() }` to generate valid UUID |
|
|
| "Connection refused" | PostgreSQL not accessible | Verify SSH tunnel: `ssh -L 5432:127.0.0.1:5432 kjh2064@178.104.200.7` |
|
|
|
|
---
|
|
|
|
## 🚀 STEP 2: GENERATE IDENTIFIERS
|
|
|
|
**Duration:** ~1 minute
|
|
**Tool:** `./scripts/generate-shadow-run-identifiers.ps1`
|
|
|
|
### Action
|
|
|
|
```powershell
|
|
.\scripts\generate-shadow-run-identifiers.ps1 -OutputPath ./phase1-versionset.json
|
|
```
|
|
|
|
### Expected Output
|
|
|
|
```
|
|
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
Phase 1: Generate Shadow Run Identifiers
|
|
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
|
|
[1/3] Generating cryptographic UUIDs...
|
|
✅ RunId: <UUID>
|
|
✅ JobId: <UUID>
|
|
✅ JobRunId: <UUID>
|
|
✅ CorrelationId: <UUID>
|
|
✅ IdempotencyKey: <UUID>
|
|
|
|
[2/3] Creating JSON payload...
|
|
✅ JSON payload generated
|
|
|
|
[3/3] Writing to file: ./phase1-versionset.json
|
|
✅ File saved: C:\Job_Roomz\KArtSell.Aegis\phase1-versionset.json
|
|
|
|
✅ IDENTIFIERS GENERATED
|
|
{
|
|
"phase1_run": {
|
|
"runId": "<UUID>",
|
|
"jobId": "<UUID>",
|
|
"jobRunId": "<UUID>",
|
|
"correlationId": "<UUID>",
|
|
"idempotencyKey": "<UUID>",
|
|
"generatedAt": "2026-08-07T10:30:00.000Z",
|
|
...
|
|
}
|
|
}
|
|
|
|
Next Steps:
|
|
1. Copy the identifiers from above or read from ./phase1-versionset.json
|
|
2. Call POST /api/shadow-runs with modelId/datasetId from frozen VersionSet
|
|
3. Hangfire will enqueue Job 893 with these correlation IDs
|
|
4. Monitor logs: grep 'CorrelationId: <UUID>' app.log
|
|
```
|
|
|
|
### Save for Reference
|
|
|
|
Copy output to clipboard or save in a secure file. You'll need these IDs in STEP 3.
|
|
|
|
---
|
|
|
|
## 🚀 STEP 3: ENQUEUE SHADOW RUN JOB
|
|
|
|
**Duration:** ~1 minute
|
|
**Method:** PowerShell HTTP request
|
|
|
|
### Prerequisites
|
|
|
|
- [ ] Host running on `http://127.0.0.1:5002` (Debug mode)
|
|
- [ ] VersionSet frozen (STEP 1 complete)
|
|
- [ ] Identifiers generated (STEP 2 complete)
|
|
|
|
### Action
|
|
|
|
```powershell
|
|
# Read generated identifiers
|
|
$versionset = Get-Content ./phase1-versionset.json | ConvertFrom-Json
|
|
$correlationId = $versionset.phase1_run.correlationId
|
|
$runId = $versionset.phase1_run.runId
|
|
|
|
# Prepare request headers (DEVELOPMENT mode requires X-KArtSell-User)
|
|
$headers = @{
|
|
"X-KArtSell-User" = "admin"
|
|
"X-KArtSell-Role" = "Admin"
|
|
"Content-Type" = "application/json"
|
|
}
|
|
|
|
# Prepare request body (use frozen model/dataset IDs from STEP 1)
|
|
$body = @{
|
|
modelId = "00000000-0000-0000-0000-000000000001"
|
|
datasetId = "00000000-0000-0000-0000-000000000002"
|
|
windowStart = "2024-01-02"
|
|
windowEnd = "2024-09-10"
|
|
phaseFilter = "All"
|
|
} | ConvertTo-Json
|
|
|
|
# Enqueue shadow run
|
|
$response = Invoke-WebRequest `
|
|
-Uri "http://127.0.0.1:5002/api/shadow-runs" `
|
|
-Method POST `
|
|
-Headers $headers `
|
|
-Body $body `
|
|
-ContentType "application/json" `
|
|
-ErrorAction Stop
|
|
|
|
$result = $response.Content | ConvertFrom-Json
|
|
|
|
Write-Host "✅ Shadow run enqueued!"
|
|
Write-Host " Job ID: $($result.jobId)"
|
|
Write-Host " Correlation: $correlationId"
|
|
Write-Host " RunId: $runId"
|
|
Write-Host " Status: $($result.status)"
|
|
```
|
|
|
|
### Expected Output (HTTP 202 Accepted)
|
|
|
|
```
|
|
✅ Shadow run enqueued!
|
|
Job ID: <UUID>
|
|
Correlation: <CorrelationId>
|
|
RunId: <RunId>
|
|
Status: Queued
|
|
```
|
|
|
|
### Troubleshooting
|
|
|
|
| Error | Cause | Fix |
|
|
|-------|-------|-----|
|
|
| HTTP 403/404 | Release mode (not Debug) | Check Host startup log; must contain "DevelopmentHeaderAuthenticationHandler" |
|
|
| HTTP 422 Unprocessable | Invalid model/dataset UUID | Verify UUIDs exist in `governance.model_version_registry` via SQL: `SELECT * FROM governance.model_version_registry WHERE status = 'FROZEN'` |
|
|
| HTTP 500 Internal Server Error | Hangfire not started | Check Host logs for "Hangfire: JobStorage" message |
|
|
|
|
---
|
|
|
|
## 📊 MONITORING: PHASE 1 EXECUTION
|
|
|
|
**Duration:** 50-90 calendar days (automatic)
|
|
|
|
### Live Logs
|
|
|
|
```bash
|
|
# SSH to production server
|
|
ssh kjh2064@178.104.200.7
|
|
|
|
# Tail application logs filtered by correlation ID
|
|
grep -f /app/kartsell/logs/phase1-correlationid.txt /app/kartsell/logs/app.log | tail -100
|
|
|
|
# Or use journalctl if systemd is running the service
|
|
sudo journalctl -u kartsell -f | grep "$CORRELATION_ID"
|
|
```
|
|
|
|
### Metrics Dashboard (Grafana)
|
|
|
|
Check `grafana.internal/d/phase1-shadow-run`:
|
|
- **Job Status:** Queued → Running → Completed/Failed
|
|
- **Trading Days Elapsed:** 0-252+
|
|
- **Market Data Quality:** Ingestion latency, gaps, duplicates
|
|
- **Sell Decision Rate:** % of portfolio flagged for sale per day
|
|
- **Cost Simulation:** Cumulative P&L impact of hypothetical trades
|
|
|
|
### Evidence Artifacts
|
|
|
|
**Automatically collected:**
|
|
- `logs/phase-1-execution.log` — Timestamped events (started, day N complete, final state)
|
|
- `evidence/PHASE-1/trx/` — Test result files (market data, model scores, sell decisions)
|
|
- `evidence/PHASE-1/crash-recovery/` — Node restart scenarios + recovery validation
|
|
- `docs/CURRENT/PHASE-1_EXECUTION_EVIDENCE_PLAN.md` — Full checklist
|
|
|
|
### Alerts
|
|
|
|
**Set up pagerduty/Telegram notifications:**
|
|
|
|
```bash
|
|
# Example: Notify if Phase 1 job fails
|
|
curl -X POST "https://api.telegram.org/bot$TELEGRAM_TOKEN/sendMessage" \
|
|
-d "chat_id=$TELEGRAM_CHAT_ID" \
|
|
-d "text=⚠️ Phase 1 Job $JOB_ID failed: $ERROR_MESSAGE"
|
|
```
|
|
|
|
---
|
|
|
|
## ✅ COMPLETION: PHASE 1 EXECUTION COMPLETE
|
|
|
|
**When:**
|
|
- Job 893 reaches 252+ trading days
|
|
- All sell decisions generated + cost impact simulated
|
|
- No gaps or anomalies in market data
|
|
|
|
**What to do:**
|
|
1. Download `logs/phase-1-execution.log` (evidence of completion)
|
|
2. Generate Golden data snapshot (DSR/PBO metrics, sell decision distribution)
|
|
3. Unlock Gates 2-5 (downstream slices depend on this data)
|
|
4. Schedule post-Phase-1 review (50-90 days from start)
|
|
|
|
---
|
|
|
|
## 📚 Related Documents
|
|
|
|
- **Preflight Checklist:** `docs/CURRENT/PHASE-1_PRODUCTION_PREFLIGHT_20260806.md`
|
|
- **Architecture Decision:** `docs/DECISIONS/ADR-SEC-001.md`
|
|
- **Hangfire Jobs:** `src/KArtSell.Host/Jobs/ShadowRunJob.cs`
|
|
- **Evidence Plan:** `docs/CURRENT/PHASE-1_EXECUTION_EVIDENCE_PLAN.md`
|
|
|
|
---
|
|
|
|
## 🆘 Emergency Rollback
|
|
|
|
**If Phase 1 must be stopped:**
|
|
|
|
1. SSH to production
|
|
2. `sudo systemctl stop kartsell`
|
|
3. Kill Job 893 in Hangfire Dashboard (Admin UI)
|
|
4. Archive logs: `cp /app/kartsell/logs/phase-1-execution.log evidence/PHASE-1/rollback-$(date +%s).log`
|
|
5. Notify team (Telegram/Email)
|
|
6. Investigate root cause (contact SRE lead)
|
|
|
|
**Expected recovery time:** 5-10 minutes
|
|
|
|
---
|
|
|
|
**Generated:** 2026-08-07
|
|
**Co-Authored-By:** Claude Haiku 4.5 <noreply@anthropic.com>
|