KArtSell.Aegis/docs/CURRENT/PHASE-1_ACTIVATION_RUNBOOK.md

Phase 1 Activation Runbook

Date: 2026-08-07
Purpose: Step-by-step activation of Phase 1 shadow run (252+ trading days)
Owner: Platform SRE
Status: READY FOR EXECUTION (All tools prepared)

---

🎯 Objective

Launch Job 893 (Shadow Run) with frozen model/dataset VersionSet, generating 252+ trading days of market simulation with auditable evidence trail.

Timeline:

• Setup: ~15 minutes (this runbook)
• Execution: 50-90 calendar days (automatic, no manual intervention)
• Evidence Collection: Concurrent (logs, metrics, state snapshots)

---

📋 PRE-FLIGHT CHECKLIST

All items must be COMPLETE before proceeding to Step 1.

• 1. Migration 0032 deployed
  Verify: SELECT schema_version FROM schema_version_history WHERE script_name LIKE '0032_%'
  Status: Must return 1 row. If missing, run dotnet run --project src/KArtSell.DbMigrator

• 2. Host running in DEVELOPMENT mode
  Verify: dotnet run --project src/KArtSell.Host -c Debug --no-build
  Expected: "Now listening on: http://127.0.0.1:5002"
  Why Debug mode? DevelopmentHeaderAuthenticationHandler required for testing; Release mode uses FailClosedAuthenticationHandler (rejects all requests)

• 3. PostgreSQL accessible via SSH tunnel
  Verify: ssh -L 5432:127.0.0.1:5432 kjh2064@178.104.200.7 (keep open in separate terminal)
  Expected: No errors; tunnel stays alive

• 4. Hangfire scheduler running
  Verify: Host logs contain Hangfire: JobStorage initialized
  Expected: Startup completes without timeout

• 5. Scripts available in ./scripts/
  Verify: ls scripts/freeze-versionset.ps1 scripts/generate-shadow-run-identifiers.ps1

---

🚀 STEP 1: FREEZE VERSIONSET

Duration: ~2 minutes
Tool: ./scripts/freeze-versionset.ps1

Action

Execute with REAL, APPROVED model/dataset IDs:

cd C:\Job_Roomz\KArtSell.Aegis

$env:KARTSELL_POSTGRES = "Host=localhost;Port=5432;Database=kartsell;Username=kartsell;Password=kartsell"

.\scripts\freeze-versionset.ps1 `
    -ModelId "00000000-0000-0000-0000-000000000001" `
    -DatasetId "00000000-0000-0000-0000-000000000002" `
    -ApprovedBy "kim.jae.hyun@example.com" `
    -ConfigVersion "v1.0.0" `
    -CodeSha "acaa731b3f"

Expected Output

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Phase 1: Freeze VersionSet
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[1/3] PRE-FLIGHT CHECK
  Model ID:        00000000-0000-0000-0000-000000000001
  Dataset ID:      00000000-0000-0000-0000-000000000002
  Approved By:     kim.jae.hyun@example.com
  Config Version:  v1.0.0
  Code SHA:        acaa731b3f
  Connection:      Host=localhost;Port=5432;Database=kartsell;***

[2/3] VERIFY Migration 0032 deployed...
  ✅ Migration 0032 deployed (schema_version: 32)

[3/3] FREEZE VersionSet...
  ✅ Inserted governance.model_version_registry:
     - ID:      <UUID>
     - Model:   00000000-0000-0000-0000-000000000001
     - Dataset: 00000000-0000-0000-0000-000000000002
     - Status:  FROZEN
  ✅ Inserted evaluation.dataset_manifest:
     - ID:      <UUID>
     - Dataset: 00000000-0000-0000-0000-000000000002
     - Model:   00000000-0000-0000-0000-000000000001
     - Status:  FROZEN

✅ VersionSet FROZEN successfully
   Correlation ID: <UUID>
   Next: Run generate-shadow-run-identifiers.ps1 to create RunId/JobId

Troubleshooting

Error	Cause	Fix
"Migration 0032 NOT FOUND"	DbMigrator hasn't run yet	Run: dotnet run --project src/KArtSell.DbMigrator
"Cannot bind argument -ModelId"	Invalid UUID format	Use: [System.Guid]::NewGuid() | % { $_.ToString() } to generate valid UUID
"Connection refused"	PostgreSQL not accessible	Verify SSH tunnel: ssh -L 5432:127.0.0.1:5432 kjh2064@178.104.200.7

---

🚀 STEP 2: GENERATE IDENTIFIERS

Duration: ~1 minute
Tool: ./scripts/generate-shadow-run-identifiers.ps1

Action

.\scripts\generate-shadow-run-identifiers.ps1 -OutputPath ./phase1-versionset.json

Expected Output

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Phase 1: Generate Shadow Run Identifiers
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[1/3] Generating cryptographic UUIDs...
  ✅ RunId:             <UUID>
  ✅ JobId:             <UUID>
  ✅ JobRunId:          <UUID>
  ✅ CorrelationId:     <UUID>
  ✅ IdempotencyKey:    <UUID>

[2/3] Creating JSON payload...
  ✅ JSON payload generated

[3/3] Writing to file: ./phase1-versionset.json
  ✅ File saved: C:\Job_Roomz\KArtSell.Aegis\phase1-versionset.json

✅ IDENTIFIERS GENERATED
{
  "phase1_run": {
    "runId": "<UUID>",
    "jobId": "<UUID>",
    "jobRunId": "<UUID>",
    "correlationId": "<UUID>",
    "idempotencyKey": "<UUID>",
    "generatedAt": "2026-08-07T10:30:00.000Z",
    ...
  }
}

Next Steps:
  1. Copy the identifiers from above or read from ./phase1-versionset.json
  2. Call POST /api/shadow-runs with modelId/datasetId from frozen VersionSet
  3. Hangfire will enqueue Job 893 with these correlation IDs
  4. Monitor logs: grep 'CorrelationId: <UUID>' app.log

Save for Reference

Copy output to clipboard or save in a secure file. You'll need these IDs in STEP 3.

---

🚀 STEP 3: ENQUEUE SHADOW RUN JOB

Duration: ~1 minute
Method: PowerShell HTTP request

Prerequisites

• Host running on http://127.0.0.1:5002 (Debug mode)
• VersionSet frozen (STEP 1 complete)
• Identifiers generated (STEP 2 complete)

Action

# Read generated identifiers
$versionset = Get-Content ./phase1-versionset.json | ConvertFrom-Json
$correlationId = $versionset.phase1_run.correlationId
$runId = $versionset.phase1_run.runId

# Prepare request headers (DEVELOPMENT mode requires X-KArtSell-User)
$headers = @{
    "X-KArtSell-User" = "admin"
    "X-KArtSell-Role" = "Admin"
    "Content-Type" = "application/json"
}

# Prepare request body (use frozen model/dataset IDs from STEP 1)
$body = @{
    modelId = "00000000-0000-0000-0000-000000000001"
    datasetId = "00000000-0000-0000-0000-000000000002"
    windowStart = "2024-01-02"
    windowEnd = "2024-09-10"
    phaseFilter = "All"
} | ConvertTo-Json

# Enqueue shadow run
$response = Invoke-WebRequest `
    -Uri "http://127.0.0.1:5002/api/shadow-runs" `
    -Method POST `
    -Headers $headers `
    -Body $body `
    -ContentType "application/json" `
    -ErrorAction Stop

$result = $response.Content | ConvertFrom-Json

Write-Host "✅ Shadow run enqueued!"
Write-Host "   Job ID:       $($result.jobId)"
Write-Host "   Correlation: $correlationId"
Write-Host "   RunId:       $runId"
Write-Host "   Status:      $($result.status)"

Expected Output (HTTP 202 Accepted)

✅ Shadow run enqueued!
   Job ID:       <UUID>
   Correlation: <CorrelationId>
   RunId:       <RunId>
   Status:      Queued

Troubleshooting

Error	Cause	Fix
HTTP 403/404	Release mode (not Debug)	Check Host startup log; must contain "DevelopmentHeaderAuthenticationHandler"
HTTP 422 Unprocessable	Invalid model/dataset UUID	Verify UUIDs exist in governance.model_version_registry via SQL: SELECT * FROM governance.model_version_registry WHERE status = 'FROZEN'
HTTP 500 Internal Server Error	Hangfire not started	Check Host logs for "Hangfire: JobStorage" message

---

📊 MONITORING: PHASE 1 EXECUTION

Duration: 50-90 calendar days (automatic)

Live Logs

# SSH to production server
ssh kjh2064@178.104.200.7

# Tail application logs filtered by correlation ID
grep -f /app/kartsell/logs/phase1-correlationid.txt /app/kartsell/logs/app.log | tail -100

# Or use journalctl if systemd is running the service
sudo journalctl -u kartsell -f | grep "$CORRELATION_ID"

Metrics Dashboard (Grafana)

Check grafana.internal/d/phase1-shadow-run:

• Job Status: Queued → Running → Completed/Failed
• Trading Days Elapsed: 0-252+
• Market Data Quality: Ingestion latency, gaps, duplicates
• Sell Decision Rate: % of portfolio flagged for sale per day
• Cost Simulation: Cumulative P&L impact of hypothetical trades

Evidence Artifacts

Automatically collected:

• logs/phase-1-execution.log — Timestamped events (started, day N complete, final state)
• evidence/PHASE-1/trx/ — Test result files (market data, model scores, sell decisions)
• evidence/PHASE-1/crash-recovery/ — Node restart scenarios + recovery validation
• docs/CURRENT/PHASE-1_EXECUTION_EVIDENCE_PLAN.md — Full checklist

Alerts

Set up pagerduty/Telegram notifications:

# Example: Notify if Phase 1 job fails
curl -X POST "https://api.telegram.org/bot$TELEGRAM_TOKEN/sendMessage" \
  -d "chat_id=$TELEGRAM_CHAT_ID" \
  -d "text=⚠️  Phase 1 Job $JOB_ID failed: $ERROR_MESSAGE"

---

✅ COMPLETION: PHASE 1 EXECUTION COMPLETE

When:

• Job 893 reaches 252+ trading days
• All sell decisions generated + cost impact simulated
• No gaps or anomalies in market data

What to do:

1. Download logs/phase-1-execution.log (evidence of completion)
2. Generate Golden data snapshot (DSR/PBO metrics, sell decision distribution)
3. Unlock Gates 2-5 (downstream slices depend on this data)
4. Schedule post-Phase-1 review (50-90 days from start)

---

📚 Related Documents

• Preflight Checklist: docs/CURRENT/PHASE-1_PRODUCTION_PREFLIGHT_20260806.md
• Architecture Decision: docs/DECISIONS/ADR-SEC-001.md
• Hangfire Jobs: src/KArtSell.Host/Jobs/ShadowRunJob.cs
• Evidence Plan: docs/CURRENT/PHASE-1_EXECUTION_EVIDENCE_PLAN.md

---

🆘 Emergency Rollback

If Phase 1 must be stopped:

1. SSH to production
2. sudo systemctl stop kartsell
3. Kill Job 893 in Hangfire Dashboard (Admin UI)
4. Archive logs: cp /app/kartsell/logs/phase-1-execution.log evidence/PHASE-1/rollback-$(date +%s).log
5. Notify team (Telegram/Email)
6. Investigate root cause (contact SRE lead)

Expected recovery time: 5-10 minutes

---

Generated: 2026-08-07
Co-Authored-By: Claude Haiku 4.5 noreply@anthropic.com