# Phase 1 Activation Runbook

**Date:** 2026-08-07  
**Purpose:** Step-by-step activation of Phase 1 shadow run (252+ trading days)  
**Owner:** Platform SRE  
**Status:** READY FOR EXECUTION (All tools prepared)

---

## 🎯 Objective

Launch **Job 893 (Shadow Run)** with frozen model/dataset VersionSet, generating 252+ trading days of market simulation with auditable evidence trail.

**Timeline:**
- **Setup:** ~15 minutes (this runbook)
- **Execution:** 50-90 calendar days (automatic, no manual intervention)
- **Evidence Collection:** Concurrent (logs, metrics, state snapshots)

---

## 📋 PRE-FLIGHT CHECKLIST

**All items must be COMPLETE before proceeding to Step 1.**

- [ ] **1. Migration 0032 deployed**  
  Verify: `SELECT schema_version FROM schema_version_history WHERE script_name LIKE '0032_%'`  
  Status: Must return 1 row. If missing, run `dotnet run --project src/KArtSell.DbMigrator`

- [ ] **2. Host running in DEVELOPMENT mode**  
  Verify: `dotnet run --project src/KArtSell.Host -c Debug --no-build`  
  Expected: "Now listening on: http://127.0.0.1:5002"  
  **Why Debug mode?** `DevelopmentHeaderAuthenticationHandler` required for testing; Release mode uses `FailClosedAuthenticationHandler` (rejects all requests)

- [ ] **3. PostgreSQL accessible via SSH tunnel**  
  Verify: `ssh -L 5432:127.0.0.1:5432 kjh2064@178.104.200.7` (keep open in separate terminal)  
  Expected: No errors; tunnel stays alive

- [ ] **4. Hangfire scheduler running**  
  Verify: Host logs contain `Hangfire: JobStorage initialized`  
  Expected: Startup completes without timeout

- [ ] **5. Scripts available in ./scripts/**  
  Verify: `ls scripts/freeze-versionset.ps1 scripts/generate-shadow-run-identifiers.ps1`

---

## 🚀 STEP 1: FREEZE VERSIONSET

**Duration:** ~2 minutes  
**Tool:** `./scripts/freeze-versionset.ps1`

### Action

Execute with **REAL, APPROVED** model/dataset IDs:

```powershell
cd C:\Job_Roomz\KArtSell.Aegis

$env:KARTSELL_POSTGRES = "Host=localhost;Port=5432;Database=kartsell;Username=kartsell;Password=kartsell"

.\scripts\freeze-versionset.ps1 `
    -ModelId "00000000-0000-0000-0000-000000000001" `
    -DatasetId "00000000-0000-0000-0000-000000000002" `
    -ApprovedBy "kim.jae.hyun@example.com" `
    -ConfigVersion "v1.0.0" `
    -CodeSha "acaa731b3f"
```

### Expected Output

```
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Phase 1: Freeze VersionSet
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[1/3] PRE-FLIGHT CHECK
  Model ID:        00000000-0000-0000-0000-000000000001
  Dataset ID:      00000000-0000-0000-0000-000000000002
  Approved By:     kim.jae.hyun@example.com
  Config Version:  v1.0.0
  Code SHA:        acaa731b3f
  Connection:      Host=localhost;Port=5432;Database=kartsell;***

[2/3] VERIFY Migration 0032 deployed...
  ✅ Migration 0032 deployed (schema_version: 32)

[3/3] FREEZE VersionSet...
  ✅ Inserted governance.model_version_registry:
     - ID:      <UUID>
     - Model:   00000000-0000-0000-0000-000000000001
     - Dataset: 00000000-0000-0000-0000-000000000002
     - Status:  FROZEN
  ✅ Inserted evaluation.dataset_manifest:
     - ID:      <UUID>
     - Dataset: 00000000-0000-0000-0000-000000000002
     - Model:   00000000-0000-0000-0000-000000000001
     - Status:  FROZEN

✅ VersionSet FROZEN successfully
   Correlation ID: <UUID>
   Next: Run generate-shadow-run-identifiers.ps1 to create RunId/JobId
```

### Troubleshooting

| Error | Cause | Fix |
|-------|-------|-----|
| "Migration 0032 NOT FOUND" | DbMigrator hasn't run yet | Run: `dotnet run --project src/KArtSell.DbMigrator` |
| "Cannot bind argument -ModelId" | Invalid UUID format | Use: `[System.Guid]::NewGuid() \| % { $_.ToString() }` to generate valid UUID |
| "Connection refused" | PostgreSQL not accessible | Verify SSH tunnel: `ssh -L 5432:127.0.0.1:5432 kjh2064@178.104.200.7` |

---

## 🚀 STEP 2: GENERATE IDENTIFIERS

**Duration:** ~1 minute  
**Tool:** `./scripts/generate-shadow-run-identifiers.ps1`

### Action

```powershell
.\scripts\generate-shadow-run-identifiers.ps1 -OutputPath ./phase1-versionset.json
```

### Expected Output

```
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Phase 1: Generate Shadow Run Identifiers
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[1/3] Generating cryptographic UUIDs...
  ✅ RunId:             <UUID>
  ✅ JobId:             <UUID>
  ✅ JobRunId:          <UUID>
  ✅ CorrelationId:     <UUID>
  ✅ IdempotencyKey:    <UUID>

[2/3] Creating JSON payload...
  ✅ JSON payload generated

[3/3] Writing to file: ./phase1-versionset.json
  ✅ File saved: C:\Job_Roomz\KArtSell.Aegis\phase1-versionset.json

✅ IDENTIFIERS GENERATED
{
  "phase1_run": {
    "runId": "<UUID>",
    "jobId": "<UUID>",
    "jobRunId": "<UUID>",
    "correlationId": "<UUID>",
    "idempotencyKey": "<UUID>",
    "generatedAt": "2026-08-07T10:30:00.000Z",
    ...
  }
}

Next Steps:
  1. Copy the identifiers from above or read from ./phase1-versionset.json
  2. Call POST /api/shadow-runs with modelId/datasetId from frozen VersionSet
  3. Hangfire will enqueue Job 893 with these correlation IDs
  4. Monitor logs: grep 'CorrelationId: <UUID>' app.log
```

### Save for Reference

Copy output to clipboard or save in a secure file. You'll need these IDs in STEP 3.

---

## 🚀 STEP 3: ENQUEUE SHADOW RUN JOB

**Duration:** ~1 minute  
**Method:** PowerShell HTTP request

### Prerequisites

- [ ] Host running on `http://127.0.0.1:5002` (Debug mode)
- [ ] VersionSet frozen (STEP 1 complete)
- [ ] Identifiers generated (STEP 2 complete)

### Action

```powershell
# Read generated identifiers
$versionset = Get-Content ./phase1-versionset.json | ConvertFrom-Json
$correlationId = $versionset.phase1_run.correlationId
$runId = $versionset.phase1_run.runId

# Prepare request headers (DEVELOPMENT mode requires X-KArtSell-User)
$headers = @{
    "X-KArtSell-User" = "admin"
    "X-KArtSell-Role" = "Admin"
    "Content-Type" = "application/json"
}

# Prepare request body (use frozen model/dataset IDs from STEP 1)
$body = @{
    modelId = "00000000-0000-0000-0000-000000000001"
    datasetId = "00000000-0000-0000-0000-000000000002"
    windowStart = "2024-01-02"
    windowEnd = "2024-09-10"
    phaseFilter = "All"
} | ConvertTo-Json

# Enqueue shadow run
$response = Invoke-WebRequest `
    -Uri "http://127.0.0.1:5002/api/shadow-runs" `
    -Method POST `
    -Headers $headers `
    -Body $body `
    -ContentType "application/json" `
    -ErrorAction Stop

$result = $response.Content | ConvertFrom-Json

Write-Host "✅ Shadow run enqueued!"
Write-Host "   Job ID:       $($result.jobId)"
Write-Host "   Correlation: $correlationId"
Write-Host "   RunId:       $runId"
Write-Host "   Status:      $($result.status)"
```

### Expected Output (HTTP 202 Accepted)

```
✅ Shadow run enqueued!
   Job ID:       <UUID>
   Correlation: <CorrelationId>
   RunId:       <RunId>
   Status:      Queued
```

### Troubleshooting

| Error | Cause | Fix |
|-------|-------|-----|
| HTTP 403/404 | Release mode (not Debug) | Check Host startup log; must contain "DevelopmentHeaderAuthenticationHandler" |
| HTTP 422 Unprocessable | Invalid model/dataset UUID | Verify UUIDs exist in `governance.model_version_registry` via SQL: `SELECT * FROM governance.model_version_registry WHERE status = 'FROZEN'` |
| HTTP 500 Internal Server Error | Hangfire not started | Check Host logs for "Hangfire: JobStorage" message |

---

## 📊 MONITORING: PHASE 1 EXECUTION

**Duration:** 50-90 calendar days (automatic)

### Live Logs

```bash
# SSH to production server
ssh kjh2064@178.104.200.7

# Tail application logs filtered by correlation ID
grep -f /app/kartsell/logs/phase1-correlationid.txt /app/kartsell/logs/app.log | tail -100

# Or use journalctl if systemd is running the service
sudo journalctl -u kartsell -f | grep "$CORRELATION_ID"
```

### Metrics Dashboard (Grafana)

Check `grafana.internal/d/phase1-shadow-run`:
- **Job Status:** Queued → Running → Completed/Failed
- **Trading Days Elapsed:** 0-252+
- **Market Data Quality:** Ingestion latency, gaps, duplicates
- **Sell Decision Rate:** % of portfolio flagged for sale per day
- **Cost Simulation:** Cumulative P&L impact of hypothetical trades

### Evidence Artifacts

**Automatically collected:**
- `logs/phase-1-execution.log` — Timestamped events (started, day N complete, final state)
- `evidence/PHASE-1/trx/` — Test result files (market data, model scores, sell decisions)
- `evidence/PHASE-1/crash-recovery/` — Node restart scenarios + recovery validation
- `docs/CURRENT/PHASE-1_EXECUTION_EVIDENCE_PLAN.md` — Full checklist

### Alerts

**Set up pagerduty/Telegram notifications:**

```bash
# Example: Notify if Phase 1 job fails
curl -X POST "https://api.telegram.org/bot$TELEGRAM_TOKEN/sendMessage" \
  -d "chat_id=$TELEGRAM_CHAT_ID" \
  -d "text=⚠️  Phase 1 Job $JOB_ID failed: $ERROR_MESSAGE"
```

---

## ✅ COMPLETION: PHASE 1 EXECUTION COMPLETE

**When:**
- Job 893 reaches 252+ trading days
- All sell decisions generated + cost impact simulated
- No gaps or anomalies in market data

**What to do:**
1. Download `logs/phase-1-execution.log` (evidence of completion)
2. Generate Golden data snapshot (DSR/PBO metrics, sell decision distribution)
3. Unlock Gates 2-5 (downstream slices depend on this data)
4. Schedule post-Phase-1 review (50-90 days from start)

---

## 📚 Related Documents

- **Preflight Checklist:** `docs/CURRENT/PHASE-1_PRODUCTION_PREFLIGHT_20260806.md`
- **Architecture Decision:** `docs/DECISIONS/ADR-SEC-001.md`
- **Hangfire Jobs:** `src/KArtSell.Host/Jobs/ShadowRunJob.cs`
- **Evidence Plan:** `docs/CURRENT/PHASE-1_EXECUTION_EVIDENCE_PLAN.md`

---

## 🆘 Emergency Rollback

**If Phase 1 must be stopped:**

1. SSH to production
2. `sudo systemctl stop kartsell`
3. Kill Job 893 in Hangfire Dashboard (Admin UI)
4. Archive logs: `cp /app/kartsell/logs/phase-1-execution.log evidence/PHASE-1/rollback-$(date +%s).log`
5. Notify team (Telegram/Email)
6. Investigate root cause (contact SRE lead)

**Expected recovery time:** 5-10 minutes

---

**Generated:** 2026-08-07  
**Co-Authored-By:** Claude Haiku 4.5 <noreply@anthropic.com>
