KArtSell.Aegis/docs/PHASE_1_MONITORING_GUIDE.md

# Phase 1 (Job 893) Monitoring Guide

**Status:** Ready to monitor  
**Job ID:** 00000000-0000-0000-0000-000000000893  
**Duration:** 50-90 trading days (autonomous)  
**Updated:** 2026-08-06

---

## Prerequisites

1. **SSH Tunnel** (Terminal 1 - Keep Open)
```powershell
ssh -L 5432:127.0.0.1:5432 kjh2064@178.104.200.7
```

2. **Host Application** (Terminal 2 - Keep Open)
```powershell
cd D:\JobRoomz\KArtSell.Aegis

# Set environment
$env:KARTSELL_POSTGRES = "Host=localhost;Port=5432;Database=kartsell;Username=kartsell;Password=kartsell"
$env:KRX_OPENAPI = "<actual-krx-api-key>"  # from Gitea Secrets
$env:OPENDART_API = "<actual-opendart-api-key>"
$env:KIS_API_KEY = "<actual-kis-api-key>"

# Start in DEVELOPMENT mode (critical for testing)
dotnet run --project src/KArtSell.Host --configuration Debug --no-build
```

Expected output:
```
info: Microsoft.Hosting.Lifetime[14]
      Now listening on: http://127.0.0.1:5002
```

3. **Monitoring Dashboard** (Terminal 3 - Monitor)
```powershell
# Option A: Hangfire Dashboard (Real-time UI)
Start-Process "http://localhost:5002/hangfire"

# Option B: Database queries (Command line)
# See below
```

---

## Monitoring Methods

### Method 1: Hangfire Dashboard (Recommended)

**URL:** `http://localhost:5002/hangfire`

**What to watch:**
- **Queues:** `q-research` queue depth
- **Processing:** Active job (should be Job 893)
- **Jobs:** Completed/Failed count
- **Scheduled:** Any pending tasks

**Metrics:**
- Current queue depth
- Processing rate (rows/second)
- Average job duration
- Error count & last error

### Method 2: Database Queries

**Check Job Status:**
```bash
psql -h localhost -U kartsell -d kartsell <<EOF
SELECT 
    job_id,
    model_id,
    status,
    window_start,
    window_end,
    progress_percent,
    rows_processed,
    updated_at,
    last_error
FROM model_operations.shadow_runs
WHERE job_id = '00000000-0000-0000-0000-000000000893'
LIMIT 1;
EOF
```

**Expected output (in progress):**
```
           job_id           | model_id | status  | progress_percent | rows_processed | updated_at
---------------------+----------+---------+------------------+----------------+-------------------
 00000000-0000-0000-0000-000000000893 | 00000000-0000-0000-0000-000000000001 | Running |               35 |       1250000 | 2026-08-06 14:32:15.123456+00
```

**Check Recent Logs:**
```bash
psql -h localhost -U kartsell -d kartsell <<EOF
SELECT 
    log_time,
    log_level,
    message
FROM model_operations.job_logs
WHERE job_id = '00000000-0000-0000-0000-000000000893'
ORDER BY log_time DESC
LIMIT 10;
EOF
```

### Method 3: PowerShell Auto-Monitor (5-minute intervals)

```powershell
# Terminal 3: Run continuous monitor
$query = @"
SELECT job_id, status, progress_percent, rows_processed, updated_at
FROM model_operations.shadow_runs
WHERE job_id = '00000000-0000-0000-0000-000000000893'
LIMIT 1;
"@

1..1000 | ForEach-Object {
    Write-Host "[$(Get-Date -Format 'HH:mm:ss')]" -ForegroundColor Cyan
    psql -h localhost -U kartsell -d kartsell -c $query
    Write-Host "`n---`n"
    
    Start-Sleep -Seconds 300  # 5 minutes
}
```

---

## Expected Behavior

### Phase 1 Timeline

| Phase | Duration | Status | Actions |
|-------|----------|--------|---------|
| **Init** | 1 min | Queued → Running | Job 893 starts, row count = 0 |
| **Window 1** | 1-2 days | Running | Processes 1st trading week (5 days) |
| **Windows 2-52** | 48-52 weeks | Running | Incremental progress, 50-90% range |
| **Completion** | 1 min | Completed | Final metrics computed, status = "Completed" |

### Expected Progress

- **Start:** Progress = 0%, Rows = 0, Status = "Running"
- **After 1 week:** Progress ~2%, Rows = 500K
- **After 1 month:** Progress ~8%, Rows = 2M
- **After 3 months:** Progress ~25%, Rows = 6M
- **After 6 months:** Progress ~50%, Rows = 12M
- **After 9 months:** Progress ~75%, Rows = 18M
- **After 12 months:** Progress ~95%, Rows = 22M
- **Completion:** Progress = 100%, Status = "Completed"

### Error Handling

If `last_error` is not NULL:
1. Check `log_level = 'ERROR'` entries
2. Classify: transient (retry) vs permanent (investigate)
3. If permanent: Check AGENTS.md #20 (failures not retried blindly)

**Common Errors:**
| Error | Cause | Action |
|-------|-------|--------|
| "Network timeout" | KRX API unavailable | Auto-retry (Hangfire) |
| "Duplicate key" | Idempotency key collision | Wait for cleanup job |
| "Out of memory" | Large date range | Reduce window size |
| "Access denied" | Auth token expired | Restart Host with fresh keys |

---

## Alerts & Thresholds

**Create alerts for:**
- Status = "Failed" → Page oncall
- Progress flat for > 24 hours → Check logs
- Error rate > 5% → Investigate data quality
- Memory usage > 80% → Consider restart

**Safe to ignore:**
- Progress rate varies (weekend vs weekday)
- Occasional transient errors (network glitches)
- Queue depth spikes (normal batch processing)

---

## Monitoring Commands Reference

```powershell
# Check Host health
curl http://localhost:5002/health

# View Hangfire in browser
Start-Process "http://localhost:5002/hangfire"

# Database status (one-liner)
psql -h localhost -U kartsell -d kartsell -c "SELECT status, progress_percent, rows_processed FROM model_operations.shadow_runs WHERE job_id = '00000000-0000-0000-0000-000000000893'"

# Stop Host gracefully
# Press Ctrl+C in Host terminal

# View all Job 893 events
psql -h localhost -U kartsell -d kartsell -c "SELECT event_type, created_at, details FROM audit.job_events WHERE job_id = '00000000-0000-0000-0000-000000000893' ORDER BY created_at DESC LIMIT 20"
```

---

## Success Criteria

✅ **Job 893 Monitoring Active** when:
1. SSH tunnel is open
2. Host is listening on 127.0.0.1:5002
3. Database returns `status = 'Running'` and `progress_percent > 0`
4. Hangfire dashboard shows Job 893 in queue or processing

---

## Troubleshooting

### "Host not responding"
```powershell
# Check if process is running
Get-Process | Where-Object {$_.ProcessName -like "*Host*"}

# Restart Host
dotnet run --project src/KArtSell.Host --configuration Debug
```

### "SSH tunnel failed"
```bash
# Check SSH key permissions
ls -la ~/.ssh/id_rsa  # Should be 600

# Test SSH connection
ssh kjh2064@178.104.200.7 -v
```

### "Database connection refused"
```powershell
# Check port forwarding
Test-NetConnection -ComputerName localhost -Port 5432

# Restart SSH tunnel in Terminal 1
ssh -L 5432:127.0.0.1:5432 kjh2064@178.104.200.7
```

### "No rows in shadow_runs table"
- Job 893 may not have started yet
- Check Hangfire queue: http://localhost:5002/hangfire
- Verify Gate 3 API was called (see PHASE_1_STARTUP_GUIDE.md)

---

## 3-Terminal Setup (Recommended)

```
Terminal 1 (SSH)              Terminal 2 (Host)           Terminal 3 (Monitor)
│                             │                           │
├─ ssh -L 5432 ...            ├─ dotnet run Host          ├─ Hangfire dashboard
│  (Keep open)                │  (Keep open)              │  OR
│                             │                           ├─ PowerShell loop
│                             │                           └─ psql queries
```

Each terminal:
- Separate window/tab
- Keep open for entire Phase 1 duration
- Log output for troubleshooting
- Do NOT close during run

---

## Log Files

- **Host logs:** `logs/host-*.log` (check for errors)
- **Job logs:** `model_operations.job_logs` table (database)
- **Audit trail:** `audit.job_events` table (database)
- **Hangfire logs:** Embedded in Host logs

---

## Phase 1 Completion

When `status = 'Completed'`:

1. ✅ Check final `progress_percent = 100`
2. ✅ Verify `last_error IS NULL`
3. ✅ Record `rows_processed` (expected: 20M+)
4. ✅ Save completion timestamp
5. ✅ Proceed to Production Deployment (DEPLOY_PRODUCTION_NOW.ps1)

**Estimated completion:** November 2026 (50-90 trading days from start)

---

**Document Version:** 1.0  
**Last Updated:** 2026-08-06  
**AGENTS.md Compliance:** v16.0 ✅