KArtSell.Aegis/docs/PHASE_1_MONITORING_GUIDE.md

Phase 1 (Job 893) Monitoring Guide

Status: Ready to monitor
Job ID: 00000000-0000-0000-0000-000000000893
Duration: 50-90 trading days (autonomous)
Updated: 2026-08-06

---

Prerequisites

1. SSH Tunnel (Terminal 1 - Keep Open)

ssh -L 5432:127.0.0.1:5432 kjh2064@178.104.200.7

2. Host Application (Terminal 2 - Keep Open)

cd D:\JobRoomz\KArtSell.Aegis

# Set environment
$env:KARTSELL_POSTGRES = "Host=localhost;Port=5432;Database=kartsell;Username=kartsell;Password=kartsell"
$env:KRX_OPENAPI = "<actual-krx-api-key>"  # from Gitea Secrets
$env:OPENDART_API = "<actual-opendart-api-key>"
$env:KIS_API_KEY = "<actual-kis-api-key>"

# Start in DEVELOPMENT mode (critical for testing)
dotnet run --project src/KArtSell.Host --configuration Debug --no-build

Expected output:

info: Microsoft.Hosting.Lifetime[14]
      Now listening on: http://127.0.0.1:5002

3. Monitoring Dashboard (Terminal 3 - Monitor)

# Option A: Hangfire Dashboard (Real-time UI)
Start-Process "http://localhost:5002/hangfire"

# Option B: Database queries (Command line)
# See below

---

Monitoring Methods

Method 1: Hangfire Dashboard (Recommended)

URL: http://localhost:5002/hangfire

What to watch:

• Queues: q-research queue depth
• Processing: Active job (should be Job 893)
• Jobs: Completed/Failed count
• Scheduled: Any pending tasks

Metrics:

• Current queue depth
• Processing rate (rows/second)
• Average job duration
• Error count & last error

Method 2: Database Queries

Check Job Status:

psql -h localhost -U kartsell -d kartsell <<EOF
SELECT 
    job_id,
    model_id,
    status,
    window_start,
    window_end,
    progress_percent,
    rows_processed,
    updated_at,
    last_error
FROM model_operations.shadow_runs
WHERE job_id = '00000000-0000-0000-0000-000000000893'
LIMIT 1;
EOF

Expected output (in progress):

           job_id           | model_id | status  | progress_percent | rows_processed | updated_at
---------------------+----------+---------+------------------+----------------+-------------------
 00000000-0000-0000-0000-000000000893 | 00000000-0000-0000-0000-000000000001 | Running |               35 |       1250000 | 2026-08-06 14:32:15.123456+00

Check Recent Logs:

psql -h localhost -U kartsell -d kartsell <<EOF
SELECT 
    log_time,
    log_level,
    message
FROM model_operations.job_logs
WHERE job_id = '00000000-0000-0000-0000-000000000893'
ORDER BY log_time DESC
LIMIT 10;
EOF

Method 3: PowerShell Auto-Monitor (5-minute intervals)

# Terminal 3: Run continuous monitor
$query = @"
SELECT job_id, status, progress_percent, rows_processed, updated_at
FROM model_operations.shadow_runs
WHERE job_id = '00000000-0000-0000-0000-000000000893'
LIMIT 1;
"@

1..1000 | ForEach-Object {
    Write-Host "[$(Get-Date -Format 'HH:mm:ss')]" -ForegroundColor Cyan
    psql -h localhost -U kartsell -d kartsell -c $query
    Write-Host "`n---`n"
    
    Start-Sleep -Seconds 300  # 5 minutes
}

---

Expected Behavior

Phase 1 Timeline

Phase	Duration	Status	Actions
Init	1 min	Queued → Running	Job 893 starts, row count = 0
Window 1	1-2 days	Running	Processes 1st trading week (5 days)
Windows 2-52	48-52 weeks	Running	Incremental progress, 50-90% range
Completion	1 min	Completed	Final metrics computed, status = "Completed"

Expected Progress

• Start: Progress = 0%, Rows = 0, Status = "Running"
• After 1 week: Progress ~2%, Rows = 500K
• After 1 month: Progress ~8%, Rows = 2M
• After 3 months: Progress ~25%, Rows = 6M
• After 6 months: Progress ~50%, Rows = 12M
• After 9 months: Progress ~75%, Rows = 18M
• After 12 months: Progress ~95%, Rows = 22M
• Completion: Progress = 100%, Status = "Completed"

Error Handling

If last_error is not NULL:

1. Check log_level = 'ERROR' entries
2. Classify: transient (retry) vs permanent (investigate)
3. If permanent: Check AGENTS.md #20 (failures not retried blindly)

Common Errors:

Error	Cause	Action
"Network timeout"	KRX API unavailable	Auto-retry (Hangfire)
"Duplicate key"	Idempotency key collision	Wait for cleanup job
"Out of memory"	Large date range	Reduce window size
"Access denied"	Auth token expired	Restart Host with fresh keys

---

Alerts & Thresholds

Create alerts for:

• Status = "Failed" → Page oncall
• Progress flat for > 24 hours → Check logs
• Error rate > 5% → Investigate data quality
• Memory usage > 80% → Consider restart

Safe to ignore:

• Progress rate varies (weekend vs weekday)
• Occasional transient errors (network glitches)
• Queue depth spikes (normal batch processing)

---

Monitoring Commands Reference

# Check Host health
curl http://localhost:5002/health

# View Hangfire in browser
Start-Process "http://localhost:5002/hangfire"

# Database status (one-liner)
psql -h localhost -U kartsell -d kartsell -c "SELECT status, progress_percent, rows_processed FROM model_operations.shadow_runs WHERE job_id = '00000000-0000-0000-0000-000000000893'"

# Stop Host gracefully
# Press Ctrl+C in Host terminal

# View all Job 893 events
psql -h localhost -U kartsell -d kartsell -c "SELECT event_type, created_at, details FROM audit.job_events WHERE job_id = '00000000-0000-0000-0000-000000000893' ORDER BY created_at DESC LIMIT 20"

---

Success Criteria

✅ Job 893 Monitoring Active when:

1. SSH tunnel is open
2. Host is listening on 127.0.0.1:5002
3. Database returns status = 'Running' and progress_percent > 0
4. Hangfire dashboard shows Job 893 in queue or processing

---

Troubleshooting

"Host not responding"

# Check if process is running
Get-Process | Where-Object {$_.ProcessName -like "*Host*"}

# Restart Host
dotnet run --project src/KArtSell.Host --configuration Debug

"SSH tunnel failed"

# Check SSH key permissions
ls -la ~/.ssh/id_rsa  # Should be 600

# Test SSH connection
ssh kjh2064@178.104.200.7 -v

"Database connection refused"

# Check port forwarding
Test-NetConnection -ComputerName localhost -Port 5432

# Restart SSH tunnel in Terminal 1
ssh -L 5432:127.0.0.1:5432 kjh2064@178.104.200.7

"No rows in shadow_runs table"

• Job 893 may not have started yet
• Check Hangfire queue: http://localhost:5002/hangfire
• Verify Gate 3 API was called (see PHASE_1_STARTUP_GUIDE.md)

---

3-Terminal Setup (Recommended)

Terminal 1 (SSH)              Terminal 2 (Host)           Terminal 3 (Monitor)
│                             │                           │
├─ ssh -L 5432 ...            ├─ dotnet run Host          ├─ Hangfire dashboard
│  (Keep open)                │  (Keep open)              │  OR
│                             │                           ├─ PowerShell loop
│                             │                           └─ psql queries

Each terminal:

• Separate window/tab
• Keep open for entire Phase 1 duration
• Log output for troubleshooting
• Do NOT close during run

---

Log Files

• Host logs: logs/host-*.log (check for errors)
• Job logs: model_operations.job_logs table (database)
• Audit trail: audit.job_events table (database)
• Hangfire logs: Embedded in Host logs

---

Phase 1 Completion

When status = 'Completed':

1. ✅ Check final progress_percent = 100
2. ✅ Verify last_error IS NULL
3. ✅ Record rows_processed (expected: 20M+)
4. ✅ Save completion timestamp
5. ✅ Proceed to Production Deployment (DEPLOY_PRODUCTION_NOW.ps1)

Estimated completion: November 2026 (50-90 trading days from start)

---

Document Version: 1.0
Last Updated: 2026-08-06
AGENTS.md Compliance: v16.0 ✅