From 627e7397b497b635950cf52148efedfcba42d3ae Mon Sep 17 00:00:00 2001 From: Claude Code Date: Fri, 7 Aug 2026 14:24:37 +0900 Subject: [PATCH] docs(phase1): add comprehensive readiness validation checklist PHASE-1_READINESS_VALIDATION_CHECKLIST.md provides structured pre-execution validation across 6 sections: A. Governance & Approvals (DEC-037/038/079, VersionSet) - Validates law/compliance, calendar SLA, business sign-off B. Infrastructure & Environment (PostgreSQL, Host, Frontend) - Database connectivity, migration 0032, Host startup, Hangfire C. Tools & Scripts Validation (freeze, generate, runbook) - Script syntax, dry-run test, error handling, execution procedure D. Data Quality & State Validation (Model/Dataset, PIT queries) - Model card, dataset manifest, market data completeness, audit trail E. Monitoring & Observability (Logging, metrics, alerts) - Structured logging, Grafana dashboard, on-call setup (recommended) F. Final Readiness Sign-offs - Go/No-Go decision matrix with stakeholder approvals - Launch window, emergency contacts, expected completion timeline Features: - 40+ detailed check items across governance + technical + operations - Sign-off blanks for traceability - Error handling matrix for common blockers - Reference links to supporting docs AGENTS.md: Necessity (real validation gap), Maturity (checklist before execution), Traceability (approval audit trail), Right-Way (documented procedure vs ad-hoc). Co-Authored-By: Claude Haiku 4.5 --- .../PHASE-1_READINESS_VALIDATION_CHECKLIST.md | 557 ++++++++++++++++++ 1 file changed, 557 insertions(+) create mode 100644 docs/CURRENT/PHASE-1_READINESS_VALIDATION_CHECKLIST.md diff --git a/docs/CURRENT/PHASE-1_READINESS_VALIDATION_CHECKLIST.md b/docs/CURRENT/PHASE-1_READINESS_VALIDATION_CHECKLIST.md new file mode 100644 index 00000000..f4dd23e1 --- /dev/null +++ b/docs/CURRENT/PHASE-1_READINESS_VALIDATION_CHECKLIST.md @@ -0,0 +1,557 @@ +# Phase 1 Readiness Validation Checklist + +**Date:** 2026-08-07 +**Purpose:** Pre-execution validation of all prerequisites before Phase 1 shadow run activation +**Audience:** SRE, Platform Lead, Business Owner +**Status:** TEMPLATE (ready to execute) + +--- + +## 🎯 Overview + +**Phase 1 Shadow Run:** 252+ trading days autonomous market simulation with auditable evidence +**Setup Time:** ~2 hours (pre-checks + tool validation) +**Execution Time:** 50-90 calendar days (automatic, no manual intervention) +**Success Criteria:** All checks PASS before proceeding to activation + +--- + +## 📋 SECTION A: Governance & Approvals + +### A.1 — DEC-037: Source/License/SLA Approved + +**Owner:** Law + Data Governance +**Deadline:** 2026-08-10 +**Blocking:** YES (blocks P2-P6 automation) + +- [ ] **Source Approved:** KRX/OpenDart/Consensus data sources confirmed + - Evidence: `docs/CURRENT/AEG-X-009_DECISION_PACKAGE.md` signed-off + - Confirm: Which sources are approved for ingestion? + +- [ ] **License Verified:** All sources have compliant license terms + - Evidence: License agreement file path: ___________ + - Confirm: No GPL/AGPL (incompatible with commercial products)? + +- [ ] **Retention SLA Confirmed:** Data retention period defined (1yr/3yr/perpetual) + - Evidence: SLA document: ___________ + - Confirm: Complies with GDPR/PCI-DSS? + +- [ ] **Update Freshness SLA Confirmed:** Daily/weekly/monthly refresh rate + - Evidence: SLA document: ___________ + - Confirm: Shadow run can consume data at this frequency? + +**Sign-off:** ___________ (Law Lead) / ___________ (DataGov Lead) + +--- + +### A.2 — DEC-038: Market Calendar Source & Operator Assigned + +**Owner:** Data Governance + Ops Lead +**Deadline:** 2026-08-12 +**Blocking:** YES (blocks market simulation accuracy) + +- [ ] **Calendar Source Approved:** KRX official holidays/trading calendar + - Evidence: Data source URI: ___________ + - Confirm: 3rd-party aggregator or direct KRX API? + +- [ ] **Owner Assigned:** Named operator responsible for calendar data + - Owner Name: ___________ + - Email: ___________ + - Confirm: On-call rotation configured? + +- [ ] **Secondary Assigned:** Backup operator for calendar updates + - Secondary Name: ___________ + - Email: ___________ + - Confirm: Escalation path defined? + +- [ ] **Timezone Standardized:** Asia/Seoul or UTC chosen globally + - Timezone: ___________ + - Evidence: Config location: ___________ + - Confirm: All shadow run calculations use same timezone? + +**Sign-off:** ___________ (DataGov Lead) / ___________ (Ops Lead) + +--- + +### A.3 — DEC-079: Holiday Correction SLA & Policy + +**Owner:** Data Architecture + Ops + Legal +**Deadline:** 2026-08-12 +**Blocking:** YES (blocks ad-hoc holiday handling) + +- [ ] **Timezone Standard Confirmed:** Asia/Seoul official timezone + - Standard: ___________ + - Evidence: appsettings.json: ___________ + +- [ ] **Holiday Corrections Procedure Defined:** Request → Approve → Reflect + - Request mechanism: ___________ + - Approver(s): ___________ + - SLA (e.g., T+0, T+1, EOM): ___________ + - Evidence: Runbook path: ___________ + +- [ ] **Correction Authority Assigned:** Who can request/approve corrections? + - Request Authority: ___________ + - Approval Authority: ___________ + - Emergency escalation: ___________ + +**Sign-off:** ___________ (Ops Lead) / ___________ (Compliance) + +--- + +### A.4 — VersionSet Approved by Business + +**Owner:** Business Owner / Portfolio Manager +**Deadline:** TBD (Phase 1 start signal) +**Blocking:** YES (gates entire Phase 1) + +- [ ] **Model ID Confirmed:** UUID of model to shadow-run + - Model ID: ___________ + - Model Name: ___________ + - Model Version: ___________ + - Evidence: governance.model_version_registry query result + +- [ ] **Dataset ID Confirmed:** UUID of dataset for backtest period + - Dataset ID: ___________ + - Dataset Name: ___________ + - Coverage: ___________ to ___________ + - Evidence: evaluation.dataset_manifest query result + +- [ ] **Approval Signed:** Model approved for production shadow run + - Approved By (email): ___________ + - Approval Date: ___________ + - Confidence Level (High/Medium/Low): ___________ + - Evidence: Approval document path: ___________ + +- [ ] **Risk Sign-off:** Risk team has signed off on model usage + - Risk Lead: ___________ + - Approval Date: ___________ + - Known Risks Documented: YES / NO + - Risk Mitigation Plan: ___________ + +**Sign-off:** ___________ (Business Owner) / ___________ (Risk Lead) + +--- + +## 🏗️ SECTION B: Infrastructure & Environment + +### B.1 — PostgreSQL Database (Remote) + +**Owner:** DBA / Database Team +**Blocking:** YES (core persistence) + +- [ ] **Remote Host Accessible:** 178.104.200.7 responding to SSH + ```bash + ssh -v kjh2064@178.104.200.7 "exit" + ``` + - Result: ✅ / ❌ + - Latency (ms): ___________ + +- [ ] **SSH Port Forwarding Works:** localhost:5432 → remote PostgreSQL + ```bash + ssh -L 5432:127.0.0.1:5432 kjh2064@178.104.200.7 & + psql -h localhost -U kartsell -d kartsell -c "SELECT NOW()" + ``` + - Result: ✅ / ❌ + - Connection Time (ms): ___________ + +- [ ] **Database Connectivity:** kartsell DB accessible with test query + - Query: `SELECT COUNT(*) FROM governance.model_version_registry` + - Result: ✅ (row count: _______) / ❌ + - Last Backup: ___________ + +- [ ] **Migration 0032 Deployed:** Queued status contract present + - Query: `SELECT schema_version FROM schema_version_history WHERE script_name LIKE '0032_%'` + - Result: ✅ (version: _______) / ❌ + - Evidence: DbMigrator log timestamp: ___________ + +- [ ] **Tables Pre-checked:** + ```sql + SELECT COUNT(*) FROM governance.model_version_registry; + SELECT COUNT(*) FROM evaluation.dataset_manifest; + SELECT COUNT(*) FROM model_operations.shadow_runs; + ``` + - model_version_registry rows: _______ + - dataset_manifest rows: _______ + - shadow_runs rows: _______ + +**Sign-off:** ___________ (DBA) + +--- + +### B.2 — Host Application (.NET) + +**Owner:** Backend Lead / Platform SRE +**Blocking:** YES (API endpoint required) + +- [ ] **Build Successful:** dotnet build -c Release produces artifact + ```bash + dotnet build KArtSell.sln -c Release + ``` + - Result: ✅ (warnings: _______) / ❌ + - Build Time: _______s + - Build Date: ___________ + +- [ ] **Host Startup (DEVELOPMENT mode):** App listens on http://127.0.0.1:5002 + ```bash + dotnet run --project src/KArtSell.Host -c Debug --no-build + ``` + - Result: ✅ / ❌ + - Startup Time: _______s + - Expected Log: "Now listening on: http://127.0.0.1:5002" + +- [ ] **DevelopmentHeaderAuthenticationHandler Active:** + - Log output contains: "DevelopmentHeaderAuthenticationHandler" ✅ / ❌ + - Confirm: Debug mode enables X-KArtSell-User header acceptance + - NOT Release mode (which uses FailClosedAuthenticationHandler) ✅ / ❌ + +- [ ] **Hangfire Scheduler Initialized:** + - Log output contains: "Hangfire: JobStorage initialized" ✅ / ❌ + - Dashboard available: http://127.0.0.1:5002/admin/dashboard ✅ / ❌ + - Job queues visible: q-customer-sla, q-research ✅ / ❌ + +- [ ] **API Health Check:** + ```bash + curl -H "X-KArtSell-User: admin" -H "X-KArtSell-Role: Admin" \ + http://127.0.0.1:5002/health + ``` + - Result: HTTP 200 ✅ / ❌ + +- [ ] **Shadow Run Endpoint Accessible:** + ```bash + curl -X POST \ + -H "X-KArtSell-User: admin" \ + -H "X-KArtSell-Role: Admin" \ + -H "Content-Type: application/json" \ + -d '{"modelId":"","datasetId":"","windowStart":"2024-01-02","windowEnd":"2024-09-10","phaseFilter":"All"}' \ + http://127.0.0.1:5002/api/shadow-runs + ``` + - Result: HTTP 202 Accepted ✅ / HTTP 422 Validation Error ❌ / HTTP 5xx Server Error ❌ + - Response Job ID: ___________ + +**Sign-off:** ___________ (Backend Lead) + +--- + +### B.3 — Frontend Build & Distribution + +**Owner:** Frontend Lead +**Blocking:** NO (Phase 1 is backend-only, but validates deployment) + +- [ ] **Frontend Build Successful:** pnpm build produces dist/ + ```bash + cd frontend && pnpm build + ``` + - Result: ✅ / ❌ + - Build Time: _______s + - Bundle Size (gzip): _______kb + +- [ ] **Static Assets Copied to Host:** dist → src/KArtSell.Host/wwwroot/ + - Confirm: `ls -lh src/KArtSell.Host/wwwroot/index.html` + - Result: ✅ / ❌ + - File Size: _______kb + - Modification Time: ___________ + +- [ ] **UI Contract Markers Present:** + ```bash + grep -r "app-version" dist/ && grep -r "UI contract 4.0" dist/ + ``` + - Result: ✅ (found) / ❌ (missing) + +**Sign-off:** ___________ (Frontend Lead) + +--- + +## 🔧 SECTION C: Tools & Scripts Validation + +### C.1 — freeze-versionset.ps1 Validation + +**Owner:** SRE +**Blocking:** YES (mandatory for VersionSet freeze) + +- [ ] **Script Syntax Valid:** PowerShell parse-check succeeds + ```powershell + pwsh -NoProfile -Command ". scripts/freeze-versionset.ps1 -Help" -ErrorAction Stop + ``` + - Result: ✅ / ❌ + +- [ ] **Parameters Documented:** Help shows all 5 required params + ```powershell + Get-Help scripts/freeze-versionset.ps1 -Full + ``` + - Params found: ModelId ✅, DatasetId ✅, ApprovedBy ✅, ConfigVersion ✅, CodeSha ✅ + +- [ ] **Dry-run Test:** Script validates input without DB modification + ```powershell + scripts/freeze-versionset.ps1 ` + -ModelId "00000000-0000-0000-0000-000000000001" ` + -DatasetId "00000000-0000-0000-0000-000000000002" ` + -ApprovedBy "test@example.com" ` + -ConfigVersion "v1.0.0" ` + -CodeSha "aaaaaaaaaa" + ``` + - Pre-flight Check: ✅ Passed / ❌ Failed + - Migration 0032: ✅ Found / ❌ Not deployed + - Database Insert: ✅ Success / ❌ Failed + - Correlation ID: ___________ + +- [ ] **Error Handling:** Script fails safely if parameter missing + ```powershell + scripts/freeze-versionset.ps1 -ModelId "..." -DatasetId "..." + # Missing: -ApprovedBy, -ConfigVersion, -CodeSha + ``` + - Result: ✅ (fails immediately) / ❌ (proceeds incorrectly) + +**Sign-off:** ___________ (SRE) + +--- + +### C.2 — generate-shadow-run-identifiers.ps1 Validation + +**Owner:** SRE +**Blocking:** NO (utility; can be run anytime) + +- [ ] **Script Syntax Valid:** + ```powershell + pwsh -NoProfile -Command ". scripts/generate-shadow-run-identifiers.ps1 -Help" -ErrorAction Stop + ``` + - Result: ✅ / ❌ + +- [ ] **UUID Generation Works:** + ```powershell + scripts/generate-shadow-run-identifiers.ps1 -OutputPath ./test-versionset.json + ``` + - Result: ✅ / ❌ + - JSON Valid: ✅ / ❌ + - IDs Generated: RunId ✅, JobId ✅, CorrelationId ✅ + - File Size: _______bytes + +- [ ] **Output Format Correct:** + ```bash + jq '.phase1_run | keys' test-versionset.json + ``` + - Keys present: runId ✅, jobId ✅, jobRunId ✅, correlationId ✅, idempotencyKey ✅ + +**Sign-off:** ___________ (SRE) + +--- + +### C.3 — PHASE-1_ACTIVATION_RUNBOOK.md Validation + +**Owner:** SRE / Platform Lead +**Blocking:** YES (execution procedure) + +- [ ] **Pre-flight Checklist Complete:** + - [ ] Migration 0032 deployed ✅ + - [ ] Host running in DEVELOPMENT mode ✅ + - [ ] PostgreSQL accessible via SSH tunnel ✅ + - [ ] Hangfire scheduler running ✅ + - [ ] Scripts available in ./scripts/ ✅ + +- [ ] **3-Step Procedure Verified:** + - [ ] STEP 1: FREEZE VersionSet (2 min) — ready to execute + - [ ] STEP 2: GENERATE identifiers (1 min) — ready to execute + - [ ] STEP 3: ENQUEUE Job 893 (1 min) — ready to execute + +- [ ] **Troubleshooting Matrix Present:** + - Common errors documented ✅ + - Recovery procedures clear ✅ + +- [ ] **Monitoring Instructions Clear:** + - Log tailing command: ✅ + - Grafana dashboard: ✅ + - Alert setup: ✅ + - Emergency rollback: ✅ + +**Sign-off:** ___________ (SRE Lead) + +--- + +## 📊 SECTION D: Data Quality & State Validation + +### D.1 — Model & Dataset State + +**Owner:** Data Governance / Quant Lead +**Blocking:** YES (ensures reproducibility) + +- [ ] **Model Card Complete:** + - [ ] Model ID: ___________ + - [ ] Model Name: ___________ + - [ ] Algorithm: ___________ + - [ ] Training Data Window: ___________ to ___________ + - [ ] Last Validated: ___________ + - [ ] Known Limitations: ___________ + +- [ ] **Dataset Manifest Complete:** + - [ ] Dataset ID: ___________ + - [ ] Dataset Name: ___________ + - [ ] Features: ___________ + - [ ] Data Quality Score: ___________ + - [ ] Last Refreshed: ___________ + - [ ] Completeness: _______% (target: ≥95%) + +- [ ] **Market Data Available:** + - [ ] KRX price data: 2024-01-02 to 2024-09-10 ✅ / ❌ (gaps: _________) + - [ ] Index data: KOSPI/KOSDAQ ✅ / ❌ + - [ ] Volume data: Available ✅ / ❌ + - [ ] Corporate actions: Splits/dividends integrated ✅ / ❌ + +- [ ] **No Data Quality Anomalies:** + ```sql + SELECT COUNT(*) FROM market_data WHERE price_close <= 0 OR volume = 0; + ``` + - Bad rows: _______ (target: 0) + +**Sign-off:** ___________ (Quant Lead) + +--- + +### D.2 — PIT (Point-in-Time) Query Validation + +**Owner:** Data Architect +**Blocking:** YES (ensures audit trail) + +- [ ] **Correlation IDs Trackable:** + - Sample query passes ✅ / ❌ + - `SELECT COUNT(*) FROM outbox WHERE correlation_id = ?` + - Result: _______rows + +- [ ] **Revision History Preserved:** + - Append-only tables confirmed ✅ + - No UPDATE/DELETE allowed ✅ + - Soft deletes only ✅ + +- [ ] **Published_at Timestamp Correct:** + ```sql + SELECT COUNT(*) FROM governance.model_version_registry + WHERE published_at > NOW(); + ``` + - Result: 0 rows (no future dates) ✅ / ❌ + +**Sign-off:** ___________ (Data Architect) + +--- + +## 📈 SECTION E: Monitoring & Observability Setup + +### E.1 — Logging Configured + +**Owner:** SRE / Observability Lead +**Blocking:** NO (but strongly recommended) + +- [ ] **Structured Logging Active:** + - Log file: `/app/kartsell/logs/phase-1-execution.log` ✅ + - Format: JSON with CorrelationId ✅ + - Retention: _______ days + +- [ ] **Serilog PII Redaction Active:** + - SSN redaction: ✅ + - Credit card redaction: ✅ + - API key redaction: ✅ + +- [ ] **Log Aggregation Ready:** + - ELK / Splunk / Datadog connected: ✅ / ❌ + - Search by CorrelationId functional: ✅ / ❌ + +**Sign-off:** ___________ (Observability Lead) + +--- + +### E.2 — Metrics & Alerting + +**Owner:** SRE / Observability +**Blocking:** NO (but recommended for incident response) + +- [ ] **Grafana Dashboard:** + - Phase 1 dashboard available: https://grafana.internal/d/phase1-shadow-run ✅ / ❌ + - Key metrics: Job status, trading days elapsed, data quality, cost simulation ✅ + - Real-time refresh: 5-minute interval ✅ + +- [ ] **Alert Thresholds Configured:** + - Job failure alert: ✅ + - Data quality anomaly (>5% bad rows): ✅ + - Processing latency >30min: ✅ + +- [ ] **On-Call Escalation Path:** + - Primary: ___________ + - Secondary: ___________ + - Escalation delay: _______ minutes + +**Sign-off:** ___________ (SRE Lead) + +--- + +## 🚀 SECTION F: Final Readiness Sign-offs + +### F.1 — Technical Readiness + +**All sections B, C, D must be PASS before proceeding** + +| Section | Status | Signed Off By | Date | +|---------|--------|---------------|------| +| B.1 Database | ✅ / ❌ | ___________ | _______ | +| B.2 Host App | ✅ / ❌ | ___________ | _______ | +| B.3 Frontend | ✅ / ❌ | ___________ | _______ | +| C.1 freeze-versionset | ✅ / ❌ | ___________ | _______ | +| C.2 generate-identifiers | ✅ / ❌ | ___________ | _______ | +| C.3 Runbook | ✅ / ❌ | ___________ | _______ | +| D.1 Data State | ✅ / ❌ | ___________ | _______ | +| D.2 PIT Queries | ✅ / ❌ | ___________ | _______ | + +--- + +### F.2 — Business Readiness + +**All sections A must be PASS before proceeding** + +| Gate | Status | Signed Off By | Date | +|------|--------|---------------|------| +| A.1 DEC-037 (Source/License) | ✅ / ❌ | ___________ | _______ | +| A.2 DEC-038 (Calendar/Owner) | ✅ / ❌ | ___________ | _______ | +| A.3 DEC-079 (Timezone/Correction) | ✅ / ❌ | ___________ | _______ | +| A.4 VersionSet Approved | ✅ / ❌ | ___________ | _______ | + +--- + +### F.3 — Final Go/No-Go Decision + +**OVERALL READINESS:** + +**GO CRITERIA:** +- ✅ All Section A gates APPROVED (governance) +- ✅ All Section B-D checks PASS (technical) +- ✅ Emergency rollback procedure validated +- ✅ On-call team briefed & ready + +**NO-GO CRITERIA:** +- ❌ Any governance approval pending (A.1-A.4) +- ❌ Technical blocker unresolved (B.1-D.2) +- ❌ Critical data quality issue (>10% bad rows) +- ❌ Insufficient monitoring coverage + +**FINAL DECISION:** + +``` +Phase 1 Execution: ☐ GO (proceed to activation) / ☐ NO-GO (defer) + +Date: ___________ +Approved By: ___________ (Platform Lead) +Emergency Contact: ___________ +Backup Lead: ___________ +``` + +**Launch Window:** ___________ to ___________ (UTC) +**Expected Completion:** 2026-10-27 to 2026-11-26 (50-90 days) +**Evidence Preservation:** Phase 1 logs → evidence/PHASE-1/logs/ + +--- + +## 📚 Supporting Documents + +- **Pre-flight Reference:** `docs/CURRENT/PHASE-1_PRODUCTION_PREFLIGHT_20260806.md` +- **Activation Procedure:** `docs/CURRENT/PHASE-1_ACTIVATION_RUNBOOK.md` +- **Evidence Plan:** `docs/CURRENT/PHASE-1_EXECUTION_EVIDENCE_PLAN.md` +- **Tech Decision Log:** `docs/DECISIONS/ADR-*.md` (authentication, data contract, etc.) + +--- + +**Co-Authored-By:** Claude Haiku 4.5