diff --git a/CLAUDE.md b/CLAUDE.md index 64b5a2f8..9efd9999 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2,11 +2,100 @@ This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. +## High-Level Architecture Overview + +**QuantEngine v0.1** — A hybrid quantitative analysis and data collection system for retirement asset portfolio management, transitioning from Python (legacy) to .NET 9 (primary). + +### System Architecture Layers + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Web UI Layer (Razor Pages) │ +│ ├─ Admin Dashboard, Users, Collection, Monitoring, Ops │ +│ └─ Server-side rendering + Cookie auth (no WASM) │ +├─────────────────────────────────────────────────────────────┤ +│ .NET Web API Layer (FastEndpoints, Minimal APIs) │ +│ ├─ Collection API: /api/collection/{state,runs,snapshots} │ +│ ├─ Emergency password reset, auth endpoints │ +│ └─ Portfolio/settings endpoints │ +├─────────────────────────────────────────────────────────────┤ +│ Repository Layer (Dapper ORM, SQL-driven) │ +│ ├─ IWorkspaceRepository: workspace/settings CRUD │ +│ ├─ ICollectionRepository: KIS runs, snapshots, errors │ +│ ├─ IPortfolioRepository: asset/position data │ +│ └─ Whitelist-only table access (enum-based security) │ +├─────────────────────────────────────────────────────────────┤ +│ Infrastructure Layer (.NET) │ +│ ├─ DbUp Migrations: src/dotnet/QuantEngine.Infrastructure/ │ +│ ├─ PostgreSQL connection pooling, Npgsql 8.0 │ +│ ├─ ITokenCache (KIS token management) │ +│ ├─ PostProcessing: JSON normalization, schema validation │ +│ └─ Contracts: Parity, Provenance, Scheduling, Normalization│ +├─────────────────────────────────────────────────────────────┤ +│ KIS API Client Layer (.NET) │ +│ ├─ IKisApiClient: GetCurrentPrice, AskingPrice, DailyChart │ +│ ├─ Read-only mandate: quotations + ranking only │ +│ ├─ Governance enforcement: AssertReadOnly(path, trId) │ +│ ├─ Fallback chain: KIS → Naver → Yahoo → OpenDART │ +│ └─ Token caching: PostgreSQL-backed with 10-min skew │ +├─────────────────────────────────────────────────────────────┤ +│ Data Collection Pipeline (Python legacy + .NET new) │ +│ ├─ Python (ops:data-collect): SQLite accumulator │ +│ │ └─ tools/run_kis_data_collection_v1.py │ +│ ├─ .NET: Async collection runner, PostgreSQL persistence │ +│ ├─ Snapshot Admin (web UI): settings/account_snapshot edit │ +│ └─ Calibration Backlog: priority + change ledger │ +├─────────────────────────────────────────────────────────────┤ +│ Decision Engine & Reporting (.NET Canonical Renderer) │ +│ ├─ QuantEngine.Tools: Final decision packet + report render │ +│ ├─ JSON schema: schemas/operational_report.schema.json │ +│ ├─ Outputs: Temp/operational_report.{json,md} │ +│ └─ Contract validation via canonical JSON │ +├─────────────────────────────────────────────────────────────┤ +│ Database Layer (PostgreSQL) │ +│ ├─ Unified schema: quantenginedb / quantengine schema │ +│ ├─ Key tables: kis_tokens, kis_collection_runs/snapshots │ +│ │ workspaces, account_snapshots, settings │ +│ ├─ Auto-migrations via DbUp (V001.sql, V002.sql, ...) │ +│ └─ Documented in: docs/db/quantengine.dbml (DBML sync) │ +├─────────────────────────────────────────────────────────────┤ +│ Validation & Quality Gates (Python) │ +│ ├─ 30+ validators: specs, contracts, WBS, audits │ +│ ├─ CI pipeline: 9 parallel jobs (core + 8 independent) │ +│ └─ Release gates: full-gate (strict), ops:release (warn-ok) │ +├─────────────────────────────────────────────────────────────┤ +│ CI/CD Orchestration (Gitea Actions) │ +│ ├─ ci.yml: 10 jobs, core + contracts + reports + notify │ +│ ├─ prepare-release.yml: tag + artifact + checksum │ +│ ├─ deploy-prod.yml: SSH deploy + health checks │ +│ ├─ kis_data_collection.yml: cron (00:30 KST) validation │ +│ └─ Other workflows: snapshot admin, qualitative strategy │ +└─────────────────────────────────────────────────────────────┘ +``` + +### Key Design Decisions (SOLID + Domain-Driven) + +1. **Razor Pages (Server-Rendered)** — No WASM; Admin UI uses server-side rendering with cookie auth. Why? Simpler security, faster initial load, team familiarity. + +2. **Repository Pattern + Dapper ORM** — SQL-first, no EF. Why? Direct control over queries, performance, audit trail (raw SQL = explicit intent). + +3. **Read-Only KIS Governance** — `AssertReadOnly()` blocks all /trading/ paths + TTTC*/VTTC* TR_IDs. Why? Hard mandate: data collection only, no order placement. + +4. **PostgreSQL Single Source of Truth** — All collection runs, snapshots, tokens centralized. Why? Audit, reproducibility, real-time coordination. + +5. **Hybrid Python-to-.NET Transition** — Python handles legacy ops:data-collect; .NET does web, API, orchestration. Why? Gradual migration reduces risk; Python stays until .NET collection fully validated. + +6. **Contract-Driven Validation** — Parity, Provenance, Scheduler, Normalization contracts in CI. Why? Catch schema drift, data quality regressions before deployment. + +7. **Canonical JSON Renderer** (.NET Tools) — Single source of operational_report.json schema. Why? One contract, all tools emit to it (no format wars). + ## Project Overview **QuantEngine v0.1** — A comprehensive quantitative analysis and data collection system for retirement asset portfolio management. -- **Architecture**: .NET 9 + C# (web UI + APIs), Python (legacy data collection/analysis) +- **Architecture**: .NET 9 + C# (web UI + APIs, primary); Python (legacy data collection, gradual deprecation) +- **Critical Path**: KIS API → PostgreSQL → Admin Dashboard (read-only quotations only, no trading APIs) +- **Validation**: 30+ contract validators + 9 parallel CI jobs + manual release gates - **Web UI**: Blazor Interactive WebAssembly (MudBlazor) + ASP.NET Core Web API (API-First) - **Database**: PostgreSQL (Npgsql 8.0), single unified database - **Data Source**: KIS Open API (quotations/ranking read-only), with fallbacks @@ -676,21 +765,299 @@ See `docs/GITEA_ACTIONS_API_GUIDE.md` for complete API reference. --- -## Notes for Contributors (2026-07-11) +## Core Workflows & Common Scenarios -- **SQL Safety**: Whitelist-only table access (enum switch in Repository) -- **KIS API**: Read-only quotations/ranking; no order/trade endpoints (governance enforced) -- **Admin UI**: Server-rendered Razor Pages only; no WASM, no APIs between PageModel and Repository -- **Authentication**: Cookie-based only; no Bearer tokens; password reset via API endpoints only (no UI form) -- **Password Policy**: BCrypt hashing (auto-upgrade from SHA-256 on login); IP lockout: 3 strikes = 15 min ban -- **Database**: PostgreSQL contract maintained; Dapper ORM with raw SQL (no EF) -- **Legacy Code**: `QuantEngine.Web.Client` folder kept for reference (not in .sln, not built) -- **Newtonsoft.Json**: Known high-severity vulnerability (GHSA-5crp-9r3c-p9vr); update or replace when feasible -- **Release Authority**: Python gates (`full-gate`, `prepare-upload-zip`) remain authority; .NET Admin fully operational as of 2026-07-11 -- **Testing Requirement**: All code changes must pass local testing with SSH tunnel to remote DB before deployment (see "Local Development & Testing" above) -- **DBML Schema Sync (2026-07-12)**: DbUp 마이그레이션(`src/dotnet/QuantEngine.Infrastructure/Migrations/V*.sql`)으로 관리되는 모든 테이블은 **반드시 `docs/db/quantengine.dbml`에도 동기화**되어야 하며, 개발 시 스키마 참조는 이 DBML 파일을 기준으로 한다. 새 마이그레이션 추가 시 같은 커밋에서 DBML 갱신 필수. -- **Diagrams**: 상태전이/플로우차트/시퀀스 다이어그램은 Mermaid로 `docs/diagrams/`에 작성해 코딩 참조로 활용 (수집 파이프라인: `docs/diagrams/collection-pipeline.md`) -- **WBS Evidence Gate (2026-07-12)**: 퀀트 엔진 로드맵/WBS는 `spec/60_quant_engine_wbs.yaml`(기계 판정)로 관리. 작업 완료는 `npm run verify:task -- ` 게이트 PASS로만 인정 (BE=PG쿼리/로그/JSON, FE=Playwright+스크린샷). 전체 게이트: `npm run verify:wbs` +### Scenario 1: Day-to-Day Development (Code Change) + +1. **Make code changes** (C# Razor Pages / .NET API / Python tools) +2. **Local validation**: + ```powershell + dotnet build src/dotnet/QuantEngine.Web/QuantEngine.Web.csproj -c Release + dotnet test src/dotnet/QuantEngine.Core.Tests -c Release + ``` +3. **Test admin pages locally** (with SSH tunnel): + ```powershell + ssh -L 127.0.0.1:5432:localhost:5432 kjh2064@178.104.200.7 -N & + dotnet watch run --project QuantEngine.Web + # Verify: /Admin/Dashboard, /Admin/Users, /Admin/Collection, etc. all return 200 + ``` +4. **Commit & push**: Changes automatically trigger ci.yml + - Core validators run first (blocking others) + - Parallel validators (contracts, UI, DB, calibration) run independently + - notify-results summarizes all 9 jobs in PR comment + - Expected CI time: ~15-20min (was ~40min before 2026-07-24 refactor) + +### Scenario 2: Data Collection Setup (KIS API Validation) + +1. **Obtain KIS credentials** (real or mock account) +2. **Validate with mock account**: + ```powershell + $env:KIS_APP_Key_TEST="" + $env:KIS_APP_Secret_TEST="" + python tools/validate_kis_api_credentials_v1.py --account mock --ticker 005930 --dry-run + ``` +3. **Run real collection** (if approved): + ```powershell + $env:KIS_APP_Key="" + $env:KIS_APP_Secret="" + python tools/run_kis_data_collection_v1.py --input-json GatherTradingData.json --sqlite-db src/quant_engine/kis_data_collection.db --output-json Temp/kis_data_collection_v1.json --kis-account real + ``` +4. **Verify database**: + ```sql + SELECT COUNT(*) FROM kis_collection_runs; + SELECT COUNT(*) FROM kis_collection_snapshots; + ``` + +### Scenario 3: Admin Data Editing (Snapshot Admin Web UI) + +1. **Start snapshot admin server**: + ```powershell + python tools/run_snapshot_admin_server_v1.py --host 127.0.0.1 --port 8787 --db src/quant_engine/snapshot_admin.db --seed GatherTradingData.json + ``` +2. **Access web UI**: http://127.0.0.1:8787 +3. **Edit settings / account_snapshot** in browser (like Excel) +4. **Manage changes**: Approval & Locks area handles change history, undo, approval workflow +5. **Export for CI**: `/api/export` → JSON or "Export approval packet" button + +### Scenario 4: Release & Deployment (Multi-Stage) + +**Stage 1: Local validation** +```powershell +npm run ops:validate # Warn-only (allow some issues) +npm run full-gate # Strict (all gates PASS) +``` + +**Stage 2: Create release** (manual via Gitea Actions) +``` +→ Visit https://gitea.taxbaik.com/kjh2064/QuantEngineByItz/actions +→ Run "prepare-release.yml" workflow_dispatch + - Builds and publishes .NET + - Creates git tag (e.g., quant_20260724.0.abc1234) + - Generates Gitea Release with artifact + checksums + - Packages as .tar.gz +``` + +**Stage 3: Deploy** (manual, only after release exists) +``` +→ Run "deploy-prod.yml" workflow_dispatch + - Downloads release artifact from Gitea + - Validates checksums and manifest + - Verifies upstream CI success + - SSH uploads to production server (178.104.200.7) + - Extracts and symlinks + - Restarts systemd service + - 6-point health checks (HTTP, login page, CSS, service, release tag, DB auth) + - Reports final status +``` + +**Pre-deployment checklist** (MANDATORY): +- ✅ Local build: 0 errors, 0 warnings +- ✅ E2E tests pass: `npx playwright test` +- ✅ All admin pages tested locally (200 status, no 500) +- ✅ `git status` clean (no uncommitted changes) +- ✅ Commit pushed to main + +### Scenario 5: CI Workflow Debugging + +**Problem**: A specific validation fails in CI +1. Identify failing job from PR comment (notify-results output) +2. Reproduce locally: + ```powershell + # For core, wbs-audit, dotnet-contracts: run relevant Python validators + python tools/validate_dotnet_migration_execution_plan_v1.py + python tools/validate_dotnet_parity_contract_v1.py + # etc. + ``` +3. Fix and re-push (triggers ci.yml again) +4. Monitor in Gitea Actions dashboard + +**Problem**: Workflow syntax error +1. Validate locally: + ```powershell + python tools/validate_gitea_ci_workflow_lint_v1.py --workflow .gitea/workflows/ci.yml + ``` +2. Fix YAML and test again + +### Scenario 6: Database Schema Changes + +1. **Create migration**: `src/dotnet/QuantEngine.Infrastructure/Migrations/V003.sql` +2. **Update DBML**: `docs/db/quantengine.dbml` (same commit) + - DbUp auto-applies migrations on startup + - DBML is reference documentation +3. **Test locally** (with SSH tunnel): Migrations must apply cleanly +4. **Commit both** (SQL + DBML) together +5. **CI validates**: ci.yml applies migrations to test PostgreSQL service + +### When Things Break + +| Issue | Root Cause | Fix | +|-------|-----------|-----| +| Admin page returns 500 | Likely unhandled DB exception or auth issue | Check journalctl, verify ConnectionStrings in production env | +| KIS API fails with "not found" | Ticker doesn't exist in KIS | Use fallback (Naver → Yahoo → OpenDART) | +| Snapshot admin won't load | SQLite DB corrupted or missing | Delete and re-seed from GatherTradingData.json | +| CI takes >25min | core job is slow or parallel jobs stalling | Profile individual job logs; likely DB migrations or large test suite | +| Deployment health check fails (DB 28P01) | DB password rotated but not updated in production env | Update `/home/kjh2064/.config/quantengine.env` on server only (not in repo) | + +--- + +## Notes for Contributors (2026-07-11 — Updated 2026-07-24) + +### Code Standards + +- **SQL Safety**: Whitelist-only table access (enum switch in Repository). Never raw concatenation. +- **KIS API**: Read-only quotations/ranking; no order/trade endpoints. Enforced via `AssertReadOnly()` in client. +- **Admin UI**: Server-rendered Razor Pages only; no WASM. PageModel constructor: DI repos directly, no API hops. +- **Authentication**: Cookie-based (HttpOnly, SameSite=Lax). No Bearer tokens. Password reset via API endpoints only (no UI form). +- **Password Hashing**: BCrypt (auto-migrates from SHA-256 on login). IP lockout: 3 failed attempts → 15 min ban. +- **Database**: PostgreSQL as single source of truth. Dapper ORM + raw SQL (no Entity Framework). Auto-migrations via DbUp. +- **Error Handling**: Validate only at system boundaries (user input, external APIs). Trust internal guarantees. +- **Comments**: Minimal; only explain WHY for non-obvious constraints. Removed comments on refactor; code names explain WHAT. + +### Repository Structure Rules + +- **Legacy Code**: `QuantEngine.Web.Client` (WASM) kept for reference; excluded from .sln, not built. +- **DBML Sync (2026-07-12)**: All DbUp migrations (`V*.sql`) must sync with `docs/db/quantengine.dbml` in same commit. Future schema reads use DBML. +- **Diagrams**: Mermaid diagrams in `docs/diagrams/` for state machines, flows, sequences (e.g., collection-pipeline.md). +- **WBS Evidence (2026-07-12)**: Task completion = `npm run verify:task -- ` PASS (BE=DB queries/logs/JSON, FE=Playwright+screenshot). Full check: `npm run verify:wbs`. + +### Testing & Validation + +- **Local pre-deployment**: Build (0 errors), E2E tests, all admin pages return 200 (no 500), git status clean. +- **CI auto-validation**: 9 parallel jobs (core + 8 independent). Expected time: ~15-20min (was ~40min, parallelized 2026-07-24). + - core: CRITICAL tests + DB setup + - wbs-audit, dotnet-contracts, ui-storage, database-schema, calibration, operational-reporting, security-validation, workflow-lint: parallel + - notify-results: final PR summary +- **Release gates**: `npm run ops:validate` (warn-ok), `npm run full-gate` (strict), `npm run ops:release` (warn-ok + full DAG). + +### Deployment + +- **Mandatory checklist**: Local build, E2E pass, admin pages 200, git clean, pushed to main. +- **Release creation**: Manual workflow_dispatch → prepare-release.yml (tag + artifact). +- **Deployment**: Manual workflow_dispatch → deploy-prod.yml (SSH upload + health checks). +- **DB secrets**: Never hardcode `ConnectionStrings` in artifacts. Runtime injection via systemd EnvironmentFile (production only). +- **Health checks (6-point)**: HTTP 200, login page, CSS load, service active, release tag match, DB auth (no 28P01 errors). + +### Known Issues & Tech Debt + +- **Newtonsoft.Json**: High-severity vuln (GHSA-5crp-9r3c-p9vr); update or replace when feasible. +- **Python-to-.NET transition**: Legacy `ops:data-collect` remains until .NET collection validated in production (est. Q3 2026). +- **Gitea Actions limitation**: Act runner ↔ Gitea network issues (workaround: manual SSH deployment active). + +### Reliability & Data Quality + +- **Reproducibility**: All validation gates, WBS tasks, and CI runs must be reproducible. Use deterministic inputs (GatherTradingData.json seed). +- **Audit trail**: PostgreSQL is source of truth for all collection runs/snapshots. Raw SQL = explicit intent (no ORM magic). +- **Contract validation**: Parity, Provenance, Scheduler, Normalization contracts catch schema drift before deployment. +- **Canonical rendering**: .NET Tools (QuantEngine.Tools) is single source for operational_report.json schema. No format divergence. + +### When Making Changes + +1. **Change type determines scope**: + - Bug fix: No cleanup beyond fix itself. + - Feature: Add only what's requested; no hypothetical abstractions. + - Refactor: Break down large methods/classes; extract reusable patterns only if used 3+ times. + - Architecture change: Must update CLAUDE.md and CI contracts simultaneously. + +2. **Locality-first**: Prefer editing in place. Create new files only when genuinely new concept or isolation needed. + +3. **Testing locality**: Unit tests verify code correctness; UI testing (Playwright) verifies feature correctness. If UI can't be tested, say so. + +4. **Git commits**: New commits preferred over amend. Include co-author: `Claude Haiku 4.5 `. + +## Development Commands (Quick Reference) + +### Build & Restore +```powershell +# .NET +cd src/dotnet +dotnet restore +dotnet build -c Release +dotnet build src/dotnet/QuantEngine.Web/QuantEngine.Web.csproj -c Release + +# Python (no explicit build needed, but validate setup) +python3 --version +python3 -m pip install --quiet requests pyyaml openpyxl pytest psycopg[binary] +``` + +### Run Services Locally +```powershell +# 1. SSH tunnel for remote PostgreSQL (Terminal 1) +ssh -L 127.0.0.1:5432:localhost:5432 kjh2064@178.104.200.7 -N + +# 2. .NET service with hot-reload (Terminal 2) +cd src/dotnet +dotnet watch run --project QuantEngine.Web + +# 3. Access locally +http://localhost:5265/Account/Login # Admin UI +http://localhost:5265/api/collection/state # API +``` + +### Data Collection & Snapshot Admin +```powershell +# KIS data collection to SQLite +$env:KIS_APP_Key="" +$env:KIS_APP_Secret="" +python tools/run_kis_data_collection_v1.py --input-json GatherTradingData.json --sqlite-db src/quant_engine/kis_data_collection.db --output-json Temp/kis_data_collection_v1.json --kis-account real + +# Snapshot admin web UI (for editing settings/account_snapshot) +python tools/run_snapshot_admin_server_v1.py --host 127.0.0.1 --port 8787 --db src/quant_engine/snapshot_admin.db --seed GatherTradingData.json + +# With hot reload +npm run ops:snapshot-web-watch + +# Validate snapshot admin web UI +python tools/validate_snapshot_admin_web_v1.py +``` + +### Validation & Release Gates +```powershell +# Validation (warn-only mode) +npm run ops:validate + +# Strict validation (all gates PASS) +npm run full-gate + +# Release DAG (includes warn-only gates) +npm run ops:release + +# Package for distribution +npm run prepare-upload-zip + +# Calibration backlog rebuild +python tools/build_calibration_priority_v1.py +python tools/build_calibration_change_ledger_v4.py +python tools/build_calibration_review_report_v1.py +python tools/validate_calibration_change_ledger_v1.py +``` + +### CI/CD Monitoring +```powershell +# Monitor Gitea Actions +# Open: https://gitea.taxbaik.com/kjh2064/QuantEngineByItz/actions + +# Validate workflows locally +python3 tools/validate_gitea_ci_workflow_lint_v1.py + +# Check recent deployment runs +# UI: https://gitea.taxbaik.com/kjh2064/QuantEngineByItz/actions +``` + +### Testing +```powershell +# .NET unit tests (Core tests) +dotnet test src/dotnet/QuantEngine.Core.Tests/QuantEngine.Core.Tests.csproj -c Release + +# All .NET tests +dotnet test src/dotnet/QuantEngine.sln --configuration Release + +# Python unit tests (storage, KIS, snapshot admin) +python3 -m pytest tests/unit/test_*.py -v + +# E2E tests (Playwright) +npx playwright test +npx playwright test tests/e2e/complete-admin-flow.spec.ts +``` ## Gitea Actions Workflow Structure (2026-07-24)