352b440e8d
- Add prepare-release.yml: Manual release creation workflow * Builds code, generates appsettings.Production.json * Packages artifact (.tar.gz) * Creates git tag and Gitea Release with attached artifact - Refactor deploy-prod.yml: Release-based deployment workflow * Fetch Release stage: Query Gitea Releases, download artifact * Pre-Check stage: Verify SSH credentials and release integrity * Deploy stage: Upload, extract, symlink, restart service * Health Check stage: 5-point verification (HTTP, CSS, login, service, release) * Report stage: Final deployment status * Now triggered via workflow_dispatch with release version input * Removes on:push trigger (manual release selection required) - Update CLAUDE.md: * Document two-workflow architecture * Add release creation and deployment procedures * Update SSH key configuration with GITEA_TOKEN requirement * Clarify CI/CD-Only Deployment Mandate with release traceability * Add complete deployment flow documentation **Motivation**: - Separate build/release phase from deployment phase - Enable release tagging for version control and rollback - Reduce build time on re-deployments (use cached releases) - Improve deployment auditability via git tags and Gitea Releases - Match taxbaik-pattern release management strategy Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
651 lines
25 KiB
Markdown
651 lines
25 KiB
Markdown
# CLAUDE.md
|
|
|
|
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
|
|
## Project Overview
|
|
|
|
**QuantEngine v0.1** — A comprehensive quantitative analysis and data collection system for retirement asset portfolio management.
|
|
|
|
- **Architecture**: .NET 9 + C# (web UI + APIs), Python (legacy data collection/analysis)
|
|
- **Web UI**: Blazor Interactive WebAssembly (MudBlazor) + ASP.NET Core Web API (API-First)
|
|
- **Database**: PostgreSQL (Npgsql 8.0), single unified database
|
|
- **Data Source**: KIS Open API (quotations/ranking read-only), with fallbacks
|
|
- **Key Runtimes**: .NET 9, Python 3.9+, Node.js 16+
|
|
|
|
### Migration Phases Status (2026-07-11)
|
|
|
|
**Phase 1: Web UI Migration** ✅ 완료 (2026-07-11)
|
|
- **새로운 표준**: Razor Pages (Server-Rendered) + Cookie Authentication + Tabler UI
|
|
- **폐기 대상**: Blazor Interactive WebAssembly, MudBlazor, SmartAdmin
|
|
- **완료 기준 — Phase 1 Success Criteria**:
|
|
- ✅ Cookie 인증 구현 (AuthService + IpLockoutService + BCrypt)
|
|
- ✅ Razor Pages 렌더링 (Admin 레이아웃 + 3개 이상 기본 페이지)
|
|
- ✅ 공용 UI 컴포넌트 (4개 이상 shared partials)
|
|
- ✅ 보안: 백도어 제거, 무솔트 해시 마이그레이션, IP 잠금
|
|
- ✅ 빌드 성공: 0 errors, 0 warnings
|
|
- ✅ CLAUDE.md 업데이트 (UI 기준 + 인증 정책)
|
|
- **✅ 모든 기준 충족됨** (2026-07-11)
|
|
- **구현 완료**:
|
|
- ✅ Cookie 기반 인증 (AuthService + IpLockoutService)
|
|
- ✅ Razor Pages CRUD 레이아웃 (_AdminLayout.cshtml, shared partials)
|
|
- ✅ Admin 페이지: Dashboard, Collection, Users (기본 구조)
|
|
- ✅ 공용 UI 컴포넌트: _ValidationSummary, _Pagination, _StatusBadge, _EmptyState
|
|
- ✅ 보안 개선: BCrypt 해싱, IP 잠금, 하드코딩된 백도어 제거
|
|
- ✅ 빌드: 0 errors, 0 warnings (Newtonsoft.Json 보안 경고 제외)
|
|
- ✅ CLAUDE.md 완전 업데이트 (UI 기준, 인증, 상태 정의)
|
|
- **구현 미완료 (향후 작업)**:
|
|
- 🔄 Users 페이지: Create/Edit 폼 완성
|
|
- 🔄 Collection 페이지: 스냅샷/에러 조회 상세화
|
|
- 🔄 E2E 테스트: Playwright 스펙 업데이트
|
|
|
|
**Phase 2: KIS Data Collection Pipeline** ✅ 95% COMPLETE
|
|
- ✅ KIS API Client: Full implementation complete
|
|
- IKisApiClient interface (5 quotation methods)
|
|
- KisApiClient with real HTTP implementation + token caching
|
|
- All governance rules enforced (no trading APIs)
|
|
- Windows env var + registry fallback for credentials
|
|
- Build: 0 errors, 0 warnings
|
|
- ✅ PostgreSQL Infrastructure: Complete
|
|
- PostgresTokenCache (token management, 10-min skew)
|
|
- CollectionRepository (full CRUD + dashboard aggregations)
|
|
- Auto-creates kis_tokens, kis_collection_runs, kis_collection_snapshots, kis_collection_errors
|
|
- Dapper ORM + parameterized SQL (injection-proof)
|
|
- ✅ Web API Endpoints: Complete
|
|
- CollectionEndpoints (6 endpoints: state, runs, snapshots, errors, latest, start)
|
|
- ApiClient for Blazor consumption
|
|
- ✅ Blazor UI: Complete
|
|
- Collection.razor dashboard with real-time monitoring
|
|
- Summary cards, recent errors table, runs history
|
|
- Start/refresh functionality
|
|
- FluentSkeleton loading states
|
|
- 🔄 Pipeline Orchestration: Pending
|
|
- Python `kis_data_collection_v1.py` → .NET (data fetching + validation)
|
|
- Real KIS API data collection workflow integration
|
|
- E2E test: API → DB → UI validation
|
|
|
|
**Phase 3: Node.js→.NET CLI Tools** 📋 PLANNED
|
|
- Makefile created (npm → make mappings)
|
|
- np operations documented
|
|
|
|
**Phase 4: CI/CD Pipeline Hardening** ✅ 80% COMPLETE (2026-07-11)
|
|
- ✅ deploy-prod.yml (4-stage pipeline, 223 lines)
|
|
- Build → Pre-Deployment Check → Deploy → Post-Deployment Reporting
|
|
- SSH-based remote deployment (scp + ssh commands)
|
|
- Comprehensive health checks (10-retry with 3s intervals)
|
|
- Artifact management (.tar.gz)
|
|
- ✅ Workflow consolidation (2 active files)
|
|
- ci.yml: PR validation only (maintains 29 validators)
|
|
- deploy-prod.yml: Production deployment
|
|
- Deleted: merge-to-main.yml (non-functional), fast-validation.yml (redundant), archived/ directory
|
|
- ✅ SSH credentials: SSH_KEY registered in Gitea Secrets
|
|
- ⚠️ Gitea Actions limitation: Act runner ↔ Gitea network connectivity issues
|
|
- Workflow trigger (on:push) works ✓
|
|
- Job execution fails (network: dial tcp 172.18.0.2:3000 refused)
|
|
- **Workaround**: Manual SSH-based deployment (see "Production Deployment" below)
|
|
- 📚 Gitea API documentation: docs/GITEA_ACTIONS_API_GUIDE.md
|
|
|
|
**Phase 5: Admin UI & Deployment Optimization** ✅ COMPLETE (2026-07-11)
|
|
- ✅ Admin UI redesign (Tabler framework)
|
|
- Dashboard: stat cards, quick actions, system info
|
|
- Responsive sidebar navigation
|
|
- Professional layout (dark sidebar #2c3e50, white content)
|
|
- ✅ Build output: 0 errors, 0 warnings
|
|
- ✅ E2E tests: 8/8 passing (Playwright)
|
|
- ✅ Production deployment: Active since 2026-07-11 21:00:55 KST
|
|
- Commit: 30fb702
|
|
- HTTP 200 health check
|
|
- Service: active (running)
|
|
|
|
**Status Summary**:
|
|
- Python codebase: Operational (1,140 files)
|
|
- .NET 9 coverage: Core (✅), Infrastructure (✅), API (✅), Web UI (✅)
|
|
- Database: PostgreSQL fully migrated
|
|
- CI/CD: Manual SSH deployment (fully operational), Gitea Actions (limited by infrastructure)
|
|
- Release gates: Python gates remain authority until Phase 2 integration testing complete
|
|
|
|
## Deployment & Operations (Phase 4-5, 2026-07-11)
|
|
|
|
**Production Server**: Hetzner Cloud `178.104.200.7` (kjh2064@178.104.200.7)
|
|
|
|
Projects on server:
|
|
1. **TaxBaik** (홈페이지) — Nginx location `/taxbaik`
|
|
2. **QuantEngine** (데이터 수집/분석) — Nginx location `/quantengine`
|
|
|
|
### ⚠️ CRITICAL: CI/CD-Only Deployment Mandate
|
|
|
|
**Rule**: ALL production deployments MUST go through Gitea Actions CI/CD. Manual SSH deployments are **FORBIDDEN**.
|
|
|
|
**Why**:
|
|
- Automatic validation (build, health checks, version verification)
|
|
- Audit trail (all deployments logged in Gitea Actions)
|
|
- Consistent process (no manual errors)
|
|
- Rollback safety (deployment history retained)
|
|
- Release traceability (version control via git tags)
|
|
|
|
### Production Deployment Strategy (Release-Based)
|
|
|
|
**Architecture**: Two-Workflow System (Release Creation → Deployment)
|
|
|
|
#### Workflow 1: prepare-release.yml (Release Creation)
|
|
|
|
**Purpose**: Create a release with built artifact
|
|
|
|
**Trigger**: Manual (`workflow_dispatch`)
|
|
```bash
|
|
# Visit Gitea Actions and select prepare-release.yml
|
|
# Input version: v0.1.20260711 (or any semantic version)
|
|
```
|
|
|
|
**What it does**:
|
|
1. ✓ Build (restore, build, publish)
|
|
2. ✓ Generate `appsettings.Production.json`
|
|
3. ✓ Package artifact: `.tar.gz`
|
|
4. ✓ Create git tag: `v0.1.20260711`
|
|
5. ✓ Create Gitea Release with artifact attached
|
|
6. ✓ Notify: Release ready for deployment
|
|
|
|
**Output**: Gitea Release with downloadable artifact
|
|
|
|
#### Workflow 2: deploy-prod.yml (Deployment)
|
|
|
|
**Purpose**: Deploy a release to production
|
|
|
|
**Trigger**: Manual (`workflow_dispatch`)
|
|
```bash
|
|
# Visit Gitea Actions and select deploy-prod.yml
|
|
# Input release: v0.1.20260711 (optional — uses latest if empty)
|
|
```
|
|
|
|
**What it does**:
|
|
1. ✓ Fetch Release (from Gitea Releases)
|
|
2. ✓ Download artifact
|
|
3. ✓ Verify SSH credentials
|
|
4. ✓ Upload to production server
|
|
5. ✓ Extract and symlink
|
|
6. ✓ Restart service
|
|
7. ✓ 5-point health checks
|
|
8. ✓ Report deployment status
|
|
|
|
**Deployment Pipeline (5 Stages)**:
|
|
|
|
| Stage | Purpose | Timeout |
|
|
|-------|---------|---------|
|
|
| 1. Fetch Release | Query Gitea Releases, download artifact | 10min |
|
|
| 2. Pre-Check | Verify SSH keys, secrets, release | 5min |
|
|
| 3. Deploy | Upload, extract, symlink, restart service | 30min |
|
|
| 4. Health Check | 5-point verification (HTTP, CSS, login, service, release) | 10min |
|
|
| 5. Report | Final deployment status | Auto |
|
|
|
|
**Health Checks (Automatic)**:
|
|
- ✓ HTTP 200 on `/Account/Login`
|
|
- ✓ Login page content verification
|
|
- ✓ CSS file loads (`/css/admin.css`)
|
|
- ✓ Service status (systemctl active)
|
|
- ✓ Release verification (deployed release tag matches)
|
|
|
|
**Complete Deployment Flow**:
|
|
```
|
|
1. Code committed to main branch
|
|
2. Create release: prepare-release.yml workflow_dispatch (manual)
|
|
→ Builds code
|
|
→ Creates Gitea Release with artifact
|
|
→ Tags repository
|
|
3. Deploy release: deploy-prod.yml workflow_dispatch (manual)
|
|
→ Selects release version
|
|
→ Downloads artifact from Gitea Release
|
|
→ Deploys to production server
|
|
→ Runs health checks
|
|
→ Reports status
|
|
```
|
|
|
|
### Pre-Deployment Checklist
|
|
|
|
**Before creating a release**, verify:
|
|
1. ✅ Local build: `dotnet build src/dotnet/QuantEngine.Web/QuantEngine.Web.csproj -c Release` (0 errors, 0 warnings)
|
|
2. ✅ E2E tests pass: `npx playwright test`
|
|
3. ✅ Admin pages verified (200 status, no 500 errors)
|
|
4. ✅ All changes committed and pushed to main branch
|
|
5. ✅ No uncommitted changes: `git status`
|
|
|
|
### Release & Deployment Workflow
|
|
|
|
**Step 1: Create Release (prepare-release.yml)**
|
|
```bash
|
|
# Visit Gitea Actions
|
|
# https://gitea.taxbaik.com/kjh2064/QuantEngineByItz/actions
|
|
|
|
# Run prepare-release.yml workflow
|
|
# Input: version = v0.1.20260711
|
|
|
|
# Workflow will:
|
|
# - Build and publish
|
|
# - Package artifact
|
|
# - Create git tag
|
|
# - Create Gitea Release
|
|
# - Attach artifact
|
|
```
|
|
|
|
**Step 2: Deploy Release (deploy-prod.yml)**
|
|
```bash
|
|
# Visit Gitea Actions (same page)
|
|
# Run deploy-prod.yml workflow
|
|
# Input: release = v0.1.20260711 (leave empty for latest)
|
|
|
|
# Workflow will:
|
|
# - Download artifact from release
|
|
# - Deploy to production server
|
|
# - Run health checks
|
|
# - Report status
|
|
```
|
|
|
|
### SSH Key Configuration (Required)
|
|
|
|
**Setup (One-time)**:
|
|
1. Generate ED25519 key locally (or reuse existing):
|
|
```bash
|
|
ssh-keygen -t ed25519 -f ~/.ssh/quantengine_deploy -C "QuantEngine CI/CD"
|
|
```
|
|
|
|
2. Add public key to production server:
|
|
```bash
|
|
ssh-copy-id -i ~/.ssh/quantengine_deploy.pub kjh2064@178.104.200.7
|
|
```
|
|
|
|
3. Get private key in base64 format:
|
|
```bash
|
|
# macOS/Linux
|
|
base64 -w 0 ~/.ssh/quantengine_deploy > /tmp/key_b64.txt
|
|
cat /tmp/key_b64.txt | pbcopy
|
|
|
|
# Or Windows PowerShell
|
|
$key = Get-Content ~/.ssh/quantengine_deploy -Raw
|
|
[Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($key)) | Set-Clipboard
|
|
```
|
|
|
|
4. Configure in Gitea:
|
|
- URL: https://gitea.taxbaik.com/kjh2064/QuantEngineByItz/settings/secrets
|
|
- Add secret: `DEPLOY_SSH_KEY_B64` (base64-encoded private key)
|
|
- Or: `DEPLOY_SSH_KEY` (raw PEM format)
|
|
- Also add: `GITEA_TOKEN` (for release API access)
|
|
- Generate at: https://gitea.taxbaik.com/user/settings/applications
|
|
- Required permissions: `repo` + `read:actions`
|
|
|
|
### Deployment Monitoring
|
|
|
|
**During Deployment**:
|
|
- Watch live in Gitea Actions UI
|
|
- Jobs complete in order: Build → Pre-Check → Deploy → Health Check → Report
|
|
|
|
**After Deployment**:
|
|
```bash
|
|
# SSH into server
|
|
ssh kjh2064@178.104.200.7
|
|
|
|
# Check active deployment
|
|
readlink ~/quantengine_active
|
|
|
|
# View service status
|
|
systemctl status quantengine
|
|
|
|
# Tail live logs
|
|
journalctl -u quantengine -f
|
|
|
|
# Health check
|
|
curl -I http://127.0.0.1:5000/Account/Login
|
|
```
|
|
|
|
### Automatic Rollback (if health check fails)
|
|
|
|
If health check fails, deployment stops automatically:
|
|
1. Service restart may fail
|
|
2. Symlink update reverts to previous deployment
|
|
3. Gitea Actions marks deployment as FAILED
|
|
4. Logs include failure details
|
|
|
|
Manual rollback (if needed):
|
|
```bash
|
|
# List deployments
|
|
ls -lht ~/deployments/quantengine_*
|
|
|
|
# Revert symlink to previous version
|
|
ln -sfn /home/kjh2064/deployments/quantengine_YYYYMMDD_HHMMSS_COMMIT ~/quantengine_active
|
|
|
|
# Restart service
|
|
sudo systemctl restart quantengine
|
|
|
|
# Verify
|
|
curl http://127.0.0.1:5000/Account/Login
|
|
```
|
|
|
|
### Troubleshooting Deployment Failures
|
|
|
|
**Issue**: Build fails
|
|
- Check: `dotnet build` locally first
|
|
- Ensure: No compilation errors, 0 warnings
|
|
|
|
**Issue**: Health check timeout
|
|
- Check: Service logs: `journalctl -u quantengine -n 50`
|
|
- Check: Port 5000 listening: `ss -tlnp | grep 5000`
|
|
- Check: DB connectivity in appsettings.Production.json
|
|
|
|
**Issue**: SSH key error
|
|
- Verify: `DEPLOY_SSH_KEY_B64` or `DEPLOY_SSH_KEY` in Gitea Secrets
|
|
- Check: Public key added to `~/.ssh/authorized_keys` on server
|
|
- Test: `ssh -i ~/.ssh/key_file kjh2064@178.104.200.7 echo OK`
|
|
|
|
### Git Repository
|
|
|
|
**Gitea Server** (동일 호스트):
|
|
- **HTTP**: `https://gitea.taxbaik.com/kjh2064/QuantEngineByItz.git`
|
|
- **SSH**: `ssh://git@gitea.taxbaik.com:2222/kjh2064/QuantEngineByItz.git`
|
|
|
|
## UI Design Principles (2026-07-11 — Migrated to Razor Pages)
|
|
|
|
### Framework & Design System (NEW — 2026-07-11)
|
|
|
|
- **Primary Framework**: ASP.NET Core Razor Pages + Bootstrap 5 + Tabler UI
|
|
- **Design System**: Tabler (Bootstrap 5 기반), 밀집 레이아웃 + 전통 서버 렌더링
|
|
- **Render Mode**: **Server-side Razor Pages** — 모든 Admin UI는 서버에서 렌더링, Cookie 기반 인증 (API-First WASM 폐기)
|
|
- **Authentication**: Cookie Authentication (HttpOnly) + BCrypt password hashing + IP lockout (3 strikes, 15-min)
|
|
- **Deprecation**: **Blazor Interactive WebAssembly 폐기**, **MudBlazor 컴포넌트 폐기** (2026-07-11), **SmartAdmin 폐기**. 기존 WASM 코드는 `/QuantEngine.Web.Client` 폴더에 참고용으로만 보관 (`.sln`에서 제외)
|
|
|
|
### Component Development Rules (NEW)
|
|
|
|
1. **All Admin UI Development** (New + Refactored):
|
|
- Use **Razor Pages** (.cshtml + .cshtml.cs PageModel) exclusively for admin
|
|
- UI는 Repository/Service를 생성자 DI로 직접 호출 (API 홉 없음)
|
|
- Bootstrap 5 + Tabler UI CSS classes for styling
|
|
- **Form Validation**: DataAnnotations DTO + FluentValidation IValidator<T> 이중 검증
|
|
- HTML `<form>` + tag helpers (`asp-for`, `asp-action`, `asp-page`)
|
|
|
|
2. **Authentication & Authorization**:
|
|
- Cookie name: `QuantEngine.Admin.Auth` (HttpOnly, SameSite=Lax)
|
|
- Session duration: 12 hours (sliding expiration)
|
|
- Folder-level `[Authorize]` via `AuthorizeFolder("/Admin")` convention (per-page 반복 금지)
|
|
- Login: `/Account/Login` (Razor Page, NO WASM)
|
|
- Password: BCrypt-hashed (auto-migrates existing SHA-256 hashes on first login)
|
|
- IP Lockout: 3 failed attempts → 15-minute lockout
|
|
|
|
3. **Data & Form Patterns**:
|
|
- PageModel constructor: `public IndexModel(IWorkspaceRepository repo, ILogger<IndexModel> logger)`
|
|
- Form submission: `OnPostAsync()` / `OnPostDeleteAsync()` (multi-handler pattern)
|
|
- Validation failures: return `Page()` (re-render with ModelState errors)
|
|
- Pagination: `PaginationModel` record (Page, TotalPages, Func<int,string> BuildPageUrl)
|
|
- Empty states: `<PartialView name="_EmptyState" model="message" />`
|
|
|
|
4. **Component Mapping** (Bootstrap 5 + Tabler):
|
|
|
|
| UI Element | Component | Notes |
|
|
|-----------|-----------|-------|
|
|
| Button | `<button class="btn btn-primary">` | — |
|
|
| Input field | `<input asp-for="Property" class="form-control">` | tag helper |
|
|
| Dropdown | HTML `<select asp-for="Property">` | tag helper |
|
|
| Data grid | HTML `<table class="table">` | plain, no virtualization |
|
|
| Card | `<div class="card">` | Bootstrap card |
|
|
| Badge/Status | `<span class="badge bg-success">Active</span>` | Bootstrap badge |
|
|
| Layout container | `<div class="container-xl">` / `<div class="row">` | Bootstrap grid |
|
|
| Navigation | HTML navbar in `_AdminLayout.cshtml` | sidebar + topbar |
|
|
| Loading | N/A (server-rendered) | no loading states needed |
|
|
| Icons | Bootstrap Icons (`<i class="bi bi-*"></i>`) | CDN |
|
|
| Modal/Dialog | Bootstrap modal or inline `confirm()` | avoid unnecessary modals |
|
|
| Validation msg | `<span asp-validation-for="Property" class="d-block alert alert-danger mt-2">` | tag helper |
|
|
|
|
## Development Commands (Phase 1 + 2)
|
|
|
|
### Python / Node.js (Legacy & Release Gates)
|
|
```powershell
|
|
npm install
|
|
npm run ops:validate # Warn-only validation
|
|
npm run full-gate # Strict validation (all gates PASS)
|
|
npm run ops:data-collect # KIS collection (Python subprocess)
|
|
npm run ops:release # Full release DAG
|
|
```
|
|
|
|
### .NET (Primary - Phase 1 + 2)
|
|
```powershell
|
|
cd src/dotnet
|
|
dotnet restore
|
|
dotnet build # Debug build (0 errors, 0 warnings)
|
|
dotnet build -c Release # Release build
|
|
dotnet watch run --project QuantEngine.Web # Hot-reload (http://localhost:5265)
|
|
dotnet run --project QuantEngine.Web # Run API server
|
|
```
|
|
|
|
### Collection Pipeline Testing (Phase 2)
|
|
```powershell
|
|
# Set KIS credentials (sandbox account)
|
|
$env:KIS_APP_Key_TEST = "your_kis_test_key"
|
|
$env:KIS_APP_Secret_TEST = "your_kis_test_secret"
|
|
|
|
# Start web server (http://localhost:5265)
|
|
dotnet run --project QuantEngine.Web
|
|
|
|
# Verify Collection dashboard
|
|
# Navigate to http://localhost:5265/collection
|
|
# - Click "Start Collection" to trigger async run
|
|
# - Backend uses PostgreSQL-backed data storage
|
|
# - Dashboard updates with run status, snapshots, errors
|
|
|
|
# Verify API endpoints
|
|
curl http://localhost:5265/api/collection/state
|
|
curl http://localhost:5265/api/collection/runs
|
|
curl "http://localhost:5265/api/collection/latest/005930"
|
|
```
|
|
|
|
## API Endpoints (Phase 1 + 2)
|
|
|
|
### Workspace & History (Phase 1)
|
|
All endpoints prefixed with `/api/`:
|
|
|
|
| Route | Purpose |
|
|
|-------|---------|
|
|
| `GET /state` | Full UI state snapshot |
|
|
| `GET /tables` | Browsable tables list |
|
|
| `GET /table-rows` | Paginated rows |
|
|
| `POST /settings/save` | Save settings |
|
|
| `POST /account-snapshot/save` | Save snapshots |
|
|
| `POST /bootstrap` | Seed DB from JSON |
|
|
| `POST /account-snapshot/import-tsv` | Import TSV |
|
|
| `POST /autofix` | Auto-correct data |
|
|
|
|
### Collection Pipeline (Phase 2)
|
|
| Route | Purpose |
|
|
|-------|---------|
|
|
| `GET /collection/state` | Dashboard summary (runs, snapshots, errors) |
|
|
| `GET /collection/runs` | Recent collection runs (paginated) |
|
|
| `GET /collection/runs/{runId}/snapshots` | Snapshots from a run |
|
|
| `GET /collection/runs/{runId}/errors` | Errors from a run |
|
|
| `GET /collection/latest/{ticker}` | Latest snapshots for ticker |
|
|
| `POST /collection/run` | Start new collection run (async) |
|
|
|
|
### Collection Run Status Values
|
|
| Status | Meaning | UI Badge | Transitions |
|
|
|--------|---------|----------|------------|
|
|
| `running` | Collection in progress | <span class="badge bg-warning">진행 중</span> | → completed or failed |
|
|
| `completed` | Collection finished (may have errors) | <span class="badge bg-success">완료</span> | (final) |
|
|
| `failed` | Collection crashed/aborted | <span class="badge bg-danger">실패</span> | (final) |
|
|
| `pending` | Queued, not yet started | <span class="badge bg-secondary">대기 중</span> | → running |
|
|
|
|
### Collection Run Success Criteria
|
|
**Success** is defined as:
|
|
- Status = `completed` (not `failed`)
|
|
- `TotalSnapshots > 0` (at least one snapshot captured)
|
|
- `TotalErrors == 0` OR `TotalErrors < TotalSnapshots * 0.1` (error rate < 10%)
|
|
|
|
**Partial Success** (warning state):
|
|
- Status = `completed`
|
|
- `TotalSnapshots > 0` (some data captured)
|
|
- `TotalErrors > 0` (has errors, but not total loss)
|
|
|
|
**Failure**:
|
|
- Status = `failed` OR
|
|
- Status = `completed` + `TotalSnapshots == 0` (no data captured)
|
|
|
|
UI: `Pages/Admin/Collection/Index.cshtml` — status 값에 따라 배지 색상 결정, 향후 TotalSnapshots/TotalErrors로 상세 상태 표시
|
|
|
|
## KIS API Client Security (Phase 2)
|
|
|
|
### Governance Enforcement
|
|
- **Read-Only Mandate**: `AssertReadOnly(path, trId)` blocks all trading-related endpoints
|
|
- **Forbidden Paths**: `/trading/` substring triggers 🚫 immediate exception
|
|
- **Forbidden TR_IDs**: TTTC* / VTTC* prefixes (buy/sell order codes) blocked
|
|
- **Source**: `governance/rules/06_no_direct_api_trading.yaml`
|
|
|
|
### Token Management
|
|
- **ITokenCache** abstraction: PostgreSQL-backed in production
|
|
- **Credential Loading**:
|
|
- Windows environment variables: `KIS_APP_Key`, `KIS_APP_Secret`, `KIS_APP_Key_TEST`, `KIS_APP_Secret_TEST`
|
|
- Fallback: `HKCU\Environment` registry (Windows only)
|
|
- Account modes: `"real"` (prod) vs `"mock"` (sandbox)
|
|
|
|
### Quotation Methods (All Read-Only)
|
|
1. **GetCurrentPriceAsync** (FHKST01010100) — Current price inquiry
|
|
2. **GetAskingPrice10LevelAsync** (FHKST01010200) — Order book (10-level)
|
|
3. **GetDailyShortSaleAsync** (FHPST04830000) — Short-sale trends
|
|
4. **GetDailyItemChartPriceAsync** (FHKST03010100) — Daily OHLCV data
|
|
5. **GetInvestorTrendAsync** (FHKST01010900) — Investor sentiment (개인/외국인/기관)
|
|
|
|
## Local Development & Testing (2026-07-11)
|
|
|
|
### ⚠️ CRITICAL: SSH Tunnel for Remote Database Access
|
|
|
|
**Never use Docker locally.** Always use SSH tunneling to connect to remote PostgreSQL:
|
|
|
|
```powershell
|
|
# 1. Setup SSH tunnel (Terminal 1) — forwards local 5432 to remote DB
|
|
ssh -L 127.0.0.1:5432:localhost:5432 kjh2064@178.104.200.7 -N
|
|
|
|
# 2. Configure appsettings.Development.json
|
|
{
|
|
"ConnectionStrings": {
|
|
"DefaultConnection": "Host=127.0.0.1;Database=quantenginedb;Username=quantengine_app;Password=quantengine_app;Search Path=quantengine;"
|
|
}
|
|
}
|
|
|
|
# 3. Start service locally (Terminal 2)
|
|
cd src/dotnet
|
|
dotnet watch run --project QuantEngine.Web
|
|
|
|
# 4. Access locally
|
|
http://localhost:5265/Account/Login
|
|
```
|
|
|
|
### Mandatory Pre-Deployment Checklist
|
|
|
|
**EVERY code change must pass:**
|
|
|
|
1. ✅ **Local build (0 errors, 0 warnings)**
|
|
```powershell
|
|
dotnet build src/dotnet/QuantEngine.Web/QuantEngine.Web.csproj -c Release
|
|
```
|
|
|
|
2. ✅ **Local service startup with SSH tunnel**
|
|
- Service must start without DB connection errors
|
|
- DbUp migrations must succeed
|
|
|
|
3. ✅ **Login test (admin/quant123!)**
|
|
- `/Account/Login` must return 200
|
|
- Authentication flow must complete
|
|
- Cookie must be set
|
|
|
|
4. ✅ **All Admin pages must load**
|
|
- `/Admin/Dashboard` → 200 (NOT 500)
|
|
- `/Admin/Users` → 200 (NOT 500)
|
|
- `/Admin/Collection` → 200 (NOT 500)
|
|
- `/Admin/Monitoring` → 200 (NOT 500)
|
|
- `/Admin/Operations` → 200 (NOT 500)
|
|
- **No 500 errors in response body**
|
|
|
|
5. ✅ **Playwright E2E tests pass**
|
|
```powershell
|
|
npx playwright test tests/e2e/complete-admin-flow.spec.ts
|
|
```
|
|
|
|
### Deployment Gates
|
|
|
|
**NEVER deploy without:**
|
|
- ❌ Local testing complete
|
|
- ❌ All Admin pages verified (200 status, no 500 errors)
|
|
- ❌ E2E tests passing
|
|
- ❌ Authorization Policy configured (if changes made to Program.cs)
|
|
|
|
**Deployment failure is better than service outage.** Halt and investigate if local tests fail.
|
|
|
|
### Gitea Actions Workflows
|
|
|
|
**Active Workflows**:
|
|
1. **deploy-prod.yml** — Production deployment (on:push main, workflow_dispatch)
|
|
- 5 stages: Build → Pre-Check → Deploy → Health Check → Report
|
|
- Enhanced health checks (5-point verification)
|
|
- SSH-based deployment with artifact validation
|
|
|
|
2. **ci.yml** — PR validation (on:pull_request)
|
|
- 29 validators for code quality
|
|
- Runs on every pull request
|
|
|
|
**Accessing Gitea Actions**:
|
|
- Web UI: https://gitea.taxbaik.com/kjh2064/QuantEngineByItz/actions
|
|
- Runs API: https://gitea.taxbaik.com/api/v1/repos/kjh2064/QuantEngineByItz/actions/runs
|
|
|
|
### API Monitoring (CLI)
|
|
|
|
Monitor deployment status from command line:
|
|
|
|
```powershell
|
|
# Setup (one-time)
|
|
$env:GITEA_TOKEN_TAXBAIK = "your_gitea_personal_token"
|
|
|
|
# List recent deployment runs
|
|
$token = $env:GITEA_TOKEN_TAXBAIK
|
|
$response = Invoke-WebRequest `
|
|
-Uri "https://gitea.taxbaik.com/api/v1/repos/kjh2064/QuantEngineByItz/actions/runs?limit=5" `
|
|
-Headers @{ "Authorization" = "token $token" }
|
|
($response.Content | ConvertFrom-Json).workflow_runs | ForEach-Object {
|
|
Write-Host "Run #$($_.id): $($_.display_title) [$($_.conclusion)]"
|
|
}
|
|
|
|
# Get specific run details
|
|
$run_id = 1234 # Replace with actual run ID
|
|
$response = Invoke-WebRequest `
|
|
-Uri "https://gitea.taxbaik.com/api/v1/repos/kjh2064/QuantEngineByItz/actions/runs/$run_id" `
|
|
-Headers @{ "Authorization" = "token $token" }
|
|
$run = $response.Content | ConvertFrom-Json
|
|
Write-Host "Commit: $($run.head_sha)"
|
|
Write-Host "Status: $($run.status) / $($run.conclusion)"
|
|
```
|
|
|
|
See `docs/GITEA_ACTIONS_API_GUIDE.md` for complete API reference.
|
|
|
|
### Deployment Secrets Configuration
|
|
|
|
**Required Secrets** (Gitea Repository Settings → Secrets):
|
|
|
|
| Secret | Type | Purpose |
|
|
|--------|------|---------|
|
|
| `DEPLOY_SSH_KEY_B64` | Base64 (recommended) | ED25519 private key for SSH |
|
|
| `DEPLOY_SSH_KEY` | PEM (alternative) | Raw private key format |
|
|
| `DEPLOY_HOST` | Text | Production server IP (178.104.200.7) |
|
|
| `DEPLOY_USER` | Text | SSH username (kjh2064) |
|
|
|
|
**How to add secrets**:
|
|
1. Go to: https://gitea.taxbaik.com/kjh2064/QuantEngineByItz/settings/secrets
|
|
2. Click "Add Secret"
|
|
3. Name: `DEPLOY_SSH_KEY_B64`
|
|
4. Value: `base64 -w 0 ~/.ssh/deploy_key | pbcopy` (macOS) or `certutil -encode deploy_key deploy_key.b64` (Windows)
|
|
5. Save
|
|
|
|
---
|
|
|
|
## Notes for Contributors (2026-07-11)
|
|
|
|
- **SQL Safety**: Whitelist-only table access (enum switch in Repository)
|
|
- **KIS API**: Read-only quotations/ranking; no order/trade endpoints (governance enforced)
|
|
- **Admin UI**: Server-rendered Razor Pages only; no WASM, no APIs between PageModel and Repository
|
|
- **Authentication**: Cookie-based only; no Bearer tokens; password reset via API endpoints only (no UI form)
|
|
- **Password Policy**: BCrypt hashing (auto-upgrade from SHA-256 on login); IP lockout: 3 strikes = 15 min ban
|
|
- **Database**: PostgreSQL contract maintained; Dapper ORM with raw SQL (no EF)
|
|
- **Legacy Code**: `QuantEngine.Web.Client` folder kept for reference (not in .sln, not built)
|
|
- **Newtonsoft.Json**: Known high-severity vulnerability (GHSA-5crp-9r3c-p9vr); update or replace when feasible
|
|
- **Release Authority**: Python gates (`full-gate`, `prepare-upload-zip`) remain authority; .NET Admin fully operational as of 2026-07-11
|
|
- **Testing Requirement**: All code changes must pass local testing with SSH tunnel to remote DB before deployment (see "Local Development & Testing" above)
|