5589a0432b3936b63f016f0c6ffa91cd461b765e
2 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
5589a0432b |
feat(collection): wire KIS collection end-to-end, add price-history pipeline (WBS QE-M0/M1/M2)
Critical re-review of the QuantEngine WBS evidence system found several
regressions of the "no fake gates" discipline established by M0, plus a
still-unwired M1 collection path. This closes 10 more WBS tasks
(QE-M1-01..06, QE-M2-01/02/04/05/06 — see spec/60_quant_engine_wbs.yaml)
with real, gate-verified evidence (18/34 total).
M1 — real KIS data now lands in PostgreSQL end-to-end:
- SchedulerService: load ticker universe from GatherTradingData.json instead
of a hardcoded array; fix a Hangfire scoped-service resolution bug.
- KisDataCollectionOrchestrator: restore logging on the lineage-event write
path (was a bare `catch {}` swallowing all failures silently); persist
daily OHLCV bars into quantengine.price_history_daily per run.
- Verified live: POST /api/collection/run -> Hangfire -> orchestrator ->
KIS mock API -> PostgreSQL, with Playwright DOM/API parity evidence.
M2 — historical price-history pipeline:
- CollectionRepository: SavePriceHistoryDailyAsync (idempotent upsert),
GetPriceHistorySummaryAsync (per-ticker aggregation) + a new
DateOnlyTypeHandler registered globally, since Dapper has no built-in
System.DateOnly support in either direction (write threw
NotSupportedException, read threw a constructor-mismatch
InvalidOperationException — found by exercising both paths live).
- tools/validate_price_history_integrity_v1.py: gap-freeness (vs KIS
trading calendar) + price-sanity gate over collected history.
- Admin Collection page: new "히스토리 현황" summary table +
GET /api/collection/history-summary, with Playwright evidence.
Governance/gate fixes:
- validate_market_time_series_schema_v1.py mislabeled its own output
"runtime_database_query": "DATA_GATED" despite never opening a DB
connection (pure file/regex check) — relabeled "check_scope":
"STATIC_STRUCTURAL_ONLY" and wired the node into the release DAG so it
isn't only reachable from ci.yml, matching every other validator.
Live-data authority for the same claim stays with QE-M2-01's pg_query
gate (spec/60), documented in spec/64.
- Fixed a WBS log_pattern check (QE-M1-06) that couldn't match its own
multi-line target; loosened two depends_on edges (QE-M1-05/06,
QE-M2-04/05) that encoded "needs X verified" when the real requirement
was only "needs X's code merged."
- Discovered and fixed admin-pages.spec.ts logging in with the wrong
seeded password (admin/admin instead of admin/quant123!, per CLAUDE.md)
— every test in that suite had been silently failing at the login step.
Deferred: QE-M2-03 (2-year backfill) — the KIS mock/VTS token endpoint
started returning 403 after the first successful call this session; looks
like a token-issuance rate limit or credential issue on KIS's side, not a
code defect. Backfilling at scale right now would just generate more 403s,
so left QE-M2-03 PENDING pending KIS account/console verification.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
||
|
|
66f75d9014 |
feat(core): 데이터 수집 파이프라인 Core 인터페이스 추가 및 Makefile 생성
Quant Engine CI/CD Pipeline / validate-core (push) Failing after 11s
WBS-9.3 - NULL Policy CI Gate / NULL Policy Validation (push) Failing after 8s
Quant Engine CI/CD Pipeline / validate-ui-and-storage (push) Has been skipped
Snapshot Admin Deployment / build-and-deploy (push) Failing after 1m3s
Deploy to Production / Build & Deploy to Production (push) Successful in 1m25s
**Stage 2 (Python → .NET) 진행:** - ITokenCache.cs: KIS API 토큰 캐싱 추상화 - 기존 Python sqlite3 로직 → PostgreSQL 기반으로 마이그레이션 - GetCachedTokenAsync(), SaveTokenAsync(), ClearExpiredTokensAsync() - IDataCollectionStore.cs: 데이터 수집 저장소 추상화 계약 - Python data_collection_store_v1.py 계약 매핑 - UpsertRun/Snapshot/Error, Fetch 메서드 - CollectionRunRecord, CollectionSnapshotRecord, CollectionErrorRecord DTO - CollectionDashboardStateRecord 대시보드 상태 모델 - ICollectionRepository.cs: 웹 API용 데이터 수집 저장소 인터페이스 - 높은 수준의 추상화 (Dapper + PostgreSQL) - SaveRun, UpdateRunStatus, SaveSnapshot, SaveError - GetRecentRuns, GetRunSnapshots, GetRunErrors, GetDashboardState - GetLatestSnapshotsForTicker **Stage 3 (Node.js → .NET) 완료:** - Makefile: npm scripts를 make 타겟으로 변환 - ops:prepare, ops:validate, ops:data-collect 등 주요 작업 - dotnet:build, dotnet:run, dotnet:watch 개발 명령어 - 단계: Python 도구 호출 유지 (Phase 2 완료까지) **다음 단계:** - CollectionRepository PostgreSQL 구현체 (Dapper) - TokenCache PostgreSQL 구현체 - DataCollectionStore PostgreSQL 구현체 (필요시) - Program.cs DI 등록 - Web API Collection 엔드포인트 추가 Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> |