feat(collection): wire KIS collection end-to-end, add price-history pipeline (WBS QE-M0/M1/M2)
Critical re-review of the QuantEngine WBS evidence system found several
regressions of the "no fake gates" discipline established by M0, plus a
still-unwired M1 collection path. This closes 10 more WBS tasks
(QE-M1-01..06, QE-M2-01/02/04/05/06 — see spec/60_quant_engine_wbs.yaml)
with real, gate-verified evidence (18/34 total).
M1 — real KIS data now lands in PostgreSQL end-to-end:
- SchedulerService: load ticker universe from GatherTradingData.json instead
of a hardcoded array; fix a Hangfire scoped-service resolution bug.
- KisDataCollectionOrchestrator: restore logging on the lineage-event write
path (was a bare `catch {}` swallowing all failures silently); persist
daily OHLCV bars into quantengine.price_history_daily per run.
- Verified live: POST /api/collection/run -> Hangfire -> orchestrator ->
KIS mock API -> PostgreSQL, with Playwright DOM/API parity evidence.
M2 — historical price-history pipeline:
- CollectionRepository: SavePriceHistoryDailyAsync (idempotent upsert),
GetPriceHistorySummaryAsync (per-ticker aggregation) + a new
DateOnlyTypeHandler registered globally, since Dapper has no built-in
System.DateOnly support in either direction (write threw
NotSupportedException, read threw a constructor-mismatch
InvalidOperationException — found by exercising both paths live).
- tools/validate_price_history_integrity_v1.py: gap-freeness (vs KIS
trading calendar) + price-sanity gate over collected history.
- Admin Collection page: new "히스토리 현황" summary table +
GET /api/collection/history-summary, with Playwright evidence.
Governance/gate fixes:
- validate_market_time_series_schema_v1.py mislabeled its own output
"runtime_database_query": "DATA_GATED" despite never opening a DB
connection (pure file/regex check) — relabeled "check_scope":
"STATIC_STRUCTURAL_ONLY" and wired the node into the release DAG so it
isn't only reachable from ci.yml, matching every other validator.
Live-data authority for the same claim stays with QE-M2-01's pg_query
gate (spec/60), documented in spec/64.
- Fixed a WBS log_pattern check (QE-M1-06) that couldn't match its own
multi-line target; loosened two depends_on edges (QE-M1-05/06,
QE-M2-04/05) that encoded "needs X verified" when the real requirement
was only "needs X's code merged."
- Discovered and fixed admin-pages.spec.ts logging in with the wrong
seeded password (admin/admin instead of admin/quant123!, per CLAUDE.md)
— every test in that suite had been silently failing at the login step.
Deferred: QE-M2-03 (2-year backfill) — the KIS mock/VTS token endpoint
started returning 403 after the first successful call this session; looks
like a token-issuance rate limit or credential issue on KIS's side, not a
code defect. Backfilling at scale right now would just generate more 403s,
so left QE-M2-03 PENDING pending KIS account/console verification.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
+17
-34
@@ -859,21 +859,6 @@ dag:
|
||||
strict: false
|
||||
timeout_sec: 180
|
||||
warn_only: true
|
||||
build_schema_models:
|
||||
artifact_policy: keep
|
||||
cache_key: build_schema_models_v1
|
||||
command:
|
||||
- python
|
||||
- tools/generate_models_from_schema.py
|
||||
depends_on: []
|
||||
id: build_schema_models
|
||||
inputs:
|
||||
- tools/generate_models_from_schema.py
|
||||
- schemas/generated
|
||||
outputs:
|
||||
- Temp/schema_model_generation_v1.json
|
||||
strict: true
|
||||
timeout_sec: 30
|
||||
build_sector_flow_history_progress:
|
||||
artifact_policy: keep
|
||||
cache_key: build_sector_flow_history_progress_v1
|
||||
@@ -1112,7 +1097,6 @@ dag:
|
||||
- validate_low_capability_pipeline_todo_v2
|
||||
- validate_golden_coverage
|
||||
- validate_calibration
|
||||
- validate_schema_model
|
||||
- validate_gas_adapter
|
||||
- build_gas_bundle
|
||||
- validate_gas_adapter_contract
|
||||
@@ -1158,7 +1142,6 @@ dag:
|
||||
- build_artifact_chain_hash
|
||||
- build_report
|
||||
- build_bundle
|
||||
- build_schema_models
|
||||
- build_architecture_boundaries
|
||||
- validate_decision_trace
|
||||
- validate_factor_conflicts
|
||||
@@ -1881,6 +1864,22 @@ dag:
|
||||
- Temp/low_capability_pipeline_todo_validation_v2.json
|
||||
strict: true
|
||||
timeout_sec: 30
|
||||
validate_market_time_series_schema:
|
||||
artifact_policy: keep
|
||||
cache_key: validate_market_time_series_schema_v1
|
||||
command:
|
||||
- python
|
||||
- tools/validate_market_time_series_schema_v1.py
|
||||
depends_on: []
|
||||
id: validate_market_time_series_schema
|
||||
inputs:
|
||||
- tools/validate_market_time_series_schema_v1.py
|
||||
- src/dotnet/QuantEngine.Infrastructure/Migrations/V6__Add_Market_Time_Series.sql
|
||||
- docs/db/quantengine.dbml
|
||||
outputs:
|
||||
- Temp/market_time_series_schema_v1.json
|
||||
strict: true
|
||||
timeout_sec: 30
|
||||
validate_metric_alias_collision:
|
||||
artifact_policy: keep
|
||||
cache_key: validate_metric_alias_collision_v1
|
||||
@@ -2231,21 +2230,6 @@ dag:
|
||||
outputs: []
|
||||
strict: true
|
||||
timeout_sec: 30
|
||||
validate_schema_model:
|
||||
artifact_policy: keep
|
||||
cache_key: validate_schema_model_v1
|
||||
command:
|
||||
- python
|
||||
- tools/validate_schema_model_generation_v1.py
|
||||
depends_on:
|
||||
- build_schema_models
|
||||
id: validate_schema_model
|
||||
inputs:
|
||||
- tools/validate_schema_model_generation_v1.py
|
||||
- Temp/schema_model_generation_v1.json
|
||||
outputs: []
|
||||
strict: true
|
||||
timeout_sec: 30
|
||||
validate_sector_flow_history_progress:
|
||||
artifact_policy: keep
|
||||
cache_key: validate_sector_flow_history_progress_v1
|
||||
@@ -2323,7 +2307,6 @@ execution_order:
|
||||
- build_module_io_coverage
|
||||
- build_operating_cadence_signal
|
||||
- build_profit_giveback_ratchet
|
||||
- build_schema_models
|
||||
- build_shadow_ledger
|
||||
- convert_xlsx
|
||||
- validate_active_manifest
|
||||
@@ -2341,6 +2324,7 @@ execution_order:
|
||||
- validate_gas_adapter_contract
|
||||
- validate_golden_coverage
|
||||
- validate_live_activation
|
||||
- validate_market_time_series_schema
|
||||
- validate_metric_alias_collision
|
||||
- validate_packaged_refs
|
||||
- validate_property_invariants
|
||||
@@ -2374,7 +2358,6 @@ execution_order:
|
||||
- validate_no_replay_live_mix
|
||||
- validate_realized_performance
|
||||
- validate_rule_lifecycle
|
||||
- validate_schema_model
|
||||
wave_2:
|
||||
- build_smart_cash_recovery_v3
|
||||
- build_time_stop_forecast
|
||||
|
||||
Reference in New Issue
Block a user