feat(collection): wire KIS collection end-to-end, add price-history pipeline (WBS QE-M0/M1/M2)
Validators (Pushes and Pull Requests) / validate-ui-and-storage (push) Successful in 16s
Validators (Pushes and Pull Requests) / validate-core (push) Failing after 56s

Critical re-review of the QuantEngine WBS evidence system found several
regressions of the "no fake gates" discipline established by M0, plus a
still-unwired M1 collection path. This closes 10 more WBS tasks
(QE-M1-01..06, QE-M2-01/02/04/05/06 — see spec/60_quant_engine_wbs.yaml)
with real, gate-verified evidence (18/34 total).

M1 — real KIS data now lands in PostgreSQL end-to-end:
- SchedulerService: load ticker universe from GatherTradingData.json instead
  of a hardcoded array; fix a Hangfire scoped-service resolution bug.
- KisDataCollectionOrchestrator: restore logging on the lineage-event write
  path (was a bare `catch {}` swallowing all failures silently); persist
  daily OHLCV bars into quantengine.price_history_daily per run.
- Verified live: POST /api/collection/run -> Hangfire -> orchestrator ->
  KIS mock API -> PostgreSQL, with Playwright DOM/API parity evidence.

M2 — historical price-history pipeline:
- CollectionRepository: SavePriceHistoryDailyAsync (idempotent upsert),
  GetPriceHistorySummaryAsync (per-ticker aggregation) + a new
  DateOnlyTypeHandler registered globally, since Dapper has no built-in
  System.DateOnly support in either direction (write threw
  NotSupportedException, read threw a constructor-mismatch
  InvalidOperationException — found by exercising both paths live).
- tools/validate_price_history_integrity_v1.py: gap-freeness (vs KIS
  trading calendar) + price-sanity gate over collected history.
- Admin Collection page: new "히스토리 현황" summary table +
  GET /api/collection/history-summary, with Playwright evidence.

Governance/gate fixes:
- validate_market_time_series_schema_v1.py mislabeled its own output
  "runtime_database_query": "DATA_GATED" despite never opening a DB
  connection (pure file/regex check) — relabeled "check_scope":
  "STATIC_STRUCTURAL_ONLY" and wired the node into the release DAG so it
  isn't only reachable from ci.yml, matching every other validator.
  Live-data authority for the same claim stays with QE-M2-01's pg_query
  gate (spec/60), documented in spec/64.
- Fixed a WBS log_pattern check (QE-M1-06) that couldn't match its own
  multi-line target; loosened two depends_on edges (QE-M1-05/06,
  QE-M2-04/05) that encoded "needs X verified" when the real requirement
  was only "needs X's code merged."
- Discovered and fixed admin-pages.spec.ts logging in with the wrong
  seeded password (admin/admin instead of admin/quant123!, per CLAUDE.md)
  — every test in that suite had been silently failing at the login step.

Deferred: QE-M2-03 (2-year backfill) — the KIS mock/VTS token endpoint
started returning 403 after the first successful call this session; looks
like a token-issuance rate limit or credential issue on KIS's side, not a
code defect. Backfilling at scale right now would just generate more 403s,
so left QE-M2-03 PENDING pending KIS account/console verification.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-12 21:07:53 +09:00
parent f9a0ba3690
commit 5589a0432b
25 changed files with 3903 additions and 77 deletions
@@ -0,0 +1,29 @@
using System.Data;
using Dapper;
namespace QuantEngine.Infrastructure.Data;
/// <summary>
/// Dapper has no built-in type handler for System.DateOnly: writing a DateOnly parameter
/// throws NotSupportedException, and reading a DATE column into a DateOnly property throws
/// InvalidCastException. Register once at startup (SqlMapper.AddTypeHandler) to fix both
/// directions everywhere in the codebase.
/// </summary>
public class DateOnlyTypeHandler : SqlMapper.TypeHandler<DateOnly>
{
public override void SetValue(IDbDataParameter parameter, DateOnly value)
{
parameter.DbType = DbType.Date;
parameter.Value = value.ToDateTime(TimeOnly.MinValue);
}
public override DateOnly Parse(object value)
{
return value switch
{
DateOnly d => d,
DateTime dt => DateOnly.FromDateTime(dt),
_ => DateOnly.Parse(value.ToString()!)
};
}
}
@@ -156,6 +156,42 @@ namespace QuantEngine.Infrastructure.Repositories
)).ToList();
}
public async Task SavePriceHistoryDailyAsync(PriceHistoryDailyRecord record)
{
using var conn = _connectionFactory.CreateConnection();
await conn.ExecuteAsync(@"
INSERT INTO quantengine.price_history_daily (ticker, trade_date, open, high, low, close, volume, source, provenance)
VALUES (@Ticker, @TradeDate, @Open, @High, @Low, @Close, @Volume, @Source, @Provenance::jsonb)
ON CONFLICT (ticker, trade_date) DO NOTHING",
new
{
record.Ticker,
// Dapper has no built-in type handler for System.DateOnly (throws
// NotSupportedException) — pass as DateTime; the DATE column truncates the time part.
TradeDate = record.TradeDate.ToDateTime(TimeOnly.MinValue),
record.Open,
record.High,
record.Low,
record.Close,
record.Volume,
record.Source,
Provenance = record.ProvenanceJson ?? "{}"
}
);
}
public async Task<List<PriceHistorySummaryRecord>> GetPriceHistorySummaryAsync()
{
using var conn = _connectionFactory.CreateConnection();
return (await conn.QueryAsync<PriceHistorySummaryRecord>(@"
SELECT ticker AS Ticker, count(*)::int AS RowCount, min(trade_date) AS FirstDate, max(trade_date) AS LastDate
FROM quantengine.price_history_daily
GROUP BY ticker
ORDER BY ticker",
new { }
)).ToList();
}
private async Task EnsureTablesAsync()
{
using var conn = _connectionFactory.CreateConnection();