kjh2064/KArtSell.Aegis

Files

T

kjh2064 f680579134 feat: Complete VS-03 DOMAIN - Market Data Ingestion (Batch 2 - 3/7)

Implements market data validation and normalization:

✅ GOV: Market data ingestion specification
   - KRX/OpenDart data sources
   - Daily scheduling (9:00 KST)
   - Quality SLAs (99.5% availability)

✅ DATA: PIT-compliant schema (4 tables)
   - daily_prices: OHLCV with versioning
   - indices: Market indices snapshots
   - companies: Master data
   - ingestion_jobs: Audit trail

✅ DOMAIN: Policy logic (12 tests, 12/12 PASS)
   - ValidatePrice: OHLC constraints, date checks
   - IsDuplicate: Prevent redundant entries
   - NormalizePrice: Rounding, filtering
   - ClassifyQualityIssue: Quality scoring (0-100)
   - ValidateBatch: Aggregate metrics

AGENTS.md v16.0 compliance:
✅ Necessity: WBS Phase 2 Batch 2
✅ Simplicity: Pure validation logic, no I/O
✅ Idempotency: By (symbol, trading_date)
✅ Safety: Immutable history with versioning
✅ Quality gates: Data quality scoring

Phase 2 Progress: 1/4 Batches (VS-03 GOV+DATA+DOMAIN COMPLETE)

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

2026-08-05 21:16:29 +09:00

3.6 KiB

Raw Blame History

VS-03: Market Data Ingestion - Vertical Slice Specification

Slice ID: VS-03
Batch: 2 (depends on VS-00, VS-02, which are complete)
Status: 📋 SPECIFICATION
Created: 2026-08-05

Executive Summary

Establish Market Data Ingestion system that pulls stock prices, indices, and financial data from external sources (KRX, OpenDart) and normalizes them for downstream signal generation.

User Goal: Automated, daily market data collection from Korean exchanges with minimal latency and maximum reliability.

Non-Goal:

Real-time tick data (use Bloomberg/Refinitiv for that)
Cryptocurrency data
Forex integration

Acceptance Criteria

1. Data Sources ✅

KRX OpenAPI: Stock prices, indices, trading volumes
OpenDart API: Financial statements, disclosure documents
Fallback: Stub data (for testing/demo)

2. Data Model ✅

Market Daily (PIT): Date, symbol, open, high, low, close, volume
Indices: KRX 200, KOSPI, KOSDAQ snapshots
Company Info: Sector, industry classification, listing status

3. Ingestion Pipeline ✅

Schedule: Daily 9:00 KST (before market open)
Retry: Exponential backoff (3 attempts)
Validation: Schema conformance, duplicate detection
Idempotency: By date + symbol (upsert)
Audit: Correlation ID, row count, error logs

4. API Contracts ✅

Endpoint: POST /api/market/ingest

Request: { dataSource: "KRX|OpenDart", fromDate: "2026-01-01", toDate: "2026-12-31" }
Response: 202 Accepted { jobId, expectedRowCount, status }

Endpoint: GET /api/market/ingest/{jobId}

Response: 200 { status, rowsProcessed, rowsFailed, completedAt }

5. Data Quality Checks ✅

No NULL prices (OHLCV)
Volume >= 0
High >= Low >= Open >= Close (within reason)
No future dates
Deduplication by (date, symbol)

Failure Modes & Recovery

Scenario	Expected	Recovery
API timeout	503, retry in 30s	Auto-retry, exponential backoff
Bad data format	DQ quarantine	Manual review, adjust parser
Duplicate rows	Idempotent upsert	No effect (already stored)
Partial ingestion	Rollback, log error	Retry entire day's batch

Performance SLAs

Metric	Target
Daily ingestion latency	<60 seconds
Data freshness	<= 1 trading day old
Availability	99.5% (allow 1 failure/week)
Max rows/day	100,000 (stocks + indices)

Dependencies

Inbound (Blocked By)

✅ VS-00: Platform foundation (complete)
✅ VS-02: Permission model (complete)

Outbound (Unblocks)

🔄 VS-04: Trade Execution (uses VS-03's price data)
🔄 VS-05: Signal Generation (consumes VS-03 data)
🔄 VS-06: Portfolio Optimization (requires clean price history)

Component Breakdown (7 items)

Component	Status
GOV	📋 This spec
DATA	⏳ Next: PIT schema
DOMAIN	⏳ Data validation + normalization
BE	⏳ Ingestion API
ASYNC	⏳ Hangfire scheduler + event publishing
FE	⏳ Ingestion status dashboard
TESTOPS	⏳ Data quality tests

Total Duration: ~6 hours (wall-clock 1 day)

Branching Strategy

All work on Phase-2-Batch-2 branch, squash to main.

Commits:

GOV + DATA (spec + contract)
DOMAIN (validation logic)
BE + ASYNC (API + scheduler)
FE + TESTOPS (dashboard + tests)

Sign-Off

Role	Status	Date
Architect	✅ Draft	2026-08-05
Data Quality	⏳ Review	TBD

3.6 KiB Raw Blame History