kjh2064
f680579134
Implements market data validation and normalization: ✅ GOV: Market data ingestion specification - KRX/OpenDart data sources - Daily scheduling (9:00 KST) - Quality SLAs (99.5% availability) ✅ DATA: PIT-compliant schema (4 tables) - daily_prices: OHLCV with versioning - indices: Market indices snapshots - companies: Master data - ingestion_jobs: Audit trail ✅ DOMAIN: Policy logic (12 tests, 12/12 PASS) - ValidatePrice: OHLC constraints, date checks - IsDuplicate: Prevent redundant entries - NormalizePrice: Rounding, filtering - ClassifyQualityIssue: Quality scoring (0-100) - ValidateBatch: Aggregate metrics AGENTS.md v16.0 compliance: ✅ Necessity: WBS Phase 2 Batch 2 ✅ Simplicity: Pure validation logic, no I/O ✅ Idempotency: By (symbol, trading_date) ✅ Safety: Immutable history with versioning ✅ Quality gates: Data quality scoring Phase 2 Progress: 1/4 Batches (VS-03 GOV+DATA+DOMAIN COMPLETE) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
137 lines
3.6 KiB
Markdown
137 lines
3.6 KiB
Markdown
# VS-03: Market Data Ingestion - Vertical Slice Specification
|
|
|
|
**Slice ID:** VS-03
|
|
**Batch:** 2 (depends on VS-00, VS-02, which are complete)
|
|
**Status:** 📋 SPECIFICATION
|
|
**Created:** 2026-08-05
|
|
|
|
---
|
|
|
|
## Executive Summary
|
|
|
|
Establish **Market Data Ingestion** system that pulls stock prices, indices, and financial data from external sources (KRX, OpenDart) and normalizes them for downstream signal generation.
|
|
|
|
**User Goal:** Automated, daily market data collection from Korean exchanges with minimal latency and maximum reliability.
|
|
|
|
**Non-Goal:**
|
|
- Real-time tick data (use Bloomberg/Refinitiv for that)
|
|
- Cryptocurrency data
|
|
- Forex integration
|
|
|
|
---
|
|
|
|
## Acceptance Criteria
|
|
|
|
### 1. Data Sources ✅
|
|
|
|
- **KRX OpenAPI:** Stock prices, indices, trading volumes
|
|
- **OpenDart API:** Financial statements, disclosure documents
|
|
- **Fallback:** Stub data (for testing/demo)
|
|
|
|
### 2. Data Model ✅
|
|
|
|
- **Market Daily (PIT):** Date, symbol, open, high, low, close, volume
|
|
- **Indices:** KRX 200, KOSPI, KOSDAQ snapshots
|
|
- **Company Info:** Sector, industry classification, listing status
|
|
|
|
### 3. Ingestion Pipeline ✅
|
|
|
|
- **Schedule:** Daily 9:00 KST (before market open)
|
|
- **Retry:** Exponential backoff (3 attempts)
|
|
- **Validation:** Schema conformance, duplicate detection
|
|
- **Idempotency:** By date + symbol (upsert)
|
|
- **Audit:** Correlation ID, row count, error logs
|
|
|
|
### 4. API Contracts ✅
|
|
|
|
**Endpoint: POST /api/market/ingest**
|
|
```
|
|
Request: { dataSource: "KRX|OpenDart", fromDate: "2026-01-01", toDate: "2026-12-31" }
|
|
Response: 202 Accepted { jobId, expectedRowCount, status }
|
|
```
|
|
|
|
**Endpoint: GET /api/market/ingest/{jobId}**
|
|
```
|
|
Response: 200 { status, rowsProcessed, rowsFailed, completedAt }
|
|
```
|
|
|
|
### 5. Data Quality Checks ✅
|
|
|
|
- No NULL prices (OHLCV)
|
|
- Volume >= 0
|
|
- High >= Low >= Open >= Close (within reason)
|
|
- No future dates
|
|
- Deduplication by (date, symbol)
|
|
|
|
---
|
|
|
|
## Failure Modes & Recovery
|
|
|
|
| Scenario | Expected | Recovery |
|
|
|----------|----------|----------|
|
|
| API timeout | 503, retry in 30s | Auto-retry, exponential backoff |
|
|
| Bad data format | DQ quarantine | Manual review, adjust parser |
|
|
| Duplicate rows | Idempotent upsert | No effect (already stored) |
|
|
| Partial ingestion | Rollback, log error | Retry entire day's batch |
|
|
|
|
---
|
|
|
|
## Performance SLAs
|
|
|
|
| Metric | Target |
|
|
|--------|--------|
|
|
| Daily ingestion latency | <60 seconds |
|
|
| Data freshness | <= 1 trading day old |
|
|
| Availability | 99.5% (allow 1 failure/week) |
|
|
| Max rows/day | 100,000 (stocks + indices) |
|
|
|
|
---
|
|
|
|
## Dependencies
|
|
|
|
### Inbound (Blocked By)
|
|
- ✅ **VS-00:** Platform foundation (complete)
|
|
- ✅ **VS-02:** Permission model (complete)
|
|
|
|
### Outbound (Unblocks)
|
|
- 🔄 **VS-04:** Trade Execution (uses VS-03's price data)
|
|
- 🔄 **VS-05:** Signal Generation (consumes VS-03 data)
|
|
- 🔄 **VS-06:** Portfolio Optimization (requires clean price history)
|
|
|
|
---
|
|
|
|
## Component Breakdown (7 items)
|
|
|
|
| Component | Status |
|
|
|-----------|--------|
|
|
| **GOV** | 📋 This spec |
|
|
| **DATA** | ⏳ Next: PIT schema |
|
|
| **DOMAIN** | ⏳ Data validation + normalization |
|
|
| **BE** | ⏳ Ingestion API |
|
|
| **ASYNC** | ⏳ Hangfire scheduler + event publishing |
|
|
| **FE** | ⏳ Ingestion status dashboard |
|
|
| **TESTOPS** | ⏳ Data quality tests |
|
|
|
|
**Total Duration:** ~6 hours (wall-clock 1 day)
|
|
|
|
---
|
|
|
|
## Branching Strategy
|
|
|
|
All work on `Phase-2-Batch-2` branch, squash to main.
|
|
|
|
**Commits:**
|
|
1. GOV + DATA (spec + contract)
|
|
2. DOMAIN (validation logic)
|
|
3. BE + ASYNC (API + scheduler)
|
|
4. FE + TESTOPS (dashboard + tests)
|
|
|
|
---
|
|
|
|
## Sign-Off
|
|
|
|
| Role | Status | Date |
|
|
|------|--------|------|
|
|
| Architect | ✅ Draft | 2026-08-05 |
|
|
| Data Quality | ⏳ Review | TBD |
|