feat: Complete VS-03 DOMAIN - Market Data Ingestion (Batch 2 - 3/7)
Implements market data validation and normalization: ✅ GOV: Market data ingestion specification - KRX/OpenDart data sources - Daily scheduling (9:00 KST) - Quality SLAs (99.5% availability) ✅ DATA: PIT-compliant schema (4 tables) - daily_prices: OHLCV with versioning - indices: Market indices snapshots - companies: Master data - ingestion_jobs: Audit trail ✅ DOMAIN: Policy logic (12 tests, 12/12 PASS) - ValidatePrice: OHLC constraints, date checks - IsDuplicate: Prevent redundant entries - NormalizePrice: Rounding, filtering - ClassifyQualityIssue: Quality scoring (0-100) - ValidateBatch: Aggregate metrics AGENTS.md v16.0 compliance: ✅ Necessity: WBS Phase 2 Batch 2 ✅ Simplicity: Pure validation logic, no I/O ✅ Idempotency: By (symbol, trading_date) ✅ Safety: Immutable history with versioning ✅ Quality gates: Data quality scoring Phase 2 Progress: 1/4 Batches (VS-03 GOV+DATA+DOMAIN COMPLETE) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,136 @@
|
||||
# VS-03: Market Data Ingestion - Vertical Slice Specification
|
||||
|
||||
**Slice ID:** VS-03
|
||||
**Batch:** 2 (depends on VS-00, VS-02, which are complete)
|
||||
**Status:** 📋 SPECIFICATION
|
||||
**Created:** 2026-08-05
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
Establish **Market Data Ingestion** system that pulls stock prices, indices, and financial data from external sources (KRX, OpenDart) and normalizes them for downstream signal generation.
|
||||
|
||||
**User Goal:** Automated, daily market data collection from Korean exchanges with minimal latency and maximum reliability.
|
||||
|
||||
**Non-Goal:**
|
||||
- Real-time tick data (use Bloomberg/Refinitiv for that)
|
||||
- Cryptocurrency data
|
||||
- Forex integration
|
||||
|
||||
---
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
### 1. Data Sources ✅
|
||||
|
||||
- **KRX OpenAPI:** Stock prices, indices, trading volumes
|
||||
- **OpenDart API:** Financial statements, disclosure documents
|
||||
- **Fallback:** Stub data (for testing/demo)
|
||||
|
||||
### 2. Data Model ✅
|
||||
|
||||
- **Market Daily (PIT):** Date, symbol, open, high, low, close, volume
|
||||
- **Indices:** KRX 200, KOSPI, KOSDAQ snapshots
|
||||
- **Company Info:** Sector, industry classification, listing status
|
||||
|
||||
### 3. Ingestion Pipeline ✅
|
||||
|
||||
- **Schedule:** Daily 9:00 KST (before market open)
|
||||
- **Retry:** Exponential backoff (3 attempts)
|
||||
- **Validation:** Schema conformance, duplicate detection
|
||||
- **Idempotency:** By date + symbol (upsert)
|
||||
- **Audit:** Correlation ID, row count, error logs
|
||||
|
||||
### 4. API Contracts ✅
|
||||
|
||||
**Endpoint: POST /api/market/ingest**
|
||||
```
|
||||
Request: { dataSource: "KRX|OpenDart", fromDate: "2026-01-01", toDate: "2026-12-31" }
|
||||
Response: 202 Accepted { jobId, expectedRowCount, status }
|
||||
```
|
||||
|
||||
**Endpoint: GET /api/market/ingest/{jobId}**
|
||||
```
|
||||
Response: 200 { status, rowsProcessed, rowsFailed, completedAt }
|
||||
```
|
||||
|
||||
### 5. Data Quality Checks ✅
|
||||
|
||||
- No NULL prices (OHLCV)
|
||||
- Volume >= 0
|
||||
- High >= Low >= Open >= Close (within reason)
|
||||
- No future dates
|
||||
- Deduplication by (date, symbol)
|
||||
|
||||
---
|
||||
|
||||
## Failure Modes & Recovery
|
||||
|
||||
| Scenario | Expected | Recovery |
|
||||
|----------|----------|----------|
|
||||
| API timeout | 503, retry in 30s | Auto-retry, exponential backoff |
|
||||
| Bad data format | DQ quarantine | Manual review, adjust parser |
|
||||
| Duplicate rows | Idempotent upsert | No effect (already stored) |
|
||||
| Partial ingestion | Rollback, log error | Retry entire day's batch |
|
||||
|
||||
---
|
||||
|
||||
## Performance SLAs
|
||||
|
||||
| Metric | Target |
|
||||
|--------|--------|
|
||||
| Daily ingestion latency | <60 seconds |
|
||||
| Data freshness | <= 1 trading day old |
|
||||
| Availability | 99.5% (allow 1 failure/week) |
|
||||
| Max rows/day | 100,000 (stocks + indices) |
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
|
||||
### Inbound (Blocked By)
|
||||
- ✅ **VS-00:** Platform foundation (complete)
|
||||
- ✅ **VS-02:** Permission model (complete)
|
||||
|
||||
### Outbound (Unblocks)
|
||||
- 🔄 **VS-04:** Trade Execution (uses VS-03's price data)
|
||||
- 🔄 **VS-05:** Signal Generation (consumes VS-03 data)
|
||||
- 🔄 **VS-06:** Portfolio Optimization (requires clean price history)
|
||||
|
||||
---
|
||||
|
||||
## Component Breakdown (7 items)
|
||||
|
||||
| Component | Status |
|
||||
|-----------|--------|
|
||||
| **GOV** | 📋 This spec |
|
||||
| **DATA** | ⏳ Next: PIT schema |
|
||||
| **DOMAIN** | ⏳ Data validation + normalization |
|
||||
| **BE** | ⏳ Ingestion API |
|
||||
| **ASYNC** | ⏳ Hangfire scheduler + event publishing |
|
||||
| **FE** | ⏳ Ingestion status dashboard |
|
||||
| **TESTOPS** | ⏳ Data quality tests |
|
||||
|
||||
**Total Duration:** ~6 hours (wall-clock 1 day)
|
||||
|
||||
---
|
||||
|
||||
## Branching Strategy
|
||||
|
||||
All work on `Phase-2-Batch-2` branch, squash to main.
|
||||
|
||||
**Commits:**
|
||||
1. GOV + DATA (spec + contract)
|
||||
2. DOMAIN (validation logic)
|
||||
3. BE + ASYNC (API + scheduler)
|
||||
4. FE + TESTOPS (dashboard + tests)
|
||||
|
||||
---
|
||||
|
||||
## Sign-Off
|
||||
|
||||
| Role | Status | Date |
|
||||
|------|--------|------|
|
||||
| Architect | ✅ Draft | 2026-08-05 |
|
||||
| Data Quality | ⏳ Review | TBD |
|
||||
@@ -0,0 +1,260 @@
|
||||
# VS-03: Market Data Ingestion - Data Contract
|
||||
|
||||
**Slice ID:** VS-03
|
||||
**Phase:** Data Layer (write model)
|
||||
**Status:** Specification Ready
|
||||
|
||||
---
|
||||
|
||||
## Write Model (Normalized, 3NF)
|
||||
|
||||
### Table: `market_data.daily_prices` (Core)
|
||||
|
||||
```sql
|
||||
CREATE TABLE market_data.daily_prices (
|
||||
-- Identity
|
||||
price_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
symbol VARCHAR(20) NOT NULL,
|
||||
trading_date DATE NOT NULL,
|
||||
|
||||
-- OHLCV
|
||||
open_price DECIMAL(10, 2) NOT NULL CHECK (open_price > 0),
|
||||
high_price DECIMAL(10, 2) NOT NULL CHECK (high_price > 0),
|
||||
low_price DECIMAL(10, 2) NOT NULL CHECK (low_price > 0),
|
||||
close_price DECIMAL(10, 2) NOT NULL CHECK (close_price > 0),
|
||||
adjusted_close DECIMAL(10, 2),
|
||||
volume BIGINT NOT NULL CHECK (volume >= 0),
|
||||
|
||||
-- PIT (Point-in-Time) Compliance
|
||||
published_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
revision INT NOT NULL DEFAULT 1,
|
||||
|
||||
-- Audit
|
||||
data_source VARCHAR(50) NOT NULL, -- 'KRX', 'OpenDart', 'Stub'
|
||||
ingestion_job_id UUID,
|
||||
correlation_id UUID,
|
||||
|
||||
-- Soft-delete (never delete, only version)
|
||||
removed_at TIMESTAMP,
|
||||
|
||||
CONSTRAINT unique_daily_price UNIQUE (symbol, trading_date, revision),
|
||||
CONSTRAINT valid_prices CHECK (low_price <= open_price AND open_price <= high_price)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_daily_prices_symbol_date ON market_data.daily_prices(symbol, trading_date DESC);
|
||||
CREATE INDEX idx_daily_prices_published ON market_data.daily_prices(published_at DESC);
|
||||
```
|
||||
|
||||
### Table: `market_data.indices` (Supplementary)
|
||||
|
||||
```sql
|
||||
CREATE TABLE market_data.indices (
|
||||
-- Identity
|
||||
index_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
index_code VARCHAR(20) NOT NULL, -- 'KOSPI', 'KRX200', 'KOSDAQ'
|
||||
trading_date DATE NOT NULL,
|
||||
|
||||
-- OHLCV
|
||||
open_value DECIMAL(10, 2) NOT NULL,
|
||||
high_value DECIMAL(10, 2) NOT NULL,
|
||||
low_value DECIMAL(10, 2) NOT NULL,
|
||||
close_value DECIMAL(10, 2) NOT NULL,
|
||||
change_percent DECIMAL(5, 2),
|
||||
volume BIGINT,
|
||||
|
||||
-- PIT
|
||||
published_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
revision INT NOT NULL DEFAULT 1,
|
||||
|
||||
-- Audit
|
||||
data_source VARCHAR(50) NOT NULL,
|
||||
correlation_id UUID,
|
||||
|
||||
removed_at TIMESTAMP,
|
||||
|
||||
CONSTRAINT unique_index UNIQUE (index_code, trading_date, revision)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_indices_code_date ON market_data.indices(index_code, trading_date DESC);
|
||||
```
|
||||
|
||||
### Table: `market_data.companies` (Master)
|
||||
|
||||
```sql
|
||||
CREATE TABLE market_data.companies (
|
||||
-- Identity
|
||||
company_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
symbol VARCHAR(20) NOT NULL UNIQUE,
|
||||
|
||||
-- Master Data
|
||||
korean_name VARCHAR(100) NOT NULL,
|
||||
english_name VARCHAR(100),
|
||||
sector VARCHAR(50),
|
||||
industry VARCHAR(100),
|
||||
listing_date DATE,
|
||||
|
||||
-- Status
|
||||
listing_status VARCHAR(20) NOT NULL DEFAULT 'Active', -- Active, Suspended, Delisted
|
||||
market VARCHAR(20) NOT NULL, -- 'KOSPI', 'KOSDAQ', 'KONEX'
|
||||
|
||||
-- PIT
|
||||
published_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
revision INT NOT NULL DEFAULT 1,
|
||||
removed_at TIMESTAMP,
|
||||
|
||||
-- Audit
|
||||
last_updated TIMESTAMP,
|
||||
data_source VARCHAR(50),
|
||||
|
||||
CONSTRAINT unique_company UNIQUE (symbol, revision)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_companies_symbol ON market_data.companies(symbol);
|
||||
```
|
||||
|
||||
### Table: `market_data.ingestion_jobs` (Audit)
|
||||
|
||||
```sql
|
||||
CREATE TABLE market_data.ingestion_jobs (
|
||||
-- Identity
|
||||
job_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
job_run_id UUID NOT NULL, -- Hangfire RunId
|
||||
|
||||
-- Input
|
||||
data_source VARCHAR(50) NOT NULL,
|
||||
from_date DATE NOT NULL,
|
||||
to_date DATE NOT NULL,
|
||||
|
||||
-- Progress
|
||||
status VARCHAR(50) NOT NULL DEFAULT 'Queued', -- Queued, Running, Completed, Failed
|
||||
rows_processed INT DEFAULT 0,
|
||||
rows_failed INT DEFAULT 0,
|
||||
rows_skipped INT DEFAULT 0,
|
||||
|
||||
-- Timing
|
||||
started_at TIMESTAMP,
|
||||
completed_at TIMESTAMP,
|
||||
duration_seconds INT,
|
||||
|
||||
-- Error Handling
|
||||
last_error_message TEXT,
|
||||
retry_count INT DEFAULT 0,
|
||||
|
||||
-- Traceability
|
||||
correlation_id UUID NOT NULL,
|
||||
triggered_by VARCHAR(100), -- 'Scheduler', 'Manual', 'API'
|
||||
|
||||
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
|
||||
CONSTRAINT unique_job_run UNIQUE (job_run_id)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_ingestion_jobs_status ON market_data.ingestion_jobs(status);
|
||||
CREATE INDEX idx_ingestion_jobs_dates ON market_data.ingestion_jobs(from_date, to_date);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Read Model (Denormalized Projections)
|
||||
|
||||
### View: `market_data.latest_prices` (Cache)
|
||||
|
||||
```sql
|
||||
CREATE VIEW market_data.latest_prices AS
|
||||
SELECT DISTINCT ON (symbol)
|
||||
symbol,
|
||||
trading_date,
|
||||
close_price,
|
||||
volume,
|
||||
published_at
|
||||
FROM market_data.daily_prices
|
||||
WHERE removed_at IS NULL
|
||||
AND published_at <= CURRENT_TIMESTAMP
|
||||
ORDER BY symbol, trading_date DESC;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## PIT (Point-in-Time) Query Pattern
|
||||
|
||||
```sql
|
||||
-- Fetch prices as of 2026-06-30
|
||||
SELECT symbol, open_price, close_price, volume
|
||||
FROM market_data.daily_prices
|
||||
WHERE trading_date <= '2026-06-30'
|
||||
AND published_at <= '2026-06-30'::timestamp
|
||||
AND removed_at IS NULL
|
||||
ORDER BY symbol, trading_date DESC
|
||||
LIMIT 1 PER symbol;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
1. **0033_market_data_schema.sql**
|
||||
- Create market_data schema
|
||||
- Define daily_prices, indices, companies, ingestion_jobs tables
|
||||
- Add PK, FK, constraints
|
||||
|
||||
2. **0034_market_data_indexes.sql**
|
||||
- Create performance indexes
|
||||
- Partition by year (optional, if 10M+ rows/year)
|
||||
|
||||
3. **0035_market_data_audit.sql**
|
||||
- Create audit trigger (log all writes)
|
||||
- Set up row-level security (market access control)
|
||||
|
||||
---
|
||||
|
||||
## Data Dictionary
|
||||
|
||||
| Column | Type | Purpose |
|
||||
|--------|------|---------|
|
||||
| symbol | VARCHAR(20) | Stock ticker (e.g., '005930' for Samsung) |
|
||||
| trading_date | DATE | Market trading date (YYYY-MM-DD) |
|
||||
| open_price | DECIMAL(10,2) | Opening price |
|
||||
| close_price | DECIMAL(10,2) | Closing price |
|
||||
| volume | BIGINT | Trading volume (shares) |
|
||||
| published_at | TIMESTAMP | PIT anchor (when row became "true") |
|
||||
| revision | INT | Version number (immutable history) |
|
||||
| removed_at | TIMESTAMP | Soft-delete marker (NULL = active) |
|
||||
| correlation_id | UUID | Trace this data ingestion back to job |
|
||||
|
||||
---
|
||||
|
||||
## Idempotency & Upsert Strategy
|
||||
|
||||
**Idempotency Key:** `(symbol, trading_date)`
|
||||
|
||||
**Upsert SQL:**
|
||||
```sql
|
||||
INSERT INTO market_data.daily_prices (symbol, trading_date, open_price, high_price, low_price, close_price, volume, published_at, revision, correlation_id, data_source)
|
||||
VALUES (@symbol, @date, @open, @high, @low, @close, @volume, CURRENT_TIMESTAMP, 1, @corrId, @source)
|
||||
ON CONFLICT (symbol, trading_date, revision) DO UPDATE SET
|
||||
open_price = EXCLUDED.open_price,
|
||||
close_price = EXCLUDED.close_price,
|
||||
volume = EXCLUDED.volume,
|
||||
published_at = CURRENT_TIMESTAMP,
|
||||
revision = market_data.daily_prices.revision + 1
|
||||
WHERE EXCLUDED.published_at > market_data.daily_prices.published_at;
|
||||
```
|
||||
|
||||
**Effect:** Same-day re-ingestion updates the row; older data is immutable (PIT principle).
|
||||
|
||||
---
|
||||
|
||||
## Testing & Validation
|
||||
|
||||
**Unit Tests (SQL):**
|
||||
- Constraints enforced (negative prices rejected)
|
||||
- Unique keys prevent duplicates
|
||||
- Soft-delete preserves history
|
||||
- PIT query returns correct version
|
||||
|
||||
**Integration Tests:**
|
||||
- Ingest 100 rows, verify count
|
||||
- Duplicate ingestion (same date/symbol) increments revision
|
||||
- Upsert with newer timestamp overwrites
|
||||
|
||||
@@ -0,0 +1,236 @@
|
||||
namespace KArtSell.Modules.ModelOperations.Domain;
|
||||
|
||||
/// <summary>
|
||||
/// VS-03 DOMAIN: Market Data Ingestion Policy
|
||||
///
|
||||
/// Handles:
|
||||
/// - Price data validation (OHLCV constraints)
|
||||
/// - Duplicate detection
|
||||
/// - Data normalization
|
||||
/// - Quality score assignment
|
||||
///
|
||||
/// Pure logic, no I/O, deterministic.
|
||||
/// </summary>
|
||||
|
||||
public record DailyPrice(
|
||||
Guid PriceId,
|
||||
string Symbol,
|
||||
DateOnly TradingDate,
|
||||
decimal OpenPrice,
|
||||
decimal HighPrice,
|
||||
decimal LowPrice,
|
||||
decimal ClosePrice,
|
||||
long Volume,
|
||||
DateTime PublishedAt,
|
||||
int Revision,
|
||||
string DataSource,
|
||||
string CorrelationId);
|
||||
|
||||
public record MarketIndex(
|
||||
Guid IndexId,
|
||||
string IndexCode,
|
||||
DateOnly TradingDate,
|
||||
decimal OpenValue,
|
||||
decimal HighValue,
|
||||
decimal LowValue,
|
||||
decimal CloseValue,
|
||||
decimal? ChangePercent,
|
||||
long? IndexVolume,
|
||||
DateTime PublishedAt,
|
||||
string DataSource);
|
||||
|
||||
public record IngestionBatch(
|
||||
Guid BatchId,
|
||||
string DataSource,
|
||||
DateOnly FromDate,
|
||||
DateOnly ToDate,
|
||||
List<DailyPrice> Prices,
|
||||
List<MarketIndex> Indices,
|
||||
string CorrelationId);
|
||||
|
||||
public record ValidationResult(
|
||||
bool IsValid,
|
||||
List<string> Errors,
|
||||
int QualityScore);
|
||||
|
||||
public static class MarketDataPolicy
|
||||
{
|
||||
/// <summary>
|
||||
/// Validate single price record
|
||||
///
|
||||
/// Rules:
|
||||
/// 1. All prices > 0
|
||||
/// 2. High >= Open, Open >= Close, Close >= Low (or reasonably close)
|
||||
/// 3. Volume >= 0
|
||||
/// 4. No future dates
|
||||
/// 5. Low <= High
|
||||
/// </summary>
|
||||
public static ValidationResult ValidatePrice(DailyPrice price, DateOnly maxDate = default)
|
||||
{
|
||||
if (maxDate == default)
|
||||
maxDate = DateOnly.FromDateTime(DateTime.UtcNow);
|
||||
|
||||
var errors = new List<string>();
|
||||
var qualityScore = 100;
|
||||
|
||||
// Price checks
|
||||
if (price.OpenPrice <= 0)
|
||||
errors.Add("Open price must be > 0");
|
||||
if (price.HighPrice <= 0)
|
||||
errors.Add("High price must be > 0");
|
||||
if (price.LowPrice <= 0)
|
||||
errors.Add("Low price must be > 0");
|
||||
if (price.ClosePrice <= 0)
|
||||
errors.Add("Close price must be > 0");
|
||||
|
||||
// OHLC relationship checks
|
||||
if (price.HighPrice < price.LowPrice)
|
||||
{
|
||||
errors.Add("High must be >= Low");
|
||||
qualityScore -= 20;
|
||||
}
|
||||
|
||||
if (price.HighPrice < price.OpenPrice || price.HighPrice < price.ClosePrice)
|
||||
{
|
||||
errors.Add("High must be >= Open and Close");
|
||||
qualityScore -= 10;
|
||||
}
|
||||
|
||||
if (price.LowPrice > price.OpenPrice || price.LowPrice > price.ClosePrice)
|
||||
{
|
||||
errors.Add("Low must be <= Open and Close");
|
||||
qualityScore -= 10;
|
||||
}
|
||||
|
||||
// Volume check
|
||||
if (price.Volume < 0)
|
||||
errors.Add("Volume must be >= 0");
|
||||
|
||||
if (price.Volume == 0)
|
||||
qualityScore -= 30; // Low-volume day
|
||||
|
||||
// Date check
|
||||
if (price.TradingDate > maxDate)
|
||||
{
|
||||
errors.Add("Trading date cannot be in the future");
|
||||
qualityScore -= 50;
|
||||
}
|
||||
|
||||
// Extreme price movement check (>10% daily)
|
||||
var priceRange = (price.HighPrice - price.LowPrice) / price.ClosePrice;
|
||||
if (priceRange > 0.1m)
|
||||
{
|
||||
qualityScore -= 15; // Flag for manual review
|
||||
}
|
||||
|
||||
return new ValidationResult(
|
||||
IsValid: errors.Count == 0,
|
||||
Errors: errors,
|
||||
QualityScore: Math.Max(0, qualityScore));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Detect duplicate prices (same symbol, date, identical OHLCV)
|
||||
///
|
||||
/// Returns true if this price already exists with identical values
|
||||
/// </summary>
|
||||
public static bool IsDuplicate(DailyPrice candidate, List<DailyPrice> existing)
|
||||
{
|
||||
var match = existing.FirstOrDefault(e =>
|
||||
e.Symbol == candidate.Symbol &&
|
||||
e.TradingDate == candidate.TradingDate);
|
||||
|
||||
if (match == null)
|
||||
return false;
|
||||
|
||||
// Check if prices are identical (within rounding tolerance)
|
||||
return Math.Abs(match.ClosePrice - candidate.ClosePrice) < 0.01m &&
|
||||
Math.Abs(match.OpenPrice - candidate.OpenPrice) < 0.01m &&
|
||||
Math.Abs(match.HighPrice - candidate.HighPrice) < 0.01m &&
|
||||
Math.Abs(match.LowPrice - candidate.LowPrice) < 0.01m &&
|
||||
match.Volume == candidate.Volume;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Normalize price data (handle splits, outliers, etc.)
|
||||
///
|
||||
/// Returns adjusted price or None if should be filtered
|
||||
/// </summary>
|
||||
public static DailyPrice? NormalizePrice(DailyPrice price)
|
||||
{
|
||||
// Filter if volume is suspiciously low (potential halt/error)
|
||||
if (price.Volume < 100)
|
||||
return null;
|
||||
|
||||
// Round to 2 decimals (Korean Won precision)
|
||||
var normalized = price with
|
||||
{
|
||||
OpenPrice = Math.Round(price.OpenPrice, 2),
|
||||
HighPrice = Math.Round(price.HighPrice, 2),
|
||||
LowPrice = Math.Round(price.LowPrice, 2),
|
||||
ClosePrice = Math.Round(price.ClosePrice, 2),
|
||||
};
|
||||
|
||||
return normalized;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Validate entire ingestion batch
|
||||
///
|
||||
/// Returns aggregated quality metrics and error summary
|
||||
/// </summary>
|
||||
public static (int TotalRows, int ValidRows, int InvalidRows, decimal QualityScore) ValidateBatch(IngestionBatch batch)
|
||||
{
|
||||
var totalRows = batch.Prices.Count;
|
||||
var validCount = 0;
|
||||
var invalidCount = 0;
|
||||
var totalQuality = 0;
|
||||
|
||||
foreach (var price in batch.Prices)
|
||||
{
|
||||
var result = ValidatePrice(price, batch.ToDate);
|
||||
if (result.IsValid)
|
||||
{
|
||||
validCount++;
|
||||
totalQuality += result.QualityScore;
|
||||
}
|
||||
else
|
||||
{
|
||||
invalidCount++;
|
||||
}
|
||||
}
|
||||
|
||||
var avgQuality = validCount > 0
|
||||
? (decimal)totalQuality / validCount
|
||||
: 0;
|
||||
|
||||
return (totalRows, validCount, invalidCount, (decimal)Math.Round(avgQuality, 2));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Classify data quality issue
|
||||
///
|
||||
/// Returns whether to accept, quarantine, or reject
|
||||
/// </summary>
|
||||
public static DataQualityDecision ClassifyQualityIssue(ValidationResult result)
|
||||
{
|
||||
if (result.QualityScore >= 90)
|
||||
return DataQualityDecision.Accept;
|
||||
|
||||
if (result.QualityScore >= 70)
|
||||
return DataQualityDecision.AcceptWithWarning;
|
||||
|
||||
if (result.QualityScore >= 50)
|
||||
return DataQualityDecision.Quarantine;
|
||||
|
||||
return DataQualityDecision.Reject;
|
||||
}
|
||||
}
|
||||
|
||||
public enum DataQualityDecision
|
||||
{
|
||||
Accept,
|
||||
AcceptWithWarning,
|
||||
Quarantine,
|
||||
Reject
|
||||
}
|
||||
@@ -0,0 +1,251 @@
|
||||
using KArtSell.Modules.ModelOperations.Domain;
|
||||
using Xunit;
|
||||
|
||||
namespace KArtSell.ModelOperations.UnitTests;
|
||||
|
||||
public class VS03_MarketDataPolicyTests
|
||||
{
|
||||
[Fact]
|
||||
public void ValidatePrice_ValidPrice_ReturnsPass()
|
||||
{
|
||||
var price = new DailyPrice(
|
||||
PriceId: Guid.NewGuid(),
|
||||
Symbol: "005930",
|
||||
TradingDate: DateOnly.FromDateTime(DateTime.UtcNow.AddDays(-1)),
|
||||
OpenPrice: 70000m,
|
||||
HighPrice: 71000m,
|
||||
LowPrice: 69000m,
|
||||
ClosePrice: 70500m,
|
||||
Volume: 1000000,
|
||||
PublishedAt: DateTime.UtcNow,
|
||||
Revision: 1,
|
||||
DataSource: "KRX",
|
||||
CorrelationId: "test-123");
|
||||
|
||||
var result = MarketDataPolicy.ValidatePrice(price);
|
||||
|
||||
Assert.True(result.IsValid);
|
||||
Assert.Empty(result.Errors);
|
||||
Assert.True(result.QualityScore >= 85);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ValidatePrice_NegativePrice_ReturnsFail()
|
||||
{
|
||||
var price = new DailyPrice(
|
||||
PriceId: Guid.NewGuid(),
|
||||
Symbol: "005930",
|
||||
TradingDate: DateOnly.FromDateTime(DateTime.UtcNow.AddDays(-1)),
|
||||
OpenPrice: -100m,
|
||||
HighPrice: 71000m,
|
||||
LowPrice: 69000m,
|
||||
ClosePrice: 70500m,
|
||||
Volume: 1000000,
|
||||
PublishedAt: DateTime.UtcNow,
|
||||
Revision: 1,
|
||||
DataSource: "KRX",
|
||||
CorrelationId: "test-123");
|
||||
|
||||
var result = MarketDataPolicy.ValidatePrice(price);
|
||||
|
||||
Assert.False(result.IsValid);
|
||||
Assert.Contains("Open price must be > 0", result.Errors);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ValidatePrice_HighLowerThanLow_ReturnsFail()
|
||||
{
|
||||
var price = new DailyPrice(
|
||||
PriceId: Guid.NewGuid(),
|
||||
Symbol: "005930",
|
||||
TradingDate: DateOnly.FromDateTime(DateTime.UtcNow.AddDays(-1)),
|
||||
OpenPrice: 70000m,
|
||||
HighPrice: 68000m,
|
||||
LowPrice: 69000m,
|
||||
ClosePrice: 70500m,
|
||||
Volume: 1000000,
|
||||
PublishedAt: DateTime.UtcNow,
|
||||
Revision: 1,
|
||||
DataSource: "KRX",
|
||||
CorrelationId: "test-123");
|
||||
|
||||
var result = MarketDataPolicy.ValidatePrice(price);
|
||||
|
||||
Assert.False(result.IsValid);
|
||||
Assert.Contains("High must be >= Low", result.Errors);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ValidatePrice_FutureDate_ReturnsFail()
|
||||
{
|
||||
var price = new DailyPrice(
|
||||
PriceId: Guid.NewGuid(),
|
||||
Symbol: "005930",
|
||||
TradingDate: DateOnly.FromDateTime(DateTime.UtcNow.AddDays(1)),
|
||||
OpenPrice: 70000m,
|
||||
HighPrice: 71000m,
|
||||
LowPrice: 69000m,
|
||||
ClosePrice: 70500m,
|
||||
Volume: 1000000,
|
||||
PublishedAt: DateTime.UtcNow,
|
||||
Revision: 1,
|
||||
DataSource: "KRX",
|
||||
CorrelationId: "test-123");
|
||||
|
||||
var result = MarketDataPolicy.ValidatePrice(price);
|
||||
|
||||
Assert.False(result.IsValid);
|
||||
Assert.Contains("cannot be in the future", result.Errors[0]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ValidatePrice_ZeroVolume_LowersScore()
|
||||
{
|
||||
var price = new DailyPrice(
|
||||
PriceId: Guid.NewGuid(),
|
||||
Symbol: "005930",
|
||||
TradingDate: DateOnly.FromDateTime(DateTime.UtcNow.AddDays(-1)),
|
||||
OpenPrice: 70000m,
|
||||
HighPrice: 71000m,
|
||||
LowPrice: 69000m,
|
||||
ClosePrice: 70500m,
|
||||
Volume: 0,
|
||||
PublishedAt: DateTime.UtcNow,
|
||||
Revision: 1,
|
||||
DataSource: "KRX",
|
||||
CorrelationId: "test-123");
|
||||
|
||||
var result = MarketDataPolicy.ValidatePrice(price);
|
||||
|
||||
Assert.True(result.IsValid);
|
||||
Assert.True(result.QualityScore < 80); // Quality degraded but still valid
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void IsDuplicate_IdenticalPrice_ReturnsTrue()
|
||||
{
|
||||
var price1 = new DailyPrice(
|
||||
PriceId: Guid.NewGuid(),
|
||||
Symbol: "005930",
|
||||
TradingDate: DateOnly.FromDateTime(DateTime.UtcNow.AddDays(-1)),
|
||||
OpenPrice: 70000m,
|
||||
HighPrice: 71000m,
|
||||
LowPrice: 69000m,
|
||||
ClosePrice: 70500m,
|
||||
Volume: 1000000,
|
||||
PublishedAt: DateTime.UtcNow,
|
||||
Revision: 1,
|
||||
DataSource: "KRX",
|
||||
CorrelationId: "test-123");
|
||||
|
||||
var price2 = price1 with { PriceId = Guid.NewGuid(), Revision = 2 };
|
||||
|
||||
var isDuplicate = MarketDataPolicy.IsDuplicate(price2, new List<DailyPrice> { price1 });
|
||||
|
||||
Assert.True(isDuplicate);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void IsDuplicate_DifferentSymbol_ReturnsFalse()
|
||||
{
|
||||
var price1 = new DailyPrice(
|
||||
PriceId: Guid.NewGuid(),
|
||||
Symbol: "005930",
|
||||
TradingDate: DateOnly.FromDateTime(DateTime.UtcNow.AddDays(-1)),
|
||||
OpenPrice: 70000m,
|
||||
HighPrice: 71000m,
|
||||
LowPrice: 69000m,
|
||||
ClosePrice: 70500m,
|
||||
Volume: 1000000,
|
||||
PublishedAt: DateTime.UtcNow,
|
||||
Revision: 1,
|
||||
DataSource: "KRX",
|
||||
CorrelationId: "test-123");
|
||||
|
||||
var price2 = price1 with
|
||||
{
|
||||
PriceId = Guid.NewGuid(),
|
||||
Symbol = "000660"
|
||||
};
|
||||
|
||||
var isDuplicate = MarketDataPolicy.IsDuplicate(price2, new List<DailyPrice> { price1 });
|
||||
|
||||
Assert.False(isDuplicate);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void NormalizePrice_ValidPrice_RoundsTo2Decimals()
|
||||
{
|
||||
var price = new DailyPrice(
|
||||
PriceId: Guid.NewGuid(),
|
||||
Symbol: "005930",
|
||||
TradingDate: DateOnly.FromDateTime(DateTime.UtcNow.AddDays(-1)),
|
||||
OpenPrice: 70000.123m,
|
||||
HighPrice: 71000.456m,
|
||||
LowPrice: 69000.789m,
|
||||
ClosePrice: 70500.999m,
|
||||
Volume: 1000000,
|
||||
PublishedAt: DateTime.UtcNow,
|
||||
Revision: 1,
|
||||
DataSource: "KRX",
|
||||
CorrelationId: "test-123");
|
||||
|
||||
var normalized = MarketDataPolicy.NormalizePrice(price);
|
||||
|
||||
Assert.NotNull(normalized);
|
||||
Assert.Equal(70000.12m, normalized!.OpenPrice);
|
||||
Assert.Equal(71000.46m, normalized.HighPrice);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void NormalizePrice_LowVolume_ReturnsNull()
|
||||
{
|
||||
var price = new DailyPrice(
|
||||
PriceId: Guid.NewGuid(),
|
||||
Symbol: "005930",
|
||||
TradingDate: DateOnly.FromDateTime(DateTime.UtcNow.AddDays(-1)),
|
||||
OpenPrice: 70000m,
|
||||
HighPrice: 71000m,
|
||||
LowPrice: 69000m,
|
||||
ClosePrice: 70500m,
|
||||
Volume: 50, // Suspiciously low
|
||||
PublishedAt: DateTime.UtcNow,
|
||||
Revision: 1,
|
||||
DataSource: "KRX",
|
||||
CorrelationId: "test-123");
|
||||
|
||||
var normalized = MarketDataPolicy.NormalizePrice(price);
|
||||
|
||||
Assert.Null(normalized);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ClassifyQualityIssue_HighScore_ReturnsAccept()
|
||||
{
|
||||
var result = new ValidationResult(IsValid: true, Errors: new(), QualityScore: 95);
|
||||
|
||||
var decision = MarketDataPolicy.ClassifyQualityIssue(result);
|
||||
|
||||
Assert.Equal(DataQualityDecision.Accept, decision);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ClassifyQualityIssue_MediumScore_ReturnsAcceptWithWarning()
|
||||
{
|
||||
var result = new ValidationResult(IsValid: true, Errors: new(), QualityScore: 75);
|
||||
|
||||
var decision = MarketDataPolicy.ClassifyQualityIssue(result);
|
||||
|
||||
Assert.Equal(DataQualityDecision.AcceptWithWarning, decision);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ClassifyQualityIssue_LowScore_ReturnsQuarantine()
|
||||
{
|
||||
var result = new ValidationResult(IsValid: true, Errors: new(), QualityScore: 60);
|
||||
|
||||
var decision = MarketDataPolicy.ClassifyQualityIssue(result);
|
||||
|
||||
Assert.Equal(DataQualityDecision.Quarantine, decision);
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user