KArtSell.Aegis/docs/CURRENT/CATALOGS/source-catalog.md

Data Source Catalog

Purpose: Master reference for all data sources, APIs, and lineage
Owner: Data Governance Team
Version: 1.0
Date: 2026-08-06

---

📊 Source Systems Summary

Source	Type	Frequency	Availability SLA	Consumers	Retention
KRX OpenAPI	External REST	Daily (T+0)	99.5%	prices, signals, portfolio	5 years
OpenDart API	External REST	T+2	99.0%	disclosure, models, recommendations	7 years
Portfolio (User Input)	Internal Form	Real-time	100% (manual)	rebalance, risk, holdings	5 years
Shadow Run Output	Computed (Hangfire)	252+ days	99.9%	evidence, PBO/DSR, activation	10 years
Audit Events	Internal Database	Real-time (write)	99.99%	compliance, security, tracing	7 years

---

🔗 Data Lineage Map

KRX Market Data Flow

┌─────────────────────────────────────────────────────────────┐
│                    KRX OpenAPI (External)                   │
│  Endpoint: /svc/apis/idx/krx_dd_trd, /svc/apis/sco/...     │
│  Auth: AUTH_KEY header                                       │
│  Frequency: Daily (T+0, end of business)                    │
└──────────────────────────────┬──────────────────────────────┘
                               │
                               ↓
┌──────────────────────────────────────────────────────────────┐
│           market_data.prices (PostgreSQL)                    │
│  Schema: price_id, symbol, trade_date, OHLCV, volume        │
│  PIT: published_at, correlation_id, revision                │
│  Validation: No nulls, volume ≥ 0, high ≥ low ≤ close      │
└──────────────────────────────┬───────────────────────────────┘
                               │
                    ┌──────────┴──────────┐
                    ↓                     ↓
        ┌────────────────────┐  ┌────────────────────┐
        │  signal_engine     │  │  portfolio.holdings│
        │   (Signals)        │  │   (Analysis)       │
        └────────┬───────────┘  └────────┬───────────┘
                 │                       │
                 └───────────┬───────────┘
                             ↓
                ┌────────────────────────┐
                │  sell_decision_engine  │
                │  (Final Output)        │
                └────────────────────────┘

OpenDart Financial Disclosure Flow

┌──────────────────────────────────────────────────────────┐
│            OpenDart API (Financial Supervisory Service)  │
│  Endpoint: /api/list.json (공시정보, DS001)             │
│  Auth: crtfc_key (certificate key)                       │
│  Frequency: T+2 (regulatory reporting)                   │
└──────────────────────────┬───────────────────────────────┘
                           │
                           ↓
┌──────────────────────────────────────────────────────────┐
│      model_operations.disclosures (PostgreSQL)           │
│  Schema: filing_id, corp_code, report_type, filed_date  │
│  PIT: published_at, correlation_id, revision            │
│  Validation: Non-null corp_code, valid FSS report types │
└──────────────────────────┬───────────────────────────────┘
                           │
                           ↓
┌──────────────────────────────────────────────────────────┐
│        model_operations.models (Policy Input)            │
│  Lifecycle: Freeze→Mature→Score→...→ManualActivation   │
└──────────────────────────────────────────────────────────┘

Shadow Run Batch Processing

┌─────────────────────────────────────┐
│    PHASE-1-SHADOW-RUN (Job 976)    │
│   Duration: 252+ trading days      │
│   Auto-runs (Hangfire)             │
└──────────────┬──────────────────────┘
               │
               ├─→ Input: models.* + prices.* + holdings.*
               │   (PIT-queried at cutoff dates)
               │
               └─→ Processing:
                   1. Load model (published_at ≤ cutoff)
                   2. Fetch price history (T to T+252 days)
                   3. Simulate rebalance decisions
                   4. Compute P&L metrics
                   5. Calculate OOS (out-of-sample) performance
                   6. Compute PBO/DSR evidence
               │
               ↓
┌─────────────────────────────────────┐
│   shadow_run_results (PostgreSQL)   │
│   Schema: job_id, model_id,         │
│   window_start, window_end,         │
│   pbo_score, dsr_score, oos_return  │
│   PIT: published_at, revision       │
└──────────────┬──────────────────────┘
               │
               ↓
┌─────────────────────────────────────┐
│  model_operations.models (Update)   │
│  Status: Review → ManualActivation  │
│  Attach: PBO/DSR evidence proof     │
└─────────────────────────────────────┘

---

📋 API Contract Details

KRX OpenAPI

Service: Korea Exchange (KRX) Market Data
Base URL: https://openapi.krx.co.kr
Authentication: AUTH_KEY header
Rate Limit: 1000 req/day (typical)

Endpoints Used:

Endpoint	Method	Purpose	Frequency
/svc/apis/idx/krx_dd_trd	POST	Index data (KOSPI, KOSDAQ)	Daily
/svc/apis/sco/stk_bnd_isfl	POST	Stock trading volume	Daily

Request Payload:

{
  "basDd": "20260801",
  "isuCd": "005930",
  "gubun": "ALL"
}

Response Schema:

{
  "block_begin": "...",
  "OutBlock_1": [
    {
      "IDX_IND_CD": "KOSPI",
      "TRD_DD": "20260801",
      "CLSPRC_IDX": "2750.50",
      "OPNPRC_IDX": "2745.00",
      "HGPRC_IDX": "2760.00",
      "LWPRC_IDX": "2740.00",
      "ACC_TRDVOL": "1234567890"
    }
  ]
}

Error Handling:

• Transient: Retry with exponential backoff (3 attempts)
• Permanent: Log + alert + fallback to LKG (last-known-good)

---

OpenDart API

Service: Financial Supervisory Service Disclosure
Base URL: https://opendart.fss.or.kr
Authentication: crtfc_key query parameter
Rate Limit: 100 req/hour (typical)

Endpoints Used:

Endpoint	Method	Purpose	Frequency
/api/list.json	GET	Disclosure search	On-demand (T+2)
/api/document.json	GET	Document metadata	On-demand

Request Example:

GET /api/list.json?crtfc_key=KEY&corp_code=00126380&bgn_de=20260101&end_de=20260831

Response Schema:

{
  "status": "000",
  "message": "정상",
  "list": [
    {
      "corp_code": "00126380",
      "corp_name": "Samsung Electronics",
      "stock_code": "005930",
      "report_nm": "분기보고서",
      "report_code": "11013",
      "accept_dt": "20260501",
      "report_dt": "20260501",
      "rm": ""
    }
  ]
}

Error Handling:

• Queue for retry if 401/403 (certificate issues)
• Fallback to cache if 429 (rate limit)

---

🔒 Data Quality Rules by Source

KRX Prices

Completeness:

• Every KOSPI/KOSDAQ stock must have OHLCV for every trading day
• No nulls allowed in: symbol, trade_date, close_price, volume

Accuracy:

• Prices must match official KRX reporting (daily reconciliation)
• Volume > 0 for liquid stocks (> 1000 shares/day)
• OHLC ordering: low ≤ open, close ≤ high

Timeliness:

• Published T+0 (end of business day)
• Ingested within 1 hour of market close

Retention: 5 years

---

OpenDart Disclosures

Completeness:

• corp_code + filing_date must be non-null
• report_type must match FSS enum

Accuracy:

• Must match official FSS repository
• No synthetic/inferred filings

Timeliness:

• Published T+2 (regulatory requirement)

Retention: 7 years (regulatory)

---

Portfolio (User Input)

Completeness:

• quantity ≥ 0
• cost_basis > 0 (if quantity > 0)
• acquisition_date ≤ today()

Accuracy:

• User responsibility; audit trail required
• Cross-check with broker statements monthly

Timeliness:

• Real-time (synchronous input)

Retention: 5 years

---

📈 Consumption Matrix

Which Slices Consume Which Sources?

Source	VS-01	VS-02	VS-03	VS-04	VS-05+
KRX Prices	✅	✅	✅	✅	✅
OpenDart	✅	⚪	⚪	⚪	✅
Portfolio	⚪	✅	⚪	✅	✅
Shadow Run	⚪	⚪	⚪	⚪	✅
Audit Events	✅	✅	✅	✅	✅

Legend: ✅ = Primary consumer, ⚪ = Secondary/Optional

---

⚠️ Failure Modes & Remediation

Scenario	Detection	Mitigation	Recovery
KRX API down	503 from endpoint	Use LKG prices (cache)	Retry next market day
OpenDart rate limit	429 response	Queue for retry (Hangfire)	Exponential backoff
Portfolio stale	> 5 days since update	Alert user	Manual refresh
Shadow run timeout	Job > 1 day	Extend deadline	Resume from checkpoint
Data quality fail	DQ rule violation	Quarantine + alert	Manual review

---

📚 References

• KRX OpenAPI: https://openapi.krx.co.kr (requires registration)
• OpenDart API: https://opendart.fss.or.kr
• Data Contract: contracts/data/platform-data-contract.v1.json
• DQ Rules: docs/dq-lineage-rules.md
• Source Systems Table: audit.source_systems (audit log)

---

Owner: Data Governance
Last Updated: 2026-08-06
Status: ✅ APPROVED FOR OPERATIONS