KArtSell.Aegis/docs/CURRENT/CATALOGS/source-catalog.md

Data Source Catalog

Purpose: Master reference for all data sources, APIs, SLAs, and lineage
Owner: Data Governance Team
Version: 2.0
Date: 2026-08-07
Status: CONSOLIDATED (AEG-X-009)

---

📊 Source Systems Summary

Source	Type	Frequency	Availability SLA	Import Delay	Consumers	Retention	Owner
KRX OpenAPI	External REST	Daily (T+0)	99.5%	<4 hours (EoD)	prices, signals, portfolio	5 years	KRX
OpenDart API	External REST	T+2 business	99.0%	+2 calendar days	disclosure, models, recommendations	7 years	FSS
KIS API	External REST	Real-time	99.2%	<1 minute	trading, orders, execution	3 years	Korea Investment & Securities
Portfolio (User Input)	Internal Form	Real-time	100% (manual)	Immediate	rebalance, risk, holdings	5 years	Internal
Shadow Run Output	Computed (Hangfire)	252+ days	99.9%	Async (Job 976)	evidence, PBO/DSR, activation	10 years	Internal
Audit Events	Internal Database	Real-time (write)	99.99%	Immediate	compliance, security, tracing	7 years	Internal

---

🔗 Data Lineage Map

KRX Market Data Flow

┌─────────────────────────────────────────────────────────────┐
│                    KRX OpenAPI (External)                   │
│  Endpoint: /svc/apis/idx/krx_dd_trd, /svc/apis/sco/...     │
│  Auth: AUTH_KEY header                                       │
│  Frequency: Daily (T+0, end of business)                    │
└──────────────────────────────┬──────────────────────────────┘
                               │
                               ↓
┌──────────────────────────────────────────────────────────────┐
│           market_data.prices (PostgreSQL)                    │
│  Schema: price_id, symbol, trade_date, OHLCV, volume        │
│  PIT: published_at, correlation_id, revision                │
│  Validation: No nulls, volume ≥ 0, high ≥ low ≤ close      │
└──────────────────────────────┬───────────────────────────────┘
                               │
                    ┌──────────┴──────────┐
                    ↓                     ↓
        ┌────────────────────┐  ┌────────────────────┐
        │  signal_engine     │  │  portfolio.holdings│
        │   (Signals)        │  │   (Analysis)       │
        └────────┬───────────┘  └────────┬───────────┘
                 │                       │
                 └───────────┬───────────┘
                             ↓
                ┌────────────────────────┐
                │  sell_decision_engine  │
                │  (Final Output)        │
                └────────────────────────┘

OpenDart Financial Disclosure Flow

┌──────────────────────────────────────────────────────────┐
│            OpenDart API (Financial Supervisory Service)  │
│  Endpoint: /api/list.json (공시정보, DS001)             │
│  Auth: crtfc_key (certificate key)                       │
│  Frequency: T+2 (regulatory reporting)                   │
└──────────────────────────┬───────────────────────────────┘
                           │
                           ↓
┌──────────────────────────────────────────────────────────┐
│      model_operations.disclosures (PostgreSQL)           │
│  Schema: filing_id, corp_code, report_type, filed_date  │
│  PIT: published_at, correlation_id, revision            │
│  Validation: Non-null corp_code, valid FSS report types │
└──────────────────────────┬───────────────────────────────┘
                           │
                           ↓
┌──────────────────────────────────────────────────────────┐
│        model_operations.models (Policy Input)            │
│  Lifecycle: Freeze→Mature→Score→...→ManualActivation   │
└──────────────────────────────────────────────────────────┘

Shadow Run Batch Processing

┌─────────────────────────────────────┐
│    PHASE-1-SHADOW-RUN (Job 976)    │
│   Duration: 252+ trading days      │
│   Auto-runs (Hangfire)             │
└──────────────┬──────────────────────┘
               │
               ├─→ Input: models.* + prices.* + holdings.*
               │   (PIT-queried at cutoff dates)
               │
               └─→ Processing:
                   1. Load model (published_at ≤ cutoff)
                   2. Fetch price history (T to T+252 days)
                   3. Simulate rebalance decisions
                   4. Compute P&L metrics
                   5. Calculate OOS (out-of-sample) performance
                   6. Compute PBO/DSR evidence
               │
               ↓
┌─────────────────────────────────────┐
│   shadow_run_results (PostgreSQL)   │
│   Schema: job_id, model_id,         │
│   window_start, window_end,         │
│   pbo_score, dsr_score, oos_return  │
│   PIT: published_at, revision       │
└──────────────┬──────────────────────┘
               │
               ↓
┌─────────────────────────────────────┐
│  model_operations.models (Update)   │
│  Status: Review → ManualActivation  │
│  Attach: PBO/DSR evidence proof     │
└─────────────────────────────────────┘

---

📋 API Contract Details

KRX OpenAPI

Service: Korea Exchange (KRX) Market Data
Base URL: https://openapi.krx.co.kr
Authentication: AUTH_KEY header
Rate Limit: 1000 req/day (typical)

Endpoints Used:

Endpoint	Method	Purpose	Frequency
/svc/apis/idx/krx_dd_trd	POST	Index data (KOSPI, KOSDAQ)	Daily
/svc/apis/sco/stk_bnd_isfl	POST	Stock trading volume	Daily

Request Payload:

{
  "basDd": "20260801",
  "isuCd": "005930",
  "gubun": "ALL"
}

Response Schema:

{
  "block_begin": "...",
  "OutBlock_1": [
    {
      "IDX_IND_CD": "KOSPI",
      "TRD_DD": "20260801",
      "CLSPRC_IDX": "2750.50",
      "OPNPRC_IDX": "2745.00",
      "HGPRC_IDX": "2760.00",
      "LWPRC_IDX": "2740.00",
      "ACC_TRDVOL": "1234567890"
    }
  ]
}

Error Handling:

• Transient: Retry with exponential backoff (3 attempts)
• Permanent: Log + alert + fallback to LKG (last-known-good)

---

OpenDart API

Service: Financial Supervisory Service Disclosure
Base URL: https://opendart.fss.or.kr
Authentication: crtfc_key query parameter
Rate Limit: 100 req/hour (typical)

Endpoints Used:

Endpoint	Method	Purpose	Frequency
/api/list.json	GET	Disclosure search	On-demand (T+2)
/api/document.json	GET	Document metadata	On-demand

Request Example:

GET /api/list.json?crtfc_key=KEY&corp_code=00126380&bgn_de=20260101&end_de=20260831

Response Schema:

{
  "status": "000",
  "message": "정상",
  "list": [
    {
      "corp_code": "00126380",
      "corp_name": "Samsung Electronics",
      "stock_code": "005930",
      "report_nm": "분기보고서",
      "report_code": "11013",
      "accept_dt": "20260501",
      "report_dt": "20260501",
      "rm": ""
    }
  ]
}

Error Handling:

• Queue for retry if 401/403 (certificate issues)
• Fallback to cache if 429 (rate limit)

---

🔒 Data Quality Rules by Source

KRX Prices

Completeness:

• Every KOSPI/KOSDAQ stock must have OHLCV for every trading day
• No nulls allowed in: symbol, trade_date, close_price, volume

Accuracy:

• Prices must match official KRX reporting (daily reconciliation)
• Volume > 0 for liquid stocks (> 1000 shares/day)
• OHLC ordering: low ≤ open, close ≤ high

Timeliness:

• Published T+0 (end of business day)
• Ingested within 1 hour of market close

Retention: 5 years

---

OpenDart Disclosures

Completeness:

• corp_code + filing_date must be non-null
• report_type must match FSS enum

Accuracy:

• Must match official FSS repository
• No synthetic/inferred filings

Timeliness:

• Published T+2 (regulatory requirement)

Retention: 7 years (regulatory)

---

Portfolio (User Input)

Completeness:

• quantity ≥ 0
• cost_basis > 0 (if quantity > 0)
• acquisition_date ≤ today()

Accuracy:

• User responsibility; audit trail required
• Cross-check with broker statements monthly

Timeliness:

• Real-time (synchronous input)

Retention: 5 years

---

📈 Consumption Matrix

Which Slices Consume Which Sources?

Source	VS-01	VS-02	VS-03	VS-04	VS-05+
KRX Prices	✅	✅	✅	✅	✅
OpenDart	✅	⚪	⚪	⚪	✅
Portfolio	⚪	✅	⚪	✅	✅
Shadow Run	⚪	⚪	⚪	⚪	✅
Audit Events	✅	✅	✅	✅	✅

Legend: ✅ = Primary consumer, ⚪ = Secondary/Optional

---

⚠️ Failure Modes & Remediation

Scenario	Detection	Mitigation	Recovery
KRX API down	503 from endpoint	Use LKG prices (cache)	Retry next market day
OpenDart rate limit	429 response	Queue for retry (Hangfire)	Exponential backoff
Portfolio stale	> 5 days since update	Alert user	Manual refresh
Shadow run timeout	Job > 1 day	Extend deadline	Resume from checkpoint
Data quality fail	DQ rule violation	Quarantine + alert	Manual review

---

📚 References

• KRX OpenAPI: https://openapi.krx.co.kr (requires registration)
• OpenDart API: https://opendart.fss.or.kr
• Data Contract: contracts/data/platform-data-contract.v1.json
• DQ Rules: docs/dq-lineage-rules.md
• Source Systems Table: audit.source_systems (audit log)

---

---

🔑 KIS API (Korea Investment & Securities)

Service: Korea Investment & Securities Trading API
Base URL: https://openapivts.kbopenplatform.com (KIS VTS) or https://openapi.kbopenplatform.com
Authentication: APP_KEY + APP_SECRET (OAuth2, JWT)
Rate Limit: 5000 req/minute (varies by tier)

Endpoints Used:

Endpoint	Method	Purpose	Frequency
/uapi/trading-order	POST	Place order	Real-time
/uapi/trading-cancel-order	POST	Cancel order	Real-time
/uapi/domestic-stock-cash-daily	GET	Account balance	Daily EOD

Authentication Flow:

1. Get OAuth2 token: POST /oauth2/authorize + refresh_token
2. Call trading endpoint: X-APP-KEY + Authorization: Bearer <token>
3. Retry on 401: Refresh token if expired

Fallback Strategy:

• Primary: Live API
• Secondary: Last Known Good (LKG) state from DB
• Tertiary: Cached execution snapshot from previous day

---

⏱️ SLA & Retry Policy

Service Level Agreements

Source	Availability	Support Hours	Incident Contact	Escalation
KRX	99.5%	Weekdays 9 AM-5 PM KST	support@krx.co.kr	→ Operations Manager
OpenDart	99.0%	Business hours only	FSS Helpdesk	→ Data Governance Lead
KIS	99.2%	24/5 (trading hours)	api-support@kimconsulting.com	→ Backend Lead

Error Classification & Retry

Error	Classification	Retry Delay	Max Attempts	Action
Network timeout	Transient	30s exponential backoff	5	Retry immediately
429 (Rate limit)	Transient	60s + random jitter	3	Queue to Hangfire
401 (Auth expired)	Transient	Refresh token, retry	2	Obtain new credentials
400 (Bad request)	Permanent	None	0	Log error, alert ops
503 (Service unavailable)	Transient	5min + exponential	10	Use fallback (cache)
Data quality rule fail	Permanent	None	0	Quarantine + manual review

Fallback & Recovery

When Primary Source Fails:

1. KRX API down: Use LKG prices from cache (up to 1 trading day old)
2. OpenDart rate limit: Queue job for retry (Hangfire q-backfill)
3. KIS trading timeout: Use cached balance, resume next market open
4. Shadow run interrupted: Resume from last checkpoint (idempotent)

---

📋 Data Retention Policy

Source	Cold Storage	Archive Retention	Purge Policy
KRX Prices	After 2 years	5 years (compliance)	After 5 years
OpenDart	After 3 years	7 years (regulatory)	After 7 years
KIS Trading	After 1 year	3 years (audit)	After 3 years
Shadow Run	Never	10 years (evidence)	Never (immutable)

---

Owner: Data Governance
Last Updated: 2026-08-07 (AEG-X-009 Consolidated)
Status: ✅ APPROVED FOR OPERATIONS