KArtSell.Aegis/docs/CURRENT/CATALOGS/source-catalog.md

# Data Source Catalog

**Purpose:** Master reference for all data sources, APIs, SLAs, and lineage  
**Owner:** Data Governance Team  
**Version:** 2.0  
**Date:** 2026-08-07  
**Status:** CONSOLIDATED (AEG-X-009)

---

## 📊 Source Systems Summary

| Source | Type | Frequency | Availability SLA | Import Delay | Consumers | Retention | Owner |
|--------|------|-----------|------------------|--------------|-----------|-----------|-------|
| **KRX OpenAPI** | External REST | Daily (T+0) | 99.5% | <4 hours (EoD) | prices, signals, portfolio | 5 years | KRX |
| **OpenDart API** | External REST | T+2 business | 99.0% | +2 calendar days | disclosure, models, recommendations | 7 years | FSS |
| **KIS API** | External REST | Real-time | 99.2% | <1 minute | trading, orders, execution | 3 years | Korea Investment & Securities |
| **Portfolio (User Input)** | Internal Form | Real-time | 100% (manual) | Immediate | rebalance, risk, holdings | 5 years | Internal |
| **Shadow Run Output** | Computed (Hangfire) | 252+ days | 99.9% | Async (Job 976) | evidence, PBO/DSR, activation | 10 years | Internal |
| **Audit Events** | Internal Database | Real-time (write) | 99.99% | Immediate | compliance, security, tracing | 7 years | Internal |

---

## 🔗 Data Lineage Map

### KRX Market Data Flow

```
┌─────────────────────────────────────────────────────────────┐
│                    KRX OpenAPI (External)                   │
│  Endpoint: /svc/apis/idx/krx_dd_trd, /svc/apis/sco/...     │
│  Auth: AUTH_KEY header                                       │
│  Frequency: Daily (T+0, end of business)                    │
└──────────────────────────────┬──────────────────────────────┘
                               │
                               ↓
┌──────────────────────────────────────────────────────────────┐
│           market_data.prices (PostgreSQL)                    │
│  Schema: price_id, symbol, trade_date, OHLCV, volume        │
│  PIT: published_at, correlation_id, revision                │
│  Validation: No nulls, volume ≥ 0, high ≥ low ≤ close      │
└──────────────────────────────┬───────────────────────────────┘
                               │
                    ┌──────────┴──────────┐
                    ↓                     ↓
        ┌────────────────────┐  ┌────────────────────┐
        │  signal_engine     │  │  portfolio.holdings│
        │   (Signals)        │  │   (Analysis)       │
        └────────┬───────────┘  └────────┬───────────┘
                 │                       │
                 └───────────┬───────────┘
                             ↓
                ┌────────────────────────┐
                │  sell_decision_engine  │
                │  (Final Output)        │
                └────────────────────────┘
```

### OpenDart Financial Disclosure Flow

```
┌──────────────────────────────────────────────────────────┐
│            OpenDart API (Financial Supervisory Service)  │
│  Endpoint: /api/list.json (공시정보, DS001)             │
│  Auth: crtfc_key (certificate key)                       │
│  Frequency: T+2 (regulatory reporting)                   │
└──────────────────────────┬───────────────────────────────┘
                           │
                           ↓
┌──────────────────────────────────────────────────────────┐
│      model_operations.disclosures (PostgreSQL)           │
│  Schema: filing_id, corp_code, report_type, filed_date  │
│  PIT: published_at, correlation_id, revision            │
│  Validation: Non-null corp_code, valid FSS report types │
└──────────────────────────┬───────────────────────────────┘
                           │
                           ↓
┌──────────────────────────────────────────────────────────┐
│        model_operations.models (Policy Input)            │
│  Lifecycle: Freeze→Mature→Score→...→ManualActivation   │
└──────────────────────────────────────────────────────────┘
```

### Shadow Run Batch Processing

```
┌─────────────────────────────────────┐
│    PHASE-1-SHADOW-RUN (Job 976)    │
│   Duration: 252+ trading days      │
│   Auto-runs (Hangfire)             │
└──────────────┬──────────────────────┘
               │
               ├─→ Input: models.* + prices.* + holdings.*
               │   (PIT-queried at cutoff dates)
               │
               └─→ Processing:
                   1. Load model (published_at ≤ cutoff)
                   2. Fetch price history (T to T+252 days)
                   3. Simulate rebalance decisions
                   4. Compute P&L metrics
                   5. Calculate OOS (out-of-sample) performance
                   6. Compute PBO/DSR evidence
               │
               ↓
┌─────────────────────────────────────┐
│   shadow_run_results (PostgreSQL)   │
│   Schema: job_id, model_id,         │
│   window_start, window_end,         │
│   pbo_score, dsr_score, oos_return  │
│   PIT: published_at, revision       │
└──────────────┬──────────────────────┘
               │
               ↓
┌─────────────────────────────────────┐
│  model_operations.models (Update)   │
│  Status: Review → ManualActivation  │
│  Attach: PBO/DSR evidence proof     │
└─────────────────────────────────────┘
```

---

## 📋 API Contract Details

### KRX OpenAPI

**Service:** Korea Exchange (KRX) Market Data  
**Base URL:** `https://openapi.krx.co.kr`  
**Authentication:** `AUTH_KEY` header  
**Rate Limit:** 1000 req/day (typical)

**Endpoints Used:**

| Endpoint | Method | Purpose | Frequency |
|----------|--------|---------|-----------|
| `/svc/apis/idx/krx_dd_trd` | POST | Index data (KOSPI, KOSDAQ) | Daily |
| `/svc/apis/sco/stk_bnd_isfl` | POST | Stock trading volume | Daily |

**Request Payload:**
```json
{
  "basDd": "20260801",
  "isuCd": "005930",
  "gubun": "ALL"
}
```

**Response Schema:**
```json
{
  "block_begin": "...",
  "OutBlock_1": [
    {
      "IDX_IND_CD": "KOSPI",
      "TRD_DD": "20260801",
      "CLSPRC_IDX": "2750.50",
      "OPNPRC_IDX": "2745.00",
      "HGPRC_IDX": "2760.00",
      "LWPRC_IDX": "2740.00",
      "ACC_TRDVOL": "1234567890"
    }
  ]
}
```

**Error Handling:**
- Transient: Retry with exponential backoff (3 attempts)
- Permanent: Log + alert + fallback to LKG (last-known-good)

---

### OpenDart API

**Service:** Financial Supervisory Service Disclosure  
**Base URL:** `https://opendart.fss.or.kr`  
**Authentication:** `crtfc_key` query parameter  
**Rate Limit:** 100 req/hour (typical)

**Endpoints Used:**

| Endpoint | Method | Purpose | Frequency |
|----------|--------|---------|-----------|
| `/api/list.json` | GET | Disclosure search | On-demand (T+2) |
| `/api/document.json` | GET | Document metadata | On-demand |

**Request Example:**
```
GET /api/list.json?crtfc_key=KEY&corp_code=00126380&bgn_de=20260101&end_de=20260831
```

**Response Schema:**
```json
{
  "status": "000",
  "message": "정상",
  "list": [
    {
      "corp_code": "00126380",
      "corp_name": "Samsung Electronics",
      "stock_code": "005930",
      "report_nm": "분기보고서",
      "report_code": "11013",
      "accept_dt": "20260501",
      "report_dt": "20260501",
      "rm": ""
    }
  ]
}
```

**Error Handling:**
- Queue for retry if 401/403 (certificate issues)
- Fallback to cache if 429 (rate limit)

---

## 🔒 Data Quality Rules by Source

### KRX Prices

**Completeness:**
- Every KOSPI/KOSDAQ stock must have OHLCV for every trading day
- No nulls allowed in: symbol, trade_date, close_price, volume

**Accuracy:**
- Prices must match official KRX reporting (daily reconciliation)
- Volume > 0 for liquid stocks (> 1000 shares/day)
- OHLC ordering: low ≤ open, close ≤ high

**Timeliness:**
- Published T+0 (end of business day)
- Ingested within 1 hour of market close

**Retention:** 5 years

---

### OpenDart Disclosures

**Completeness:**
- corp_code + filing_date must be non-null
- report_type must match FSS enum

**Accuracy:**
- Must match official FSS repository
- No synthetic/inferred filings

**Timeliness:**
- Published T+2 (regulatory requirement)

**Retention:** 7 years (regulatory)

---

### Portfolio (User Input)

**Completeness:**
- quantity ≥ 0
- cost_basis > 0 (if quantity > 0)
- acquisition_date ≤ today()

**Accuracy:**
- User responsibility; audit trail required
- Cross-check with broker statements monthly

**Timeliness:**
- Real-time (synchronous input)

**Retention:** 5 years

---

## 📈 Consumption Matrix

### Which Slices Consume Which Sources?

| Source | VS-01 | VS-02 | VS-03 | VS-04 | VS-05+ |
|--------|-------|-------|-------|-------|--------|
| KRX Prices | ✅ | ✅ | ✅ | ✅ | ✅ |
| OpenDart | ✅ | ⚪ | ⚪ | ⚪ | ✅ |
| Portfolio | ⚪ | ✅ | ⚪ | ✅ | ✅ |
| Shadow Run | ⚪ | ⚪ | ⚪ | ⚪ | ✅ |
| Audit Events | ✅ | ✅ | ✅ | ✅ | ✅ |

Legend: ✅ = Primary consumer, ⚪ = Secondary/Optional

---

## ⚠️ Failure Modes & Remediation

| Scenario | Detection | Mitigation | Recovery |
|----------|-----------|-----------|----------|
| **KRX API down** | 503 from endpoint | Use LKG prices (cache) | Retry next market day |
| **OpenDart rate limit** | 429 response | Queue for retry (Hangfire) | Exponential backoff |
| **Portfolio stale** | > 5 days since update | Alert user | Manual refresh |
| **Shadow run timeout** | Job > 1 day | Extend deadline | Resume from checkpoint |
| **Data quality fail** | DQ rule violation | Quarantine + alert | Manual review |

---

## 📚 References

- **KRX OpenAPI:** https://openapi.krx.co.kr (requires registration)
- **OpenDart API:** https://opendart.fss.or.kr
- **Data Contract:** `contracts/data/platform-data-contract.v1.json`
- **DQ Rules:** `docs/dq-lineage-rules.md`
- **Source Systems Table:** `audit.source_systems` (audit log)

---

---

## 🔑 KIS API (Korea Investment & Securities)

**Service:** Korea Investment & Securities Trading API  
**Base URL:** `https://openapivts.kbopenplatform.com` (KIS VTS) or `https://openapi.kbopenplatform.com`  
**Authentication:** `APP_KEY` + `APP_SECRET` (OAuth2, JWT)  
**Rate Limit:** 5000 req/minute (varies by tier)

**Endpoints Used:**

| Endpoint | Method | Purpose | Frequency |
|----------|--------|---------|-----------|
| `/uapi/trading-order` | POST | Place order | Real-time |
| `/uapi/trading-cancel-order` | POST | Cancel order | Real-time |
| `/uapi/domestic-stock-cash-daily` | GET | Account balance | Daily EOD |

**Authentication Flow:**
```
1. Get OAuth2 token: POST /oauth2/authorize + refresh_token
2. Call trading endpoint: X-APP-KEY + Authorization: Bearer <token>
3. Retry on 401: Refresh token if expired
```

**Fallback Strategy:**
- **Primary:** Live API
- **Secondary:** Last Known Good (LKG) state from DB
- **Tertiary:** Cached execution snapshot from previous day

---

## ⏱️ SLA & Retry Policy

### Service Level Agreements

| Source | Availability | Support Hours | Incident Contact | Escalation |
|--------|--------------|----------------|------------------|------------|
| **KRX** | 99.5% | Weekdays 9 AM-5 PM KST | `support@krx.co.kr` | → Operations Manager |
| **OpenDart** | 99.0% | Business hours only | FSS Helpdesk | → Data Governance Lead |
| **KIS** | 99.2% | 24/5 (trading hours) | `api-support@kimconsulting.com` | → Backend Lead |

### Error Classification & Retry

| Error | Classification | Retry Delay | Max Attempts | Action |
|-------|-----------------|------------|--------------|--------|
| **Network timeout** | Transient | 30s exponential backoff | 5 | Retry immediately |
| **429 (Rate limit)** | Transient | 60s + random jitter | 3 | Queue to Hangfire |
| **401 (Auth expired)** | Transient | Refresh token, retry | 2 | Obtain new credentials |
| **400 (Bad request)** | Permanent | None | 0 | Log error, alert ops |
| **503 (Service unavailable)** | Transient | 5min + exponential | 10 | Use fallback (cache) |
| **Data quality rule fail** | Permanent | None | 0 | Quarantine + manual review |

### Fallback & Recovery

**When Primary Source Fails:**
1. **KRX API down:** Use LKG prices from cache (up to 1 trading day old)
2. **OpenDart rate limit:** Queue job for retry (Hangfire q-backfill)
3. **KIS trading timeout:** Use cached balance, resume next market open
4. **Shadow run interrupted:** Resume from last checkpoint (idempotent)

---

## 📋 Data Retention Policy

| Source | Cold Storage | Archive Retention | Purge Policy |
|--------|--------------|-------------------|--------------|
| **KRX Prices** | After 2 years | 5 years (compliance) | After 5 years |
| **OpenDart** | After 3 years | 7 years (regulatory) | After 7 years |
| **KIS Trading** | After 1 year | 3 years (audit) | After 3 years |
| **Shadow Run** | Never | 10 years (evidence) | Never (immutable) |

---

**Owner:** Data Governance  
**Last Updated:** 2026-08-07 (AEG-X-009 Consolidated)  
**Status:** ✅ **APPROVED FOR OPERATIONS**