KArtSell.Aegis/docs/CURRENT/CATALOGS/source-catalog.md

# Data Source Catalog

**Purpose:** Master reference for all data sources, APIs, and lineage  
**Owner:** Data Governance Team  
**Version:** 1.0  
**Date:** 2026-08-06

---

## 📊 Source Systems Summary

| Source | Type | Frequency | Availability SLA | Consumers | Retention |
|--------|------|-----------|------------------|-----------|-----------|
| **KRX OpenAPI** | External REST | Daily (T+0) | 99.5% | prices, signals, portfolio | 5 years |
| **OpenDart API** | External REST | T+2 | 99.0% | disclosure, models, recommendations | 7 years |
| **Portfolio (User Input)** | Internal Form | Real-time | 100% (manual) | rebalance, risk, holdings | 5 years |
| **Shadow Run Output** | Computed (Hangfire) | 252+ days | 99.9% | evidence, PBO/DSR, activation | 10 years |
| **Audit Events** | Internal Database | Real-time (write) | 99.99% | compliance, security, tracing | 7 years |

---

## 🔗 Data Lineage Map

### KRX Market Data Flow

```
┌─────────────────────────────────────────────────────────────┐
│                    KRX OpenAPI (External)                   │
│  Endpoint: /svc/apis/idx/krx_dd_trd, /svc/apis/sco/...     │
│  Auth: AUTH_KEY header                                       │
│  Frequency: Daily (T+0, end of business)                    │
└──────────────────────────────┬──────────────────────────────┘
                               │
                               ↓
┌──────────────────────────────────────────────────────────────┐
│           market_data.prices (PostgreSQL)                    │
│  Schema: price_id, symbol, trade_date, OHLCV, volume        │
│  PIT: published_at, correlation_id, revision                │
│  Validation: No nulls, volume ≥ 0, high ≥ low ≤ close      │
└──────────────────────────────┬───────────────────────────────┘
                               │
                    ┌──────────┴──────────┐
                    ↓                     ↓
        ┌────────────────────┐  ┌────────────────────┐
        │  signal_engine     │  │  portfolio.holdings│
        │   (Signals)        │  │   (Analysis)       │
        └────────┬───────────┘  └────────┬───────────┘
                 │                       │
                 └───────────┬───────────┘
                             ↓
                ┌────────────────────────┐
                │  sell_decision_engine  │
                │  (Final Output)        │
                └────────────────────────┘
```

### OpenDart Financial Disclosure Flow

```
┌──────────────────────────────────────────────────────────┐
│            OpenDart API (Financial Supervisory Service)  │
│  Endpoint: /api/list.json (공시정보, DS001)             │
│  Auth: crtfc_key (certificate key)                       │
│  Frequency: T+2 (regulatory reporting)                   │
└──────────────────────────┬───────────────────────────────┘
                           │
                           ↓
┌──────────────────────────────────────────────────────────┐
│      model_operations.disclosures (PostgreSQL)           │
│  Schema: filing_id, corp_code, report_type, filed_date  │
│  PIT: published_at, correlation_id, revision            │
│  Validation: Non-null corp_code, valid FSS report types │
└──────────────────────────┬───────────────────────────────┘
                           │
                           ↓
┌──────────────────────────────────────────────────────────┐
│        model_operations.models (Policy Input)            │
│  Lifecycle: Freeze→Mature→Score→...→ManualActivation   │
└──────────────────────────────────────────────────────────┘
```

### Shadow Run Batch Processing

```
┌─────────────────────────────────────┐
│    PHASE-1-SHADOW-RUN (Job 976)    │
│   Duration: 252+ trading days      │
│   Auto-runs (Hangfire)             │
└──────────────┬──────────────────────┘
               │
               ├─→ Input: models.* + prices.* + holdings.*
               │   (PIT-queried at cutoff dates)
               │
               └─→ Processing:
                   1. Load model (published_at ≤ cutoff)
                   2. Fetch price history (T to T+252 days)
                   3. Simulate rebalance decisions
                   4. Compute P&L metrics
                   5. Calculate OOS (out-of-sample) performance
                   6. Compute PBO/DSR evidence
               │
               ↓
┌─────────────────────────────────────┐
│   shadow_run_results (PostgreSQL)   │
│   Schema: job_id, model_id,         │
│   window_start, window_end,         │
│   pbo_score, dsr_score, oos_return  │
│   PIT: published_at, revision       │
└──────────────┬──────────────────────┘
               │
               ↓
┌─────────────────────────────────────┐
│  model_operations.models (Update)   │
│  Status: Review → ManualActivation  │
│  Attach: PBO/DSR evidence proof     │
└─────────────────────────────────────┘
```

---

## 📋 API Contract Details

### KRX OpenAPI

**Service:** Korea Exchange (KRX) Market Data  
**Base URL:** `https://openapi.krx.co.kr`  
**Authentication:** `AUTH_KEY` header  
**Rate Limit:** 1000 req/day (typical)

**Endpoints Used:**

| Endpoint | Method | Purpose | Frequency |
|----------|--------|---------|-----------|
| `/svc/apis/idx/krx_dd_trd` | POST | Index data (KOSPI, KOSDAQ) | Daily |
| `/svc/apis/sco/stk_bnd_isfl` | POST | Stock trading volume | Daily |

**Request Payload:**
```json
{
  "basDd": "20260801",
  "isuCd": "005930",
  "gubun": "ALL"
}
```

**Response Schema:**
```json
{
  "block_begin": "...",
  "OutBlock_1": [
    {
      "IDX_IND_CD": "KOSPI",
      "TRD_DD": "20260801",
      "CLSPRC_IDX": "2750.50",
      "OPNPRC_IDX": "2745.00",
      "HGPRC_IDX": "2760.00",
      "LWPRC_IDX": "2740.00",
      "ACC_TRDVOL": "1234567890"
    }
  ]
}
```

**Error Handling:**
- Transient: Retry with exponential backoff (3 attempts)
- Permanent: Log + alert + fallback to LKG (last-known-good)

---

### OpenDart API

**Service:** Financial Supervisory Service Disclosure  
**Base URL:** `https://opendart.fss.or.kr`  
**Authentication:** `crtfc_key` query parameter  
**Rate Limit:** 100 req/hour (typical)

**Endpoints Used:**

| Endpoint | Method | Purpose | Frequency |
|----------|--------|---------|-----------|
| `/api/list.json` | GET | Disclosure search | On-demand (T+2) |
| `/api/document.json` | GET | Document metadata | On-demand |

**Request Example:**
```
GET /api/list.json?crtfc_key=KEY&corp_code=00126380&bgn_de=20260101&end_de=20260831
```

**Response Schema:**
```json
{
  "status": "000",
  "message": "정상",
  "list": [
    {
      "corp_code": "00126380",
      "corp_name": "Samsung Electronics",
      "stock_code": "005930",
      "report_nm": "분기보고서",
      "report_code": "11013",
      "accept_dt": "20260501",
      "report_dt": "20260501",
      "rm": ""
    }
  ]
}
```

**Error Handling:**
- Queue for retry if 401/403 (certificate issues)
- Fallback to cache if 429 (rate limit)

---

## 🔒 Data Quality Rules by Source

### KRX Prices

**Completeness:**
- Every KOSPI/KOSDAQ stock must have OHLCV for every trading day
- No nulls allowed in: symbol, trade_date, close_price, volume

**Accuracy:**
- Prices must match official KRX reporting (daily reconciliation)
- Volume > 0 for liquid stocks (> 1000 shares/day)
- OHLC ordering: low ≤ open, close ≤ high

**Timeliness:**
- Published T+0 (end of business day)
- Ingested within 1 hour of market close

**Retention:** 5 years

---

### OpenDart Disclosures

**Completeness:**
- corp_code + filing_date must be non-null
- report_type must match FSS enum

**Accuracy:**
- Must match official FSS repository
- No synthetic/inferred filings

**Timeliness:**
- Published T+2 (regulatory requirement)

**Retention:** 7 years (regulatory)

---

### Portfolio (User Input)

**Completeness:**
- quantity ≥ 0
- cost_basis > 0 (if quantity > 0)
- acquisition_date ≤ today()

**Accuracy:**
- User responsibility; audit trail required
- Cross-check with broker statements monthly

**Timeliness:**
- Real-time (synchronous input)

**Retention:** 5 years

---

## 📈 Consumption Matrix

### Which Slices Consume Which Sources?

| Source | VS-01 | VS-02 | VS-03 | VS-04 | VS-05+ |
|--------|-------|-------|-------|-------|--------|
| KRX Prices | ✅ | ✅ | ✅ | ✅ | ✅ |
| OpenDart | ✅ | ⚪ | ⚪ | ⚪ | ✅ |
| Portfolio | ⚪ | ✅ | ⚪ | ✅ | ✅ |
| Shadow Run | ⚪ | ⚪ | ⚪ | ⚪ | ✅ |
| Audit Events | ✅ | ✅ | ✅ | ✅ | ✅ |

Legend: ✅ = Primary consumer, ⚪ = Secondary/Optional

---

## ⚠️ Failure Modes & Remediation

| Scenario | Detection | Mitigation | Recovery |
|----------|-----------|-----------|----------|
| **KRX API down** | 503 from endpoint | Use LKG prices (cache) | Retry next market day |
| **OpenDart rate limit** | 429 response | Queue for retry (Hangfire) | Exponential backoff |
| **Portfolio stale** | > 5 days since update | Alert user | Manual refresh |
| **Shadow run timeout** | Job > 1 day | Extend deadline | Resume from checkpoint |
| **Data quality fail** | DQ rule violation | Quarantine + alert | Manual review |

---

## 📚 References

- **KRX OpenAPI:** https://openapi.krx.co.kr (requires registration)
- **OpenDart API:** https://opendart.fss.or.kr
- **Data Contract:** `contracts/data/platform-data-contract.v1.json`
- **DQ Rules:** `docs/dq-lineage-rules.md`
- **Source Systems Table:** `audit.source_systems` (audit log)

---

**Owner:** Data Governance  
**Last Updated:** 2026-08-06  
**Status:** ✅ **APPROVED FOR OPERATIONS**