KArtSell.Aegis/docs/CURRENT/AEG-X-009_AUTOMATION_PROPOSAL.md

# AEG-X-009 Data/Model Proposal Automation — Design Proposal

## Status and traceability

- WBS: `AEG-X-009`
- Requirement: `REQ-DATA-SOURCE`
- Evidence class: `SOURCE+DESIGN_PROPOSAL`
- Status: `DESIGN_PROPOSAL`; not approved implementation
- Source: `contracts/schedules/model-operations.v3.json`, `contracts/schedules/execution-assurance.v1.json`, `contracts/model-governance/evaluation-promotion.v2.json`, `src/KArtSell.BuildingBlocks/Versioning/VersionSet.cs`, live read-only schema inspection on 2026-08-06
- Assumption: source ingestion and model evaluation are allowed to create immutable proposal/evidence records when their mode is `EVALUATION_ONLY` or `PROPOSAL_ONLY`.
- Unknown: approved source owners, source licenses/SLA values, model training implementation, retention period, and operator/secondary assignments.
- Decision Required: approve the proposal schema, job ownership, source allow-list, promotion review roles, and retention/alert contracts before implementation.

## Non-negotiable boundary

Automation may:

1. discover and validate an approved source;
2. ingest immutable raw records and create a content-addressed dataset manifest;
3. run deterministic evaluation against a frozen server-side VersionSet;
4. create EvidenceSnapshot and a human-review proposal;
5. notify the maker/checker queue and expose status/metrics.

Automation must never:

- activate or promote a model;
- mutate thresholds, policy, configuration, or source code;
- rollback a model automatically;
- publish to clients;
- submit an order or KIS request.

## Required state flow

```text
SOURCE_CANDIDATE
  -> SOURCE_APPROVED (human owner + license/SLA/timezone/unit)
  -> INGESTION_EVALUATION_ONLY
  -> DATASET_QUARANTINED | DATASET_FROZEN
  -> MODEL_EVALUATION_ONLY
  -> EVIDENCE_SNAPSHOT_CREATED
  -> PROPOSAL_ONLY_REVIEW
  -> HUMAN_APPROVED | HUMAN_REJECTED | EXPIRED
  -> HUMAN_CHANGE_APPLIED (separate release, never by scheduler)
```

`DATASET_QUARANTINED`, missing evidence, hash mismatch, PIT violation, or VersionSet drift is a terminal hold for that run. It is not a retryable transient failure.

## Required immutable records

### Source catalog entry

```text
source_id
source_version
owner / steward / secondary
license_reference
availability_sla / freshness_sla
timezone / calendar
unit / currency
schema_contract_version
approved_at / approved_by
status: CANDIDATE | APPROVED | SUSPENDED | RETIRED
```

### Dataset manifest

Use the existing `evaluation.dataset_manifest` table. A row is eligible for evaluation only when:

```text
status = FROZEN
dataset_id and content_hash are non-blank
source_catalog_version is approved
lineage_hash is present
frozen_at and approved_at are present
published_at/revision/PIT rules pass
```

### Evaluation VersionSet

Use the existing `VersionSet` contract. It must be loaded server-side and contain:

```text
DatasetId, DataHash, ModelVersion, ConfigVersion, CodeSha, ContractVersion
```

The client may submit scope and requested window only. The client must not submit evidence, hashes, model versions, or configuration versions as authoritative values.

### Proposal packet

The proposal must reference, without copying or mutating, the EvidenceSnapshot and VersionSet. It must contain:

```text
proposal_id / idempotency_key / scope_key / job_run_id
version_set / evidence_id / dataset_id / input_hash / output_hash
policy_id / policy_trace_schema_version / decision_contract_version
evaluation windows and metric definition versions
PBO / DSR / frozen OOS / double-cost / false-exit-reentry evidence
maker / checker / expiry / disposition
```

## Existing schedule mapping

Do not add a new schedule until ADR/Issue approval. Use the existing contract entries as follows:

| Existing job | Mode | Automated responsibility | Forbidden result |
|---|---|---|---|
| J25 SourceContractDriftCheck | EVALUATION_ONLY | detect source contract/license/SLA drift | no source activation |
| J26 MarketCalendarCompletenessCheck | EVALUATION_ONLY | detect calendar/timezone/unit gaps | no threshold mutation |
| J27 EvidenceChainAudit | EVALUATION_ONLY | validate lineage/hash/PIT chain | no evidence repair by overwrite |
| J28 ProjectionFreshnessCheck | EVALUATION_ONLY | validate read-model freshness | no client publication |
| J30 ReleaseEvidenceAssemble | PROPOSAL_ONLY | assemble a review packet | no release or activation |

The missing business flow is not a new automatic promotion job. It is the contract and application boundary that creates a frozen dataset and proposal packet for the existing review process.

## Repository catalog mapping

The following mapping is grounded in the current catalog and data contracts. It is a design mapping, not an authorization to ingest.

| Domain | Current catalog/source | Current logical tables/contracts | Automation entry condition | Current status |
|---|---|---|---|---|
| Market data | KRX OpenAPI | `market_data.prices`, `VS-03_DATA_CONTRACT.md` | source approval + calendar/unit/SLA + PIT/hash checks | CANDIDATE |
| Corporate/fundamental data | OpenDart API | `model_operations.disclosures`, `VS-05_DATA_CONTRACT.md` | license/redistribution approval + filing schema/DQ | CANDIDATE |
| Portfolio | User input | `portfolio.holdings`, `VS-04_DATA_CONTRACT.md` | authenticated owner input + audit + PIT | CANDIDATE |
| Model operations | computed/evaluation output | `evaluation.dataset_manifest`, `governance.model_version_registry`, `signal_engine.evidence_snapshot` | frozen dataset and approved model/config/code contract | BLOCKED until seed/approval |
| Shadow evaluation | Hangfire/shadow run | `model_operations.shadow_run`, result/evidence contracts | server-side VersionSet + EVALUATION_ONLY capability | BLOCKED until VersionSet |

The source catalog's logical table descriptions must be reconciled with active runtime SQL and the live schema before a migration or ingestion implementation. The catalog itself is not a substitute for runtime schema evidence.

## Existing debt and decision linkage

This proposal directly addresses, but does not close, the following open items:

- `TD-044`: approved Dataset Manifest and Model Registry initial data absent;
- `TD-063`: total-return/delisting/corporate-action golden data incomplete;
- `TD-099` / `TD-105`: market calendar/timezone source and SLA not approved;
- `TD-132`: current total-return source not approved;
- `DEC-037`, `DEC-038`, `DEC-079`: source/license/SLA and calendar ownership decisions required.

These items remain OPEN/DECISION_REQUIRED until their evidence is attached. No automation job may treat the catalog row as approved merely because the row exists.

## Proposed WBS decomposition (proposal only)

These rows must be approved before being added to `WBS_MASTER.csv`:

| Proposed ID | Scope | Acceptance evidence |
|---|---|---|
| AEG-X-009-P1 | Source allow-list and approval record | unapproved source cannot enter ingestion |
| AEG-X-009-P2 | Dataset manifest freeze command | same input produces same dataset/content hash; append-only |
| AEG-X-009-P3 | Server-side VersionSet resolver | client-supplied evidence/version values ignored |
| AEG-X-009-P4 | Evaluation/Proposal orchestration | idempotent JobRun/Watermark; modes fail closed |
| AEG-X-009-P5 | Human review packet/API/UI | maker-checker, expiry, reject, audit trail |
| AEG-X-009-P6 | Replay/failure/observability evidence | quarantine, replay hash, alert, runbook, rollback/stop evidence |

## Gate progression

| Gate | Required before next gate |
|---|---|
| G0 | contract, source owner, data semantics, WBS approval |
| G1 | approved source catalog + isolated fresh/upgrade/re-run rehearsal |
| G2 | frozen dataset + VersionSet resolver + golden/replay evidence |
| G3 | evaluation-only execution and EvidenceSnapshot proof |
| G4 | proposal packet + maker/checker review evidence |
| G5 | separate human change approval; no scheduler activation |

## Immediate decision package

Before code or migration work, approve these six values explicitly:

1. source allow-list and owner/steward;
2. license, SLA, timezone, calendar, unit, and currency contracts;
3. dataset freeze status and retention policy;
4. model evaluation metric definition versions and population/window rules;
5. maker/checker roles and proposal expiry;
6. alert, stop, runbook, and secondary owner.

Until these are approved, the correct behavior is `BLOCKED`/`QUARANTINED`, not synthetic data/model creation.