KArtSell.Aegis/docs/CURRENT/AEG-X-009_AUTOMATION_PROPOSAL.md

AEG-X-009 Data/Model Proposal Automation — Design Proposal

Status and traceability

• WBS: AEG-X-009
• Requirement: REQ-DATA-SOURCE
• Evidence class: SOURCE+DESIGN_PROPOSAL
• Status: DESIGN_PROPOSAL; not approved implementation
• Source: contracts/schedules/model-operations.v3.json, contracts/schedules/execution-assurance.v1.json, contracts/model-governance/evaluation-promotion.v2.json, src/KArtSell.BuildingBlocks/Versioning/VersionSet.cs, live read-only schema inspection on 2026-08-06
• Assumption: source ingestion and model evaluation are allowed to create immutable proposal/evidence records when their mode is EVALUATION_ONLY or PROPOSAL_ONLY.
• Unknown: approved source owners, source licenses/SLA values, model training implementation, retention period, and operator/secondary assignments.
• Decision Required: approve the proposal schema, job ownership, source allow-list, promotion review roles, and retention/alert contracts before implementation.

Non-negotiable boundary

Automation may:

1. discover and validate an approved source;
2. ingest immutable raw records and create a content-addressed dataset manifest;
3. run deterministic evaluation against a frozen server-side VersionSet;
4. create EvidenceSnapshot and a human-review proposal;
5. notify the maker/checker queue and expose status/metrics.

Automation must never:

• activate or promote a model;
• mutate thresholds, policy, configuration, or source code;
• rollback a model automatically;
• publish to clients;
• submit an order or KIS request.

Required state flow

SOURCE_CANDIDATE
  -> SOURCE_APPROVED (human owner + license/SLA/timezone/unit)
  -> INGESTION_EVALUATION_ONLY
  -> DATASET_QUARANTINED | DATASET_FROZEN
  -> MODEL_EVALUATION_ONLY
  -> EVIDENCE_SNAPSHOT_CREATED
  -> PROPOSAL_ONLY_REVIEW
  -> HUMAN_APPROVED | HUMAN_REJECTED | EXPIRED
  -> HUMAN_CHANGE_APPLIED (separate release, never by scheduler)

DATASET_QUARANTINED, missing evidence, hash mismatch, PIT violation, or VersionSet drift is a terminal hold for that run. It is not a retryable transient failure.

Required immutable records

Source catalog entry

source_id
source_version
owner / steward / secondary
license_reference
availability_sla / freshness_sla
timezone / calendar
unit / currency
schema_contract_version
approved_at / approved_by
status: CANDIDATE | APPROVED | SUSPENDED | RETIRED

Dataset manifest

Use the existing evaluation.dataset_manifest table. A row is eligible for evaluation only when:

status = FROZEN
dataset_id and content_hash are non-blank
source_catalog_version is approved
lineage_hash is present
frozen_at and approved_at are present
published_at/revision/PIT rules pass

Evaluation VersionSet

Use the existing VersionSet contract. It must be loaded server-side and contain:

DatasetId, DataHash, ModelVersion, ConfigVersion, CodeSha, ContractVersion

The client may submit scope and requested window only. The client must not submit evidence, hashes, model versions, or configuration versions as authoritative values.

Proposal packet

The proposal must reference, without copying or mutating, the EvidenceSnapshot and VersionSet. It must contain:

proposal_id / idempotency_key / scope_key / job_run_id
version_set / evidence_id / dataset_id / input_hash / output_hash
policy_id / policy_trace_schema_version / decision_contract_version
evaluation windows and metric definition versions
PBO / DSR / frozen OOS / double-cost / false-exit-reentry evidence
maker / checker / expiry / disposition

Existing schedule mapping

Do not add a new schedule until ADR/Issue approval. Use the existing contract entries as follows:

Existing job	Mode	Automated responsibility	Forbidden result
J25 SourceContractDriftCheck	EVALUATION_ONLY	detect source contract/license/SLA drift	no source activation
J26 MarketCalendarCompletenessCheck	EVALUATION_ONLY	detect calendar/timezone/unit gaps	no threshold mutation
J27 EvidenceChainAudit	EVALUATION_ONLY	validate lineage/hash/PIT chain	no evidence repair by overwrite
J28 ProjectionFreshnessCheck	EVALUATION_ONLY	validate read-model freshness	no client publication
J30 ReleaseEvidenceAssemble	PROPOSAL_ONLY	assemble a review packet	no release or activation

The missing business flow is not a new automatic promotion job. It is the contract and application boundary that creates a frozen dataset and proposal packet for the existing review process.

Repository catalog mapping

The following mapping is grounded in the current catalog and data contracts. It is a design mapping, not an authorization to ingest.

Domain	Current catalog/source	Current logical tables/contracts	Automation entry condition	Current status
Market data	KRX OpenAPI	market_data.prices, VS-03_DATA_CONTRACT.md	source approval + calendar/unit/SLA + PIT/hash checks	CANDIDATE
Corporate/fundamental data	OpenDart API	model_operations.disclosures, VS-05_DATA_CONTRACT.md	license/redistribution approval + filing schema/DQ	CANDIDATE
Portfolio	User input	portfolio.holdings, VS-04_DATA_CONTRACT.md	authenticated owner input + audit + PIT	CANDIDATE
Model operations	computed/evaluation output	evaluation.dataset_manifest, governance.model_version_registry, signal_engine.evidence_snapshot	frozen dataset and approved model/config/code contract	BLOCKED until seed/approval
Shadow evaluation	Hangfire/shadow run	model_operations.shadow_run, result/evidence contracts	server-side VersionSet + EVALUATION_ONLY capability	BLOCKED until VersionSet

The source catalog's logical table descriptions must be reconciled with active runtime SQL and the live schema before a migration or ingestion implementation. The catalog itself is not a substitute for runtime schema evidence.

Existing debt and decision linkage

This proposal directly addresses, but does not close, the following open items:

• TD-044: approved Dataset Manifest and Model Registry initial data absent;
• TD-063: total-return/delisting/corporate-action golden data incomplete;
• TD-099 / TD-105: market calendar/timezone source and SLA not approved;
• TD-132: current total-return source not approved;
• DEC-037, DEC-038, DEC-079: source/license/SLA and calendar ownership decisions required.

These items remain OPEN/DECISION_REQUIRED until their evidence is attached. No automation job may treat the catalog row as approved merely because the row exists.

Proposed WBS decomposition (proposal only)

These rows must be approved before being added to WBS_MASTER.csv:

Proposed ID	Scope	Acceptance evidence
AEG-X-009-P1	Source allow-list and approval record	unapproved source cannot enter ingestion
AEG-X-009-P2	Dataset manifest freeze command	same input produces same dataset/content hash; append-only
AEG-X-009-P3	Server-side VersionSet resolver	client-supplied evidence/version values ignored
AEG-X-009-P4	Evaluation/Proposal orchestration	idempotent JobRun/Watermark; modes fail closed
AEG-X-009-P5	Human review packet/API/UI	maker-checker, expiry, reject, audit trail
AEG-X-009-P6	Replay/failure/observability evidence	quarantine, replay hash, alert, runbook, rollback/stop evidence

Gate progression

Gate	Required before next gate
G0	contract, source owner, data semantics, WBS approval
G1	approved source catalog + isolated fresh/upgrade/re-run rehearsal
G2	frozen dataset + VersionSet resolver + golden/replay evidence
G3	evaluation-only execution and EvidenceSnapshot proof
G4	proposal packet + maker/checker review evidence
G5	separate human change approval; no scheduler activation

Immediate decision package

Before code or migration work, approve these six values explicitly:

1. source allow-list and owner/steward;
2. license, SLA, timezone, calendar, unit, and currency contracts;
3. dataset freeze status and retention policy;
4. model evaluation metric definition versions and population/window rules;
5. maker/checker roles and proposal expiry;
6. alert, stop, runbook, and secondary owner.

Until these are approved, the correct behavior is BLOCKED/QUARANTINED, not synthetic data/model creation.