kjh2064
70824c2afb
Consolidates duplicate KIS API client implementations (governance tests were exercising an unused class instead of the one actually running in production), closes a SQL injection path in the DB admin page, fixes a migration that used MySQL-only syntax and had never actually applied (confirmed against production), resyncs docs/db/quantengine.dbml with all migrations, and removes a duplicate OMS·WMS·ERP frontend tree in favor of src/frontend/. Also corrects several unverifiable/inflated claims in the OMS planning docs and realigns CI/CD and architecture documentation with what's actually in the repo. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
108 lines
6.4 KiB
Markdown
108 lines
6.4 KiB
Markdown
# QuantEngine CI/CD Pipeline Structure
|
|
|
|
Full Gitea Actions workflow structure, extracted from CLAUDE.md (2026-07-30) to keep the main
|
|
file within the character budget.
|
|
|
|
## Workflow Architecture Refactoring (2026-07-24)
|
|
|
|
**2026-07-24 refactoring**: Single-job ci.yml (30+ steps, ~40min runtime) → **9-job parallel pipeline** (~15-20min runtime).
|
|
|
|
## CI Pipeline Jobs (ci.yml)
|
|
|
|
| Job | Dependencies | Purpose | Parallelizable |
|
|
|-----|--------------|---------|---|
|
|
| **core** | — | CRITICAL: .NET tests, API trading gate, KIS creds, DB migrations | ✗ (blocks others) |
|
|
| **wbs-audit** | core | WBS validation, platform migration, coverage audits | ✓ |
|
|
| **dotnet-contracts** | core | .NET parity, provenance, scheduler, normalization contracts | ✓ |
|
|
| **ui-storage** | — | Admin UI, storage backend, integration tests | ✓ |
|
|
| **database-schema** | — | DB pipeline, PostgreSQL schema, history contracts | ✓ |
|
|
| **calibration-pipeline** | core | Calibration priority, change ledger, qualitative sell strategy | ✓ |
|
|
| **operational-reporting** | calibration | Decision packet, operational report, performance metrics | ✗ (depends on calibration) |
|
|
| **security-validation** | — | Secrets contract, workflow validation | ✓ |
|
|
| **workflow-lint** | — | CI workflow structure, secrets contract | ✓ |
|
|
| **notify-results** | ALL | PR notification with job status summary | — |
|
|
|
|
**Dependency Graph**:
|
|
```
|
|
core ─┬─> wbs-audit ─────────────────────┐
|
|
├─> dotnet-contracts ─────────────┤
|
|
└─> calibration-pipeline ────────┤
|
|
└─> operational-reporting ─┤
|
|
└─> notify-results
|
|
ui-storage ────────────────────────────────┘
|
|
database-schema ──────────────────────────┘
|
|
security-validation ───────────────────────┘
|
|
workflow-lint ─────────────────────────────┘
|
|
```
|
|
|
|
## Other Workflow Files
|
|
|
|
| File | Trigger | Purpose | Status |
|
|
|------|---------|---------|--------|
|
|
| **kis_data_collection.yml** | cron (00:30 KST M-F) + dispatch | Validate KIS credentials & PostgreSQL pipeline | ✓ 2026-07-24 |
|
|
| **qualitative_sell_strategy.yml** | cron (00:15 KST M-F) + push + dispatch | Validate sell strategy pipeline & store | ✓ 2026-07-24 |
|
|
| **ci_lint.yml** | push (.gitea/workflows/) + dispatch | Lint all workflow files, validate job dependencies, secrets contract | ✓ 2026-07-24 |
|
|
| **snapshot_admin.yml** | push (snapshot_admin_*) + dispatch | Validate snapshot admin workflow & UI (2 jobs) | ✓ 2026-07-24 |
|
|
| **ci-frontend.yml** | push (main/master/feature/**) + PR | 8-step `src/frontend/` pipeline: install, typecheck, import-boundary lint, unit test, enterprise CRUD contract parity, Vite build, Playwright E2E, npm audit | ✓ (undocumented until 2026-07-30) |
|
|
| **t20_ledger.yml** | cron (17:00 KST M-F) + dispatch | Build `tools/build_operational_t20_outcome_ledger_v1.py` daily T+20 outcome ledger | ✓ (undocumented until 2026-07-30) |
|
|
| **prepare-release.yml** | workflow_run (ci.yml success) + dispatch | Build, tag, create Gitea Release with artifact + checksums | — |
|
|
| **deploy-prod.yml** | dispatch | Deploy release, run health checks, report status (3 jobs) | — |
|
|
|
|
**Note (2026-07-30)**: An earlier version of this table claimed `ci_lint.yml` had been renamed to
|
|
`workflow_lint.yml`. That rename was never actually carried out — the file on disk is still
|
|
`ci_lint.yml`. Corrected here after direct verification against `.gitea/workflows/`.
|
|
|
|
## Performance Improvements (2026-07-24)
|
|
|
|
**ci.yml refactoring results**:
|
|
- **Before**: 1 job, 30+ sequential steps, ~40min runtime
|
|
- **After**: 9 jobs, 7 in parallel, ~15-20min total runtime
|
|
- **Speedup**: ~2-2.5x faster CI feedback (core branch blocks only downstream, others parallel)
|
|
- **Fault isolation**: Single validation failure no longer blocks unrelated checks
|
|
|
|
**Key changes**:
|
|
1. **Setup consolidation**: Database migrations, Python, .NET setup in `core` job only
|
|
2. **Parallel validation groups**: 7 jobs run independently from core (ui-storage, database-schema, security-validation, workflow-lint, etc.)
|
|
3. **Dependency clarity**: `needs:` explicitly defines blocking relationships
|
|
4. **Error reporting**: `notify-results` summarizes all 9 job statuses in PR comment
|
|
|
|
## Workflow Maintenance Checklist
|
|
|
|
When modifying workflows (.gitea/workflows/*.yml):
|
|
|
|
1. ✅ Update `ci_lint.yml` if adding new triggers or job dependencies
|
|
2. ✅ Test locally with `python3 tools/validate_gitea_ci_workflow_lint_v1.py`
|
|
3. ✅ Verify all `needs:` references point to existing jobs
|
|
4. ✅ Document new jobs in this section above
|
|
5. ✅ Validate YAML syntax: `python3 -m yaml < .gitea/workflows/new.yml`
|
|
6. ✅ Ensure no hardcoded secrets in workflow files (env vars only)
|
|
|
|
## Troubleshooting Workflows
|
|
|
|
**Symptom**: CI job timeout
|
|
- **Check**: Does your job need PostgreSQL? Only `core` provides it; others must be independent.
|
|
- **Fix**: Add `services: postgres:` block or restructure to parallel-safe job.
|
|
|
|
**Symptom**: Cascading failure (multiple jobs fail)
|
|
- **Check**: Does your job have missing dependencies? Review `needs:` and dependency graph above.
|
|
- **Fix**: Add explicit `needs: [job_name]` if job depends on another's output.
|
|
|
|
**Symptom**: "job not found" error in notify-results
|
|
- **Check**: Job name typo in `notify-results.needs` list.
|
|
- **Fix**: Match job name exactly (case-sensitive).
|
|
|
|
## Workflow Trigger Schedule (2026-07-24)
|
|
|
|
| Time (KST) | Workflow | Trigger | Purpose |
|
|
|-----------|----------|---------|---------|
|
|
| 00:15 | qualitative_sell_strategy.yml | cron (M-F) | Validate sell strategy before daily operations |
|
|
| 00:30 | kis_data_collection.yml | cron (M-F) | Validate KIS API & DB pipeline before data collection |
|
|
| Push | ci.yml | on:push (main) | Validate code on every push to main |
|
|
| PR | ci.yml | on:pull_request | Gate PR merges with full validation suite |
|
|
| Manual | prepare-release.yml | workflow_dispatch | Create release tag & artifact |
|
|
| Manual | deploy-prod.yml | workflow_dispatch | Deploy release to production |
|
|
|
|
**Dependencies**:
|
|
- Release creation (prepare-release.yml) is gated by ci.yml success (workflow_run trigger)
|
|
- Deployment (deploy-prod.yml) is manual — only after release artifact exists
|