fix: security, data-integrity, and doc-drift findings from repo audit
Consolidates duplicate KIS API client implementations (governance tests were exercising an unused class instead of the one actually running in production), closes a SQL injection path in the DB admin page, fixes a migration that used MySQL-only syntax and had never actually applied (confirmed against production), resyncs docs/db/quantengine.dbml with all migrations, and removes a duplicate OMS·WMS·ERP frontend tree in favor of src/frontend/. Also corrects several unverifiable/inflated claims in the OMS planning docs and realigns CI/CD and architecture documentation with what's actually in the repo. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,137 @@
|
||||
# QuantEngine Development Workflows & Common Scenarios
|
||||
|
||||
Full day-to-day workflow walkthroughs, extracted from CLAUDE.md (2026-07-30) to keep the main
|
||||
file within the character budget.
|
||||
|
||||
## Scenario 1: Day-to-Day Development (Code Change)
|
||||
|
||||
1. **Make code changes** (C# Razor Pages / .NET API / Python tools)
|
||||
2. **Local validation**:
|
||||
```powershell
|
||||
dotnet build src/dotnet/QuantEngine.Web/QuantEngine.Web.csproj -c Release
|
||||
dotnet test src/dotnet/QuantEngine.Core.Tests -c Release
|
||||
```
|
||||
3. **Test admin pages locally** (with SSH tunnel):
|
||||
```powershell
|
||||
ssh -L 127.0.0.1:5432:localhost:5432 kjh2064@178.104.200.7 -N &
|
||||
dotnet watch run --project QuantEngine.Web
|
||||
# Verify: /Admin/Dashboard, /Admin/Users, /Admin/Collection, etc. all return 200
|
||||
```
|
||||
4. **Commit & push**: Changes automatically trigger ci.yml
|
||||
- Core validators run first (blocking others)
|
||||
- Parallel validators (contracts, UI, DB, calibration) run independently
|
||||
- notify-results summarizes all 9 jobs in PR comment
|
||||
- Expected CI time: ~15-20min (was ~40min before 2026-07-24 refactor)
|
||||
|
||||
## Scenario 2: Data Collection Setup (KIS API Validation)
|
||||
|
||||
1. **Obtain KIS credentials** (real or mock account)
|
||||
2. **Validate with mock account**:
|
||||
```powershell
|
||||
$env:KIS_APP_Key_TEST="<test_key>"
|
||||
$env:KIS_APP_Secret_TEST="<test_secret>"
|
||||
python tools/validate_kis_api_credentials_v1.py --account mock --ticker 005930 --dry-run
|
||||
```
|
||||
3. **Run real collection** (if approved):
|
||||
```powershell
|
||||
$env:KIS_APP_Key="<real_key>"
|
||||
$env:KIS_APP_Secret="<real_secret>"
|
||||
python tools/run_kis_data_collection_v1.py --input-json GatherTradingData.json --sqlite-db src/quant_engine/kis_data_collection.db --output-json Temp/kis_data_collection_v1.json --kis-account real
|
||||
```
|
||||
4. **Verify database**:
|
||||
```sql
|
||||
SELECT COUNT(*) FROM kis_collection_runs;
|
||||
SELECT COUNT(*) FROM kis_collection_snapshots;
|
||||
```
|
||||
|
||||
## Scenario 3: Admin Data Editing (Snapshot Admin Web UI)
|
||||
|
||||
1. **Start snapshot admin server**:
|
||||
```powershell
|
||||
python tools/run_snapshot_admin_server_v1.py --host 127.0.0.1 --port 8787 --db src/quant_engine/snapshot_admin.db --seed GatherTradingData.json
|
||||
```
|
||||
2. **Access web UI**: http://127.0.0.1:8787
|
||||
3. **Edit settings / account_snapshot** in browser (like Excel)
|
||||
4. **Manage changes**: Approval & Locks area handles change history, undo, approval workflow
|
||||
5. **Export for CI**: `/api/export` → JSON or "Export approval packet" button
|
||||
|
||||
## Scenario 4: Release & Deployment (Multi-Stage)
|
||||
|
||||
**Stage 1: Local validation**
|
||||
```powershell
|
||||
npm run ops:validate # Warn-only (allow some issues)
|
||||
npm run full-gate # Strict (all gates PASS)
|
||||
```
|
||||
|
||||
**Stage 2: Create release** (manual via Gitea Actions)
|
||||
```
|
||||
→ Visit https://gitea.taxbaik.com/kjh2064/QuantEngineByItz/actions
|
||||
→ Run "prepare-release.yml" workflow_dispatch
|
||||
- Builds and publishes .NET
|
||||
- Creates git tag (e.g., quant_20260724.0.abc1234)
|
||||
- Generates Gitea Release with artifact + checksums
|
||||
- Packages as .tar.gz
|
||||
```
|
||||
|
||||
**Stage 3: Deploy** (manual, only after release exists)
|
||||
```
|
||||
→ Run "deploy-prod.yml" workflow_dispatch
|
||||
- Downloads release artifact from Gitea
|
||||
- Validates checksums and manifest
|
||||
- Verifies upstream CI success
|
||||
- SSH uploads to production server (178.104.200.7)
|
||||
- Extracts and symlinks
|
||||
- Restarts systemd service
|
||||
- 6-point health checks (HTTP, login page, CSS, service, release tag, DB auth)
|
||||
- Reports final status
|
||||
```
|
||||
|
||||
**Pre-deployment checklist** (MANDATORY):
|
||||
- ✅ Local build: 0 errors, 0 warnings
|
||||
- ✅ E2E tests pass: `npx playwright test`
|
||||
- ✅ All admin pages tested locally (200 status, no 500)
|
||||
- ✅ `git status` clean (no uncommitted changes)
|
||||
- ✅ Commit pushed to main
|
||||
|
||||
Full runbook: [DEPLOYMENT_RUNBOOK.md](DEPLOYMENT_RUNBOOK.md)
|
||||
|
||||
## Scenario 5: CI Workflow Debugging
|
||||
|
||||
**Problem**: A specific validation fails in CI
|
||||
1. Identify failing job from PR comment (notify-results output)
|
||||
2. Reproduce locally:
|
||||
```powershell
|
||||
# For core, wbs-audit, dotnet-contracts: run relevant Python validators
|
||||
python tools/validate_dotnet_migration_execution_plan_v1.py
|
||||
python tools/validate_dotnet_parity_contract_v1.py
|
||||
# etc.
|
||||
```
|
||||
3. Fix and re-push (triggers ci.yml again)
|
||||
4. Monitor in Gitea Actions dashboard
|
||||
|
||||
**Problem**: Workflow syntax error
|
||||
1. Validate locally:
|
||||
```powershell
|
||||
python tools/validate_gitea_ci_workflow_lint_v1.py --workflow .gitea/workflows/ci.yml
|
||||
```
|
||||
2. Fix YAML and test again
|
||||
|
||||
## Scenario 6: Database Schema Changes
|
||||
|
||||
1. **Create migration**: `src/dotnet/QuantEngine.Infrastructure/Migrations/V003.sql`
|
||||
2. **Update DBML**: `docs/db/quantengine.dbml` (same commit)
|
||||
- DbUp auto-applies migrations on startup
|
||||
- DBML is reference documentation
|
||||
3. **Test locally** (with SSH tunnel): Migrations must apply cleanly
|
||||
4. **Commit both** (SQL + DBML) together
|
||||
5. **CI validates**: ci.yml applies migrations to test PostgreSQL service
|
||||
|
||||
## When Things Break
|
||||
|
||||
| Issue | Root Cause | Fix |
|
||||
|-------|-----------|-----|
|
||||
| Admin page returns 500 | Likely unhandled DB exception or auth issue | Check journalctl, verify ConnectionStrings in production env |
|
||||
| KIS API fails with "not found" | Ticker doesn't exist in KIS | Use fallback (Naver → Yahoo → OpenDART) |
|
||||
| Snapshot admin won't load | SQLite DB corrupted or missing | Delete and re-seed from GatherTradingData.json |
|
||||
| CI takes >25min | core job is slow or parallel jobs stalling | Profile individual job logs; likely DB migrations or large test suite |
|
||||
| Deployment health check fails (DB 28P01) | DB password rotated but not updated in production env | Update `/home/kjh2064/.config/quantengine.env` on server only (not in repo) |
|
||||
Reference in New Issue
Block a user