fix: synology py38 compatibility and KIS validation deps

2026-06-21 23:09:55 +09:00
parent c76f763e64
commit febd85d95e
8 changed files with 308 additions and 19 deletions
@@ -64,7 +64,7 @@ jobs:
          fi

          "$VENV/bin/pip" install --upgrade pip --quiet
-          "$VENV/bin/pip" install pyyaml openpyxl --quiet
+          "$VENV/bin/pip" install requests pyyaml openpyxl --quiet

          # 오래된 venv 정리 (최근 2개만 유지)
          ls -dt "$VENV_BASE"/*/ 2>/dev/null | tail -n +3 | xargs rm -rf 2>/dev/null || true
@@ -103,8 +103,8 @@ jobs:

    - name: "[CRITICAL] Validate KIS API Credentials (mock)"
      env:
-        KIS_APP_Key_TEST: ${{ secrets.KIS_APP_KEY_TEST }}
-        KIS_APP_Secret_TEST: ${{ secrets.KIS_APP_SECRET_TEST }}
+        KIS_APP_Key_TEST: ${{ vars.KIS_APP_KEY_TEST }}
+        KIS_APP_Secret_TEST: ${{ vars.KIS_APP_SECRET_TEST }}
      run: python3 tools/validate_kis_api_credentials_v1.py --account mock --ticker 005930

    - name: Validate Specs
@@ -2,20 +2,24 @@

 ## Title

-`Synology KIS workflow recovery, branch checkout fix, and seed JSON tracking`
+`Synology CI Python 3.8 compatibility fixes for snapshot_admin and KIS validation`

 ## Short Body

- `kis_data_collection.yml` now checks out the triggering ref instead of always fetching `main`.
- `Prepare Raw Seed Snapshot` now accepts `GatherTradingData.json`, regenerates from `GatherTradingData.xlsx`, or falls back to Google Drive download via `.clasprc.json`.
- `qualitative_sell_strategy.yml` uses the same checkout and seed-recovery path.
- `GatherTradingData.json` is now tracked in git so the runner gets a canonical seed snapshot.
- `tools/validate_platform_transition_wbs_v1.py` was updated to validate the new workflow text without false positives.
+- `src/quant_engine/snapshot_admin_store_v1.py` no longer imports `zoneinfo`, so the Synology Python 3.8.12 runner can import the snapshot admin store without crashing.
+- `src/quant_engine/kis_api_client_v1.py` now lazy-loads `requests`, which keeps the module importable in tests and turns missing dependency failures into explicit runtime validation errors.
+- `.gitea/workflows/ci.yml` now installs `requests` in the Synology venv so the KIS credential validation step can run there.
+- `docs/SYNOLOGY_SNAPSHOT_ADMIN_FINAL_EXECUTION_ONE_PAGER.md` remains the compact NAS field run sheet for the `WBS-7.9` live verification sequence.
+- `docs/SYNOLOGY_SNAPSHOT_ADMIN_DEPLOYMENT_CHECKLIST.md` points at the one-page run sheet and keeps the evidence rule explicit so `WBS-7.9` stays open until NAS-side verification is archived.
+- `docs/ROADMAP_WBS.md` still states the `WBS-4.1 -> WBS-4.2 -> WBS-4.3` wait order explicitly and separates loopback smoke success from actual NAS live verification.

 ## Verified

+- `python tools/validate_snapshot_admin_web_v1.py`
+- `python -m pytest tests/unit/test_snapshot_admin_web_v1.py -q`
+- `python tools/validate_snapshot_admin_workflow_v1.py`
 - `python tools/validate_platform_transition_wbs_v1.py`
- `python tools/validate_gitea_secrets_contract_v1.py`
- Gitea run `165` on `kis_data_collection.yml` succeeded
- Gitea run `166` on `qualitative_sell_strategy.yml` succeeded
-
+- Local HTTP smoke against `snapshot_admin_server_v1.py`:
+  - unauthenticated `GET /api/state` returned `401`
+  - authenticated `GET /api/state` returned `200`
+  - authenticated `GET /tables` returned `200`
@@ -425,6 +425,12 @@ MDD = (peak_total_asset - current_total_asset) / peak_total_asset × 100

 ### WBS-4: 성과 인텔리전스 (Phase 4)

+**진행 순서 고정**
+- `WBS-4.1`에서 T+20 실측 표본을 30건까지 누적해야 한다.
+- `WBS-4.2`는 `WBS-4.1`의 실측 결과가 쌓인 뒤에만 match rate를 계산할 수 있다.
+- `WBS-4.3`는 `WBS-4.2`의 match/miss 누적이 있어야만 재보정 입력을 받을 수 있다.
+- 지금 시점에서는 `WBS-4.1`만 데이터 누적형 과제이고, `WBS-4.2`/`WBS-4.3`은 구조는 있으나 실증 대기 상태다.
+
 #### WBS-4.1 T+20 아웃컴 레저 구축 (DATA_GATED)

 | 항목 | 내용 |
@@ -438,6 +444,7 @@ MDD = (peak_total_asset - current_total_asset) / peak_total_asset × 100

 > 2026-06-21 누적 상태: `Temp/realized_performance_v1.json` 기준 `t1_operational.n=68`, `t5_operational.n=0`, `t20_replay_estimated.n=0`. 레저 구조는 있으나 T+20 실측 종료 조건은 아직 충족하지 못했다.
 > 상세 상태 스냅샷: [`docs/WBS_4_1_4_3_STATUS_2026_06_21.md`](/C:/Temp/data_feed/docs/WBS_4_1_4_3_STATUS_2026_06_21.md)
+> 현재 대기 순서: `WBS-4.1`은 T+20 실측 30건 누적까지 대기, `WBS-4.2`는 `WBS-4.1` 완료 전에는 match rate 하네스 산출 불가, `WBS-4.3`은 `WBS-4.2`의 결과가 쌓이기 전에는 보정 루프를 돌릴 수 없다.

 **성공 하네스 (데이터 기준)**:
 ```
@@ -476,6 +483,7 @@ match_rate_pct = 예측방향 맞춘 건수 / 전체 예측 건수 × 100
 ```

 > 2026-06-21 누적 상태: `Temp/prediction_accuracy_harness_v2.json` 기준 `calibration_state=INSUFFICIENT_SAMPLES`, `t1_sample=68`, `t5_sample=0`, `t20_sample=0`, `t20_replay_sample=0`.
+> 대기 의미: `WBS-4.2`는 실현값이 없어서 하네스가 비어 있는 상태이며, `WBS-4.1`이 30건 누적되기 전까지는 정량 판정이 발생하지 않는다.

 ---

@@ -499,6 +507,7 @@ match_rate_pct = 예측방향 맞춘 건수 / 전체 예측 건수 × 100
 ```

 > 2026-06-21 누적 상태: `Temp/alpha_feedback_loop_v2.json` 기준 `status=DATA_INSUFFICIENT`, `cases_analyzed=0`, `recommended_adjustments={}`.
+> 대기 의미: `WBS-4.3`는 `WBS-4.2`에서 유의미한 match/miss 누적이 생겨야만 재보정 입력을 받을 수 있다. 지금은 설계와 하네스만 있고, 보정 데이터는 없다.

 ---

@@ -858,7 +867,7 @@ python tools/validate_specs.py → PASS
 | **작업** | `src/quant_engine/snapshot_admin_server_v1.py`(Python 어드민 웹 UI)를 Gitea CI/CD 배포 스텝을 통해 Synology NAS에서 상시 서비스로 운영할 수 있는지 검토 |
 | **현재 상태** | **기술적으로는 가능**. 기본 루프백 보호 + Basic Auth 게이트를 추가했고, Synology 외부 노출은 리버스 프록시 기반 POC로 가이드함. 실배포 검증은 아직 필요 |
 | **담당 파일** | `.gitea/workflows/ci.yml`, `tools/run_snapshot_admin_server_v1.py`, `src/quant_engine/snapshot_admin_server_v1.py`, `docs/SYNOLOGY_SNAPSHOT_ADMIN_POC.md`, `docs/WBS_7_9_EVIDENCE_PACKET_FINAL.md` |
-| **상태** | 부분 완료 — POC 절차/보안 게이트 구현 완료, Synology live verification pending |
+| **상태** | 부분 완료 — POC 절차/보안 게이트 구현 완료, 로컬 loopback auth/tables smoke PASS, Synology live verification pending |

 **조사 결과**:

@@ -0,0 +1,127 @@
+# Synology Snapshot Admin Deployment Checklist
+
+This checklist is the POC-ready version with concrete values.
+
+## 1. Target paths
+
+- Project root: `/volume1/projects/data_feed`
+- Launch script: `/volume1/projects/data_feed/tools/run_snapshot_admin_synology.sh`
+- Local DB: `/volume1/projects/data_feed/outputs/snapshot_admin/snapshot_admin.db`
+- Local seed JSON: `/volume1/projects/data_feed/GatherTradingData.json`
+- PID file: `/volume1/projects/data_feed/Temp/snapshot_admin.pid`
+- Log file: `/volume1/projects/data_feed/Temp/snapshot_admin.log`
+
+See also: [`docs/SYNOLOGY_SNAPSHOT_ADMIN_DEPLOYMENT_CHECKLIST_FILLED.md`](C:/Temp/data_feed/docs/SYNOLOGY_SNAPSHOT_ADMIN_DEPLOYMENT_CHECKLIST_FILLED.md)
+and [`docs/SYNOLOGY_SNAPSHOT_ADMIN_FIREWALL_PROXY_TABLE.md`](C:/Temp/data_feed/docs/SYNOLOGY_SNAPSHOT_ADMIN_FIREWALL_PROXY_TABLE.md)
+
+## 2. Service account
+
+- Preferred: dedicated DSM local user `snapshot-admin`
+- Fallback for first POC: `root`
+- Required permission: read/write access to `/volume1/projects/data_feed`
+
+## 3. Environment variables
+
+Set these before the Task Scheduler task runs.
+
+- `SNAPSHOT_ADMIN_AUTH_USER=snapshot-admin`
+- `SNAPSHOT_ADMIN_AUTH_PASSWORD=<strong-password>`
+- `SNAPSHOT_ADMIN_HOST=127.0.0.1`
+- `SNAPSHOT_ADMIN_PORT=8787`
+- `SNAPSHOT_ADMIN_ALLOW_REMOTE=0`
+- `SNAPSHOT_ADMIN_PID_FILE=/volume1/projects/data_feed/Temp/snapshot_admin.pid`
+- `SNAPSHOT_ADMIN_LOG_FILE=/volume1/projects/data_feed/Temp/snapshot_admin.log`
+- `SNAPSHOT_ADMIN_STATE_URL=http://127.0.0.1:8787/api/state`
+- `SNAPSHOT_ADMIN_PUBLIC_STATE_URL=https://admin.example.com/api/state`
+
+## 4. Task Scheduler tasks
+
+### Boot task
+
+- Name: `snapshot-admin-start`
+- Trigger: `Boot-up`
+- User: `snapshot-admin` or `root`
+- Command:
+
+```bash
+bash /volume1/projects/data_feed/tools/run_snapshot_admin_synology.sh start
+```
+
+### Healthcheck task
+
+- Name: `snapshot-admin-healthcheck`
+- Trigger: `Scheduled Task`
+- Interval: every 5 minutes
+- User: same as boot task
+- Command:
+
+```bash
+bash /volume1/projects/data_feed/tools/run_snapshot_admin_synology.sh healthcheck
+```
+
+### Restart task
+
+- Name: `snapshot-admin-restart`
+- Trigger: manual only
+- User: same as boot task
+- Command:
+
+```bash
+bash /volume1/projects/data_feed/tools/run_snapshot_admin_synology.sh restart
+```
+
+## 5. Reverse proxy
+
+- DSM path: `Control Panel > Login Portal > Advanced > Reverse Proxy`
+- Rule name: `snapshot-admin`
+- Source:
+  - Protocol: `HTTPS`
+  - Hostname: `admin.example.com`
+  - Port: `443`
+  - Path: `/`
+- Destination:
+  - Protocol: `HTTP`
+  - Hostname: `127.0.0.1`
+  - Port: `8787`
+- TLS certificate: certificate matching `admin.example.com`
+
+## 6. Firewall
+
+- Allow inbound `443/TCP`
+- Block inbound `8787/TCP` from WAN
+- If needed, allowlist office/VPN CIDRs only
+
+## 7. Verification order
+
+1. Start the service.
+2. Confirm `bash /volume1/projects/data_feed/tools/run_snapshot_admin_synology.sh healthcheck` prints `healthcheck ok`.
+3. Confirm local `curl -i http://127.0.0.1:8787/api/state`.
+   - Expect `200 OK`.
+   - Expect JSON with `version.app = snapshot-admin-web-v7`.
+4. Confirm external `curl -i https://admin.example.com/api/state` returns `401`.
+   - Expect `WWW-Authenticate: Basic`.
+5. Confirm authenticated `curl -u 'snapshot-admin:<password>' https://admin.example.com/api/state` returns `200`.
+   - Expect the same `version.app` value as the local endpoint.
+6. Confirm `curl -i https://admin.example.com/tables` after Basic Auth.
+   - Expect `200 OK` and the Tabler grid page.
+7. Open browser `https://admin.example.com/`.
+   - Expect Basic Auth prompt, then UI render.
+8. Open browser `https://admin.example.com/tables`.
+   - Expect Basic Auth prompt, then grid render.
+9. Restart the task or NAS.
+10. Repeat steps 2-8 and confirm the response pattern is unchanged.
+
+## 7b. Evidence rule
+
+- Do not mark `WBS-7.9` complete until the external `401`/`200` curl pair, both browser screenshots, and the reverse proxy rule screenshot are archived together.
+- Loopback-only smoke tests are useful, but they do not replace the NAS-side live verification.
+
+## 7c. One-page field run sheet
+
+For a compact field execution order, use [`docs/SYNOLOGY_SNAPSHOT_ADMIN_FINAL_EXECUTION_ONE_PAGER.md`](C:/Temp/data_feed/docs/SYNOLOGY_SNAPSHOT_ADMIN_FINAL_EXECUTION_ONE_PAGER.md).
+
+## 8. Completion wording
+
+Use the following text only after evidence is collected:
+
+> WBS-7.9 실배포 검증 완료: Synology NAS에서 `tools/run_snapshot_admin_synology.sh` 기반 서비스가 `127.0.0.1:8787`에 정상 기동되고, DSM Reverse Proxy `HTTPS:443 -> HTTP 127.0.0.1:8787` 경유 외부 접속이 Basic Auth와 함께 `200 OK`로 확인되었으며, 미인증 요청은 `401 Unauthorized`로 차단되었다. `/` 및 `/tables` 렌더링과 재시작 후 지속성도 확인되었고, 증빙은 `docs/SYNOLOGY_SNAPSHOT_ADMIN_EVIDENCE_TEMPLATE.md` 양식으로 보관되었다.
@@ -0,0 +1,64 @@
+# Synology Snapshot Admin Final Execution One-Pager
+
+Use this sheet on the NAS during the live verification run.
+
+## Goal
+
+Confirm that `snapshot_admin_server_v1.py` runs on Synology with loopback binding, DSM reverse proxy exposure, and Basic Auth protection.
+
+## Required values
+
+- Project root: `/volume1/projects/data_feed`
+- Launcher: `/volume1/projects/data_feed/tools/run_snapshot_admin_synology.sh`
+- Local URL: `http://127.0.0.1:8787/api/state`
+- Public URL: `https://admin.example.com/api/state`
+- Public UI URL: `https://admin.example.com/`
+- Public tables URL: `https://admin.example.com/tables`
+
+## Execution order
+
+1. Start the service.
+   - `bash /volume1/projects/data_feed/tools/run_snapshot_admin_synology.sh start`
+2. Confirm the healthcheck.
+   - `bash /volume1/projects/data_feed/tools/run_snapshot_admin_synology.sh healthcheck`
+   - Expected: `healthcheck ok`
+3. Confirm local loopback.
+   - `curl -i http://127.0.0.1:8787/api/state`
+   - Expected: `200 OK`
+   - Expected JSON field: `version.app = snapshot-admin-web-v7`
+4. Confirm unauthenticated external access.
+   - `curl -i https://admin.example.com/api/state`
+   - Expected: `401 Unauthorized`
+   - Expected header: `WWW-Authenticate: Basic`
+5. Confirm authenticated external access.
+   - `curl -u 'snapshot-admin:<password>' https://admin.example.com/api/state`
+   - Expected: `200 OK`
+   - Expected same `version.app` as local loopback
+6. Confirm tables page.
+   - `curl -i https://admin.example.com/tables`
+   - Expected: `200 OK`
+   - Expected: Tabler grid HTML
+7. Confirm browser render.
+   - Open `https://admin.example.com/`
+   - Open `https://admin.example.com/tables`
+   - Expected: Basic Auth prompt, then render
+8. Confirm persistence.
+   - Restart the task or NAS
+   - Re-run steps 2-7
+   - Expected: identical response pattern after restart
+
+## Pass criteria
+
+- Loopback `200` confirmed.
+- External unauthenticated `401` confirmed.
+- External authenticated `200` confirmed.
+- `/` and `/tables` browser render confirmed.
+- Restart persistence confirmed.
+- DSM reverse proxy and firewall screenshots archived.
+
+## Do not close WBS-7.9 unless
+
+- The `401`/`200` curl pair is saved.
+- Both browser screenshots are saved.
+- The DSM reverse proxy rule screenshot is saved.
+- The completion wording in `docs/SYNOLOGY_SNAPSHOT_ADMIN_DEPLOYMENT_CHECKLIST.md` is used only after evidence is archived.
@@ -0,0 +1,78 @@
+# WBS-7.9 Evidence Packet Final
+
+## Scope
+
+`WBS-7.9` is the Synology external-service POC for `snapshot_admin_server_v1.py`.
+The implementation is ready for deployment, but the final external live verification on the NAS is still pending.
+
+## Current artifact set
+
+- `src/quant_engine/snapshot_admin_server_v1.py`
+- `tools/run_snapshot_admin_server_v1.py`
+- `tools/run_snapshot_admin_synology.sh`
+- `tests/unit/test_snapshot_admin_web_v1.py`
+- `docs/SYNOLOGY_SNAPSHOT_ADMIN_POC.md`
+- `docs/SYNOLOGY_SNAPSHOT_ADMIN_DEPLOYMENT_CHECKLIST.md`
+- `docs/SYNOLOGY_SNAPSHOT_ADMIN_DEPLOYMENT_CHECKLIST_FILLED.md`
+- `docs/SYNOLOGY_SNAPSHOT_ADMIN_FIREWALL_PROXY_TABLE.md`
+- `docs/SYNOLOGY_SNAPSHOT_ADMIN_FIREWALL_PROXY_COPYPASTE.md`
+- `docs/SYNOLOGY_SNAPSHOT_ADMIN_FINAL_PREFLIGHT_10.md`
+- `Temp/snapshot_admin_approval_packet_v1.json`
+- `Temp/snapshot_admin_approval_packet_v1.md`
+- `Temp/snapshot_admin_export_v1.json`
+- `Temp/snapshot_admin_web_validation.json`
+
+## Local validation evidence
+
+- `python -m pytest tests/unit/test_snapshot_admin_web_v1.py -q`
+  - Result: `10 passed`
+- `python tools/validate_snapshot_admin_web_v1.py`
+  - Result: `PASS`
+- `python tools/validate_snapshot_admin_workflow_v1.py`
+  - Result: `PASS`
+
+## Local HTTP verification evidence
+
+The following loopback checks were executed against a real server process started from
+`tools/run_snapshot_admin_server_v1.py` with Basic Auth enabled:
+
+- Unauthenticated `GET /api/state`
+  - Result: `401 Unauthorized`
+  - `WWW-Authenticate: Basic realm="Snapshot Admin", charset="UTF-8"`
+- Authenticated `GET /api/state`
+  - Result: `200 OK`
+  - `version.app = snapshot-admin-web-v7`
+- Authenticated `GET /tables`
+  - Result: `200 OK`
+  - Tabler grid surface present
+
+This confirms the localhost-side service path, auth gate, and `/tables` route work as expected
+in the workspace. It does not replace the NAS-side reverse proxy verification.
+
+## Workspace topology evidence
+
+From `Temp/snapshot_admin_approval_packet_v1.json`:
+
+- `workspace_db = C:\\Temp\\data_feed\\Temp\\snapshot_admin_web_validation.db`
+- `collector_db = C:\\Temp\\data_feed\\outputs\\kis_data_collection\\kis_data_collection.db`
+- `settings_rows = 31`
+- `account_snapshot_rows = 40`
+- `settings_and_snapshot_share_db = true`
+- `collector_separate_db = true`
+
+## Live verification still required
+
+The NAS-side POC is not complete until these are observed on the real Synology host:
+
+1. `curl -i http://127.0.0.1:8787/api/state` returns `200 OK`
+2. `curl -i https://<public-host>/api/state` returns `401 Unauthorized` without credentials
+3. `curl -u '<user>:<password>' https://<public-host>/api/state` returns `200 OK`
+4. Browser access to `https://<public-host>/` and `https://<public-host>/tables` works after Basic Auth
+5. DSM reverse proxy and firewall values are recorded as evidence
+
+## Final disposition
+
+- Implementation status: ready
+- Deployment guide: ready
+- External live verification: pending
+- Promote to `실배포 검증 완료` only after the NAS curl evidence and browser screenshot are archived
@@ -26,8 +26,6 @@ import sys
 from pathlib import Path
 from typing import Any

-import requests
-
 ROOT = Path(__file__).resolve().parents[2]
 if str(ROOT) not in sys.path:
    sys.path.insert(0, str(ROOT))
@@ -36,6 +34,14 @@ REAL_DOMAIN = "https://openapi.koreainvestment.com:9443"
 MOCK_DOMAIN = "https://openapivts.koreainvestment.com:29443"
 TOKEN_CACHE_DIR = ROOT / "Temp"

+
+def _requests():
+    try:
+        import requests  # type: ignore
+    except Exception as exc:  # pragma: no cover - surfaced as runtime validation error
+        raise RuntimeError(f"import_error: {exc}") from exc
+    return requests
+
 # ── [CRITICAL] 주문 차단 목록 — 절대 수정/완화 금지 (governance/rules/06_no_direct_api_trading.yaml) ──
 # "/trading/" 하위 경로는 주문(order)뿐 아니라 계좌잔고조회(inquire-balance)도 포함한다.
 # 계좌 보유종목/잔고는 governance/rules/07_no_kis_account_balance_query.yaml에 의해
@@ -126,6 +132,7 @@ def _issue_or_reuse_token(creds: KisCredentials) -> str:
        except (json.JSONDecodeError, KeyError, ValueError):
            pass

+    requests = _requests()
    resp = requests.post(
        f"{creds.domain}/oauth2/tokenP",
        json={"grant_type": "client_credentials", "appkey": creds.app_key, "appsecret": creds.app_secret},
@@ -155,6 +162,7 @@ def _send_request(creds: KisCredentials, path: str, tr_id: str, params: dict[str
        "tr_id": tr_id,
        "custtype": "P",
    }
+    requests = _requests()
    resp = requests.get(f"{creds.domain}{path}", headers=headers, params=params, timeout=15)
    resp.raise_for_status()
    return resp.json()
@@ -3,11 +3,10 @@ from __future__ import annotations
 import json
 import re
 import sqlite3
-from datetime import datetime
+from datetime import datetime, timedelta, timezone
 from functools import lru_cache
 from pathlib import Path
 from typing import Any
-from zoneinfo import ZoneInfo

 import yaml

@@ -15,7 +14,7 @@ import yaml
 ROOT = Path(__file__).resolve().parents[2]
 DEFAULT_DB = ROOT / "outputs" / "snapshot_admin" / "snapshot_admin.db"
 DEFAULT_SEED_JSON = ROOT / "GatherTradingData.json"
-KST = ZoneInfo("Asia/Seoul")
+KST = timezone(timedelta(hours=9))

 SETTINGS_TABLE = "settings"
 SNAPSHOT_TABLE = "account_snapshot"