kjh2064
4059828abf
The SCP/DbMigrator deploy to the production target (178.104.200.7)
failed today with `relation "model_operations.models" does not
exist` at migration 0036 — the exact same failure I'd already hit
against a local test database, confirming this isn't environment
drift but a real, deterministic bug: no migration ever created
model_operations.models, only referenced it via FK (0036, 0038) and
queried it directly (OpenDartDailyBatchJob.cs). Migration 0037 had
the same class of bug for the `compliance` schema itself.
- Add 0035_model_operations_models.sql (must sort before 0036).
Scope is intentionally minimal — id/ticker/published_at/
correlation_id/revision, i.e. only what's actually referenced
today. The full Model Card/lifecycle schema is separate, larger
work and isn't guessed at here.
- Add `CREATE SCHEMA IF NOT EXISTS compliance;` to 0037, plus
IF NOT EXISTS on its indexes for re-run idempotency (matching the
rest of this migration set).
- Verified: full chain 0000->0040 applies to a fresh DB
("Upgrade successful") and re-run is a clean no-op
("No new scripts need to be executed").
Fixing the schema far enough to actually run queries against it
surfaced 3 more real, previously untested bugs in already-merged
code (none reachable before because the tables/schema didn't exist):
- Dapper was never configured for snake_case<->PascalCase column
mapping (`Dapper.DefaultTypeMap.MatchNamesWithUnderscores`), so
every Sql class's result-set queries were silently returning
null/default for every property instead of throwing. Fixed once,
centrally, via a `[ModuleInitializer]` in
KArtSell.BuildingBlocks/Data/DapperBootstrap.cs so it's set before
the first query regardless of entry point (Host/DbMigrator/tests).
- jsonb/inet columns written without an explicit cast
(`42804: column "x" is of type jsonb but expression is of type
text`) in AuditSql (details, ip_address), TradeSql (kis_response),
SellDecisionSql (oos_performance) — fixed with `::jsonb`/`::inet`
casts. AuditSql's jsonb read-back into Dictionary<string,object>
also needed a raw-DTO + JsonSerializer.Deserialize mapping.
- AuditSql.RedactAuditEventDetailsAsync had a literal duplicate
`SET details = ... details = ...` (invalid SQL) — nested the two
jsonb_set calls into one assignment.
Verified: dotnet build 0/0; architecture 13/13; unit 54/54+18/18;
Host boots cleanly and registers all 34 endpoints against the
now-complete schema.
New tech debt recorded: DEBT-020 (this fix), DEBT-021 (Dapper
snake_case fix), DEBT-022 (jsonb/inet casts, partial — not yet
audited beyond what surfaced), DEBT-023 (ApprovalSql.
InsertProposalAsync still fails on a raw DateOnly parameter — same
class of issue as DEBT-021, not yet fixed), DEBT-024 (TradeExecution
tests don't insert FK parent rows; one pure-logic ranker test
returned 1000 instead of 950 under the full suite, not yet
root-caused; DbUpMigrationTests fail locally on a Postgres role
permission gap unrelated to this fix).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
16 KiB
16 KiB
Tech Debt Register
Governance: Tracked per AGENTS.md v16.0. Quarterly paydown target: 20% by Impact.
Summary
| Status | Count | Total Impact |
|---|---|---|
| Backlog | 5 | 9 pts |
| In Progress | 0 | 0 pts |
| Completed | 2 | 3 pts |
| No Action | 1 | 1 pt |
| Deferred | 5 | 7 pts |
| Accepted | 1 | 2 pts |
Registry
Code Analysis Suppressions
| ID | Category | Impact | Effort | Status | Notes | Owner | ADR |
|---|---|---|---|---|---|---|---|
| DEBT-001 | CA1822 (static hints) | Low (1) | Low (1) | Completed | Applied static to GetNextDueAt, Evaluate, Plan methods; removed DI registrations. |
@claude | PR 4b |
| DEBT-002 | CA1873 (array logging) | Low (1) | Low (1) | No Action | Already compliant: all logging uses LoggerMessage delegates. Verified PR 4b build with CA1873 enabled: 0 warnings. | @claude | Verified |
| DEBT-003 | CA1305 (culture) | Low (1) | Low (1) | Deferred | Locale-specific formatting. Accept as-is for Serilog; breaking change if fixed. Revisit if conditions change. | @claude | PR 4d |
| DEBT-004 | CA1707 (test naming) | Low (1) | Low (1) | Deferred | xUnit underscores in test names. Convention; no fix needed. Revisit if conditions change. | @claude | PR 4d |
| DEBT-005 | CA1861 (array overhead) | Low (1) | Low (1) | Deferred | Static readonly array allocations. Negligible perf; accept trade-off for readability. Revisit if conditions change. | @claude | PR 4d |
| DEBT-006 | xUnit2031 (filter) | Low (1) | Low (1) | Deferred | Use overload instead of .Where() for Assert.Single. Analyzer nit; defer. Revisit if conditions change. | @claude | PR 4d |
Gate 3 Simplified Analytics (Deferred per v16.0)
| ID | Category | Impact | Effort | Status | Notes | Owner | ADR |
|---|---|---|---|---|---|---|---|
| DEBT-009 | PBO/Sharpe calculation | High (3) | High (3) | Backlog | MetricsCalculator.cs:148,170 use simplified percentile formulas. Need proper CSCV-based PBO and DSR methodology. Required for production Sharpe baseline. Gate 3 rehearsal will use simplified version; full implementation deferred to separate work. | @claude | Gate 3 Rehearsal Scope |
| DEBT-010 | Model prediction logic | High (3) | High (3) | Backlog | ReplayEngine.cs:90,163 predict fixed quantities (100 units). Need actual position-sizing algorithm. Required for realistic cost simulation. Gate 3 uses fixed quantities; full implementation deferred. | @claude | Gate 3 Rehearsal Scope |
| DEBT-011 | Cost 2x simulation | High (3) | High (3) | Backlog | ShadowRunJob.cs:132 uses linear approximation (TotalReturn * 0.5m). Need full re-simulation with actual fee/slippage impact. Required for realistic scenario analysis. Gate 3 uses linear model; full implementation deferred. | @claude | Gate 3 Rehearsal Scope |
| DEBT-012 | False-exit analysis | High (3) | High (3) | Backlog | ShadowRunJob.cs:136-139, FalseExitAnalyzer.cs always returns 0. Unimplemented feature. Required for accurate sell-reason attribution. Gate 3 rehearsal does not include false-exit analysis; deferred to separate work. | @claude | Gate 3 Rehearsal Scope |
| DEBT-013 | Credentials in appsettings | High (3) | Low (1) | Deferred | Host/tests appsettings.json contains plaintext DB password. Deferred: not in v16.0 scope. Revisit if security compliance requirements change. | @claude | Deferred |
| DEBT-014 | Duplicate & reconciliation tracking | Medium (2) | Medium (2) | Backlog | MetricsSql.cs GetDuplicateDetectionAsync/GetReconciliationBreaksAsync return null placeholders. Requires operation_audit_trail population by job consumers + OutboxPollerJob hooks. Non-blocking; dashboard degrades gracefully. | @claude | Observability Enhancement |
| DEBT-015 | Hangfire distributed lock timeout resilience | Medium (2) | High (3) | Completed | Applied consistent try/catch(Timeout) guard to all 6 Hangfire RecurringJob registrations: line 216 (RegisterModelOperationsSchedules), 260 (OpenDartDaily), 267 (DailyRecommendation), 273 (WeeklyRecommendation), 279 (MonthlyRecommendation). Prevents silent infinite wait; logs WARN and continues if lock times out. Resolves Host startup hangs when Hangfire schema initialization contentions occur. | @claude | PR Session commit 8b1c2f1 |
Deferred Refactoring
| ID | Category | Impact | Effort | Status | Notes | Owner | ADR |
|---|---|---|---|---|---|---|---|
| DEBT-007 | Newtonsoft.Json override | Medium (2) | Medium (2) | Completed | Fixed in 88ea5ed: CA1848/CA1859 actual implementation. LoggerMessage + HashSet/Dictionary. |
@claude | - |
| DEBT-008 | Namespace consistency | Medium (2) | Low (1) | Accepted | All projects use RootNamespace=KArtSell.Aegis; AssemblyName retained per-project for DLL clarity. Trade-off accepted: DLL clarity > namespace alignment. No action. | @claude | PR 4d |
| DEBT-016 | VS-02 mislabeled domain | Medium (2) | Low (1) | Backlog | Existing code VS02_SyncSecurityMasterEndpoint.cs, VS02_SecurityMasterJobs.cs, VS02_SecurityMasterPolicy.cs implement RBAC rule synchronization (access control), not financial security master data (listing/delisting/product structure). Dead code: endpoints disabled (DISABLED comment), schema security_master.rules table never migrated, never deployed. Correct domain documented in docs/CURRENT/SLICE_SPECS/VS-02-SLICE_SPEC.md (financial PIT). Removal decision deferred pending architect review (PR recommended). |
@claude | docs/CURRENT/SLICE_SPECS/VS-02-SLICE_SPEC.md |
| DEBT-017 | Duplicate VS-03 Approval Workflow implementation | High (3) | Medium (2) | Backlog | Two independent, functionally-identical VS-03 maker-checker slices exist: ApprovalWorkflow/ (Workstream H, own ApprovalProposal/IClock/IOutbox types) and Features/ApprovalWorkflow/ (Workstream G, matches documented Features/<Slice>/ convention). Both mapped the same routes (/approvals, /approvals/{id}, /approvals/{id}/approve), which crashed Host startup with a duplicate-route/missing-DI error the first time the app was actually booted (2026-08-07 — apparently never booted successfully before). Old set annotated [DontRegister] (FastEndpoints) 2026-08-07 to unblock boot; code and its test file (ApprovalWorkflowTests.cs) kept for now. Needs an architect decision: delete the old slice entirely (and its test) or intentionally keep both for a reason not yet documented. |
@claude | Session 2026-08-07 (Phase 3 J/K/L hardening) |
| DEBT-018 | Outbox write not co-transactional with entity write | Medium (2) | Medium (2) | Backlog | TradeExecution/TradeHandlers.cs (TradeOutboxPublisher) and PortfolioReconciliation/ReconcileTradeHandler.cs open a second, separate connection/transaction to write the outbox message after the trade/holding write already committed on its own connection. A crash between the two leaves the entity updated but no outbox event emitted (silent, non-atomic). Proper fix: thread a shared NpgsqlTransaction through TradeSql/ReconciliationSql mutation methods so entity insert + outbox insert commit together, matching DapperModelOperationRequestRepository's pattern. |
@claude | Session 2026-08-07 (Phase 3 J/K/L hardening) |
| DEBT-019 | Multiple duplicate cross-cutting abstractions (IClock, IOutboxWriter, IKrxDataService) |
Medium (2) | Low (1) | Completed (partial) | Found and collapsed 3 separate cases where a slice reinvented an abstraction that already existed in KArtSell.BuildingBlocks: a second IKrxDataService (deleted, ShadowRun.Services), a second IOutboxWriter/WriteAsync<T> in ReconcileTradeHandler.cs (removed, switched to BuildingBlocks.Reliability.IOutboxWriter), and a second IClock/SystemClock in ApprovalWorkflow/ApprovalPolicy.cs (removed, switched to BuildingBlocks.Time.IClock). Root cause: successive sessions implementing a slice without searching BuildingBlocks first. Recommend a pre-implementation checklist step ("does this abstraction already exist in BuildingBlocks?") for future slices. |
@claude | Session 2026-08-07 (Phase 3 J/K/L hardening) |
| DEBT-020 | model_operations.models and compliance schema never created by any migration |
High (3) | Low (1) | Completed | 0036/0038 reference model_operations.models(id) via FK and OpenDartDailyBatchJob.cs queries it directly, but no migration ever ran CREATE TABLE model_operations.models; 0037 wrote to compliance.* tables without CREATE SCHEMA compliance. Any fresh database — including the actual deploy target (178.104.200.7), confirmed via a live failed SCP/DbMigrator deploy on 2026-08-07 — failed at migration 0036/0037. Fixed via new 0035_model_operations_models.sql (minimal: id/ticker/published_at/correlation_id/revision only — full Model Card schema is separate future work) and CREATE SCHEMA IF NOT EXISTS compliance; added to 0037. Full chain 0000→0040 now verified fresh-install + idempotent re-run clean. |
@claude | Session 2026-08-07 (deploy failure triage) |
| DEBT-021 | Dapper never configured for snake_case↔PascalCase column mapping | High (3) | Low (1) | Completed | Dapper.DefaultTypeMap.MatchNamesWithUnderscores was never set anywhere in the codebase, so every QueryAsync<T>/QuerySingleOrDefaultAsync<T> result-mapping onto a snake_case DB column (e.g. event_type → EventType) silently returned null/default for that property instead of throwing — masking the bug in every Sql class across every module. Confirmed via ApprovalWorkflowTests.InsertAndRetrieveProposal_RoundTrips and AuditTrailTests.InsertAuditEvent_CreatesImmutableRecord both getting real rows back with null fields. Fixed centrally via a [ModuleInitializer] in KArtSell.BuildingBlocks/Data/DapperBootstrap.cs (runs once per process regardless of entry point — Host/DbMigrator/tests). |
@claude | Session 2026-08-07 (deploy failure triage) |
| DEBT-022 | jsonb/inet columns written as plain text without an explicit cast | Medium (2) | Low (1) | Completed (partial) | Dapper does not know to cast a string parameter to jsonb/inet for Npgsql; AuditSql.InsertAuditEventAsync (details, ip_address), AuditSql.RedactAuditEventDetailsAsync (duplicate SET details = assignment, separately fixed), TradeSql.InsertTradeAsync/UpdateTradeStatusAsync (kis_response), and SellDecisionSql.InsertDecisionAsync (oos_performance) all failed with 42804: column "x" is of type jsonb but expression is of type text the first time they were run against a real schema. Fixed with explicit ::jsonb/::inet casts at each call site (mechanical, no behavior change). AuditSql's jsonb read-back (Dictionary<string,object> from a jsonb column) also needed a raw-DTO + JsonSerializer.Deserialize mapping since Dapper has no built-in jsonb→Dictionary conversion either. Not yet checked: PortfolioReconciliation/ApprovalWorkflow Sql classes for the same pattern beyond what surfaced in this session's test runs — a full audit of jsonb/inet columns across all Sql classes is still open. |
@claude | Session 2026-08-07 (deploy failure triage) |
| DEBT-023 | ApprovalSql.InsertProposalAsync fails on DateOnly parameter |
Medium (2) | Low (1) | Backlog | ApprovalWorkflowTests.InsertAndRetrieveProposal_RoundTrips fails with System.NotSupportedException: The member effectiveAt of type System.DateOnly cannot be used as a parameter value — Dapper's LookupDbType doesn't recognize DateOnly without an explicit type map (SqlMapper.AddTypeMap/custom TypeHandler). Likely affects every other DateOnly-typed Dapper parameter in the codebase, not just this one; needs a similar centralized fix to DEBT-021 rather than a per-call-site patch. Discovered but not fixed in this session (scope cut to unblock the live deploy). |
@claude | Session 2026-08-07 (deploy failure triage) |
| DEBT-024 | New integration tests don't insert FK parent rows / one pure-logic test flakes under full-suite run | Low (1) | Low (1) | Backlog | TradeExecutionTests constructs Trade with a random sellDecisionId that was never inserted into sell_decisions, so every insert now correctly fails its FK constraint (trades_sell_decision_id_fkey) once the schema was actually complete (see DEBT-020) — test-only gap, not a production code defect; needs the tests updated to insert a parent models+sell_decisions row first. Separately, SellPriorityRankerTests.CalculateScore_HardImpairment_ReturnsLowestScore (pure logic, no DB) passed in isolation but returned 1000 instead of the expected 950 (age-boost not applied) when run as part of the full suite — not yet root-caused; may be test-order/parallelization state leakage rather than a SellPriorityRanker bug. Also, DbUpMigrationTests.* (pre-existing, unrelated to this session) fail locally with 42501: must be owner of database kartsell_migration_test — a local Postgres role permission gap, not a code issue. |
@claude | Session 2026-08-07 (deploy failure triage) |
Impact/Effort Matrix (Updated: PR 4)
Low Effort High Effort
High Impact QUICK WINS ROADMAP
(DEBT-007✓) (none currently)
Low Impact QUICK WINS MONITOR
(DEBT-001/002) (DEBT-003/004/005/006/008)
Quick Wins — Q3 2026 (Completed)
Rationale (per AGENTS.md v16.0 "Paydown Target: 20% quarterly"):
- ✅ DEBT-001 (CA1822): static method hints — Completed in PR 4b. Applied
staticto ScheduleOccurrencePlanner.GetNextDueAt, PromotionGateEvaluator.Evaluate, EvaluationWindowPlanner.Plan; removed unnecessary DI registrations (+1 pt). - ✅ DEBT-002 (CA1873): array logging — Already compliant: all logging uses LoggerMessage delegates. Verified in PR 4b build with CA1873 enabled: 0 warnings. No action needed (+0 pts, marked "No Action").
- Result: +1 pt resolved (25% of 4pt target). Target rate achievable by completing additional small-effort items from remaining backlog.
Batch During Feature Work
DEBT-001,DEBT-002— Moving to Quick Wins (PR 4 priority)
Monitor & Defer (No Action)
Rationale (per AGENTS.md "Keep in backlog; revisit if conditions change"):
- DEBT-003 (CA1305 culture): Locale formatting. Accept as-is for Serilog. Breaking change risk > benefit. Status: Permanently defer
- DEBT-004 (CA1707 test naming): xUnit convention (underscores). No fix needed; convention not a defect. Status: Permanently defer
- DEBT-005 (CA1861 array overhead): Static readonly arrays. Negligible perf; readability priority. Status: Permanently defer
- DEBT-006 (xUnit2031 filter): Assert.Single overload vs .Where(). Style preference, not safety. Status: Permanently defer
- DEBT-008 (Namespace consistency): Per-project AssemblyName intentional (DLL clarity). No action needed. Status: Accepted
Paydown Tracking
Q3 2026 (Current)
- Target: 20% of total impact resolved = 4 pts
- Completed: DEBT-007 (2 pts) — 50% of target achieved
- PR 4 Plan: DEBT-001 + DEBT-002 (2 pts) — Complete 100% of target ✅
- PR 4a: Evaluation & finalization (this commit)
- PR 4b: DEBT-001 — CA1822 static methods implementation
- PR 4c: DEBT-002 — CA1873 array logging optimization
- PR 4d: Permanent defer decisions for DEBT-003~006
Q4 2026
- Target: 20% = 4 pts (cumulative: 8 pts / 40% debt)
- Plan: TBD after Q3 completion
Q1 2027
- Target: 20% = 4 pts (cumulative: 12 pts / 60% debt)
- Plan: TBD
How to Resolve Tech Debt
- Identify: Find in this register or add new entry with Impact/Effort estimate
- Estimate: Low (1) / Medium (2) / High (3) for each dimension
- Schedule: Pick based on matrix above
- Implement: Separate PR, reference Debt ID in commit message (e.g.,
TECH-007: Fix CA1848) - Verify: Update register (move to Completed, record date + ADR link)
- Retrospective: Review in sprint retro; aim for 20% quarterly paydown