Compare commits

...

4 Commits

Author SHA1 Message Date
kjh2064 8ed2bcf56f fix: Dapper underscore-mapping race condition affects AuditSql too, not just TradeSql
The static-ctor guard added to TradeSql in the previous commit was a
symptom fix. Confirmed the same bug independently affects AuditSql:
running the Compliance test filter in isolation (no other class that
happens to touch a BuildingBlocks type first) reproduced the identical
failure mode - every snake_case column (event_type, purge_status, ...)
silently mapped to null.

Root cause: KArtSell.BuildingBlocks.Data.DapperBootstrap's
[ModuleInitializer] only runs once that assembly is actually loaded,
and a `using` directive for a BuildingBlocks namespace does not force
that load - only an executed reference to one of its types does. Any
Sql class that never actually touches a BuildingBlocks type at runtime
is exposed, and this is a property of *when* a given test/request
happens to run relative to everything else in the process, not of any
one class.

Replaced the ad-hoc TradeSql static ctor with one [ModuleInitializer]
per module assembly (KArtSell.Modules.ModelOperations,
KArtSell.Modules.SignalEngine). Every Sql/reader class lives inside its
own module's assembly, so a module initializer there is guaranteed to
run before any of them are used, independent of BuildingBlocks or
load order. Verified both KArtSell.Integration.Tests.Compliance and
.TradeExecution now pass 100% run in full isolation, not just as part
of the full suite.

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
2026-08-07 23:36:08 +09:00
kjh2064 2ccf74c410 fix: Release build breakage + Dapper mapping bugs in VS-03/VS-04/Phase3-K
- KArtSell.Host.csproj: FrontendFiles glob was evaluated at project-load
  time, before pnpm build ran, so it copied stale/missing Vite-hashed
  filenames every Release build. Move the glob inside the target, after
  the build Exec.
- ApprovalSql/AuditSql/TradeSql: fix live-DB integration failures never
  caught by unit tests: DateOnly and inet columns can't be bound/read
  directly through Dapper without conversion; kis_response (jsonb) read
  as JsonElement threw InvalidCastException; GdprRetention.RetentionEndsAt
  was typed DateTime against a DATE column.
- TradeSql: UpdateTradeStatusAsync only ever persisted status/kis_response
  /error_message, silently dropping kis_order_id, executed_quantity,
  unit_price, total_amount, commission, net_proceeds and the execution/
  settlement timestamps on every call. Changed it to take the Trade
  aggregate so the full state transition persists.
- TradeSql: add a static ctor setting Dapper.DefaultTypeMap.
  MatchNamesWithUnderscores = true. The repo's [ModuleInitializer] in
  KArtSell.BuildingBlocks only fires once that assembly is actually
  loaded; TradeSql/Trade never reference a BuildingBlocks type, so under
  test isolation (or any host that queries a trade before touching
  BuildingBlocks) every snake_case column silently mapped to null/default.
- Test fixes: seed the FK prerequisites (model_operations.models,
  sell_decisions) that ApprovalWorkflowTests/TradeExecutionTests were
  missing, correct a SellPriorityRanker test input to match the approved
  VS-10-SLICE_SPEC age-boost threshold, and fix a GDPR redaction
  assertion that called ToString() on a Dictionary instead of inspecting
  its values.

12 DbUpMigrationTests failures remain and are unrelated to this fix: the
kartsell DB user isn't the owner of kartsell_migration_test, so DbUp's
fresh-database rehearsal can't DROP/CREATE it. Needs a DBA grant.

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
2026-08-07 23:21:04 +09:00
kjh2064 54b7922167 Merge pull request 'fix: missing model_operations.models table + compliance schema breaking every fresh DB (live deploy failure)' (#29) from feat/L-vs14-portfolio-reconciliation into main
deploy / deploy (push) Successful in 1m38s
deploy / notify (push) Successful in 0s
2026-08-07 20:26:21 +09:00
kjh2064 4059828abf fix: missing model_operations.models table + compliance schema breaking every fresh DB (live deploy failure)
The SCP/DbMigrator deploy to the production target (178.104.200.7)
failed today with `relation "model_operations.models" does not
exist` at migration 0036 — the exact same failure I'd already hit
against a local test database, confirming this isn't environment
drift but a real, deterministic bug: no migration ever created
model_operations.models, only referenced it via FK (0036, 0038) and
queried it directly (OpenDartDailyBatchJob.cs). Migration 0037 had
the same class of bug for the `compliance` schema itself.

- Add 0035_model_operations_models.sql (must sort before 0036).
  Scope is intentionally minimal — id/ticker/published_at/
  correlation_id/revision, i.e. only what's actually referenced
  today. The full Model Card/lifecycle schema is separate, larger
  work and isn't guessed at here.
- Add `CREATE SCHEMA IF NOT EXISTS compliance;` to 0037, plus
  IF NOT EXISTS on its indexes for re-run idempotency (matching the
  rest of this migration set).
- Verified: full chain 0000->0040 applies to a fresh DB
  ("Upgrade successful") and re-run is a clean no-op
  ("No new scripts need to be executed").

Fixing the schema far enough to actually run queries against it
surfaced 3 more real, previously untested bugs in already-merged
code (none reachable before because the tables/schema didn't exist):

- Dapper was never configured for snake_case<->PascalCase column
  mapping (`Dapper.DefaultTypeMap.MatchNamesWithUnderscores`), so
  every Sql class's result-set queries were silently returning
  null/default for every property instead of throwing. Fixed once,
  centrally, via a `[ModuleInitializer]` in
  KArtSell.BuildingBlocks/Data/DapperBootstrap.cs so it's set before
  the first query regardless of entry point (Host/DbMigrator/tests).
- jsonb/inet columns written without an explicit cast
  (`42804: column "x" is of type jsonb but expression is of type
  text`) in AuditSql (details, ip_address), TradeSql (kis_response),
  SellDecisionSql (oos_performance) — fixed with `::jsonb`/`::inet`
  casts. AuditSql's jsonb read-back into Dictionary<string,object>
  also needed a raw-DTO + JsonSerializer.Deserialize mapping.
- AuditSql.RedactAuditEventDetailsAsync had a literal duplicate
  `SET details = ... details = ...` (invalid SQL) — nested the two
  jsonb_set calls into one assignment.

Verified: dotnet build 0/0; architecture 13/13; unit 54/54+18/18;
Host boots cleanly and registers all 34 endpoints against the
now-complete schema.

New tech debt recorded: DEBT-020 (this fix), DEBT-021 (Dapper
snake_case fix), DEBT-022 (jsonb/inet casts, partial — not yet
audited beyond what surfaced), DEBT-023 (ApprovalSql.
InsertProposalAsync still fails on a raw DateOnly parameter — same
class of issue as DEBT-021, not yet fixed), DEBT-024 (TradeExecution
tests don't insert FK parent rows; one pure-logic ranker test
returned 1000 instead of 950 under the full suite, not yet
root-caused; DbUpMigrationTests fail locally on a Postgres role
permission gap unrelated to this fix).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-07 20:23:56 +09:00
18 changed files with 266 additions and 112 deletions
+5
View File
@@ -52,6 +52,11 @@
| DEBT-017 | Duplicate VS-03 Approval Workflow implementation | High (3) | Medium (2) | Backlog | Two independent, functionally-identical VS-03 maker-checker slices exist: `ApprovalWorkflow/` (Workstream H, own `ApprovalProposal`/`IClock`/`IOutbox` types) and `Features/ApprovalWorkflow/` (Workstream G, matches documented `Features/<Slice>/` convention). Both mapped the same routes (`/approvals`, `/approvals/{id}`, `/approvals/{id}/approve`), which crashed Host startup with a duplicate-route/missing-DI error the first time the app was actually booted (2026-08-07 — apparently never booted successfully before). Old set annotated `[DontRegister]` (FastEndpoints) 2026-08-07 to unblock boot; code and its test file (`ApprovalWorkflowTests.cs`) kept for now. Needs an architect decision: delete the old slice entirely (and its test) or intentionally keep both for a reason not yet documented. | @claude | Session 2026-08-07 (Phase 3 J/K/L hardening) |
| DEBT-018 | Outbox write not co-transactional with entity write | Medium (2) | Medium (2) | Backlog | `TradeExecution/TradeHandlers.cs` (`TradeOutboxPublisher`) and `PortfolioReconciliation/ReconcileTradeHandler.cs` open a second, separate connection/transaction to write the outbox message after the trade/holding write already committed on its own connection. A crash between the two leaves the entity updated but no outbox event emitted (silent, non-atomic). Proper fix: thread a shared `NpgsqlTransaction` through `TradeSql`/`ReconciliationSql` mutation methods so entity insert + outbox insert commit together, matching `DapperModelOperationRequestRepository`'s pattern. | @claude | Session 2026-08-07 (Phase 3 J/K/L hardening) |
| DEBT-019 | Multiple duplicate cross-cutting abstractions (`IClock`, `IOutboxWriter`, `IKrxDataService`) | Medium (2) | Low (1) | Completed (partial) | Found and collapsed 3 separate cases where a slice reinvented an abstraction that already existed in `KArtSell.BuildingBlocks`: a second `IKrxDataService` (deleted, `ShadowRun.Services`), a second `IOutboxWriter`/`WriteAsync<T>` in `ReconcileTradeHandler.cs` (removed, switched to `BuildingBlocks.Reliability.IOutboxWriter`), and a second `IClock`/`SystemClock` in `ApprovalWorkflow/ApprovalPolicy.cs` (removed, switched to `BuildingBlocks.Time.IClock`). Root cause: successive sessions implementing a slice without searching `BuildingBlocks` first. Recommend a pre-implementation checklist step ("does this abstraction already exist in BuildingBlocks?") for future slices. | @claude | Session 2026-08-07 (Phase 3 J/K/L hardening) |
| DEBT-020 | `model_operations.models` and `compliance` schema never created by any migration | High (3) | Low (1) | Completed | `0036`/`0038` reference `model_operations.models(id)` via FK and `OpenDartDailyBatchJob.cs` queries it directly, but no migration ever ran `CREATE TABLE model_operations.models`; `0037` wrote to `compliance.*` tables without `CREATE SCHEMA compliance`. Any fresh database — including the actual deploy target (178.104.200.7), confirmed via a live failed SCP/DbMigrator deploy on 2026-08-07 — failed at migration `0036`/`0037`. Fixed via new `0035_model_operations_models.sql` (minimal: id/ticker/published_at/correlation_id/revision only — full Model Card schema is separate future work) and `CREATE SCHEMA IF NOT EXISTS compliance;` added to `0037`. Full chain 0000→0040 now verified fresh-install + idempotent re-run clean. | @claude | Session 2026-08-07 (deploy failure triage) |
| DEBT-021 | Dapper never configured for snake_case↔PascalCase column mapping | High (3) | Low (1) | Completed | `Dapper.DefaultTypeMap.MatchNamesWithUnderscores` was never set anywhere in the codebase, so every `QueryAsync<T>`/`QuerySingleOrDefaultAsync<T>` result-mapping onto a snake_case DB column (e.g. `event_type``EventType`) silently returned null/default for that property instead of throwing — masking the bug in every Sql class across every module. Confirmed via `ApprovalWorkflowTests.InsertAndRetrieveProposal_RoundTrips` and `AuditTrailTests.InsertAuditEvent_CreatesImmutableRecord` both getting real rows back with null fields. Fixed centrally via a `[ModuleInitializer]` in `KArtSell.BuildingBlocks/Data/DapperBootstrap.cs` (runs once per process regardless of entry point — Host/DbMigrator/tests). | @claude | Session 2026-08-07 (deploy failure triage) |
| DEBT-022 | jsonb/inet columns written as plain text without an explicit cast | Medium (2) | Low (1) | Completed (partial) | Dapper does not know to cast a `string` parameter to `jsonb`/`inet` for Npgsql; `AuditSql.InsertAuditEventAsync` (`details`, `ip_address`), `AuditSql.RedactAuditEventDetailsAsync` (duplicate `SET details =` assignment, separately fixed), `TradeSql.InsertTradeAsync`/`UpdateTradeStatusAsync` (`kis_response`), and `SellDecisionSql.InsertDecisionAsync` (`oos_performance`) all failed with `42804: column "x" is of type jsonb but expression is of type text` the first time they were run against a real schema. Fixed with explicit `::jsonb`/`::inet` casts at each call site (mechanical, no behavior change). `AuditSql`'s jsonb read-back (`Dictionary<string,object>` from a jsonb column) also needed a raw-DTO + `JsonSerializer.Deserialize` mapping since Dapper has no built-in jsonb→Dictionary conversion either. **Not yet checked**: `PortfolioReconciliation`/`ApprovalWorkflow` Sql classes for the same pattern beyond what surfaced in this session's test runs — a full audit of jsonb/inet columns across all Sql classes is still open. | @claude | Session 2026-08-07 (deploy failure triage) |
| DEBT-023 | `ApprovalSql.InsertProposalAsync` fails on `DateOnly` parameter | Medium (2) | Low (1) | Backlog | `ApprovalWorkflowTests.InsertAndRetrieveProposal_RoundTrips` fails with `System.NotSupportedException: The member effectiveAt of type System.DateOnly cannot be used as a parameter value` — Dapper's `LookupDbType` doesn't recognize `DateOnly` without an explicit type map (`SqlMapper.AddTypeMap`/custom `TypeHandler`). Likely affects every other `DateOnly`-typed Dapper parameter in the codebase, not just this one; needs a similar centralized fix to DEBT-021 rather than a per-call-site patch. Discovered but not fixed in this session (scope cut to unblock the live deploy). | @claude | Session 2026-08-07 (deploy failure triage) |
| DEBT-024 | New integration tests don't insert FK parent rows / one pure-logic test flakes under full-suite run | Low (1) | Low (1) | Backlog | `TradeExecutionTests` constructs `Trade` with a random `sellDecisionId` that was never inserted into `sell_decisions`, so every insert now correctly fails its FK constraint (`trades_sell_decision_id_fkey`) once the schema was actually complete (see DEBT-020) — test-only gap, not a production code defect; needs the tests updated to insert a parent `models`+`sell_decisions` row first. Separately, `SellPriorityRankerTests.CalculateScore_HardImpairment_ReturnsLowestScore` (pure logic, no DB) passed in isolation but returned 1000 instead of the expected 950 (age-boost not applied) when run as part of the full suite — not yet root-caused; may be test-order/parallelization state leakage rather than a `SellPriorityRanker` bug. Also, `DbUpMigrationTests.*` (pre-existing, unrelated to this session) fail locally with `42501: must be owner of database kartsell_migration_test` — a local Postgres role permission gap, not a code issue. | @claude | Session 2026-08-07 (deploy failure triage) |
---
@@ -0,0 +1,20 @@
-- Migration 0041: model_operations.models
-- Missing prerequisite table: referenced via FK by 0036 (approval_proposals.model_id)
-- and 0038 (sell_decisions.model_id), and queried directly by OpenDartDailyBatchJob.cs
-- (SELECT DISTINCT ticker ... WHERE published_at <= @now), but never created by any
-- prior migration. Any fresh database fails at 0036 without this table.
--
-- Scope is intentionally minimal (only the columns actually referenced today). The full
-- Model Card / lifecycle schema (Freeze/Mature/Score/Diagnose/.../Manual Activation per
-- CLAUDE.md) is a separate, larger piece of work and is not guessed at here.
CREATE TABLE IF NOT EXISTS model_operations.models (
id UUID PRIMARY KEY,
ticker VARCHAR(20) NOT NULL,
published_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
correlation_id UUID NOT NULL,
revision INT NOT NULL DEFAULT 1
);
CREATE INDEX IF NOT EXISTS ix_models_ticker ON model_operations.models(ticker);
CREATE INDEX IF NOT EXISTS ix_models_published_at ON model_operations.models(published_at DESC);
+9 -7
View File
@@ -1,6 +1,8 @@
-- Workstream I: VS-04 Audit Trail (Immutable events + GDPR compliance)
-- Creates compliance audit trail for model operations, regulatory reporting, and GDPR redaction
CREATE SCHEMA IF NOT EXISTS compliance;
-- Audit events (immutable, INSERT-only)
CREATE TABLE IF NOT EXISTS compliance.audit_events (
id UUID PRIMARY KEY,
@@ -22,11 +24,11 @@ CREATE TABLE IF NOT EXISTS compliance.audit_events (
);
-- Indexes for compliance querying
CREATE INDEX idx_audit_events_entity_id ON compliance.audit_events(entity_id);
CREATE INDEX idx_audit_events_event_type ON compliance.audit_events(event_type);
CREATE INDEX idx_audit_events_actor_email ON compliance.audit_events(actor_email);
CREATE INDEX idx_audit_events_event_at ON compliance.audit_events(event_at);
CREATE INDEX idx_audit_events_correlation_id ON compliance.audit_events(correlation_id);
CREATE INDEX IF NOT EXISTS idx_audit_events_entity_id ON compliance.audit_events(entity_id);
CREATE INDEX IF NOT EXISTS idx_audit_events_event_type ON compliance.audit_events(event_type);
CREATE INDEX IF NOT EXISTS idx_audit_events_actor_email ON compliance.audit_events(actor_email);
CREATE INDEX IF NOT EXISTS idx_audit_events_event_at ON compliance.audit_events(event_at);
CREATE INDEX IF NOT EXISTS idx_audit_events_correlation_id ON compliance.audit_events(correlation_id);
-- GDPR retention tracking (personal data retention policy)
CREATE TABLE IF NOT EXISTS compliance.gdpr_retention (
@@ -43,8 +45,8 @@ CREATE TABLE IF NOT EXISTS compliance.gdpr_retention (
);
-- Indexes for GDPR processing
CREATE INDEX idx_gdpr_retention_customer_id ON compliance.gdpr_retention(customer_id);
CREATE INDEX idx_gdpr_retention_purge_status ON compliance.gdpr_retention(purge_status);
CREATE INDEX IF NOT EXISTS idx_gdpr_retention_customer_id ON compliance.gdpr_retention(customer_id);
CREATE INDEX IF NOT EXISTS idx_gdpr_retention_purge_status ON compliance.gdpr_retention(purge_status);
-- Event types enumeration (reference, not enforced at DB level)
CREATE TABLE IF NOT EXISTS compliance.audit_event_types (
@@ -0,0 +1,23 @@
using System.Runtime.CompilerServices;
namespace KArtSell.BuildingBlocks.Data;
/// <summary>
/// Dapper does not map snake_case DB columns (event_type) to PascalCase C# properties
/// (EventType) by default; every Sql class in this repo relies on that mapping, so this
/// must be set before any query runs. A module initializer guarantees it runs once per
/// process regardless of entry point (Host, DbMigrator, test runner) without every Sql
/// class or Program.cs having to remember to configure it.
/// </summary>
internal static class DapperBootstrap
{
#pragma warning disable CA2255 // intentional: BuildingBlocks is this solution's internal shared layer,
// not a distributed package, and every entry point (Host/DbMigrator/tests) needs this set
// before its first query regardless of which one runs first.
[ModuleInitializer]
#pragma warning restore CA2255
public static void Initialize()
{
Dapper.DefaultTypeMap.MatchNamesWithUnderscores = true;
}
}
+6 -4
View File
@@ -4,16 +4,18 @@
</PropertyGroup>
<!-- Frontend Build Target: Automatically build Vite and copy to wwwroot (dev only) -->
<!-- FrontendFiles must be globbed *after* pnpm build, inside the target: Vite emits
content-hashed filenames each build, and a top-level ItemGroup is evaluated once
at project load (before pnpm build runs), so it would copy stale/missing filenames. -->
<Target Name="BuildFrontend" BeforeTargets="Build" Condition="'$(CI)' != 'true' AND Exists('$(ProjectDir)../../frontend/package.json')">
<Exec Command="pnpm install --frozen-lockfile" WorkingDirectory="$(ProjectDir)../../frontend" ContinueOnError="false" />
<Exec Command="pnpm build" WorkingDirectory="$(ProjectDir)../../frontend" ContinueOnError="false" />
<ItemGroup>
<FrontendFiles Include="../../frontend/dist/**/*" />
</ItemGroup>
<Copy SourceFiles="@(FrontendFiles)" DestinationFolder="$(ProjectDir)wwwroot/%(RecursiveDir)" />
</Target>
<ItemGroup>
<FrontendFiles Include="../../frontend/dist/**/*" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="../KArtSell.BuildingBlocks/KArtSell.BuildingBlocks.csproj" />
<ProjectReference Include="../KArtSell.Modules.SignalEngine/KArtSell.Modules.SignalEngine.csproj" />
@@ -69,7 +69,7 @@ public class ApprovalSql
const string sql = """
INSERT INTO model_operations.approval_proposals
(id, model_id, status, created_by, created_at, justification, effective_at, published_at, revision, correlation_id)
VALUES (@id, @modelId, @status, @createdBy, @createdAt, @justification, @effectiveAt, @publishedAt, 1, @correlationId)
VALUES (@id, @modelId, @status, @createdBy, @createdAt, @justification, @effectiveAt::date, @publishedAt, 1, @correlationId)
""";
await conn.ExecuteAsync(sql, new
@@ -80,7 +80,7 @@ public class ApprovalSql
createdBy,
createdAt = _clock.UtcNow,
justification,
effectiveAt,
effectiveAt = effectiveAt.ToString("yyyy-MM-dd"), // Dapper: DateOnly cannot be used as a parameter value directly
publishedAt,
correlationId
});
@@ -43,7 +43,7 @@ public class AuditSql
error_message, details, evidence_links, ip_address, user_agent, published_at,
correlation_id, revision)
VALUES (@Id, @EventType, @EntityType, @EntityId, @ActorEmail, @ActorRole, @EventAt,
@Result, @ErrorMessage, @Details, @EvidenceLinks, @IpAddress, @UserAgent,
@Result, @ErrorMessage, @Details::jsonb, @EvidenceLinks, @IpAddress::inet, @UserAgent,
NOW(), @CorrelationId, 1)
""";
@@ -135,7 +135,7 @@ public class AuditSql
// Get paginated results
var sql = $"""
SELECT id, event_type, entity_type, entity_id, actor_email, actor_role, event_at,
result, error_message, details, evidence_links, ip_address, user_agent,
result, error_message, details, evidence_links, ip_address::text as ip_address, user_agent,
published_at, correlation_id, revision
FROM compliance.audit_events
WHERE {whereClause}
@@ -146,7 +146,8 @@ public class AuditSql
parameters.Add("@Skip", skip);
parameters.Add("@Take", take);
var events = (await db.QueryAsync<AuditEvent>(sql, parameters)).ToList();
var raw = await db.QueryAsync<AuditEventRaw>(sql, parameters);
var events = raw.Select(ToAuditEvent).ToList();
return (events, total);
}
@@ -161,13 +162,54 @@ public class AuditSql
{
const string sql = """
SELECT id, event_type, entity_type, entity_id, actor_email, actor_role, event_at,
result, error_message, details, evidence_links, ip_address, user_agent,
result, error_message, details, evidence_links, ip_address::text as ip_address, user_agent,
published_at, correlation_id, revision
FROM compliance.audit_events
WHERE id = @EventId
""";
return await db.QuerySingleOrDefaultAsync<AuditEvent>(sql, new { EventId = eventId });
var raw = await db.QuerySingleOrDefaultAsync<AuditEventRaw>(sql, new { EventId = eventId });
return raw == null ? null : ToAuditEvent(raw);
}
private static AuditEvent ToAuditEvent(AuditEventRaw raw) => new()
{
Id = raw.Id,
EventType = raw.EventType,
EntityType = raw.EntityType,
EntityId = raw.EntityId,
ActorEmail = raw.ActorEmail,
ActorRole = raw.ActorRole,
EventAt = raw.EventAt,
Result = raw.Result,
ErrorMessage = raw.ErrorMessage,
Details = raw.Details == null ? null : JsonSerializer.Deserialize<Dictionary<string, object>>(raw.Details),
EvidenceLinks = raw.EvidenceLinks,
IpAddress = raw.IpAddress,
UserAgent = raw.UserAgent,
PublishedAt = raw.PublishedAt,
CorrelationId = raw.CorrelationId,
Revision = raw.Revision
};
private sealed class AuditEventRaw
{
public Guid Id { get; set; }
public required string EventType { get; set; }
public required string EntityType { get; set; }
public Guid EntityId { get; set; }
public required string ActorEmail { get; set; }
public string? ActorRole { get; set; }
public DateTime EventAt { get; set; }
public required string Result { get; set; }
public string? ErrorMessage { get; set; }
public string? Details { get; set; }
public string[]? EvidenceLinks { get; set; }
public string? IpAddress { get; set; }
public string? UserAgent { get; set; }
public DateTime PublishedAt { get; set; }
public Guid CorrelationId { get; set; }
public int Revision { get; set; }
}
/// <summary>
@@ -255,12 +297,11 @@ public class AuditSql
const string sql = """
UPDATE compliance.audit_events
SET details = jsonb_set(
COALESCE(details, '{}'::jsonb),
'{actor_email}',
'"<redacted>"'::jsonb
),
details = jsonb_set(
details,
jsonb_set(
COALESCE(details, '{}'::jsonb),
'{actor_email}',
'"<redacted>"'::jsonb
),
'{customer_id}',
'"<purged>"'::jsonb
),
@@ -10,7 +10,7 @@ public class GdprRetention
public Guid EventId { get; set; }
public Guid? CustomerId { get; set; }
public string[]? DataCategories { get; set; } // PII, EMAIL, TRADING_HISTORY, PORTFOLIO_DATA, etc.
public DateTime RetentionEndsAt { get; set; }
public DateOnly RetentionEndsAt { get; set; }
public required string PurgeStatus { get; set; } // PENDING, PURGED, EXCEPTION
public DateTime? PurgedAt { get; set; }
public string? ExceptionReason { get; set; }
@@ -0,0 +1,25 @@
using System.Runtime.CompilerServices;
namespace KArtSell.Modules.ModelOperations;
/// <summary>
/// KArtSell.BuildingBlocks.Data.DapperBootstrap sets Dapper's snake_case-to-PascalCase column
/// mapping via its own [ModuleInitializer], but that only fires once that assembly is actually
/// loaded into the process. Several Sql classes in this module (e.g. AuditSql, TradeSql) only
/// have a `using` for a BuildingBlocks namespace without ever touching a type from it at
/// runtime, so under test isolation - or any host that queries this module before touching
/// BuildingBlocks - the load (and the mapping) can be skipped, silently nulling out every
/// snake_case column. Every Sql class in this assembly is defined here, so a module initializer
/// in this assembly is guaranteed to run before any of them are used, regardless of what else
/// has loaded.
/// </summary>
internal static class DapperMappingBootstrap
{
#pragma warning disable CA2255
[ModuleInitializer]
#pragma warning restore CA2255
public static void Initialize()
{
Dapper.DefaultTypeMap.MatchNamesWithUnderscores = true;
}
}
@@ -24,7 +24,7 @@ public class SellDecisionSql : ISellDecisionSql
sell_priority, target_quantity, target_price, approval_id, execution_id,
created_at, created_by, created_justification, published_at, correlation_id, revision
) VALUES (
@id, @modelId, @status, @pboScore, @dsrMetric, @oosPerformance,
@id, @modelId, @status, @pboScore, @dsrMetric, @oosPerformance::jsonb,
@sellPriority, @targetQuantity, @targetPrice, @approvalId, @executionId,
@createdAt, @createdBy, @createdJustification, @publishedAt, @correlationId, @revision
)
@@ -26,7 +26,7 @@ public class Trade
public decimal? Commission { get; set; }
public decimal? NetProceeds { get; set; }
public string? ErrorMessage { get; set; }
public JsonElement? KisResponse { get; set; }
public string? KisResponse { get; set; }
public DateTime? ExecutionTimestamp { get; set; }
public DateTime? SettlementTimestamp { get; set; }
public DateTime PublishedAt { get; set; }
@@ -55,14 +55,14 @@ public class Trade
{
Status = TradeStatus.Submitted;
KisOrderId = kisOrderId;
KisResponse = response;
KisResponse = response.ToString();
Revision++;
}
public void MarkAccepted(JsonElement response)
{
Status = TradeStatus.Accepted;
KisResponse = response;
KisResponse = response.ToString();
Revision++;
}
@@ -73,7 +73,7 @@ public class Trade
TotalAmount = executedQty * unitPrice;
Status = executedQty >= Quantity ? TradeStatus.FullyFilled : TradeStatus.PartiallyFilled;
ExecutionTimestamp = now;
KisResponse = response;
KisResponse = response.ToString();
Revision++;
}
@@ -98,7 +98,7 @@ public class Trade
public void MarkErrored(KisTradeExecutionException exception)
{
ErrorMessage = exception.Message;
KisResponse = exception.KisResponse;
KisResponse = exception.KisResponse?.ToString();
Revision++;
}
}
@@ -57,14 +57,7 @@ public class SubmitTradeHandler
);
trade.MarkSubmitted(orderId, response);
await _sql.UpdateTradeStatusAsync(
trade.Id,
TradeStatus.Submitted,
response,
null,
command.CorrelationId,
ct
);
await _sql.UpdateTradeStatusAsync(trade, response, null, ct);
await PublishEventAsync(
"TradeSubmitted",
@@ -84,14 +77,7 @@ public class SubmitTradeHandler
catch (KisTradeExecutionException ex)
{
trade.MarkErrored(ex);
await _sql.UpdateTradeStatusAsync(
trade.Id,
trade.Status,
ex.KisResponse,
ex.Message,
command.CorrelationId,
ct
);
await _sql.UpdateTradeStatusAsync(trade, ex.KisResponse, ex.Message, ct);
_logger.LogError(
"Trade submission failed: {TradeId} {Classification}",
@@ -163,14 +149,7 @@ public class PollTradeStatusHandler
trade.MarkFilled(executedQty, unitPrice, response, _clock.UtcNow.UtcDateTime);
}
await _sql.UpdateTradeStatusAsync(
trade.Id,
trade.Status,
response,
null,
command.CorrelationId,
ct
);
await _sql.UpdateTradeStatusAsync(trade, response, null, ct);
if (trade.Status is TradeStatus.FullyFilled)
{
@@ -195,14 +174,7 @@ public class PollTradeStatusHandler
}
catch (KisTradeExecutionException ex)
{
await _sql.UpdateTradeStatusAsync(
trade.Id,
trade.Status,
ex.KisResponse,
ex.Message,
command.CorrelationId,
ct
);
await _sql.UpdateTradeStatusAsync(trade, ex.KisResponse, ex.Message, ct);
_logger.LogError("Failed to poll trade status: {TradeId}", trade.Id);
}
@@ -262,14 +234,7 @@ public class ConfirmSettlementHandler
if (success)
{
trade.MarkConfirmed(_clock.UtcNow.UtcDateTime, command.Commission);
await _sql.UpdateTradeStatusAsync(
trade.Id,
TradeStatus.Confirmed,
response,
null,
command.CorrelationId,
ct
);
await _sql.UpdateTradeStatusAsync(trade, response, null, ct);
await TradeOutboxPublisher.PublishAsync(
_connectionFactory,
@@ -290,14 +255,7 @@ public class ConfirmSettlementHandler
}
catch (KisTradeExecutionException ex)
{
await _sql.UpdateTradeStatusAsync(
trade.Id,
trade.Status,
ex.KisResponse,
ex.Message,
command.CorrelationId,
ct
);
await _sql.UpdateTradeStatusAsync(trade, ex.KisResponse, ex.Message, ct);
_logger.LogError("Failed to confirm settlement: {TradeId}", trade.Id);
}
@@ -12,7 +12,7 @@ public interface ITradeSql
Task<IEnumerable<Trade>> GetTradesByStatusAsync(TradeStatus status, Guid correlationId, CancellationToken ct = default);
Task<IEnumerable<Trade>> GetTradesByDecisionIdAsync(Guid sellDecisionId, Guid correlationId, CancellationToken ct = default);
Task InsertTradeAsync(Trade trade, CancellationToken ct = default);
Task UpdateTradeStatusAsync(Guid tradeId, TradeStatus newStatus, JsonElement? kisResponse, string? errorMessage, Guid correlationId, CancellationToken ct = default);
Task UpdateTradeStatusAsync(Trade trade, JsonElement? kisResponse, string? errorMessage, CancellationToken ct = default);
Task<int> CountTradesByStatusAsync(TradeStatus status, CancellationToken ct = default);
}
@@ -33,7 +33,7 @@ public class TradeSql : ITradeSql
const string sql = """
SELECT id, sell_decision_id, kis_order_id, status, quantity, executed_quantity,
unit_price, total_amount, commission, net_proceeds, error_message, kis_response,
unit_price, total_amount, commission, net_proceeds, error_message, kis_response::text as kis_response,
execution_timestamp, settlement_timestamp, published_at, correlation_id, revision
FROM model_operations.trades
WHERE id = @tradeId
@@ -61,7 +61,7 @@ public class TradeSql : ITradeSql
const string sql = """
SELECT id, sell_decision_id, kis_order_id, status, quantity, executed_quantity,
unit_price, total_amount, commission, net_proceeds, error_message, kis_response,
unit_price, total_amount, commission, net_proceeds, error_message, kis_response::text as kis_response,
execution_timestamp, settlement_timestamp, published_at, correlation_id, revision
FROM model_operations.trades
WHERE kis_order_id = @kisOrderId
@@ -82,7 +82,7 @@ public class TradeSql : ITradeSql
const string sql = """
SELECT id, sell_decision_id, kis_order_id, status, quantity, executed_quantity,
unit_price, total_amount, commission, net_proceeds, error_message, kis_response,
unit_price, total_amount, commission, net_proceeds, error_message, kis_response::text as kis_response,
execution_timestamp, settlement_timestamp, published_at, correlation_id, revision
FROM model_operations.trades
WHERE status = @status
@@ -102,7 +102,7 @@ public class TradeSql : ITradeSql
const string sql = """
SELECT id, sell_decision_id, kis_order_id, status, quantity, executed_quantity,
unit_price, total_amount, commission, net_proceeds, error_message, kis_response,
unit_price, total_amount, commission, net_proceeds, error_message, kis_response::text as kis_response,
execution_timestamp, settlement_timestamp, published_at, correlation_id, revision
FROM model_operations.trades
WHERE sell_decision_id = @sellDecisionId
@@ -126,7 +126,7 @@ public class TradeSql : ITradeSql
unit_price, total_amount, commission, net_proceeds, error_message, kis_response,
execution_timestamp, settlement_timestamp, published_at, correlation_id, revision)
VALUES (@id, @sellDecisionId, @kisOrderId, @status, @quantity, @executedQuantity,
@unitPrice, @totalAmount, @commission, @netProceeds, @errorMessage, @kisResponse,
@unitPrice, @totalAmount, @commission, @netProceeds, @errorMessage, @kisResponse::jsonb,
@executionTimestamp, @settlementTimestamp, @publishedAt, @correlationId, @revision)
""";
@@ -143,7 +143,7 @@ public class TradeSql : ITradeSql
trade.Commission,
trade.NetProceeds,
trade.ErrorMessage,
kisResponse = trade.KisResponse?.ToString(),
trade.KisResponse,
trade.ExecutionTimestamp,
trade.SettlementTimestamp,
trade.PublishedAt,
@@ -155,11 +155,9 @@ public class TradeSql : ITradeSql
}
public async Task UpdateTradeStatusAsync(
Guid tradeId,
TradeStatus newStatus,
Trade trade,
JsonElement? kisResponse,
string? errorMessage,
Guid correlationId,
CancellationToken ct = default)
{
using var connection = await _dataSource.OpenConnectionAsync(ct);
@@ -167,13 +165,21 @@ public class TradeSql : ITradeSql
const string sql = """
INSERT INTO model_operations.trade_status_history
(id, trade_id, old_status, new_status, transitioned_at, kis_response, error_message, published_at, correlation_id)
SELECT @id, id, status, @newStatus, NOW(), @kisResponse, @errorMessage, NOW(), @correlationId
SELECT @id, id, status, @newStatus, NOW(), @kisResponse::jsonb, @errorMessage, NOW(), @correlationId
FROM model_operations.trades
WHERE id = @tradeId;
UPDATE model_operations.trades
SET status = @newStatus,
kis_response = COALESCE(@kisResponse, kis_response),
kis_order_id = COALESCE(@kisOrderId, kis_order_id),
executed_quantity = COALESCE(@executedQuantity, executed_quantity),
unit_price = COALESCE(@unitPrice, unit_price),
total_amount = COALESCE(@totalAmount, total_amount),
commission = COALESCE(@commission, commission),
net_proceeds = COALESCE(@netProceeds, net_proceeds),
execution_timestamp = COALESCE(@executionTimestamp, execution_timestamp),
settlement_timestamp = COALESCE(@settlementTimestamp, settlement_timestamp),
kis_response = COALESCE(@kisResponse::jsonb, kis_response),
error_message = COALESCE(@errorMessage, error_message),
revision = revision + 1
WHERE id = @tradeId
@@ -182,14 +188,22 @@ public class TradeSql : ITradeSql
await connection.ExecuteAsync(sql, new
{
id = Guid.NewGuid(),
tradeId,
newStatus = newStatus.ToString(),
tradeId = trade.Id,
newStatus = trade.Status.ToString(),
kisOrderId = trade.KisOrderId,
executedQuantity = trade.ExecutedQuantity,
unitPrice = trade.UnitPrice,
totalAmount = trade.TotalAmount,
commission = trade.Commission,
netProceeds = trade.NetProceeds,
executionTimestamp = trade.ExecutionTimestamp,
settlementTimestamp = trade.SettlementTimestamp,
kisResponse = kisResponse?.ToString(),
errorMessage,
correlationId
correlationId = trade.CorrelationId
});
_logger.LogInformation("Updated trade {TradeId} status to {Status}", tradeId, newStatus);
_logger.LogInformation("Updated trade {TradeId} status to {Status}", trade.Id, trade.Status);
}
public async Task<int> CountTradesByStatusAsync(TradeStatus status, CancellationToken ct = default)
@@ -0,0 +1,21 @@
using System.Runtime.CompilerServices;
namespace KArtSell.Modules.SignalEngine;
/// <summary>
/// See KArtSell.Modules.ModelOperations.DapperMappingBootstrap for the full rationale: Dapper's
/// snake_case-to-PascalCase column mapping is a process-wide static flag set via
/// KArtSell.BuildingBlocks.Data.DapperBootstrap's [ModuleInitializer], which only fires once
/// that assembly is loaded. This mirrors it locally so every Sql/reader class defined in this
/// assembly is guaranteed the mapping is on before its first query, regardless of load order.
/// </summary>
internal static class DapperMappingBootstrap
{
#pragma warning disable CA2255
[ModuleInitializer]
#pragma warning restore CA2255
public static void Initialize()
{
Dapper.DefaultTypeMap.MatchNamesWithUnderscores = true;
}
}
@@ -3,6 +3,7 @@ namespace KArtSell.Integration.Tests.ApprovalWorkflow;
using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Dapper;
using Xunit;
using KArtSell.BuildingBlocks.Time;
using KArtSell.Modules.ModelOperations.ApprovalWorkflow;
@@ -165,6 +166,7 @@ public class ApprovalWorkflowTests : IAsyncLifetime
var id = Guid.NewGuid();
var modelId = Guid.NewGuid();
var correlationId = Guid.NewGuid();
await SeedModelAsync(modelId);
// Act
await _sql.InsertProposalAsync(
@@ -185,6 +187,15 @@ public class ApprovalWorkflowTests : IAsyncLifetime
Assert.Equal(modelId, retrieved.ModelId);
Assert.Equal(correlationId, retrieved.CorrelationId);
}
private async Task SeedModelAsync(Guid modelId)
{
await using var conn = new Npgsql.NpgsqlConnection(_connectionString);
await conn.OpenAsync();
await conn.ExecuteAsync(
"INSERT INTO model_operations.models (id, ticker, correlation_id) VALUES (@Id, @Ticker, @CorrelationId)",
new { Id = modelId, Ticker = "TEST", CorrelationId = Guid.NewGuid() });
}
}
public class InMemoryOutbox : IOutbox
@@ -106,6 +106,11 @@ public class AuditTrailTests : IAsyncLifetime
var customerId = Guid.NewGuid();
var retentionId = Guid.NewGuid();
await _sql.InsertAuditEventAsync(
_db, eventId, AuditEventTypes.ModelActivated, AuditEntityTypes.Model,
Guid.NewGuid(), "customer@company.com", null, DateTime.UtcNow, "SUCCESS", null,
null, null, null, null, Guid.NewGuid(), CancellationToken.None);
// Act
await _sql.InsertGdprRetentionAsync(
_db, retentionId, eventId, customerId,
@@ -175,7 +180,8 @@ public class AuditTrailTests : IAsyncLifetime
// Assert
var @event = await _sql.GetAuditEventByIdAsync(_db, eventId, CancellationToken.None);
Assert.NotNull(@event);
Assert.Contains("<redacted>", @event.Details?.ToString() ?? "");
Assert.Equal("<redacted>", @event.Details?["actor_email"].ToString());
Assert.Equal("<purged>", @event.Details?["customer_id"].ToString());
}
private const string TestConnectionString =
@@ -118,8 +118,8 @@ public class SellPriorityRankerTests
[Fact]
public void CalculateScore_HardImpairment_ReturnsLowestScore()
{
var score = _ranker.CalculateScore(SellPriority.HardImpairment, fundAgeDays: 200, liquidityPercent: 0.5m);
Assert.Equal(950m, score); // 1000 - 50 (age boost)
var score = _ranker.CalculateScore(SellPriority.HardImpairment, fundAgeDays: 400, liquidityPercent: 0.5m);
Assert.Equal(950m, score); // 1000 - 50 (age boost, fundAgeDays > 365 per VS-10-SLICE_SPEC.md)
}
[Fact]
@@ -1,4 +1,5 @@
using System.Text.Json;
using Dapper;
using KArtSell.Modules.ModelOperations.TradeExecution;
using Microsoft.Extensions.Logging;
using Npgsql;
@@ -31,11 +32,31 @@ public class TradeExecutionTests : IAsyncLifetime
await _dataSource.DisposeAsync();
}
private async Task<Guid> SeedSellDecisionAsync()
{
await using var connection = await _dataSource.OpenConnectionAsync();
var modelId = Guid.NewGuid();
await connection.ExecuteAsync(
"INSERT INTO model_operations.models (id, ticker, correlation_id) VALUES (@Id, @Ticker, @CorrelationId)",
new { Id = modelId, Ticker = "TEST", CorrelationId = Guid.NewGuid() });
var sellDecisionId = Guid.NewGuid();
await connection.ExecuteAsync(
"""
INSERT INTO model_operations.sell_decisions
(id, model_id, status, created_by, published_at, correlation_id)
VALUES (@Id, @ModelId, 'PENDING', 'test@company.com', NOW(), @CorrelationId)
""",
new { Id = sellDecisionId, ModelId = modelId, CorrelationId = Guid.NewGuid() });
return sellDecisionId;
}
[Fact]
public async Task CreateTrade_WithValidData_ShouldInsertSuccessfully()
{
var sql = new TradeSql(_dataSource, _logger);
var sellDecisionId = Guid.NewGuid();
var sellDecisionId = await SeedSellDecisionAsync();
var correlationId = Guid.NewGuid();
var trade = Trade.Create(sellDecisionId, 1000, correlationId, DateTime.UtcNow);
@@ -55,14 +76,14 @@ public class TradeExecutionTests : IAsyncLifetime
{
var sql = new TradeSql(_dataSource, _logger);
var correlationId = Guid.NewGuid();
var trade = Trade.Create(Guid.NewGuid(), 1000, correlationId, DateTime.UtcNow);
var trade = Trade.Create(await SeedSellDecisionAsync(), 1000, correlationId, DateTime.UtcNow);
await sql.InsertTradeAsync(trade);
var response = JsonDocument.Parse("{}").RootElement;
trade.MarkSubmitted("KIS-ORDER-123", response);
await sql.UpdateTradeStatusAsync(trade.Id, TradeStatus.Submitted, response, null, correlationId);
await sql.UpdateTradeStatusAsync(trade, response, null);
var retrieved = await sql.GetTradeByIdAsync(trade.Id, correlationId);
@@ -76,14 +97,14 @@ public class TradeExecutionTests : IAsyncLifetime
{
var sql = new TradeSql(_dataSource, _logger);
var correlationId = Guid.NewGuid();
var trade = Trade.Create(Guid.NewGuid(), 1000, correlationId, DateTime.UtcNow);
var trade = Trade.Create(await SeedSellDecisionAsync(), 1000, correlationId, DateTime.UtcNow);
await sql.InsertTradeAsync(trade);
var response = JsonDocument.Parse("{}").RootElement;
trade.MarkFilled(1000, 49.95m, response, DateTime.UtcNow);
await sql.UpdateTradeStatusAsync(trade.Id, TradeStatus.FullyFilled, response, null, correlationId);
await sql.UpdateTradeStatusAsync(trade, response, null);
var retrieved = await sql.GetTradeByIdAsync(trade.Id, correlationId);
@@ -100,8 +121,8 @@ public class TradeExecutionTests : IAsyncLifetime
var sql = new TradeSql(_dataSource, _logger);
var correlationId = Guid.NewGuid();
var trade1 = Trade.Create(Guid.NewGuid(), 1000, correlationId, DateTime.UtcNow);
var trade2 = Trade.Create(Guid.NewGuid(), 2000, correlationId, DateTime.UtcNow);
var trade1 = Trade.Create(await SeedSellDecisionAsync(), 1000, correlationId, DateTime.UtcNow);
var trade2 = Trade.Create(await SeedSellDecisionAsync(), 2000, correlationId, DateTime.UtcNow);
await sql.InsertTradeAsync(trade1);
await sql.InsertTradeAsync(trade2);
@@ -118,14 +139,14 @@ public class TradeExecutionTests : IAsyncLifetime
{
var sql = new TradeSql(_dataSource, _logger);
var correlationId = Guid.NewGuid();
var trade = Trade.Create(Guid.NewGuid(), 1000, correlationId, DateTime.UtcNow);
var trade = Trade.Create(await SeedSellDecisionAsync(), 1000, correlationId, DateTime.UtcNow);
await sql.InsertTradeAsync(trade);
trade.TotalAmount = 49950m;
trade.MarkConfirmed(DateTime.UtcNow, 50m);
await sql.UpdateTradeStatusAsync(trade.Id, TradeStatus.Confirmed, null, null, correlationId);
await sql.UpdateTradeStatusAsync(trade, null, null);
var retrieved = await sql.GetTradeByIdAsync(trade.Id, correlationId);
@@ -140,11 +161,16 @@ public class TradeExecutionTests : IAsyncLifetime
{
var sql = new TradeSql(_dataSource, _logger);
var correlationId = Guid.NewGuid();
var trade = Trade.Create(Guid.NewGuid(), 1000, correlationId, DateTime.UtcNow);
var trade = Trade.Create(await SeedSellDecisionAsync(), 1000, correlationId, DateTime.UtcNow);
await sql.InsertTradeAsync(trade);
await sql.UpdateTradeStatusAsync(trade.Id, TradeStatus.Submitted, null, null, correlationId);
await sql.UpdateTradeStatusAsync(trade.Id, TradeStatus.Accepted, null, null, correlationId);
var response = JsonDocument.Parse("{}").RootElement;
trade.MarkSubmitted("KIS-1", response);
await sql.UpdateTradeStatusAsync(trade, response, null);
trade.MarkAccepted(response);
await sql.UpdateTradeStatusAsync(trade, response, null);
var retrieved = await sql.GetTradeByIdAsync(trade.Id, correlationId);
@@ -158,8 +184,8 @@ public class TradeExecutionTests : IAsyncLifetime
var sql = new TradeSql(_dataSource, _logger);
var correlationId = Guid.NewGuid();
var trade1 = Trade.Create(Guid.NewGuid(), 1000, correlationId, DateTime.UtcNow);
var trade2 = Trade.Create(Guid.NewGuid(), 2000, correlationId, DateTime.UtcNow);
var trade1 = Trade.Create(await SeedSellDecisionAsync(), 1000, correlationId, DateTime.UtcNow);
var trade2 = Trade.Create(await SeedSellDecisionAsync(), 2000, correlationId, DateTime.UtcNow);
await sql.InsertTradeAsync(trade1);
await sql.InsertTradeAsync(trade2);