Architecture

This page describes the internal model of the MemDb engine — useful for reasoning about durability, recovery, or performance before depending on it. None of it is required to use MemDb.

Component overview

  flowchart TB
    Caller["Application code"]

    subgraph MemDbInstance["MemDb instance — per process, per dataset"]
        Cache["In-memory cache<br/>List of MemDbRecord"]
        IdxId["Identity index<br/>Dictionary id to position"]
        IdxApplied["Applied indexes<br/>hash + sorted list per index"]
        Cloner["IMemDbCloner"]
        Persister["Persister<br/>insert queue + state-mod queue"]
        Serializer["IMemDbSerializer"]
        Encryptor["AES-256-CBC encryptor"]
        Timer["Flush timer"]
    end

    subgraph Disk["On-disk files"]
        DbFile[("data file<br/>htl.NAME.db")]
        MapFile[("pointer map<br/>htl.NAME.map")]
        LockFile[("lock file")]
    end

    Caller -->|Insert / Update / Delete| Cache
    Caller -->|Find / Query / Count| Cache

    Cache <--> IdxId
    Cache <--> IdxApplied
    Cache <--> Cloner

    Cache -->|enqueue| Persister
    Timer -->|tick| Persister
    Persister --> Serializer
    Persister --> Encryptor
    Persister --> DbFile
    Persister --> MapFile

    MemDbInstance -.holds.-> LockFile

The MemDb<T> instance is the public facade. Behind it sit a single in-memory cache, two optional index structures, a persister with two write queues, and a small set of pluggable interfaces (serializer, cloner, encryptor) that determine how records are encoded and copied.

File layout

A persisted dataset consists of three files in the configured directory:

  flowchart LR
    subgraph DbF["htl.{name}.db"]
        direction TB
        R1[Record 1 bytes]
        R2[Record 2 bytes]
        R3[...]
        Rn[Record N bytes]
    end

    subgraph MapF["htl.{name}.map"]
        direction TB
        Hdr["Header (12 bytes)<br/>4: count · 8: lastId"]
        P1[Pointer 1 — 39 bytes]
        P2[Pointer 2 — 39 bytes]
        P3[...]
        Pn[Pointer N — 39 bytes]
    end

    subgraph LockF["~$htl.{name}.lock"]
        L[OS-enforced cross-process lock]
    end

    P1 -.points to.-> R1
    P2 -.points to.-> R2
    Pn -.points to.-> Rn

Pointer layout (39 bytes each):

The pointer format is intentionally versioned — future revisions can be added as MemDbPointerV1, V2, etc., and selected at deserialization time based on the version byte.

For an encrypted record, the pointer’s Length field stores the unencrypted payload length. The actual on-disk size is computed at read time from the AES block size and IV length. This keeps stale/deleted size statistics meaningful regardless of encryption state.

Record lifecycle

  stateDiagram-v2
    [*] --> Fresh: Insert (cloned in)
    Fresh --> Stale: Update (new Fresh appended,<br/>original marked Stale)
    Fresh --> Deleted: Delete
    Stale --> [*]: PurgeCache / Defrag
    Deleted --> [*]: PurgeCache / Defrag

• Fresh — the current authoritative version of a record. Reads return only fresh records.
• Stale — superseded by a later update. Still present in the cache and on disk; ignored by reads. Reclaimable by PurgeCache (RAM) and Defrag (disk).
• Deleted — explicitly deleted. Same reclaim path as stale.

This log-structured approach has two practical consequences:

1. Repeated updates to the same record accumulate stale versions until purge/defrag runs.
2. Each update yields a distinct, traceable record version that can be archived on defrag — enabling point-in-time restore.

Write path

  sequenceDiagram
    participant Caller
    participant Cache
    participant Cloner
    participant InsertQ as Insert queue
    participant Timer as Flush timer
    participant Persister
    participant Disk

    Caller->>Cache: Insert(record)
    Cache->>Cache: GetNextId() (locked)
    Cache->>Cloner: DeepCopy(record)
    Cloner-->>Cache: cloned record
    Cache->>Cache: append to in-memory list
    Cache->>Cache: update identity + applied indexes
    Cache->>InsertQ: enqueue cloned record
    Cache-->>Caller: return (id captured via callback)

    Note over Timer,Persister: Timer fires every N seconds
    Timer->>Persister: Flush()
    Persister->>InsertQ: drain
    Persister->>Disk: serialize, encrypt (if flagged),<br/>append to .db, update .map

The cache is updated synchronously and is immediately consistent for subsequent reads in the same process. Disk writes are deferred to the persister’s queue and committed by the flush timer, by an explicit db.Flush(), or on dispose.

The persister maintains two queues:

• Insert queue — records pending an initial write to the .db file
• State-mod queue — records pending a Stale or Deleted state transition in the .map file

State changes for a record cannot run ahead of that record’s initial insert. The state-mod queue checks the head of the queue against the record’s MapIndex; a state change is held until the corresponding insert has been written and assigned a map index.

Read path

  sequenceDiagram
    participant Caller
    participant Cache
    participant Cloner

    alt Find by id with identity index
        Caller->>Cache: Find(id)
        Cache->>Cache: O(1) Dictionary lookup to cache position
        Cache->>Cloner: DeepCopy(record)
        Cloner-->>Caller: cloned record
    else Find by id without identity index
        Caller->>Cache: Find(id)
        Cache->>Cache: full scan for matching fresh record
        Cache->>Cloner: DeepCopy(record)
        Cloner-->>Caller: cloned record
    else QueryViaIndex with operator
        Caller->>Cache: index lookup
        Cache->>Cache: Dictionary lookup to pointer set
        Cache->>Cloner: DeepCopy(matching records)
        Cloner-->>Caller: cloned record array
    else Query with predicate
        Caller->>Cache: full scan with predicate
        Cache->>Cache: filter fresh records
        Cache->>Cloner: DeepCopy(matching records)
        Cloner-->>Caller: cloned record array
    end

All reads return clones — never references into the cache. Select and SelectDistinct skip the clone when the projected type is a string or value type.

Concurrency model

MemDb uses a single coarse-grained Lock to serialize all cache mutations within a process — not a sharded or lock-free design.

• Reads acquire the lock briefly to capture a consistent snapshot of matching records, then release it before cloning.
• Index lookups (Find by id with identity index, QueryViaIndex with any operator) share the same cache lock — they are faster than a full scan but not lock-free.
• Writes hold the lock for the duration of the cache-side operation (append, index update, queue enqueue).
• The persister has its own internal locks for the two queues and a separate flush lock that serializes disk-touching operations.
• PurgeCache is thread-safe and acquires the same cache lock during the rebuild — concurrent operations are briefly blocked until the purge completes.

For workloads where cache-side throughput becomes a contention bottleneck, consider partitioning into multiple datasets keyed on a domain attribute and opening them as separate MemDb<T> instances in the same process.

Durability model

MemDb does not implement a write-ahead log. Durability is determined entirely by flush cadence and clean disposal. For workloads that cannot tolerate the default 5-second window of exposure on abnormal termination, reduce the flush interval, call db.Flush() explicitly at critical checkpoints, or both.