Docs
View as markdown

Storage and durability

How a write travels from a statement to disk in relate, through the WAL, checkpoints and codecs, and what each durability mode can lose in a crash.

clipythonnode

What happens to a write between the statement and the disk, and what can a crash take back? In relate, every commit takes one path: its changes are appended to the store A relate database: a directory, a single .rl file, or :memory:, opened with --database. Read more in the glossary's WAL Append-only log each commit goes to before it applies; under the default full durability mode it is synced first, so a crash loses no acknowledged write. Read more in the glossary (WAL), then applied to the graph in memory, which is where every query reads. Disk is for recovery, not for answering queries. Under the default durability mode the log is synced to disk before a commit is acknowledged, so a crash loses no acknowledged write; the two relaxed modes skip that wait and accept a stated loss window instead.

One write path

The write path step by step, what each durability mode can lose in a crash, and the parts of one .rl file.

The write path, under the default durability mode, full. One commit:

  1. The statement runs and collects its changes privately; nothing is written yet.
  2. At commit, the changes become one record, numbered by its log sequence number (LSN). It holds physical changes to nodes, relationships, labels and properties, not Cypher text.
  3. The record is appended to the open WAL segment, wal/NNNNNNNN.log.
  4. The WAL is synced to disk before the commit returns; commits waiting together share one sync.
  5. The changes are applied to the in-memory graph, where every query reads.
  6. The commit is acknowledged.

A checkpoint:

  1. The open segment closes, and new commits go to the next one, so writers do not stop.
  2. The checkpoint pins the LSN of the newest closed segment's last record.
  3. The committed state up to that LSN goes through the store's codec into snapshot files, under temporary names.
  4. The files are renamed into place, manifest.json last. A crash before the renames leaves the previous snapshot in force.
  5. Closed segments whose last record is at or below the pinned LSN are deleted.

A checkpoint runs when the open segment reaches 64 MiB or 10,000 commits, every 300 seconds by default, when a writable store closes cleanly, and on .checkpoint.

After a crash, the next open:

  1. Loads the latest snapshot: the state at its checkpoint LSN.
  2. Replays every WAL record with a higher LSN, in LSN order.
  3. Drops a record the crash cut short at the very end of the WAL. Under full that commit was never acknowledged, so every acknowledged commit is back.

What a crash can lose. The durability mode decides when a commit is acknowledged relative to the WAL sync. It is a setting of the process that opens the store (--durability-mode or RELATE_DURABILITY_MODE, default full), not of the store.

The picture's timeline: 8 writes are acknowledged in the 4.6 s after a completed checkpoint, then the process crashes. A killed process can lose the window as well as a power cut.

ModeThe WAL is syncedAcknowledged writes the crash at 4.6 s can lose
fullBefore each commit, or group of commits, is acknowledgedNone: all 8 come back
everysecondAbout once a second, in the backgroundThe 2 writes after the last sync, at 4 s
snapshotNever on commit; only a completed checkpoint is durableAll 8, every write since the checkpoint

Under everysecond, slow storage or a delayed sync makes the window longer than about a second. Under snapshot, the window runs until the next completed checkpoint: up to 300 seconds of writes under the default timer, fewer when a WAL segment fills first.

One .rl file. A single-file store keeps in one file the same tree a directory store spreads over a folder, WAL included. Its parts, in logical order rather than by byte position:

PartHolds
SuperblockThe container's version marker: magic MDBCLB01, version 1 (relate info prints superblock version: 1)
Snapshotmanifest.json and the codec's files (with the default parquet codec, the files under nodes/ and relationships/): the state at the last checkpoint
WALwal/NNNNNNNN.log segments: every commit since the last checkpoint
Space it reusesThe most space the snapshot and WAL have needed at once; later writes reuse what they no longer hold, and the file does not shrink

A commit appends its record to the open segment inside the file. A checkpoint writes the new snapshot files inside it, publishes them with manifest.json last, and deletes the segments the snapshot covers. A writer holds the file itself, with no lock files beside it, so a --read-only open is refused while a writer has it, and --wal-dir is refused for a .rl path. relate clone writes a compact copy that holds only the current state.

The write path, what each durability mode can lose in a crash and the parts of one .rl file, as relate documents themThe write path, after two commits: In memory: a statement, its commit record, and the in-memory graph (commits up to LSN 3). On disk: the WAL (00000002.log holding LSN 2, 3 (open)) and the snapshot at checkpoint LSN 1. A commit record is appended to the WAL, then applied to the graph; a checkpoint writes the graph to the snapshot and recycles the segments it covers; an open loads the snapshot and replays the WAL. What a crash 4.6 s after a checkpoint can lose: Acknowledged writes on a ten-second timeline after a completed checkpoint, with a crash at 4.6 s. Under full, all 8 writes acknowledged before the crash come back: each returned only after its WAL record was synced. Under everysecond, the last background sync was at 4 s, so the 2 writes acknowledged in the 0.6 s after it can be lost. Slow storage or a delayed sync makes the window longer. Under snapshot, nothing is synced on commit, so all 8 writes acknowledged since the last completed checkpoint can be lost. The next timed checkpoint is due at 300 s; a WAL segment that fills, or a clean close, checkpoints sooner. One .rl file, after a checkpoint: One .rl file holding, top to bottom: the superblock (MDBCLB01, version 1); the snapshot (manifest.json and the codec's files) at checkpoint LSN 3; the WAL (00000003.log with no records); and space the file keeps from parts it no longer needs and reuses for later writes. The parts are drawn in logical order, not to scale.The write path, after two commitsIN MEMORYON DISK, IN THE STOREapplyappendcheckpointopen: loadopen: replayrecyclesStatementa writeCommit recordphysical changes,not Cypher textIn-memory graphcommits up to LSN 3every query reads hereWrite-ahead log (WAL)00000002.log (open)23Snapshotmanifest.json, codec filescheckpoint LSN 1What a crash 4.6 s after a checkpoint can losecrash at 4.6 sfullno acknowledged write losteverysecondcan lose 2 writessnapshotcan lose 8 writes012345678910seconds since the last completed checkpointcompleted checkpointacknowledged, survivesacknowledged, can be lostbackground syncOne .rl file, after a checkpointteam.rl: one fileSuperblockMDBCLB01, version 1Snapshotmanifest.json and the codec's filescheckpoint LSN 3WAL00000003.log (open)no recordsspace it keeps and reusesA logical view of its parts, not to scale.The writer holds the file itself:no lock files sit beside it.A --read-only open is refusedwhile a writer holds the file.It grows to the most space itssnapshot and WAL needed at once,then reuses it; it never shrinks.relate clone writes a compact copythat holds only the current state.
The write path, what each durability mode can lose in a crash and the parts of one .rl file, as relate documents themThe write path, after two commits: In memory: a statement, its commit record, and the in-memory graph (commits up to LSN 3). On disk: the WAL (00000002.log holding LSN 2, 3 (open)) and the snapshot at checkpoint LSN 1. A commit record is appended to the WAL, then applied to the graph; a checkpoint writes the graph to the snapshot and recycles the segments it covers; an open loads the snapshot and replays the WAL. What a crash 4.6 s after a checkpoint can lose: Acknowledged writes on a ten-second timeline after a completed checkpoint, with a crash at 4.6 s. Under full, all 8 writes acknowledged before the crash come back: each returned only after its WAL record was synced. Under everysecond, the last background sync was at 4 s, so the 2 writes acknowledged in the 0.6 s after it can be lost. Slow storage or a delayed sync makes the window longer. Under snapshot, nothing is synced on commit, so all 8 writes acknowledged since the last completed checkpoint can be lost. The next timed checkpoint is due at 300 s; a WAL segment that fills, or a clean close, checkpoints sooner. One .rl file, after a checkpoint: One .rl file holding, top to bottom: the superblock (MDBCLB01, version 1); the snapshot (manifest.json and the codec's files) at checkpoint LSN 3; the WAL (00000003.log with no records); and space the file keeps from parts it no longer needs and reuses for later writes. The parts are drawn in logical order, not to scale.The write path, after two commitsIN MEMORYON DISK, IN THE STOREapplyappendcheckpointopen: loadopen: replayrecyclesStatementa writeCommit recordphysical changes,not Cypher textIn-memory graphcommits up to LSN 3every query reads hereWrite-ahead log (WAL)00000002.log (open)23Snapshotmanifest.json, codec filescheckpoint LSN 1What a crash 4.6 s after a checkpoint can losecrash at 4.6 sfullno acknowledged write losteverysecondcan lose 2 writessnapshotcan lose 8 writes012345678910seconds since the last completed checkpointcompleted checkpointacknowledged, survivesacknowledged, can be lostbackground syncOne .rl file, after a checkpointteam.rl: one fileSuperblockMDBCLB01, version 1Snapshotmanifest.json and the codec's filescheckpoint LSN 3WAL00000003.log (open)no recordsspace it keeps and reusesA logical view of its parts, not to scale.The writer holds the file itself:no lock files sit beside it.A --read-only open is refusedwhile a writer holds the file.It grows to the most space itssnapshot and WAL needed at once,then reuses it; it never shrinks.relate clone writes a compact copythat holds only the current state.
The write path, what each durability mode can lose in a crash and the parts of one .rl file, as relate documents them

Live: Commit, Checkpoint and Crash step through what relate does with the WAL, the in-memory graph and the snapshot (inside one .rl file in the file view), and a mode switch with a crash slider shades what full, everysecond and snapshot can lose; under reduced motion each step waits for Next. It is a diagram of documented behaviour: no engine runs.

A statement collects its changes privately while it runs. At commit they become one record of physical changes: a node or relationship created or deleted, a property set or removed, or a label added or removed. The record is not Cypher text, so replaying it never re-runs a query. relate appends the record to the WAL first and applies it to the in-memory graph second; the graph in memory is never the source of a commit.

A single statement outside a transaction is one commit. An explicit transaction collects all its statements and appends one record at COMMIT; a rollback appends nothing. Transactions and concurrency covers isolation and conflicts.

The write-ahead log

The WAL is a series of segment files, wal/NNNNNNNN.log in a directory store (.cypherlite/wal/ under the markdown codec) and the same tree inside a .rl file. Each record carries a checksum. One commit's record can be at most 64 MiB; a larger commit is refused before anything is written, and splitting it into smaller transactions clears it.

On open, relate loads the latest snapshot The committed graph state a checkpoint writes to disk, loaded on open before the WAL replays; not the snapshot durability mode or --read-only freshness. Read more in the glossary and replays every WAL record written after the snapshot's position in the log. Committed writes that never reached a snapshot survive this way. A crash can leave a final record cut short, which recovery does not count as a commit; Recover from a crash covers what recovery does and the errors it can report.

Checkpoints

A checkpoint The step that writes a fresh snapshot of committed state and recycles old WAL segments: every 300 seconds by default, on clean exit, and on .checkpoint. Read more in the glossary turns the log into a snapshot. It pins a position in the log, writes the committed state up to it through the store's codec, then recycles the closed WAL segments the snapshot now covers. Writers do not stop: new commits go to the next segment while the snapshot is written.

A checkpoint runs when the current segment reaches 64 MiB or 10,000 commits, every 300 seconds by default (--checkpoint-interval-secs, or RELATE_CHECKPOINT_INTERVAL_SECS for relate serve), and when a writable store closes cleanly. The CLI closes cleanly on every exit from a writable run, so a store last written from the CLI has its state in the snapshot. A process that keeps a store open, such as an application or relate serve, carries recent commits in the WAL until the next checkpoint. That is why a backup copies the WAL along with the snapshot, or uses relate clone.

An in-memory store has no files, so its checkpoints only recycle the WAL.

Codecs

A codec The on-disk encoding of a store's snapshot: Parquet (the default), JSON or markdown, fixed when the store is created. Read more in the glossary is the encoding a checkpoint writes the snapshot in. parquet, the default, writes compact columnar files and rewrites only the id ranges that changed. json rewrites one readable graph.json every time. markdown writes one note per node. The WAL's format is the same under every codec.

Queries never read the snapshot files, so the codec decides size on disk, checkpoint cost and how long an open takes to load, not how fast a statement runs. A store keeps the codec it was created with; relate clone --to-snapshot-format makes a copy in another.

Durability modes and their loss windows

The durability mode decides when a commit is acknowledged relative to the WAL sync. It is chosen per open with --durability-mode or RELATE_DURABILITY_MODE, and is not stored in the store.

ModeA commit is acknowledgedA crash can lose
full (default)After the WAL is synced to diskNo acknowledged write
everysecondAfter it is applied; the WAL is synced about once per secondAbout the last second of acknowledged writes; slow scheduling or storage can widen the window
snapshotWithout a WAL syncEvery write since the last completed checkpoint: up to 300 seconds of writes under the default timer, fewer when the WAL fills first

The relaxed modes exist for work that can be rebuilt: a bulk load from a file that still exists, or a scratch store an agent fills and throws away. Choose a durability mode covers when each fits and how to set it.

Trade-offs against a page cache design

Many databases keep their data in files on disk and cache recently used pages in memory. Their working set can be larger than RAM, at the cost of reads that sometimes wait on disk and a cache that needs warming and tuning. relate takes the other side of that trade:

relate's choiceWhat it givesWhat it costs
The whole graph in memoryReads never wait on disk; there is no cache to size or warmThe graph and its indexes must fit in memory; there is no paging to disk
Disk holds a snapshot and a logRecovery is load-then-replay, the same path as every openAn open reads the whole snapshot, so startup grows with the graph rather than with what the first query touches
Snapshots in open formatsgraph.json, markdown notes and Parquet files can be read by other tools, and JSON can be diffed in reviewA json checkpoint rewrites the whole file, so large stores favor parquet
One process owns the storeNo shared cache or lock manager between processesA second process reads a directory store with --read-only, or goes through relate serve

The reason for this shape is the job relate is built for: an embedded graph that one application or agent owns, small enough to hold in memory, where the slow part should never be a cache miss. The disk remembers the graph; memory answers the questions.

Where this shows up