WAL and checkpoints

Every commit to a relate storestore A relate database: a directory, a single .rl file, or :memory:, opened with --database. Read more in the glossary is appended as one record to the store's write-ahead logWAL Append-only log each commit goes to before it applies; under the default full durability mode it is synced first, so a crash loses no acknowledged write. Read more in the glossary (WAL) before it applies to the graph in memory. A checkpointcheckpoint The step that writes a fresh snapshot of committed state and recycles old WAL segments: every 300 seconds by default, on clean exit, and on .checkpoint. Read more in the glossary writes the committed graph to the store's snapshotsnapshot The committed graph state a checkpoint writes to disk, loaded on open before the WAL replays; not the snapshot durability mode or --read-only freshness. Read more in the glossary files and deletes the WAL segments the snapshot now covers. Opening a store loads the snapshot, then replays the WAL records written after it. The WAL's format is the same under every layout and codec. Storage and durability explains why the write path has this shape.

Synopsis

relate --database <path> [--wal-dir <dir>] [--checkpoint-interval-secs <n>]
       [--durability-mode full|everysecond|snapshot] [--read-only[=live]] [<statement>]
relate query --database <path> < <script>
relate info --database <path> [--json]

Two dot commands of the interactive shell, also accepted as lines of a script piped to relate query, act on the WAL and checkpoints:

Files on disk

The store's root also holds its own .cypherlite.lock and .cypherlite.writer. The manifest.json of the last checkpoint records the checkpoint's log sequence number (LSN) as checkpoint_lsn, with its commit_ts; Storage formats lists the snapshot files beside them.

Segment format

The WAL is a series of segment files. Exactly one, the segment with the highest id, is open for appends.

A segment that holds no records is 8 bytes long: its magic alone.

Record format

What a commit appends

The record is appended to the open segment. When that segment then holds 64 MiB or 10,000 commits, it closes and a checkpoint starts. Under the default full durability modedurability mode When a commit counts as safe on disk: full (the default) syncs the WAL first, everysecond about once a second, and snapshot only at checkpoints. Read more in the glossary the record is synced to disk before the commit is acknowledged, and commits waiting together share one sync; everysecond syncs about once a second and snapshot never syncs on commit. Choose a durability mode states each mode's loss window.

What a checkpoint does

An in-memory store has no snapshot files: its automatic checkpoints only recycle the WAL, and .checkpoint writes a snapshot that lives in the process.

When a checkpoint runs

A --read-only open never checkpoints, in either freshness. A --read-only=live reader that finds a segment it still needed recycled by the writer's checkpoint reloads the current snapshot and the WAL that remains.

What replay does on open

The torn-tail rule

A torn tail is the damage a crash leaves at the very end of a segment: a record whose writing stopped part of the way through. Its commit was never acknowledged under full durability. The rule decides which damage replay drops and which stops the open:

Dropping a torn tail prints nothing. A writable open writes the segment's intact records to a new file and renames it over the old one, so a failure part of the way through leaves the original in place. Recover from a crash rehearses a crash and its recovery.

Examples

These commands write three commits to a new directory store in one script run, then print the store's metadata, the WAL folder and its one segment:

cat > team.cypher <<'EOF'
CREATE (:Person {name: 'Ada', role: 'engineer'});
CREATE (:Person {name: 'Grace', role: 'engineer'});
CREATE (:Person {name: 'Noor', role: 'maintainer'});
EOF
relate query --database ./team < team.cypher
relate info --database ./team
ls -A team/wal
cat team/wal/00000002.log; echo

{"planCache":"miss","timings":{"elapsed_ms":1.341,"execute_ms":1.173125,"lex_parse_ms":0.004666,"lower_ms":0.003875,"parse_ms":0.16037500000000002,"plan_ms":0.001708,"result_generation_ms":0.000209,"rewrite_ms":0.00025}}
{"planCache":"miss","timings":{"elapsed_ms":0.876416,"execute_ms":0.844,"lex_parse_ms":0.002209,"lower_ms":0.002,"parse_ms":0.030042000000000003,"plan_ms":0.001333,"result_generation_ms":0.000083,"rewrite_ms":0.000167}}
{"planCache":"miss","timings":{"elapsed_ms":0.8362499999999999,"execute_ms":0.795542,"lex_parse_ms":0.002375,"lower_ms":0.003791,"parse_ms":0.038416,"plan_ms":0.001458,"result_generation_ms":0.000042,"rewrite_ms":0.000209}}
layout: native directory
codec: parquet
format version: 3
checkpoint LSN: 3
commit timestamp: 3
total bytes: 2010
wal segments past checkpoint: 1
nodes (at checkpoint LSN 3): 3
relationships (at checkpoint LSN 3): 0
indexes: 0
constraints: 0
.cypherlite.lock
.cypherlite.writer
00000002.log
MDBWAL01

The run closed the store cleanly, so its checkpoint covers all three commits: checkpoint LSN: 3. The checkpoint recycled 00000001.log, and the open segment 00000002.log holds its magic and no records. wal segments past checkpoint counts the segment files that may hold commits newer than the snapshot, so an empty open segment counts as 1.

This script commits one relationship, prints the WAL and checkpoint fields, forces a checkpoint with .checkpoint, and prints them again:

cat > checkpoint.cypher <<'EOF'
MATCH (a:Person {name: 'Ada'}), (g:Person {name: 'Grace'}) CREATE (a)-[:KNOWS]->(g);
.info
.checkpoint
.info
EOF
relate query --database ./team < checkpoint.cypher | grep -E '^(checkpoint LSN|total bytes|wal segments)'

checkpoint LSN:               3
total bytes:                  2128
wal segments past checkpoint: 1
checkpoint LSN:               4
total bytes:                  3338
wal segments past checkpoint: 1

Before .checkpoint, the commit is in the WAL only: the checkpoint LSN is still 3, and the store is larger by the new record. After it, the checkpoint LSN is 4, the snapshot holds a relationships/ data file, and the WAL is back to one empty segment.

A single statement makes 10,001 commits with CALL { … } IN TRANSACTIONS OF 1 ROWS. Commit number 10,000 fills the open segment's commit count and starts a checkpoint:

relate query --database ./batch <<'EOF' | grep -E '^(checkpoint LSN|nodes)'
UNWIND range(1, 10001) AS i CALL (i) { CREATE (:Item {i: i}) } IN TRANSACTIONS OF 1 ROWS;
.info
EOF
relate info --database ./batch | grep -E '^(checkpoint LSN|nodes)'

checkpoint LSN:               10000
nodes:                        10001
checkpoint LSN: 10001
nodes (at checkpoint LSN 10001): 10001

Inside the session the checkpoint LSN is 10,000 while the live graph holds 10,001 nodes; the last commit is in the WAL. The checkpoint at exit covers it.

Errors

The first seven errors stop the open with exit code 1. On stderr each is prefixed Failed to open Relate: , and in --mode envelope it is an error object with kind storage. <segment> is the segment's name, such as 00000002.log.

The errors at open that concern the WAL folder itself, such as a --read-only open without one or --wal-dir on a .rl path, are in the storage formats errors. The errors reference lists the storage errors with their remedies.

See also

• Storage and durability: why every commit goes through the WAL, and what checkpoints trade.
• Storage formats: the snapshot files, codecs, and version markers.
• Limits and sizing: the 64 MiB record limit and the other limits.
• Choose a durability mode: when a commit's record reaches the disk.
• Recover from a crash: replay and the torn tail, step by step.
• Upgrade a store: the checkpoint that migrates an older store.