Clone and convert a store

This guide shows you how to copy a relate storestore A relate database: a directory, a single .rl file, or :memory:, opened with --database. Read more in the glossary into a new one, change its layout or its codec on the way, and check that the copy holds the same graph. The command is relate clone <source> <dest>: the destination's path picks the layout (a path ending in .rl is one file, any other path a directory storedirectory store A store kept as a folder of snapshot files, a WAL folder and lock files; the layout for any --database path that is not :memory: and does not end in .rl. Read more in the glossary), and --to-snapshot-format parquet|json|markdown picks the codec. The source is opened read-only and never written.

Prerequisites

• relate on your PATH (Install relate).
• A store to copy. The commands on this page start from the shared example graph in a single-file store, team.rl:

cat > team.cypher <<'EOF'
CREATE (ada:Person {name: 'Ada', role: 'engineer'}),
       (grace:Person {name: 'Grace', role: 'engineer'}),
       (noor:Person {name: 'Noor', role: 'maintainer'}),
       (relate:Project {name: 'relate', language: 'Rust'}),
       (tck:Project {name: 'opencypher-tck', language: 'Gherkin'}),
       (ada)-[:WORKS_ON {since: 2024}]->(relate),
       (grace)-[:WORKS_ON {since: 2023}]->(relate),
       (grace)-[:WORKS_ON {since: 2022}]->(tck),
       (noor)-[:MAINTAINS]->(relate),
       (ada)-[:KNOWS]->(grace);
EOF
relate query --database ./team.rl < team.cypher

Decide when to convert

A store's layout and codeccodec The on-disk encoding of a store's snapshot: Parquet (the default), JSON or markdown, fixed when the store is created. Read more in the glossary are fixed when it is created: --snapshot-format applies only to an empty directory. To change either, write a new store with relate cloneclone relate clone: the command that copies a store into a new one, optionally in another codec, reading the source without writing it. Read more in the glossary:

The storage formats reference has the files each layout and codec writes.

Change the layout

Clone the .rl file into a directory, then the directory back into a second .rl file. relate clone prints nothing on success:

relate clone ./team.rl ./team
relate clone ./team ./team-copy.rl

The destination must not exist yet; relate clone never overwrites. It reads the state committed when it opens the source and does not follow later commits. A directory source can be cloned while another process writes to it; a .rl file.rl The file extension of a single-file store; a --database path that ends in .rl selects that layout. Read more in the glossary only while no process holds it open for writing.

Change the codec

Without --to-snapshot-format, the destination keeps the source's codec. With it, the destination's first checkpoint is written in the codec it names. Write a json copy and a markdown copy of the directory store:

relate clone --to-snapshot-format json ./team ./team-json
relate clone --to-snapshot-format markdown ./team ./team-md

relate info reads a store's layout and codec without opening it:

relate info --database ./team-md | head -n 2

layout: native directory
codec: markdown

The json copy holds the graph in one graph.json. The markdown copy holds one note per node, in a folder per label, with an index note per folder, a root index.md and a log.md:

ls team-json team-md team-md/Person

team-json:
graph.json
manifest.json
wal

team-md:
Person
Project
index.md
log.md

team-md/Person:
Ada.md
Grace.md
Noor.md
index.md

The markdown codec names each note after a key property of its label, name here, so every node of a label needs that property, and its value must be unique within the label. --markdown-body-properties chooses the properties written as each note's body; Open a store as an Obsidian vault covers both.

Check that the copy holds the same graph

Compare what a statement returns from the source and from the copy. This check writes every node's labels and properties and every relationship to a file, sorted, once for the source and once for a store that went from .rl to markdown and back to a parquet .rl file. The jsonl mode ends each statement with a line of its timings, which change on every run, so grep -v '"timings"' drops those lines before the comparison:

cat > graph-check.cypher <<'EOF'
MATCH (n) RETURN labels(n) AS labels, properties(n) AS props ORDER BY n.name;
MATCH (a)-[r]->(b) RETURN a.name AS from, type(r) AS type, properties(r) AS props, b.name AS to ORDER BY from, type, to;
EOF
relate clone --to-snapshot-format parquet ./team-md ./team-roundtrip.rl
relate query --database ./team.rl --read-only --mode jsonl < graph-check.cypher | grep -v '"timings"' > before.jsonl
relate query --database ./team-roundtrip.rl --read-only --mode jsonl < graph-check.cypher | grep -v '"timings"' > after.jsonl
cmp before.jsonl after.jsonl && echo "same graph: $(wc -l < after.jsonl | tr -d ' ') rows"

same graph: 10 rows

Five nodes and five relationships come back identical. The way back names its codec: in relate 0.1.2, a .rl destination refuses the body-property configuration a markdown store records, so relate clone ./team-md ./team-roundtrip.rl without --to-snapshot-format, or with --to-snapshot-format markdown, exits 1 with Failed to clone database: semantic error: markdown body properties are set but the store's snapshot codec is parquet, not markdown; the configuration would be ignored. A markdown store clones into a .rl file with --to-snapshot-format parquet or json, and into a directory store with any codec. Order the rows by a property your graph has on every node. Values without a YAML form keep their type in a markdown note: a date or a point is written as a small map tagged __type and reads back as a date or a point. One difference to expect from a markdown round trip: a node with two or more labels can return them from labels(n) in another order, so sort them before comparing such a graph.

Measure the sizes

A .rl file grows to the most space it has needed and reuses it; it does not shrink when data is deleted. relate clone writes only the current state. To see it, add 20,000 draft nodes to team.rl, delete them again, and clone it:

relate info --database ./team.rl | grep 'total bytes'
relate --database ./team.rl --mode envelope "UNWIND range(1, 20000) AS i CREATE (:Draft {id: i, text: 'draft note ' + toString(i)})"
relate --database ./team.rl --mode envelope "MATCH (d:Draft) DELETE d"
relate clone ./team.rl ./team-compact.rl

total bytes: 36864
{"columnTypes":[],"columns":[],"notifications":[],"planCache":"miss","rows":[],"timings":{"elapsed_ms":66.759917,"execute_ms":66.566792,"lex_parse_ms":0.006292,"lower_ms":0.007292,"parse_ms":0.182708,"plan_ms":0.005332999999999999,"result_generation_ms":0.000125,"rewrite_ms":0.000375},"truncated":false}
{"columnTypes":[],"columns":[],"notifications":[],"planCache":"miss","rows":[],"timings":{"elapsed_ms":21.065709,"execute_ms":20.903374999999997,"lex_parse_ms":0.002333,"lower_ms":0.0029579999999999997,"parse_ms":0.154167,"plan_ms":0.003583,"result_generation_ms":0.000417,"rewrite_ms":0.00020800000000000001},"truncated":false}

The method: relate info prints each store's size on disk as total bytes, read without opening the store. The loop prints it for the bloated file, its clone, and the three copies made in Change the codec and Change the layout:

for store in team.rl team-compact.rl team team-json team-md; do printf '%s: ' "$store"; relate info --database "./$store" | grep 'total bytes'; done

team.rl: total bytes: 3203072
team-compact.rl: total bytes: 32768
team: total bytes: 3556
team-json: total bytes: 1385
team-md: total bytes: 2926

These numbers are for this 5-node graph. At 100,000 nodes the json codec is about 3.4 times the size of parquet (measured in the formats reference), so the sizes on a small graph don't predict the order on a large one.

Verify

The converted store opens and answers statements. The markdown copy returns who works on what:

relate --database ./team-md --read-only --mode envelope "MATCH (p:Person)-[:WORKS_ON]->(proj:Project) RETURN p.name AS name, proj.name AS project ORDER BY name, project"

{"columnTypes":["STRING","STRING"],"columns":["name","project"],"notifications":[],"planCache":"miss","rows":[{"name":"Ada","project":"relate"},{"name":"Grace","project":"opencypher-tck"},{"name":"Grace","project":"relate"}],"timings":{"elapsed_ms":0.55875,"execute_ms":0.382,"lex_parse_ms":0.033417,"lower_ms":0.039709,"parse_ms":0.1085,"plan_ms":0.058542000000000004,"result_generation_ms":0.000375,"rewrite_ms":0.008291999999999999},"truncated":false}

If it goes wrong

• Error: unexpected argument to clone: --snapshot-format (exit code 2). The flag on relate clone is --to-snapshot-format:

  relate clone --snapshot-format json ./team ./team-json2

  Error: unexpected argument to clone: --snapshot-format
  Did you mean '--to-snapshot-format'?
  Usage: relate clone [--wal-dir DIR] [--to-snapshot-format F] [--markdown-body-properties LIST] SOURCE DEST
  Run 'relate clone --help' for more information.

  A --snapshot-format placed before the verb, as in relate --snapshot-format json clone …, does not change the destination's codec: the copy keeps the source's. Put --to-snapshot-format after clone.

• Failed to clone database: semantic error: markdown body properties are set but the store's snapshot codec is parquet, not markdown; the configuration would be ignored (exit code 1). The destination is a .rl path and the source is a markdown store, or the command passes --markdown-body-properties. In relate 0.1.2 a .rl destination refuses that configuration, even with --to-snapshot-format markdown. Pass --to-snapshot-format parquet or json for a .rl copy of a vault, or clone it into a directory store.

• Failed to clone database: storage error: clone destination already exists (exit code 1). relate clone never overwrites. Pick a new destination path.

• Failed to clone database: storage error: markdown vault: node <id> of label <Label> has no '<property>' property; the markdown vault requires a non-null unique '<property>' on :<Label> (exit code 1). A markdown destination names each note after a key property, and a node of that label lacks it. Set the property on every node of the label first. A failed clone removes the destination it created.

• Failed to clone database: storage error: markdown vault: two nodes of label <Label> share the key value '<value>'; the vault filename would collide (exit code 1). Two nodes of one label have the same key value. Make the values unique, or keep the store in parquet or json.

• Error: invalid --to-snapshot-format: storage error: unknown snapshot codec '<value>' (expected parquet, json, markdown or rkyv) (exit code 2). The codec name is misspelled.

• Failed to clone database: storage locked: <path> (exit code 1). The source is a .rl file that a process holds open for writing. Clone it when that process exits (storage locked).

Related

• relate clone: the command's flags and exit codes.
• Storage formats: the files each layout and codec writes, and sizes at 100,000 nodes.
• Choose a store layout: which layout and codec fit a job.
• Back up and restore a store: relate clone as the backup command.
• Single-file graph in the capability catalog.