Import a directory tree with fs-to-graph

This guide shows you how to import a directory tree into a store with fs-to-graph, a first-party extension, and which queries to run on the graph it builds. Each folder becomes a :Folder node, each file a :File node holding its text when the text is small enough, and a CONTAINS relationshiprelationship A directed, typed connection between two nodes, with properties of its own; every relationship has exactly one type and a start and an end node. Read more in the glossary links every folder to what is in it. A full-text index over file names and text makes the tree searchable.

Prerequisites

• relate on your PATH (Install relate).
• fs-to-graph installed. It is on the first-party channel, and it is a self-contained binary that needs no other runtime (Install and run an extension):

relate extension install fs-to-graph

Import a directory

The examples import this tree, team/. It holds a dotfile, a 3-byte binary file, and a build folder:

team
team/.env
team/README.md
team/docs
team/docs/roadmap.md
team/logo.bin
team/src
team/src/main.rs
team/target
team/target/debug
team/target/debug/out.txt

The run writes into the store named by -d, a new tree.rl in this example. --ignore-patterns target/ leaves out the build folder:

relate run fs-to-graph team --ignore-patterns target/ -d ./tree.rl

imported 3 folders, 4 files (6 CONTAINS edges) into the store — search it with:
  MATCH (f:File) SEARCH f IN (FULLTEXT INDEX file_content FOR 'term' LIMIT 10) RETURN f.path, f.name;

Three folders are team, team/docs, and team/src. Four files are the README, the roadmap, main.rs, and logo.bin; .env is a dotfile, so the walk skipped it.

The flags come from the extension's manifest, plus relate's own -d:

With --include-hidden and no ignore pattern, the same tree imports as 5 folders, 6 files, and 10 CONTAINS relationships.

The graph it builds

A few rules decide what lands:

• path is the folder you passed, joined with each name under it. team gives team/docs/roadmap.md; an omitted DIR gives absolute paths. path is the key that git-to-graph joins on, so import both with the same DIR.
• content is present only when the whole file fits in --max-content-bytes and its first 8 KiB hold no zero byte and are valid UTF-8. When bytes after the first 8 KiB are not valid UTF-8, content holds the text up to the first invalid byte. For other files the propertyproperty A named value stored on a node or relationship, such as name: 'Ada'; values are scalars, temporal and spatial values, vectors, or lists of them. Read more in the glossary is absent, not null.
• Symbolic links are never followed, and sockets, FIFOs, and device files are skipped.
• Every nodenode An entity in the graph: it carries zero or more labels and a map of properties, and relationships connect it to other nodes. Read more in the glossary is merged on path, so importing the same tree again leaves the counts unchanged. A file deleted since the last run keeps its node; import into a new store to drop it.
• The whole import is one transaction. A failure rolls it back, and the store holds nothing from that run.

Query the graph

The cells in this section run in your browser. The first writes the graph the run built for team/, with the same keys and values, so the queries return what they return on tree.rl:

CREATE (team:Folder {path: 'team', name: 'team', depth: 0}),
       (docs:Folder {path: 'team/docs', name: 'docs', depth: 1}),
       (src:Folder {path: 'team/src', name: 'src', depth: 1}),
       (readme:File {path: 'team/README.md', name: 'README.md', extension: 'md', size_bytes: 43,
                     content: '# team\nAda, Grace and Noor work on relate.\n'}),
       (roadmap:File {path: 'team/docs/roadmap.md', name: 'roadmap.md', extension: 'md', size_bytes: 47,
                      content: 'The planner rewrite lands in the next release.\n'}),
       (main:File {path: 'team/src/main.rs', name: 'main.rs', extension: 'rs', size_bytes: 37,
                   content: 'fn main() {\n    println!("hello");\n}\n'}),
       (logo:File {path: 'team/logo.bin', name: 'logo.bin', extension: 'bin', size_bytes: 3}),
       (team)-[:CONTAINS]->(readme), (team)-[:CONTAINS]->(docs),
       (team)-[:CONTAINS]->(logo), (team)-[:CONTAINS]->(src),
       (docs)-[:CONTAINS]->(roadmap), (src)-[:CONTAINS]->(main)

(no rows) · nodes created: 7, relationships created: 6, properties set: 28, labels added: 7

CREATE FULLTEXT INDEX file_content IF NOT EXISTS FOR (f:File) ON EACH [f.name, f.content]

(no rows) · indexes added: 1

The tree as a graph:

MATCH (folder:Folder)-[c:CONTAINS]->(child)
RETURN folder, c, child

// 6 rows
[ | (n0:Folder {depth: 0, name: "team", path: "team"}), (n0)-[r3:`CONTAINS`]->(n2), (n2:Folder {depth: 1, name: "src", path: "team/src"})]
[ | (n0), (n0)-[r2:`CONTAINS`]->(n6), (n6:File {extension: "bin", name: "logo.bin", path: "team/logo.bin", size_bytes: 3})]
[ | (n0), (n0)-[r1:`CONTAINS`]->(n1), (n1:Folder {depth: 1, name: "docs", path: "team/docs"})]
[ | (n0), (n0)-[r0:`CONTAINS`]->(n3), (n3:File {content: "# team\nAda, Grace and Noor work on relate.\n", extension: "md", name: "README.md", path: "team/README.md", size_bytes: 43})]
[ | (n1), (n1)-[r4:`CONTAINS`]->(n4), (n4:File {content: "The planner rewrite lands in the next release.\n", extension: "md", name: "roadmap.md", path: "team/docs/roadmap.md", size_bytes: 47})]
[ | (n2), (n2)-[r5:`CONTAINS`]->(n5), (n5:File {content: "fn main() {\n    println!(\"hello\");\n}\n", extension: "rs", name: "main.rs", path: "team/src/main.rs", size_bytes: 37})]

Find files by a word in their name or text with the SEARCH clausesearch clause The Cypher 25 SEARCH clause, which restricts a MATCH to the top results of a vector or full-text index and binds their score. Read more in the glossary. This is the starter query the run prints, with a term filled in:

MATCH (f:File)
SEARCH f IN (FULLTEXT INDEX file_content FOR 'planner' LIMIT 10)
RETURN f.path AS file

| file                   |
|------------------------|
| "team/docs/roadmap.md" |
1 row

Count files and bytes by extension:

MATCH (f:File)
RETURN f.extension AS extension, count(*) AS files, sum(f.size_bytes) AS bytes
ORDER BY files DESC, extension

| extension | files | bytes |
|-----------|-------|-------|
| "md"      | 2     | 90    |
| "bin"     | 1     | 3     |
| "rs"      | 1     | 37    |
3 rows

List every file under a folder, at any depth, with how many levels down it sits:

MATCH p = (:Folder {path: 'team'})-[:CONTAINS*]->(f:File)
RETURN f.path AS file, length(p) AS levels
ORDER BY file

| file                   | levels |
|------------------------|--------|
| "team/README.md"       | 1      |
| "team/docs/roadmap.md" | 2      |
| "team/logo.bin"        | 1      |
| "team/src/main.rs"     | 2      |
4 rows

Find the files whose text was not stored, because they are binary or larger than --max-content-bytes:

MATCH (f:File)
WHERE f.content IS NULL
RETURN f.path AS file, f.size_bytes AS bytes

| file            | bytes |
|-----------------|-------|
| "team/logo.bin" | 3     |
1 row

On a code checkout, fs-to-graph and git-to-graph together give one store with the files and their history; Import git history with git-to-graph shows the joined queries.

Verify

Count the nodes by label, opening the store read-only:

relate --database ./tree.rl --read-only --mode table "MATCH (n) RETURN labels(n)[0] AS label, count(*) AS nodes ORDER BY label"

+----------+-------+
| label    | nodes |
+----------+-------+
| "File"   | 4     |
| "Folder" | 3     |
+----------+-------+

The counts match the run's summary line: 4 files and 3 folders.

If it goes wrong

Error: This statement tried to commit a single WAL record of at least 64 MB. This is a safety limit protecting against memory exhaustion — the per-record WAL payload cap is 64 MB; commit the data in smaller transactions. (exit code 1). The import is one transaction, and the file text it writes went past the 64 MB limit on one write-ahead log record. Nothing from the run is committed. Store less text with a lower --max-content-bytes (or 0), or leave out large folders with --ignore-patterns.

Error: the extension child exited with 1: not a directory: team/README.md (exit code 1). DIR must be a folder.

Error: the extension child exited with 1: invalid --ignore-patterns value: '/' is empty (exit code 1). A pattern is a file or folder name, optionally followed by /.

Error: no installed action 'fs-to-graph' — first-party extension available: relate extension install fs-to-graph (exit code 2). Install the extension first; relate run fetches nothing.

Related

• Import git history with git-to-graph: commits, authors, and co-changing files, joined to this graph on path.
• Install and run an extension: trust tiers, list, remove.
• SEARCH reference: full-text search options and scoring.
• Codebase context graph: the problem this graph serves.
• Context graph for a codebase: a worked recipe over a repository's records.