Sync Data:
Negentropy and LeafSync
Sync File Chunks with LeafSync
Sync Roots with Negentropy
Merkle Tree Storage and Syncing
on Nostr
Deduplicate storage with content-addressing: never store the same file twice.
Store data as a chunked Merkle Tree. Attach files to a Nostr Note by including the file's root.
History of Merkle Trees
Hashed tree structures have a long open-source legacy. From the invention of Merkle Trees by Ralph Merkle in the 80s, later adopted by bitcoin, to Git’s use of hashed DAGs and IPFS's implementation of Merkle DAGs for file storage. However, Classic Merkle Trees cannot contain a folder of files. Merkle DAGs were developed as a solution to incorporate a directory of files into a similar Merklized structure.
However, Merkle DAGs require users to download the hash of every other file inside the folder its stored in. This becomes inefficient when dealing with folders that contain large amounts of files, or very large single files. You must download the hashes to all of the other chunks to retrieve a single file. This requirement can lead to large Merkle DAG branches that Scionic Merkle Trees avoid.
Scionic Merkle Trees exhibit logarithmic growth. This means that as the number of files in a folder increases, the size of each Scionic Merkle branch decreases. This makes Scionic Merkle Trees an efficient structure for managing large datasets, as the branches become exponentially smaller with the increasing number of files in the folder.
Best of Both Worlds: Compatible with All File Types and Folders
Scionic Merkle Tree are a new type of Merkle Tree/Merkle DAG structure for file chunking. Scionic Merkle Trees have small branches like Classic Merkle trees, the file/folder storage ability of Merkle DAGs, and numbered leaves so relays can request a range of missing leaf numbers in order to sync succinctly. Merging the advantages of Classic Merkle trees and Merkle DAGs gives H.O.R.N.E.T. Storage the best of both worlds: file storage and small branches. This will unlock a new tier of speed for a variety of decentralized apps that require large file directories.
In plant grafting, the scion is the upper part of the plant, chosen for its desirable fruits or flowers. It's grafted onto another plant's base to grow together. In a similar vein, the Scionic Merkle Tree was born from grafting together Merkle DAGs and Classic Merkle Trees. This process emphasizes why we use the term "scion" for the Scionic Merkle Trees. The name symbolizes the digital grafting of these two similar data structures, combining their strengths into one piece.
Airlock Uses Scionic Merkle Trees for GitNestr Permissions
Include Phase D here on launch day.
Scionic Merkle Tree
Merkle Tree
File Storage and Syncing
on Nostr
Deduplicate storage with content-addressing: never store the same file twice.
Store data as a chunked Merkle Tree. Attach files to a Nostr Note by including the file's root.
Scionic Merkle Branch
Scionic Merkle Tree
First, the Scionic Merkle Root of each file is synced with Negentropy's RBSR. The Merkle root of each file is inspected so the total number of leaves/chunks for that file is revealed. After, H.O.R.N.E.T Storage checks if any of those file chunks/leaf numbers are currently saved. Finally, LeafSync is initiated to request the missing file chunks by listing a range of leaf numbers, e.g. "merkle_root:1-50" that represents a request for the missing file chunks.
