A developer shares work on optimizing value block compression for the Marginalia project's index using zigzag delta encoding combined with byte-plane reshuffling and LZ4 compression. The technique sacrifices L1 cache performance for dramatically better compression ratios, saving approximately 4 TB of disk space overall with around 70% compression in standard LZ4 mode.
Background
Marginalia is a personal knowledge/annotation tool written in Zig by David Kastrup, focused on storing and retrieving marginal notes efficiently. This discussion centers on index compression techniques for the tool.
- Source
- Lobsters
- Published
- Sep 21, 2026 at 08:30 PM
- Score
- 6.0 / 10