Meta AI Open-Sources OpenZL: A Format-Aware Compression Framework with a Universal Decoder

Understanding the Target Audience

The primary audience for OpenZL encompasses data engineers, software developers, and technical managers involved in data compression and processing. Their pain points include the challenges of integrating new compression formats into existing systems, the need for higher throughput and compression ratios, and the complexity of managing multiple codecs. Their goals are to enhance data processing efficiency, reduce storage costs, and streamline the decoding process. They are interested in open-source tools with robust documentation and community support, while preferring concise, technical communication that emphasizes practical applications and performance metrics.

What’s New?

OpenZL formalizes compression as a computational graph: nodes represent codecs/graphs and edges denote typed message streams. The finalized graph is serialized with the payload, allowing any frame produced by an OpenZL compressor to be decompressed by a universal decoder, thanks to the accompanying graph specification. This design merges the benefits of domain-specific codecs with the simplicity of a single, stable decoder binary.

How Does It Work?

Developers provide a data description to OpenZL, which composes parse, group, transform, and entropy stages into a directed acyclic graph (DAG) tailored to that structure. The result is a self-describing frame consisting of compressed bytes plus the graph specification. The universal decode path follows the embedded graph, eliminating the need to ship new readers when compressors evolve.

Tooling and APIs

OpenZL features the Simple Data Description Language (SDDL), which allows users to decompose inputs into typed streams from pre-compiled data descriptions. It is available in C and Python through openzl.ext.graphs.SDDL. Additionally, the core library and bindings are open-sourced, with documentation for C/C++ and Python usage, and community bindings like Rust’s openzl-sys already being developed.

Performance Metrics

The research team indicates that OpenZL achieves superior compression ratios and speeds compared to state-of-the-art general-purpose codecs across various real-world datasets. Internal deployments at Meta have consistently shown improvements in size and speed, alongside shorter compressor development timelines. Specific numeric benchmarks are not provided; instead, results are expressed as Pareto improvements reliant on data and pipeline configuration.

Editorial Comments

OpenZL makes format-aware compression practical: compressors are expressed as DAGs, embedded as a self-describing graph within each frame, and decoded by a universal decoder. This eliminates the need for reader rollouts. Meta reports Pareto gains over zstd/xz on real datasets.

For further exploration, refer to the Paper, GitHub Page, and additional Technical Details.

Illustration