FZGPUModules | Skyler Ruiter

FZGPUModules is a CUDA C++ library for building high-throughput scientific data compression pipelines entirely on the GPU. Rather than committing to a fixed compression algorithm, the library exposes a modular directed acyclic graph (DAG) pipeline builder: stages (predictors, quantizers, coders, shufflers, transforms) are connected by the user at runtime, and the pipeline executes the full compress/decompress chain on-device with stream-ordered memory management.

Key features:

Composable pipelines — mix and match stages including Lorenzo predictors, quantizers (ABS/REL/NOA error modes), Huffman entropy coding, ANS/rANS coding, run-length encoding, bitshuffle, recursive zero elimination, zigzag and negabinary transforms, and more
CUDA Graph capture — compress passes can be captured as CUDA Graphs and replayed with near-zero CPU overhead, critical for latency-sensitive or repeatedly-invoked workflows
Smart memory management — two strategies: MINIMAL (allocate on demand, lowest peak GPU memory) and PREALLOCATE (all buffers at finalize time, enables buffer coloring to alias non-overlapping allocations)
Self-describing file format — compressed .fzm files embed full stage configuration and CRC32 checksums; pipelines can be reconstructed from a file without any external config
CLI tool — fzgmod-cli supports stage chains via --stages "lorenzo->bitshuffle->rze" or TOML preset configs, with profiling, bounds checking, and comparison reporting built in

FZGPUModules was developed as part of research into heterogeneous computing frameworks for scientific data compression. It was published at the SC ‘25 Workshops (International Conference for High Performance Computing, Networking, Storage and Analysis) and builds on prior work from the cuSZ (Argonne National Laboratory) and LC framework (Texas State University) libraries.

Tech stack: CUDA, C++17, CMake — requires CUDA 11.2+

(Ruiter et al., 2025)

Links: GitHub

arXiv

FZ Website

FZ Team Photos

FZ/ZF Joint Workshop, Washington DC. I'm pictured back middle left.

FZ/ZF Joint Workshop, Sarasota FL. I'm pictured top right.

Dr. Song's research group out for dinner.

References

2025

SC Wksp
FZModules: A Heterogeneous Computing Framework for Customizable Data Compression Pipelines

Skyler Ruiter, Jiannan Tian, and Fengguang Song

In Proceedings of the 11th International Workshop on Data Analysis and Reduction for Big Scientific Data (DRBSD-11), held in conjunction with SC’25, Nov 2025

Abs DOI arXiv Bib

Modern scientific simulations and instruments generate data volumes that overwhelm memory and storage, throttling scalability. Lossy compression mitigates this by trading controlled error for reduced footprint and throughput gains, yet optimal pipelines are highly data and objective specific, demanding compression expertise. GPU compressors supply raw throughput but often hard-code fused kernels that hinder rapid experimentation, and underperform in rate-distortion. We present FZModules, a heterogeneous framework for assembling error-bounded custom compression pipelines from high-performance modules through a concise extensible interface. We further utilize an asynchronous task-backed execution library that infers data dependencies, manages memory movement, and exposes branch and stage level concurrency for powerful asynchronous compression pipelines. Evaluating three pipelines built with FZModules on four representative scientific datasets, we show they can compare end-to-end speedup of fused-kernel GPU compressors while achieving similar rate-distortion to higher fidelity CPU or hybrid compressors, enabling rapid, domain-tailored design.
@inproceedings{ruiter2025fzmodules, title = {FZModules: A Heterogeneous Computing Framework for Customizable Data Compression Pipelines}, author = {Ruiter, Skyler and Tian, Jiannan and Song, Fengguang}, booktitle = {Proceedings of the 11th International Workshop on Data Analysis and Reduction for Big Scientific Data (DRBSD-11), held in conjunction with SC'25}, year = {2025}, month = nov, address = {St. Louis, MO, USA}, doi = {10.1145/3731599.3767376}, }