Accelerating Model Inference with On‑the‑Fly Snapshot Compression

Over the past few years, the CRIU community has been exploring how to implement efficient memory compression for container snapshots. The potential benefits of this functionality range from smaller checkpoints and lower storage requirements to fewer bytes to transfer during live migration and faster restore times during inference. These performance optimizations have a significant impact on memory-intensive workloads such as LLM inference engines, where checkpoint sizes can be hundreds of gigabytes.

A straightforward approach is to compress the checkpoint data as a separate post-processing step after it is written to disk (e.g., as a compressed tar file). This simple approach is inefficient because it requires saving the full uncompressed checkpoint, reading it back for compression, and then writing the compressed snapshot. Although this overhead may be acceptable for applications with small memory footprints, the additional disk I/O, required temporary storage, and compression time become increasingly costly for memory-intensive workloads, such as LLM inference engines.

We set out to address these challenges by designing a solution that integrates compression directly into CRIU’s checkpoint and restore pipeline. Instead of performing compression as a post-processing step, our approach compresses memory pages as they are extracted from the target process and writes them directly into the checkpoint image. Similarly, the memory pages are decompressed on the fly as they are being restored. This streaming approach eliminates the need for intermediate storage of uncompressed checkpoint data, reducing both I/O overhead and storage requirements.

The rest of this blog post describes the design and implementation of this approach, along with its performance benefits and trade-offs.

Earlier compression approaches

“This could (or should) be implemented in the future” was my answer back in 2018¹ when a user asked whether CRIU compressed its image files. What followed was a journey of ideas, experiments, and incremental improvements that eventually led us to the approach described here. Some of the initial approaches came from projects using CRIU for Java applications where large checkpoint sizes were already becoming an important problem.

Compressing memory in JVM checkpoints

One such example came from the OpenJDK Coordinated Restore at Checkpoint (CRaC) project. CRaC uses CRIU to capture an initialized Java Virtual Machine (JVM), allowing Java applications to restart without repeating their initialization work. To reduce checkpoint size, a CRIU fork added image compression as a post-processing step after checkpointing, compressing pages-*.img files with LZ4 and replacing the original uncompressed images.

Before restore, the fork expands those images into temporary files under /tmp, then restores the process from the uncompressed copies. This reduces stored checkpoint size, but adds extra disk I/O, temporary storage for the decompressed images, and an additional decompression step before restore. Those costs grow with the process’s memory footprint, making the approach less practical for applications with hundreds of gigabytes of memory state. Still, this work demonstrated the value of compressing memory pages and highlighted the limitations of applying compression as a separate step, limitations that would shape the approaches that followed.

Representing zero-filled memory without storing data

A later proposal explored a different way to reduce checkpoint size. Instead of compressing page data after it had been written, CRIU could avoid writing some of that data altogether. The proposed --skip-zero-pages option focused specifically on pages containing only zero bytes. During checkpoint, CRIU would detect such pages and omit their payloads from the pages-*.img files. During restore, it would reconstruct them as zero-filled pages without reading any payload data from storage.

This approach is particularly useful for Java applications because the JVM can allocate large memory regions without immediately using every page. In those cases, storing an entire page of zeroes adds checkpoint size and I/O without preserving any meaningful data. Although the proposal was not merged upstream, it introduced an important idea, showing that reducing checkpoint overhead did not always require compressing bytes more efficiently. In some cases, CRIU could avoid storing those bytes in the first place.

Compressing checkpoint data before end-to-end encryption

Another motivation for integrated compression emerged from our work on end-to-end encryption. As container checkpoints may contain process memory with cryptographic keys, access tokens, passwords, and other confidential data, protecting them with end-to-end encryption is important. However, since encrypted data is generally not compressible, compression must happen before encryption, making its placement in the checkpoint and restore pipelines important.

CRIUsec, presented at Linux Plumbers Conference 2023², integrated encryption of CRIU image files and memory pages directly into the checkpoint and restore pipeline rather than relying on a separate file-processing step. We later presented its use in Kubernetes clusters at CloudNativeSecurityCon 2024³ and described the design in an APSys 2024 paper⁴. That work made compression a natural next step. If checkpoint data could be transformed as it flowed through CRIU for encryption, it could also be compressed before being written, avoiding the extra storage and I/O required by post-processing approaches.

Optimizing LLM inference cold starts

Checkpoint/restore is already being applied to model serving: Dynamo Snapshot⁵ uses CRIU and cuda-checkpoint to fast-start GPU workers in Kubernetes; Cloudburst⁶ uses warmed SGLang snapshots to avoid repeated server initialization; and SwapServeLLM⁷ uses transparent GPU checkpointing to hot-swap models as demand changes. In each case, as inference demand fluctuates, model-serving replicas can come online quickly to handle additional requests.

Reusing initialized runtime state

Bringing a new model replica online involves more than loading its model weights. Python needs to import packages, the inference engine needs to create memory pools, kernels are compiled or autotuned, and CUDA graphs may be captured. A replica is ready only once this initialization is complete and it can begin serving inference requests.

Restoring from a checkpoint avoids repeating much of this work. The platform initializes the server once, captures its warmed state, and later restores new replicas from that checkpoint. In effect, initialization becomes reusable data. The faster that checkpoint can be staged, read, and reconstructed, the sooner the restored replica can begin serving inference requests.

Restoring from compressed memory

CRIU stores the checkpointed memory page contents in pages-*.img files. With NVIDIA’s current checkpoint path⁸, GPU state is first copied into host memory, after which CRIU checkpoints the resulting process state. CRIU’s compression path therefore operates on host pages rather than reading device memory directly. GPU-aware checkpoint and restore require the appropriate vendor support and CRIU plugin⁹.

For inference servers, those host-side memory images can reach tens or hundreds of gigabytes. Compressing them reduces the amount of data that must be stored, transferred, and read when bringing new replicas online. This can reduce cold-start time when the time saved by moving less checkpoint data outweighs the cost of decompression. Because LZ4 is designed for fast decompression¹⁰, this cost can remain low enough for the reduced checkpoint size to translate into faster replica starts.

Where compression is performed also affects efficiency. Compressing a completed checkpoint archive with gzip or zstd can reduce storage requirements, but it introduces an additional post-processing step and may require compressed and uncompressed artifacts to coexist. With compression built directly into CRIU, memory pages are compressed before they are written to storage and decompressed as the process’s memory is restored. This avoids the additional post-processing step and intermediate uncompressed checkpoint data.

Checkpoint size and restore latency

The latest measurements come from a baseline compression comparison and a parallel block-size comparison on an AMD EPYC 9335 host with 16 CPUs available to the benchmark and an NVIDIA RTX PRO 6000 Blackwell GPU. For both sets, SGLang ran in a Podman container and checkpoint/restore used a development build of CRIU. Each condition had one warm-up and five measured trials. Podman’s outer archive compression was disabled. The compressed condition used acceleration 1. The baseline compared compression disabled with 256 KiB blocks and serial decompression. The parallel comparison tested 4 KiB, 256 KiB, 512 KiB, and 1 MiB blocks with up to 16 threads performing decompression, including the restore thread.

The baseline measures the Podman restore command, including archive import, and reports the OCI runtime and CRIU restore_time as nested intervals. The block-size comparison also measures from the start of that command to the first streamed token. Fresh-start TTFT is shown separately as context: it pools ten pre-checkpoint launches per model, five collected alongside each baseline condition, and is not a compression-specific restore result.

Note that CRIU reads LZ4-compressed page data through buffered I/O even with --image-io-mode direct. CRIU can still use direct I/O for aligned raw and zero ranges¹¹.

On-the-fly memory compression

At Linux Plumbers Conference 2025¹², we presented the first prototype of CRIU-LZ4. In contrast to previous approaches, which compressed checkpoint images as a post-processing step, CRIU-LZ4 integrated LZ4 directly into the checkpoint/restore pipeline by extending CRIU’s pagemap metadata and image format to support compressed memory. The prototype also demonstrated that it was possible to checkpoint and restore a memory-intensive workload entirely within an in-memory filesystem, illustrating that built-in compression can make large checkpoints practical even without backing storage. During the Q&A¹³, Andrei Vagin suggested retaining compression only when it provides a worthwhile reduction. That review shifted the design from compressing every page to choosing the most useful representation for each block, as described below.

The figure below follows page data through CRIU’s compressed-memory checkpoint and restore paths. It also shows how restore handles the three possible block representations: zero, raw (stored without compression), and LZ4-compressed.

Decompressing before the restorer PIE

During the final restore phase, the CRIU process becomes the process being restored. It maps the restorer PIE into a temporary mapping within its address space that does not overlap either CRIU’s current mappings or the restored process’s final virtual memory areas (VMAs). From there, the restorer removes CRIU’s remaining mappings, recreates the saved memory layout, and resumes the saved execution state. The restorer PIE runs without a dynamic loader, so it cannot use shared libraries such as liblz4.

CRIU therefore decodes LZ4 blocks before jumping to the restorer PIE. An earlier prototype used a helper process, but after several design discussions, we moved this work into the normal page reader¹⁴. The restore path uses the normal page reader to fill the process’s private memory regions, decompressing LZ4 blocks as it reads them. Raw and zero ranges can still be populated later by the restorer. Hugetlb and external-plugin mappings cannot use this path, so CRIU stores their blocks as raw or zero.

Encoding memory blocks

The current implementation offers two ways to divide memory into LZ4 blocks:

• --compress encodes each system page independently. On the x86 benchmark host, that means one 4 KiB LZ4 block per page.
• --compress-block SIZE groups consecutive pages and encodes the group as one block. The size must be page-aligned and no larger than 4 MiB.

Page-sized blocks preserve fine-grained access and work across the widest set of CRIU paths. Multi-page blocks give LZ4 a larger history, so it can find repetitions across page boundaries and amortize codec calls and metadata. The price is coarser partial reads: asking for one page can require decoding its whole block.

Every block has one of three representations in the pagemap schema:

The writer first detects an all-zero block. Otherwise, it runs LZ4 and keeps the result only when it is smaller than seven eighths of the original. A weakly compressible block is stored raw, avoiding decode work for a marginal saving. If every block in a pagemap entry falls back to raw, CRIU drops the compression metadata for that entry and leaves restore’s ordinary contiguous fast path available. The final dump path therefore avoids expanding the page payload for incompressible data, although the compression attempt, alignment, and surrounding metadata are not free.

The blocks submessage records the layout needed to locate and decode the stored data. block_sizes[] gives each block’s stored length. total_payload_size lets the reader skip a complete pagemap entry without summing the array, while pages_per_block records the block granularity. The inventory records the image-wide compression mode and an informational block size; each pagemap entry’s blocks submessage supplies the layout used to decode that entry.

Restoring incremental checkpoint chains

An incremental checkpoint stores only pages that changed since its parent. During restore, an inherited range is passed to the parent reader, which can recurse through further generations. Each reader interprets its own pagemap, so ordinary, ZERO, RAW, and LZ4 entries can coexist in one chain without expanding the checkpoints first. The readers share one encoded-read context, keeping decompression batches within the same restore-wide working-set limit.

Large blocks can amplify sparse parent reads because restoring one inherited page may require decompressing its complete block. CRIU batches copy-on-write comparison reads in groups of at most 256 pages and aligns them to block boundaries when possible. If a request still covers only part of an LZ4 block, the parent reader caches that block. Later slices can then be copied from the cache instead of reading and decompressing the same parent block again.

Restoring variable-length blocks

CRIU does not simply read pages-*.img from beginning to end. It reconstructs private mappings, shared memory, memfd regions, copy-on-write relationships, and pages inherited from earlier checkpoints. A request can start in the middle of a pagemap entry, and a block can cross destination iovec boundaries.

The reader handles this in four steps:

1. Resolve the requested blocks and validate their counts, sizes, offsets, and expected decoded length before those values drive allocations.
2. Read one packed payload for a bounded batch.
3. Copy raw blocks and fill simple zero runs directly, then describe eligible zero and LZ4 blocks as independent jobs.
4. Decode into the final mappings, using a scratch buffer only when one block spans multiple destination vectors.

For pagemap entries with compression metadata, CRIU limits each restore batch to 32 MiB of page data after decoding. This keeps temporary input buffers and block descriptions bounded even for very large checkpoints. The 32 MiB value is an implementation limit, not part of the checkpoint format. CRIU may keep two batches active: while workers decode one batch, the caller may prefetch the next if it can acquire the second slot without waiting. The details are in the final encoded reader and prefetch path.

Decompressing blocks in parallel

--decompress-threads N controls the aggregate worker concurrency used for LZ4 blocks and eligible large zero fills:

• 1, the default, keeps each decode serial. Separate private, shared-memory, and memfd requests may still make progress independently.
• 0 asks CRIU to choose a width from its CPU affinity, the number and decoded size of blocks in the batch, and the shared restore CPU budget.
• A value greater than one is an upper bound, not a promise to start that many threads. CRIU reduces impossible requests to the available CPU count.

Small batches remain serial. Parallel work needs enough decoded data per active thread to amortize scheduling, and the encoded staging-memory bound is independent of the thread count. These constraints make 0 safe to experiment with, but they do not make it the correct default for a serving node. Eight cores spent shortening one restore may delay eight other replicas or the control plane.

Automatic width follows sched_getaffinity(), not a cgroup’s CFS CPU quota. For a quota-constrained container with a broad affinity mask, use a cpuset or an explicit cap. The shared worker budget is also local to one CRIU restore. An orchestrator still has to limit aggregate CPU demand across simultaneous restores.

The baseline comparison used the serial default, while the block-size comparison allowed up to 16 threads to perform decompression, including the restore thread. Because neither benchmark varied the worker count while holding the other variables constant, the results do not isolate the effect of parallel decompression. I would start with 1 under concurrent scale-out and benchmark 0, 2, 4, and 8 only when spare CPU is available and decompression is visible in the restore profile.

Offline memory compression

CRIT can convert an existing checkpoint after the process has been dumped. This is not generic archive compression: CRIT understands CRIU’s image format. It rewrites every task and shared-memory pages-*.img/pagemap-*.img pair and updates inventory.img. The result uses the same version 1.2 block format as on-the-fly compression and can be restored directly, without first expanding a temporary copy.

The conversion requires the Python lz4 package:

python3 -m pip install lz4
crit compress checkpoint/
crit decompress checkpoint/

crit compress validates each pagemap/page-image pair and applies the ZERO, RAW, and LZ4 rules to each present system page. It does not regroup an existing image into larger blocks. Non-zero pages in hugetlb and plugin-managed VMAs stay raw because those restore paths cannot consume LZ4 blocks.

crit decompress accepts page-sized and multi-page blocks, validates the complete image set, and expands each entry into an ordinary contiguous page payload. Both commands stage and synchronize replacements before changing live names, then roll back the complete image set on failure. Originals remain as .bak hard links by default. --in-place omits persistent backups. The operational behavior is documented in the CRIT manual.

Enabling memory compression

Until these changes land upstream, both checkpoint and restore hosts need an LZ4-enabled CRIU build. The CRIU manual documents the available compression options.

-c is the short form of --compress. It selects one LZ4 block per system page and supports page-server and image-streaming workflows. --compress-block SIZE selects an explicit block size. Page-sized blocks support page-server and image-streaming workflows; multi-page blocks currently require local images. If both options are present, the last one wins.

First check that CRIU was built with LZ4 support:

criu check --feature compress

For a local process checkpoint, my starting configuration is:

mkdir -p checkpoint
criu dump --tree "$PID" --images-dir checkpoint --compress-block 256K
criu restore --images-dir checkpoint

The compression choice is stored in inventory.img. The dump flag should not be repeated on restore. Restore-only concurrency can be selected independently:

criu restore --images-dir checkpoint --decompress-threads 0

For an OCI runtime that invokes CRIU through runc, put the equivalent settings in runc’s CRIU configuration:

# /etc/criu/runc.conf
compress-block 262144
decompress-threads 1

Use compress instead of compress-block in runc.conf to select page-sized compression.

This is the relevant integration point for Podman and for Kubernetes nodes where the CRI runtime delegates checkpoint and restore to runc. Kubernetes does not expose these CRIU tuning options through its checkpoint API, so provision the configuration on every participating node and verify that its runtime uses the expected runc and CRIU installation.

Conclusion

CRIU can now make the compression trade where pages enter and leave the memory image, without creating a second artifact or making a second pass over the checkpoint. ZERO, RAW, and LZ4 representations avoid storing bytes that do not help, while premap restore and bounded parallel decompression keep that choice inside CRIU’s existing memory path.

The measurements also show that a smaller image does not guarantee lower restore latency. Serial decompression slowed CRIU for the smaller models, while the larger models had lower nested medians with LZ4. In the parallel Qwen 4B comparison, 1 MiB had the lowest median for each reported metric. Treat any default as a starting point: larger blocks suit local, I/O-bound images, while page-sized blocks preserve page-server, streaming, and lazy-page compatibility. On memory-speed storage, compression may not help. Change the block size, worker budget, or LZ4 acceleration only after profiling the production restore path.

Acknowledgements

Thanks to Andrei Vagin and Alexander Mikhalitsyn for their detailed review and design feedback. I am grateful to Viktória Spišaková and Adrian Reber for our long-standing collaboration, including on the work that led to the CRIU-LZ4 prototype and EuroMLSys paper. I also thank my PhD supervisors, Rodrigo Bruno and Wes Armour, for their guidance and support throughout this research. Thanks to Fergus Finn, Debosmit Ray, and the teams at Doubleword and DevZero for their support in bringing this work to production environments.