Accelerating Model Inference with On‑the‑Fly Snapshot Compression

Over the past few years, the CRIU community has been exploring how to implement efficient memory compression for container snapshots. The potential benefits of this functionality range from smaller checkpoints and lower storage requirements to fewer bytes to transfer during live migration and faster restore times during inference. These performance optimizations have a significant impact on memory-intensive workloads such as LLM inference engines, where checkpoint sizes can be hundreds of gigabytes.

A straightforward approach is to compress the checkpoint data as a separate post-processing step after it is written to disk (e.g., as a compressed tar file). This simple approach is inefficient because it requires saving the full uncompressed checkpoint, reading it back for compression, and then writing the compressed snapshot. Although this overhead may be acceptable for applications with small memory footprints, the additional disk I/O, required temporary storage, and compression time become increasingly costly for memory-intensive workloads, such as LLM inference engines.

We set out to address these challenges by designing a solution that integrates compression directly into CRIU’s checkpoint and restore pipeline. Instead of performing compression as a post-processing step, our approach compresses memory pages as they are extracted from the target process and writes them directly into the checkpoint image. Similarly, the memory pages are decompressed on the fly as they are being restored. This streaming approach eliminates the need for intermediate storage of uncompressed checkpoint data, reducing both I/O overhead and storage requirements.

The rest of this blog post describes the design and implementation of this approach, along with its performance benefits and trade-offs.

Earlier compression approaches

“This could (or should) be implemented in the future” was my answer back in 2018¹ when a user asked whether CRIU compressed its image files. What followed was a journey of ideas, experiments, and incremental improvements that eventually led us to the approach described here. Some of the initial approaches came from projects using CRIU for Java applications where large checkpoint sizes were already becoming an important problem.

Compressing memory in JVM checkpoints

One such example came from the OpenJDK Coordinated Restore at Checkpoint (CRaC) project. CRaC uses CRIU to capture an initialized Java Virtual Machine (JVM), allowing Java applications to restart without repeating their initialization work. To reduce checkpoint size, a CRIU fork added image compression as a post-processing step after checkpointing, compressing pages-*.img files with LZ4² and replacing the original uncompressed images.

Before restore, the fork expands those images into temporary files under /tmp, then restores the process from the uncompressed copies. This reduces stored checkpoint size, but adds extra disk I/O, temporary storage for the decompressed images, and an additional decompression step before restore. Those costs grow with the process’s memory footprint, making the approach less practical for applications with hundreds of gigabytes of memory state. Still, this work demonstrated the value of compressing memory pages and highlighted the limitations of applying compression as a separate step, limitations that would shape the approaches that followed.

Representing zero-filled memory without storing data

A later proposal explored a different way to reduce checkpoint size. Instead of compressing page data after it had been written, CRIU could avoid writing some of that data altogether. The proposed --skip-zero-pages³ option focused specifically on pages containing only zero bytes. During checkpoint, CRIU would detect such pages and omit their payloads from the pages-*.img files. During restore, it would reconstruct them as zero-filled pages without reading any payload data from storage.

This approach is particularly useful for Java applications because the JVM can allocate large memory regions without immediately using every page. In those cases, storing an entire page of zeroes adds checkpoint size and I/O without preserving any meaningful data. Although the proposal was not merged upstream, it introduced an important idea, showing that reducing checkpoint overhead did not always require compressing bytes more efficiently. In some cases, CRIU could avoid storing those bytes in the first place.

Compressing checkpoint data before end-to-end encryption

Our work on end-to-end encryption for CRIU checkpoints⁴ also influenced the design. Checkpoints can contain cryptographic keys, access tokens, passwords, and other confidential data⁵, so they need to be protected⁶. However, encrypted data does not compress well, which means compression has to happen first.

Compression therefore needed to be applied within CRIU’s checkpoint and restore data paths. Memory pages could be compressed before being encrypted and written, avoiding the extra storage and I/O of a separate post-processing step.

Optimizing LLM inference cold starts

Checkpoint/restore is already being applied to model serving: Dynamo Snapshot⁷ uses CRIU and cuda-checkpoint to fast-start GPU workers in Kubernetes; Cloudburst⁸ uses warmed SGLang snapshots to avoid repeated server initialization; and SwapServeLLM⁹ uses transparent GPU checkpointing to hot-swap models as demand changes. In each case, as inference demand fluctuates, model-serving replicas can come online quickly to handle additional requests.

Reusing initialized runtime state

Bringing a new model replica online involves more than loading its model weights. Python needs to import packages, the inference engine needs to create memory pools, kernels are compiled or autotuned, and CUDA graphs may be captured. A replica is ready only once this initialization is complete and it can begin serving inference requests.

Restoring from a checkpoint avoids repeating much of this work. The platform initializes the server once, captures its warmed state, and later restores new replicas from that checkpoint. In effect, initialization becomes reusable data. The faster that checkpoint can be staged, read, and reconstructed, the sooner the restored replica can begin serving inference requests.

Restoring from compressed memory

CRIU stores the checkpointed memory page contents in pages-*.img files. With NVIDIA’s current checkpoint path¹⁰, GPU state is first copied into host memory, after which CRIU checkpoints the resulting process state. CRIU’s compression path therefore operates on host pages rather than reading device memory directly. GPU-aware checkpoint and restore require the appropriate vendor support and CRIU plugin¹¹.

For inference servers, those host-side memory images can reach tens or hundreds of gigabytes. Compressing them reduces the amount of data that must be stored, transferred, and read when bringing new replicas online. This can reduce cold-start time when the time saved by moving less checkpoint data outweighs the cost of decompression. Because LZ4 is designed for fast decompression¹², this cost can remain low enough for the reduced checkpoint size to translate into faster replica starts.

Modal described a related restore-side trade-off for containerized GPU inference applications¹³. Modal’s snapshot pipeline uses gVisor’s runsc and creates checkpoints without the default gzip compression because DEFLATE decompression is single-threaded and slower than the storage layers supplying the data in their system. This is an example of why reducing checkpoint size is not enough. Decompression must also be fast enough that it does not become the restore bottleneck.

Where compression is performed also affects efficiency. Compressing a completed checkpoint archive with gzip or zstd can reduce storage requirements, but it introduces an additional post-processing step and may require compressed and uncompressed artifacts to coexist. With compression built directly into CRIU, memory pages are compressed before they are written to storage and decompressed as the process’s memory is restored. This avoids the additional post-processing step and intermediate uncompressed checkpoint data.

On-the-fly memory compression

At Linux Plumbers Conference 2025¹⁴, we presented the first prototype of CRIU-LZ4. In contrast to previous approaches, which compressed checkpoint images as a post-processing step, CRIU-LZ4 integrated LZ4 directly into the checkpoint/restore pipeline by extending CRIU’s pagemap metadata and image format to support compressed memory. The prototype also demonstrated that it was possible to checkpoint and restore a memory-intensive workload entirely within an in-memory filesystem, illustrating that built-in compression can make large checkpoints practical even without backing storage. During the Q&A¹⁵, Andrei Vagin suggested retaining compression only when it provides a worthwhile reduction. That review shifted the design from compressing every page to choosing the most useful representation for each block, as described below.

The figure below follows page data through CRIU’s compressed-memory checkpoint and restore paths. It also shows how restore handles the three possible block representations: zero, raw (stored without compression), and LZ4-compressed.

Compressing memory blocks

The current implementation offers two ways to divide memory into LZ4 blocks:

• --compress uses each memory page as a separate compression block. On the x86 benchmark host, that means one 4 KiB block per page.
• --compress-block SIZE groups consecutive pages into one compression block. The size must be page-aligned and no larger than 4 MiB.

Page-sized blocks preserve fine-grained access and work across the widest set of CRIU paths. Because CRIU compresses each block independently, multi-page blocks allow LZ4 matches to cross page boundaries within its roughly 64 KiB window. They also spread per-block processing and metadata overhead across more pages. The trade-off is that asking for one page can require decoding its whole block.

Every block has one of three representations in the pagemap schema:

During checkpoint, CRIU first checks whether a block contains only zeroes. For other blocks, it runs LZ4 and keeps the compressed result only if it reduces the block size by more than 12.5%. Otherwise, CRIU stores the original bytes because the small space saving does not justify the performance overhead of decompression during restore. This per-block optimization is especially useful for LLM inference checkpoints, where blocks containing model weights may offer little size reduction with LZ4 while other memory blocks remain compressible. If every block in a pagemap entry falls back to raw, CRIU omits the block metadata and writes the entry in the standard uncompressed page-image format, so restore does not perform decompression. The final dump path therefore avoids expanding the page payload for incompressible data, although the compression attempt, alignment, and surrounding metadata are not free.

The blocks field contains the metadata needed to locate and decode the stored data. block_sizes[] gives each block’s stored length. total_payload_size lets the page reader skip a complete pagemap entry without summing the array, while pages_per_block records the block granularity. The inventory records the image-wide compression mode and an informational block size. When present, an entry’s blocks field contains its block metadata.

Decompressing before the restorer PIE

During the final restore phase, the CRIU process becomes the process being restored. It maps the restorer PIE into a temporary mapping within its address space that does not overlap either CRIU’s current mappings or the restored process’s final virtual memory areas (VMAs). From there, the restorer removes CRIU’s remaining mappings, recreates the saved virtual memory mappings, and resumes the saved execution state. The restorer PIE runs without a dynamic loader, so it cannot use shared libraries such as liblz4.

CRIU therefore decodes LZ4 blocks before jumping to the restorer PIE. An earlier prototype used a helper process, but after several design discussions, we moved this work into CRIU’s page reader, the component that uses pagemap metadata to locate memory pages in the corresponding pages-*.img file¹⁶. The restore path uses the page reader to fill the process’s private memory regions, decompressing LZ4 blocks as it reads them. Raw and zero ranges can still be populated later by the restorer. Hugetlb and external-plugin mappings cannot use this path, so CRIU stores their blocks as raw or zero.

Restoring variable-length blocks

CRIU does not simply read pages-*.img from beginning to end. It reconstructs private mappings, shared memory, memfd regions, copy-on-write relationships, and pages inherited from earlier checkpoints. A request can start in the middle of a pagemap entry, and a block can cross destination iovec boundaries. When a restore request starts and ends at block boundaries, CRIU can process it asynchronously in a bounded batch. It first resolves the requested blocks. Before allocating temporary buffers for the batch, it validates their count, stored sizes, offsets, and total uncompressed length. CRIU then reads the stored bytes for the batch. It copies RAW data and fills simple ZERO ranges directly. It creates independent jobs for LZ4 blocks with contiguous destinations. When parallel zero-filling is enabled, large ZERO ranges can also run as independent jobs.

LZ4 output is written directly to the restored mappings when the destination is contiguous. If a block spans multiple destination iovecs, CRIU uses a scratch buffer for decompression and then copies each segment to its destination. Requests that start or end within a block use a synchronous path, which reads blocks individually and caches a decompressed LZ4 block when only part is needed. For pagemap entries with compression metadata, CRIU limits each restore batch to 32 MiB of page data after decoding. This keeps temporary input buffers and block descriptions bounded even for very large checkpoints. The 32 MiB value is an implementation limit, not part of the checkpoint format. To overlap image reads with decompression, CRIU may keep two bounded batches active. While workers decompress one batch, the restore thread reads the next if a second slot is immediately available. The details are in the final block reader and prefetch path.

Note that CRIU reads LZ4-compressed page data through buffered I/O even with --image-io-mode direct¹⁷. CRIU can still use direct I/O when reading aligned raw page data, while all-zero pages are restored without reading from the pages image.

Restoring incremental checkpoints

CRIU supports incremental checkpointing, where each subsequent checkpoint stores only the memory pages that changed since the previous checkpoint. This previous checkpoint is called the parent checkpoint and may also have its own parent. During restore, CRIU follows the chain of checkpoints to find the page data. The page reader for each checkpoint interprets that checkpoint’s pagemap, so standard uncompressed entries can coexist with entries containing ZERO, RAW, or LZ4 blocks without expanding the checkpoints first. These page readers share one block-read context, keeping decompression batches within the same restore-wide working-set limit.

Large blocks can amplify sparse parent reads because restoring one inherited page may require decompressing its complete block. CRIU batches copy-on-write comparison reads in groups of at most 256 pages and aligns them to block boundaries when possible. If a request still covers only part of an LZ4 block, the parent page reader caches that block. Later slices can then be copied from the cache instead of reading and decompressing the same parent block again.

Decompressing blocks in parallel

--decompress-threads N controls the aggregate worker concurrency used for LZ4 decompression and large ZERO fills:

• 1, the default, keeps each decode serial. Separate private, shared-memory, and memfd requests may still make progress independently.
• 0 asks CRIU to choose a thread count from its CPU affinity, the number and decoded size of blocks in the batch, and the shared restore CPU budget.
• A value greater than one sets the maximum. CRIU may use fewer threads when a batch contains too little data or too few independent jobs, or when fewer CPUs are available.

Small batches remain serial. Parallel work needs enough decoded data per active thread to amortize scheduling, and the limit on temporary input buffers is independent of the thread count. These constraints make 0 safe to experiment with, but they do not make it the correct default for a serving node. Eight cores spent shortening one restore may delay eight other replicas or the control plane.

CRIU derives the automatic thread count from sched_getaffinity(), not the CPU quota configured for a cgroup. For a quota-constrained container with a broad affinity mask, use a cpuset or an explicit cap. The shared worker budget is also local to one CRIU restore. An orchestrator still has to limit aggregate CPU demand across simultaneous restores.

For concurrent scale-out, I would start with 1 and benchmark 0, 2, 4, and 8 only when spare CPU is available and decompression is visible in the restore profile.

Checkpoint size and restore latency

We evaluated CRIU memory compression with SGLang running in a Podman container on an AMD EPYC 9335 host with 16 CPUs available to the benchmark and an NVIDIA RTX PRO 6000 Blackwell GPU. For each configuration, we ran one warm-up followed by five measurements. In the results below, checkpoint size is the size of the complete exported checkpoint file, not only the CRIU memory image. No additional compression was applied when the checkpoint was exported. The LZ4 measurements used CRIU’s default compression acceleration of 1, which prioritizes a smaller output over faster compression¹⁸. In the baseline comparison, CRIU memory compression was disabled in one configuration and enabled with 256 KiB LZ4 blocks and serial decompression in the other. The parallel comparison used 4 KiB, 256 KiB, 512 KiB, and 1 MiB blocks with up to 16 threads performing decompression.

The results use three restore timing boundaries. Container restore latency covers the complete restore command, including archive import. Runtime restore latency comes from the benchmark’s runtime_restore_duration statistic. Its timer starts immediately before the container monitor (conmon) is launched and stops after the restored process and conmon PIDs are received. It includes conmon startup and the runc/CRIU restore. Archive import happens before this timer, while SGLang GPU-memory resumption, health checking, and inference happen afterward. The CRIU restore phase uses CRIU’s internal restore_time statistic. The block-size comparison also measures from the start of the restore command to the first streamed token. Cold-start TTFT shows how long the same model took to start without restoring a checkpoint. It is calculated from ten launches per model, five before the compression-off measurements and five before the LZ4 measurements.

Enabling memory compression

Memory compression is available in CRIU’s development branch (criu-dev) and is planned for the v4.3 release.

To check if CRIU was built with LZ4 support, use the following command:

criu check --feature compress

The compression functionality can be enabled during checkpointing (dump/pre-dump) with the --compress option, which uses one LZ4 block per memory page, or alternatively with --compress-block SIZE, which specifies the block size CRIU uses. When compression is enabled, the checkpoint inventory metadata stores this information. The restore command reads it automatically, so the compression options do not need to be specified again.

The following example commands create a compressed checkpoint of the process tree for the specified $PID and then restore it:

mkdir -p checkpoint
criu dump --tree "$PID" --images-dir checkpoint --compress-block 256K
criu restore --images-dir checkpoint

Page-server and image-streaming workflows require page-sized blocks. Multi-page blocks currently work only with local images.

For a container engine that uses runc for checkpoint and restore, put the equivalent setting in runc’s CRIU configuration:

# /etc/criu/runc.conf
compress-block 262144

Use compress instead of compress-block in runc.conf to select page-sized compression.

These CRIU settings can be used to enable memory compression for container engines such as Podman and in Kubernetes clusters where the container runtime uses runc for checkpoint and restore.

Offline memory compression

After CRIU creates a checkpoint, CRIT can be used to compress or decompress its contents offline. It uses the same block format as on-the-fly compression to rewrite every task and shared-memory pages-*.img/pagemap-*.img pair and update inventory.img.

This CRIT functionality requires the Python lz4 package and can be used as follows:

python3 -m pip install lz4

To compress an uncompressed checkpoint:

crit compress checkpoint/

To decompress a compressed checkpoint:

crit decompress checkpoint/

crit compress handles each present memory page independently, following the same ZERO, RAW, and LZ4 rules as on-the-fly compression. Non-zero pages in hugetlb and plugin-managed VMAs remain RAW because those restore paths do not support LZ4-compressed blocks.

crit decompress reads page-sized and multi-page blocks and writes uncompressed page data. Before replacing any checkpoint files, both commands validate every pagemap and corresponding page image in the directory, then write and synchronize all replacements.

Once the replacement files are ready, CRIT updates the checkpoint. It keeps the original image files as .bak hard links by default and restores them if the update fails. The --in-place option can be used when the backups are not needed.

Conclusion

CRIU can now compress memory pages during checkpoint and decompress them during restore, without storing an intermediate uncompressed checkpoint or adding a separate processing step. It uses ZERO blocks to represent all-zero memory pages without storing page data, LZ4 blocks to store compressed bytes, and RAW blocks to store the original bytes when LZ4 reduces the block by no more than 12.5%. With serial decompression, LZ4 reduced both CRIU restore time and runtime restore latency for large models. With smaller models such as Qwen 4B and 9B, LZ4 reduced checkpoint size by 45.5% and 32.6%, respectively, while also reducing container restore latency.

Increasing the block size can improve compression ratio and restore latency, while the optimal block size depends on the workload and restore path. Page-sized blocks are required for page servers, image streaming, and remote --lazy-pages restore. CRIU’s compression is most useful when reading the checkpoint data from storage accounts for a significant part of the restore latency. For workloads where checkpoint creation time is critical, CRIT can be used to compress the checkpoint data offline. It can also be used to decompress the checkpoint data offline before restore when fast storage is available.

Acknowledgements

Thanks to Andrei Vagin and Alexander Mikhalitsyn for their detailed review and design feedback. I am grateful to Viktória Spisaková and Adrian Reber for our long-standing collaboration, including on the work that led to the CRIU-LZ4 prototype and EuroMLSys paper. I also thank my PhD supervisors, Rodrigo Bruno and Wes Armour, for their guidance and support throughout this research. Thanks to Fergus Finn, Debosmit Ray, and the teams at Doubleword and DevZero for their support in bringing this work to production environments.