Accelerating Model Inference with On‑the‑Fly Snapshot Compression

Checkpoint/restore is already being applied to model serving: Dynamo Snapshot¹ uses CRIU and cuda-checkpoint to fast-start GPU workers in Kubernetes; Cloudburst² uses warmed SGLang snapshots to avoid repeated server initialization; and SwapServeLLM³ uses transparent GPU checkpointing to hot-swap models as demand changes.

Memory compression has been a topic the CRIU community has been exploring over the past few years. A straightforward approach has been to compress the checkpoint data after it is written to disk (as a compressed tar file). This simple approach is inefficient because it requires saving the full uncompressed checkpoint, reading it back for compression, and then writing the compressed snapshot. Although this overhead may be acceptable for applications with small memory footprints, the additional disk I/O, required temporary storage, and compression time become increasingly costly for memory-intensive workloads, such as LLM inference engines.

We set out to address these challenges by designing a solution that integrates compression directly into CRIU’s checkpoint and restore pipeline. Instead of performing compression as a post-processing step, our approach compresses memory pages as they are extracted from the target process and writes them directly into the checkpoint image. Similarly, the memory pages are decompressed on the fly as they are being restored. This streaming approach eliminates the need for intermediate storage of uncompressed checkpoint data, reducing both I/O overhead and storage requirements.

Earlier compression approaches

“This could (or should) be implemented in the future” was my answer back in 2018⁴ when a user asked whether CRIU compressed its image files.

Compressing memory in JVM checkpoints

OpenJDK’s Coordinated Restore at Checkpoint (CRaC) project uses CRIU to capture an initialized Java Virtual Machine (JVM), allowing Java applications to restart without repeating their initialization work. A CRIU fork added image compression that runs after the dump: it compresses complete pages-*.img files with LZ4 and replaces the original raw images.

Before restore, the fork expands those images into temporary files under /tmp, then restores the process from the raw copies. This reduces retained checkpoint size, but adds full-image I/O, temporary storage, and decompression to the restore path. Those costs grow with the process’s memory footprint, making the approach less practical for applications with hundreds of gigabytes of memory state.

Representing zero-filled memory without storing data

The proposed --skip-zero-pages option took a narrower approach. During checkpoint, CRIU would detect pages containing only zero bytes and omit their payloads from the page image. During restore, it would fill the omitted pages with zeroes instead of reading their data from storage.

This is particularly useful for Java applications because the JVM often allocates large memory regions without immediately using every page. The proposal did not land upstream, but it demonstrated that CRIU could reduce both stored bytes and page-image I/O by representing zero-filled pages without always writing a full payload.

Compressing checkpoint data before end-to-end encryption

Compression also reduces the work required for end-to-end encryption. Container checkpoints include process memory that may contain cryptographic keys, access tokens, passwords, and other confidential data. Compressing that memory before encryption reduces the bytes that must be encrypted, transferred, and stored without moving the confidentiality boundary. CRIUsec, presented at Linux Plumbers Conference 2023,⁵ integrated encryption of CRIU image files and memory pages into the checkpoint/restore pipeline rather than a separate file-processing pass. We later presented its use in Kubernetes clusters at CloudNativeSecurityCon 2024⁶ and described the design in an APSys 2024 paper,⁷ which identified integrated compression as an important next step.

Reducing LLM inference cold starts

Reusing initialized runtime state

An LLM replica is not ready after its model weights have merely arrived. Python must import packages, the inference engine must create memory pools, kernels may be compiled or autotuned, and CUDA graphs may be captured. A cold start includes that initialization and ends only when the new replica can serve a valid request.

Checkpoint/restore removes most of that repeated work from the cold-start path. The platform initializes the server once, captures its warmed state, and later restores new replicas from that checkpoint. Initialization becomes data: the checkpoint must be staged, read, and reconstructed quickly enough for the replica to become useful.

Restoring from compressed memory

CRIU stores the process’s anonymous and shared memory in pages-*.img. With NVIDIA’s current checkpoint path, GPU memory is first brought into system memory, and CRIU saves the resulting host mappings. CRIU’s compressor therefore operates on host pages rather than reading device memory itself. GPU-aware restore still requires the appropriate vendor support and CRIU plugin.⁸ For an inference server, these memory images can be many gigabytes.

Compressing those images reduces the bytes that must be stored, transferred, and read during scale-out. That can shorten a cold start when moving snapshot data costs more than LZ4 decompression. It can regress when storage is already fast and decompression becomes the bottleneck, so a smaller checkpoint alone does not prove a faster replica start.

Compressing a completed checkpoint archive with gzip or zstd saves capacity, but adds a separate pass around CRIU’s memory path and may require compressed and raw artifacts to coexist. CRIU’s integrated path instead compresses pages before they reach storage and decompresses them while restoring the process’s memory mappings.

Optimizing cold-start latency

CRIU memory restore is only one interval within an LLM cold start. A production timeline begins when the platform launches a replica and ends when that replica returns its first valid inference response. Between those points, the platform may fetch or import the checkpoint, create the container, restore CPU memory, reconstruct GPU state, wait for service health, and submit the first request.

The figure below isolates the CRIU memory-image portion of that path:

Those boundaries should be measured separately with one clock. Compare compressed and raw checkpoints with the same model placement, cache state, runtime configuration, and concurrency. Record readiness and first response as different endpoints: a successful health probe does not prove that the complete inference path is ready. Include checkpoint transfer or archive import whenever the production path includes it.

A smaller memory image is therefore a mechanism, not the cold-start result. Compression is a win only when the complete launch-to-response interval falls. Saved storage I/O can otherwise be offset by decompression CPU or by time in unaffected layers. The evaluation below reports the OCI runtime and CRIU memory-restore phases separately and treats application cold start as distinct context.

Evaluation

The latest results come from a baseline compression comparison and a parallel block-size comparison on an AMD EPYC 9335 host with an NVIDIA RTX PRO 6000 Blackwell Server Edition. Both benchmark sets ran SGLang under Podman with a development build of CRIU, one warm-up, and five measured trials per condition. Podman’s outer archive compression was disabled. The compressed condition used acceleration 1. The baseline compared compression disabled with 256 KiB blocks and serial decompression. The parallel comparison tested 4 KiB, 256 KiB, 512 KiB, and 1 MiB blocks with a 16-thread restore limit.

The benchmark validates a deterministic response before and after every successful restore. All 50 measured baseline observations and all 20 measurements in the complete Qwen 4B block-size comparison passed. This is a correctness and readiness check, not a throughput, tail-latency, or concurrent-load result.

Compression improves restore latency only when the time avoided by reading fewer image bytes exceeds the additional decompression work.

On an I/O-bound image path, the reduction in read time can dominate. On tmpfs, a hot page cache, or very high-bandwidth local storage, decompression may instead become the bottleneck. The benchmark metadata did not record the storage path, so these results locate the trade-off for that host rather than proving which storage regime produced it.

Compressed LZ4 blocks use buffered I/O, so --image-io-mode direct does not make the compressed path use direct I/O. Deployments that depend on direct I/O should benchmark the compressed path separately.⁹

On-the-fly memory compression

At Linux Plumbers Conference 2025,¹⁰ we presented the first prototype of CRIU-LZ4. In contrast to previous approaches, which compressed checkpoint images as a post-processing step, CRIU-LZ4 integrated LZ4 directly into the checkpoint/restore pipeline by extending CRIU’s pagemap metadata and image format to support compressed memory. The prototype also demonstrated that it was possible to checkpoint and restore a workload occupying approximately 75% of the available memory entirely within an in-memory filesystem, illustrating that built-in compression can make memory-intensive checkpoints practical even without backing storage. During the Q&A,¹¹ Andrei Vagin suggested retaining compression only when it provides a worthwhile reduction. That review shifted the design from compressing every page to choosing the most useful representation for each block, as described below.

Using LZ4 with restorer context

The restorer context is a minimal, self-contained execution environment that CRIU copies into a temporary location before the final stage of process restoration. It executes outside the target process’s address space, allowing CRIU to recreate the original memory mappings and transfer control without overwriting its own execution state. CRIU compiles this context as position-independent code (PIE) without access to the dynamic loader or shared libraries, so it cannot simply link against liblz4. Any decompression logic executed there must therefore be built into the restorer blob or be completely self-contained.

An earlier prototype used a helper process, but review showed that compressed pages could be restored through the normal page reader before PIE.¹² The restore path premaps eligible private VMAs containing real LZ4 blocks and decompresses them through the normal page reader before PIE. Raw and zero ranges can retain delayed PIE restore. Hugetlb and external-plugin mappings, which cannot use the generic premap path, are kept raw or zero.

Encoding memory blocks

CRIU now offers two ways to divide memory into LZ4 blocks:

• --compress encodes each system page independently. On the x86 benchmark host, that means one 4 KiB LZ4 block per page.
• --compress-block SIZE groups consecutive pages and encodes the group as one block. The size must be page-aligned and no larger than 4 MiB.

Per-page mode preserves fine-grained access and works across the widest set of CRIU paths. Region mode gives LZ4 a larger history, so it can find repetitions that cross page boundaries and amortize the call and metadata overhead. The price is coarser partial reads: asking for one page can require decoding its whole region.

For either mode, every block has one of three representations in the pagemap schema:

The writer first detects an all-zero block. Otherwise, it runs LZ4 and keeps the result only when it is smaller than seven eighths of the original. A weakly compressible page is stored raw, avoiding decode work for a marginal saving. If every block in a pagemap entry falls back to raw, CRIU drops the compression metadata for that entry and leaves restore’s ordinary contiguous fast path available. The final dump path therefore avoids expanding the page payload for incompressible data, although the compression attempt, alignment, and surrounding metadata are not free.

The metadata has to solve two different problems. compressed_size[] locates individual blocks by describing their variable lengths. total_compressed_size lets the reader skip a complete pagemap entry without summing the array. region_pages says whether one array element describes one page or a group of pages. The inventory records the overall mode and region size, so restore detects the representation automatically.

Restoring incremental checkpoint chains

An incremental checkpoint stores only pages that changed since its parent. During restore, an inherited range is passed to the parent reader, which can recurse through further generations. Each reader interprets its own pagemap, so ordinary, ZERO, RAW, and LZ4 entries can coexist in one chain without expanding the checkpoints first. The readers share one encoded-read context, keeping decompression batches within the same restore-wide working-set limit.

Large blocks can amplify sparse parent reads because restoring one inherited page may require decompressing its complete block. CRIU batches copy-on-write comparison reads in groups of at most 256 pages and aligns them to block boundaries when possible. If a request still covers only part of an LZ4 block, the parent reader caches that block. Later slices can then be copied from the cache instead of reading and decompressing the same parent block again.

Restoring variable-length blocks

CRIU does not simply read pages-*.img from beginning to end. It reconstructs private mappings, shared memory, memfd regions, copy-on-write relationships, and pages inherited from earlier checkpoints. A request can start in the middle of a pagemap entry, and a region can cross destination iovec boundaries.

The reader handles this in four steps:

1. Resolve the requested blocks and validate their counts, sizes, offsets, and expected decoded length before those values drive allocations.
2. Read one packed payload for a bounded batch.
3. Copy raw blocks and fill simple zero runs directly, then describe eligible zero and LZ4 blocks as independent jobs.
4. Decode into the final mappings, using a scratch buffer only when one region spans multiple destination vectors.

The implementation keeps encoded input batches to at most 32 MiB. At most two such working sets can be active: while workers decode one batch, the caller may prefetch the next payload if it can acquire the second slot without waiting. The details are in the final encoded reader and prefetch path.

Decompressing blocks in parallel

--decompress-threads N controls the aggregate worker concurrency used for LZ4 blocks and eligible large zero fills:

• 1, the default, keeps each decode serial. Separate private, shared-memory, and memfd requests may still make progress independently.
• 0 asks CRIU to choose a width from its CPU affinity, the number and decoded size of blocks in the batch, and the shared restore CPU budget.
• A value greater than one is an upper bound, not a promise to start that many threads. CRIU reduces impossible requests to the available CPU count.

Small batches remain serial. Parallel work needs enough decoded data per active thread to amortize scheduling, and the encoded staging-memory bound is independent of the thread count. These constraints make 0 safe to experiment with, but they do not make it the correct default for a serving node. Eight cores spent shortening one restore may delay eight other replicas or the control plane.

Automatic width follows sched_getaffinity(), not a cgroup’s CFS CPU quota. For a quota-constrained container with a broad affinity mask, use a cpuset or an explicit cap. The shared worker budget is also local to one CRIU restore. An orchestrator still has to limit aggregate CPU demand across simultaneous restores.

The baseline comparison used the serial default, while the block-size comparison fixed the upper bound at 16 threads. Because neither benchmark varied the worker count while holding the other variables constant, the results do not isolate the effect of parallel decompression. I would start with 1 under concurrent scale-out and benchmark 0, 2, 4, and 8 only when spare CPU is available and decompression is visible in the restore profile.

Offline memory compression

CRIT can convert an existing checkpoint after the process has been dumped. This is not generic archive compression: CRIT understands CRIU’s image format. It rewrites every task and shared-memory pages-*.img/pagemap-*.img pair and updates inventory.img. The result uses the same version 1.2 block format as on-the-fly compression and can be restored directly, without first expanding a temporary copy.

The conversion requires the Python lz4 package:

python3 -m pip install lz4
crit compress checkpoint/
crit decompress checkpoint/

crit compress validates each pagemap/page-image pair and applies the ZERO, RAW, and LZ4 rules to each present 4 KiB page. It does not regroup an existing image into larger blocks. Non-zero pages in hugetlb and plugin-managed VMAs stay raw because those restore paths cannot consume LZ4 blocks.

crit decompress accepts page-sized and multi-page blocks, validates the complete image set, and expands each entry into an ordinary contiguous page payload. Both commands stage and synchronize replacements before changing live names, then roll back the complete image set on failure. Originals remain as .bak hard links by default. --in-place omits persistent backups. The operational behavior is documented in the CRIT manual.

Configuring memory compression

Memory compression will be included in CRIU 4.3. Until that release, both checkpoint and restore hosts need an LZ4-enabled build from the criu-dev branch. The public options and limits are in the CRIU manual.

-c is the short form of --compress. It selects one LZ4 block per system page and supports page-server and image-streaming workflows. --compress-block SIZE selects larger blocks for local images. If both options are present, the last one wins.

First check that CRIU was built with LZ4 support:

criu check --feature compress

For a local process checkpoint, my starting configuration is:

mkdir -p checkpoint
criu dump --tree "$PID" --images-dir checkpoint --compress-block 256K
criu restore --images-dir checkpoint

The compression choice is stored in inventory.img. The dump flag should not be repeated on restore. Restore-only concurrency can be selected independently:

criu restore --images-dir checkpoint --decompress-threads 0

For an OCI runtime that invokes CRIU through runc, put the equivalent settings in runc’s CRIU configuration:

# /etc/criu/runc.conf
compress-block 262144
decompress-threads 1

Use compress instead of compress-block in runc.conf to select page-sized compression.

This is the relevant integration point for Podman and for Kubernetes nodes where the CRI runtime delegates checkpoint and restore to runc. Kubernetes does not expose these CRIU tuning options through its checkpoint API, so provision the configuration on every participating node and verify that its runtime uses the expected runc and CRIU installation.

Conclusion

CRIU can now make the compression trade where pages enter and leave the memory image, without creating a second artifact or making a second pass over the checkpoint. ZERO, RAW, and LZ4 representations avoid storing bytes that do not help, while premap restore and bounded parallel decompression keep that choice inside CRIU’s existing memory path.

The measurements also show that a smaller image does not guarantee lower restore latency. Serial decompression slowed CRIU for the smaller models, while the larger models benefited inside the runtime and the 16-thread Qwen 4B comparison favored 1 MiB blocks on the measured host. Treat any default as a starting point: larger blocks suit local, I/O-bound images, while page-sized blocks preserve page-server, streaming, and lazy-page compatibility. On memory-speed storage, compression may not help. Change the block size, worker budget, or LZ4 acceleration only after profiling the production restore path.

Acknowledgements

Thanks to Andrei Vagin and Alexander Mikhalitsyn for their detailed review and design feedback. I am grateful to Viktória Spišaková and Adrian Reber for our long-standing collaboration, including on the work that led to the CRIU-LZ4 prototype and EuroMLSys paper. I also thank my PhD supervisors, Rodrigo Bruno and Wes Armour, for their guidance and support throughout this research. Thanks to Fergus Finn, Debosmit Ray, and the teams at Doubleword and DevZero for their support in bringing this work to production environments.