IMAGE DEDUPLICATOR INSTALL
IMAGE DEDUPLICATOR CODE
IMAGE DEDUPLICATOR SERIES

eStargzĮStargz is a file format used by containerd for lazy image pulls. These new formats keep the metadata for each file contained in the tarball, including their checksum. Two candidates are currently supported: eStargz and zstd:chunked. The first step is to create the layers in a way that allows for individual file retrieval. The current image format used by container engines, a gzip-compressed tarball, doesn't have enough information to allow for these optimizations. Read-only files that are used by multiple layers or containers can be mapped in memory just once.Files that are present in multiple layers can be stored only once (this requires filesystem support).A container engine doesn't have to pull files that are already present locally.They can also create squashed images and not worry about deduplication. Containerfile authors do not need to worry about how the registry will store the image and optimize deduplication.

IMAGE DEDUPLICATOR SERIES

The new storage model is trying to solve a series of problems by making the following changes: Having more layers can facilitate deduplication, but it has a runtime cost because the overlay filesystem needs to scan more layers for each lookup and build a directory listing. Images can share layers, but it requires discipline. config/foo /etc/nginx contains the file /etc/nginx/fooĬurrently, users can only perform per-layer deduplication.

IMAGE DEDUPLICATOR INSTALL

RUN yum install -y nginx contains all the new files created by yum.

FROM fedora results in the base Fedora image layer.

When using the overlay backend, the container runtime fetches and extracts each layer to a different directory.Īt runtime, each layer is used as a lower layer for the final container overlay filesystem mount.Ī Containerfile (Dockerfile) made of the following lines results in a series of different layers: These layers are stored on a registry as a compressed tarball archive. Each layer contains a subset of the files in the image. Open Container Initiative (OCI) container images are distributed as a series of layers.

Let the Linux kernel know when content can be shared in memoryĬontainer administrators can do all of this by using content previously stored on disk rather than pulling and duplicating the content.

This article shows new technologies that have been merged into container tools to:

Because of the way we currently store container images, this is very common. When you run containers with images, if the same /usr/lib/libc.so.6 is in multiple different images, then the kernel gets confused and loads several versions of the same content into memory, wasting resources.

IMAGE DEDUPLICATOR CODE

For example, if you have 10 different programs running simultaneously that use libc, the /usr/lib/libc.so.6 code loads into kernel memory only once. The Linux kernel is smart enough to know that if two different processes load the same container (such as a shared library) into memory, they should load it into memory just once. These images often contain many duplicate files.Īnother thing to think about is kernel memory. Have you ever examined how much space your container images are taking on disk? Some users run out of space in their home directories just because they have pulled down hundreds or thousands of container images. Large images also lead to storage problems.

Free course: Deploying containerized applications.

A practical introduction to container terminology.

Image deduplicator