quine-global-labs/mirror-coreos-chunkah
1
0
Fork 0
mirror of https://github.com/coreos/chunkah.git synced 2026-11-03 21:49:44 -09:00
Code Issues Projects Releases Packages Wiki Activity

tree: initial commit

Browse source
This commit is contained in:
Jonathan Lebon 2025-11-09 11:13:31 -05:00
commit 11ca5eb66a
No known key found for this signature in database
33 changed files with 44566 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

43
.claude/skills/extract-oci-archive/SKILL.md Normal file
View file

@ -0,0 +1,43 @@
---
name: extract-oci-archive
description: Use when you need to extract an OCI archive to inspect its contents, get the rootfs layer, or read the image config.
---
# Extract OCI Archive
## Extract the OCI archive structure
```bash
mkdir /tmp/oci-extracted
tar -xf /path/to/image.ociarchive -C /tmp/oci-extracted
```
## Get the manifest digest from index.json
```bash
MANIFEST=$(jq -r '.manifests[0].digest' /tmp/oci-extracted/index.json \
| cut -d: -f2)
```
## Extract a layer to get the rootfs
For a single-layer image:
```bash
LAYER=$(jq -r '.layers[0].digest' \
/tmp/oci-extracted/blobs/sha256/${MANIFEST} | cut -d: -f2)
mkdir /tmp/rootfs
tar -xf /tmp/oci-extracted/blobs/sha256/${LAYER} -C /tmp/rootfs
```
For multi-layer images, extract each layer in order and overlay them.
## Read the image config
The config contains the created timestamp, entrypoint, labels, etc.:
```bash
CONFIG=$(jq -r '.config.digest' \
/tmp/oci-extracted/blobs/sha256/${MANIFEST} | cut -d: -f2)
jq . /tmp/oci-extracted/blobs/sha256/${CONFIG}
```

79
.github/workflows/ci.yml vendored Normal file
View file

@ -0,0 +1,79 @@
name: CI
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
unit-tests:
runs-on: ubuntu-latest
container: quay.io/fedora/fedora:latest
steps:
- name: Install tools
run: dnf install -y cargo clippy rustfmt just ShellCheck nodejs-npm openssl-devel git
- name: Configure git
run: git config --global --add safe.directory '*'
- name: Checkout
uses: actions/checkout@v5
- name: Install markdownlint
run: npm install -g markdownlint-cli
- name: Build
run: just build
- name: Check
run: just checkall
e2e-tests:
needs: unit-tests
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
container:
image: quay.io/fedora/fedora:latest
options: --privileged --cgroupns=host -v /var/lib/containers -v /var/tmp -v /tmp
steps:
- name: Checkout
uses: actions/checkout@v5
- name: Install tools
run: dnf install -y just buildah podman skopeo jq
- name: Build image
run: just buildimg
- name: Run tests
run: just test
- name: Push to ghcr.io
run: |
set -euo pipefail
podman login -u ${{ github.actor }} -p ${{ secrets.GITHUB_TOKEN }} ghcr.io
if [[ "${{ github.event_name }}" == "pull_request" ]]; then
podman push chunkah ghcr.io/${{ github.repository }}:pr-${{ github.event.pull_request.number }}-${{ github.sha }}
else
podman push chunkah ghcr.io/${{ github.repository }}:${{ github.sha }}
podman push chunkah ghcr.io/${{ github.repository }}:latest
fi
release:
needs: e2e-tests
if: github.event_name == 'push'
runs-on: ubuntu-latest
permissions:
packages: write
steps:
- name: Install skopeo
run: sudo apt-get update && sudo apt-get install -y skopeo
- name: Copy to quay.io
run: |
set -euo pipefail
mkdir -p ~/.docker
echo '${{ secrets.QUAY_AUTH }}' > ~/.docker/config.json
skopeo copy --src-creds=${{ github.actor }}:${{ secrets.GITHUB_TOKEN }} \
docker://ghcr.io/${{ github.repository }}:latest docker://quay.io/jlebon/chunkah:dev
skopeo copy --src-creds=${{ github.actor }}:${{ secrets.GITHUB_TOKEN }} \
docker://ghcr.io/${{ github.repository }}:latest docker://quay.io/jlebon/chunkah:latest
- name: Cleanup old images
uses: actions/delete-package-versions@v5
with:
package-name: chunkah
package-type: container
min-versions-to-keep: 20

1
.gitignore vendored Normal file
View file

@ -0,0 +1 @@
/target

85
AGENTS.md Normal file
View file

@ -0,0 +1,85 @@
# AGENTS.md
This file provides guidance to coding agents when working with code in this
repository.
## Project Overview
chunkah is an OCI building tool that takes a flat rootfs and outputs a layered
OCI image with content-based layers. It "postprocesses" images so that layers
are created to maximize layer reuse rather than reflecting Dockerfile structure.
## Build and Test Commands
```bash
just build # Build the project (dev)
just build release # Build the project (release)
just check # Run unit tests
just clippy # Run clippy linter
just fmt # Check code formatting
just markdownlint # Lint markdown files
just checkall # Run all checks (shellcheck, check, fmt, clippy, markdownlint)
just test # Run end-to-end tests (requires built container image)
just buildimg # Build chunkah container image
just buildimg nochunk # Build chunkah container image without chunking (faster)
```
Run a single test:
```bash
cargo test test_name
```
## Architecture
### Core Pipeline
1. **scan** (`src/scan.rs`) - Walks the rootfs and builds a map of paths to types
2. **components** (`src/components/`) - Determines which files belong to which
components
3. **ocibuilder** (`src/ocibuilder.rs`) - Creates OCI layers from components
4. **tar** (`src/tar.rs`) - Writes files to tar archives with proper metadata
### Component System
The `ComponentsRepo` trait (`src/components/mod.rs`) defines how different
package systems claim files:
- `rpm` - Claims files based on RPM database, groups by SRPM
- `xattr` - Claims files based on `user.component` extended attributes
Repos have priorities; higher priority repos (lower values) win when claiming
paths. Unclaimed files go to `chunkah/unclaimed`.
### Commands
- `build` (`src/cmd_build.rs`) - Main command: scans rootfs, assigns
components, builds OCI archive
- `prescan` (`src/cmd_prescan.rs`) - Hidden command: outputs component manifest
for testing
## Code Guidelines
### Rust
- Never use `expect()` or `unwrap()` for error handling. Use `Result` instead.
- Add `context()` or `with_context()` to errors (lowercase first word:
`context("opening file")`).
- After editing: `cargo fmt && just check && just clippy`
### Bash
Script header:
```bash
#!/bin/bash
set -euo pipefail
shopt -s inherit_errexit
```
Always use curly braces: `${foo}`, not `$foo`. Run `just shellcheck` after
editing.
### Markdown
Run `just markdownlint` after editing.

1
CLAUDE.md Symbolic link
View file

@ -0,0 +1 @@
AGENTS.md

1431
Cargo.lock generated Normal file
View file

File diff suppressed because it is too large Load diff

31
Cargo.toml Normal file
View file

@ -0,0 +1,31 @@
[package]
name = "chunkah"
version = "0.1.0"
description = "An OCI building tool for content-based layers"
authors = ["Jonathan Lebon <jonathan@jlebon.com>"]
repository = "https://github.com/jlebon/chunkah"
license = "MIT OR Apache-2.0"
edition = "2024"
publish = false
[dependencies]
anyhow = "1"
camino = { version = "1", features = ["serde1"] }
cap-std = { version = "3", default-features = false, features = ["fs_utf8"] }
cap-std-ext = { version = "4", default-features = false, features = ["fs_utf8"] }
chrono = "0.4"
clap = { version = "4", default-features = false, features = ["derive", "std", "help", "usage", "error-context", "env"] }
ctrlc = "3.5.1"
flate2 = "1"
indexmap = "2"
libc = "0.2"
ocidir = "0.6"
rpm-qa = { git = "https://github.com/jlebon/rpm-qa-rs" }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
tar = "0.4"
[dev-dependencies]
fs-set-times = "0.20.3"
maplit = "1"
tempfile = "3"

32
Containerfile Normal file
View file

@ -0,0 +1,32 @@
# Use `just buildimg` to build this, or:
#
# buildah build --skip-unused-stages=false -t chunkah .
ARG BASE=quay.io/fedora/fedora-minimal:43
ARG FINAL_FROM=oci-archive:out.ociarchive
FROM ${BASE} AS builder
RUN --mount=type=cache,rw,id=dnf,target=/var/cache/libdnf5 \
dnf install -y cargo rust pkg-config openssl-devel
WORKDIR /build
COPY Cargo.toml Cargo.lock ./
COPY src ./src
RUN --mount=type=cache,rw,id=cargo,target=/root/.cargo \
--mount=type=cache,rw,id=target,target=/build/target \
cargo build --release && cp /build/target/release/chunkah /usr/bin
FROM ${BASE} AS rootfs
RUN --mount=type=cache,id=dnf,target=/mnt \
cp -a /mnt /var/cache/libdnf5 && \
dnf install -y openssl skopeo && rm -rf /var/cache/*
COPY --from=builder /usr/bin/chunkah /usr/bin/chunkah
FROM rootfs AS rechunk
COPY --from=rootfs / /rootfs
RUN --mount=type=bind,target=/run/src,rw \
chunkah build --rootfs /rootfs > /run/src/out.ociarchive
FROM ${FINAL_FROM}
ENTRYPOINT ["/usr/bin/chunkah"]
ENV CHUNKAH_ROOTFS=/chunkah
WORKDIR /srv

16
Containerfile.splitter Normal file
View file

@ -0,0 +1,16 @@
# This file is used to split existing images using `buildah build`. See
# README.md for how to use it.
ARG CHUNKAH=quay.io/jlebon/chunkah:latest
ARG CHUNKAH_ARGS=""
ARG CHUNKAH_CONFIG_STR
FROM overridden AS target
FROM ${CHUNKAH} AS chunkah
ARG CHUNKAH_CONFIG_STR
ARG CHUNKAH_ARGS
RUN --mount=type=bind,target=/run/src,rw \
--mount=from=target,target=/chunkah \
chunkah build ${CHUNKAH_ARGS} > /run/src/out.ociarchive
FROM oci-archive:out.ociarchive

44
Justfile Normal file
View file

@ -0,0 +1,44 @@
# Build the project
build profile="dev":
cargo build --profile {{ profile }}
# Check code formatting
fmt:
cargo fmt --check
# Run unit tests
check:
cargo test
# Run clippy linter
clippy:
cargo clippy -- -D warnings
# Lint shell scripts
shellcheck:
shellcheck --external-sources --enable=all $(git ls-files '*.sh')
# Lint markdown files
markdownlint:
markdownlint $(git ls-files '*.md')
# Run all checks (shellcheck, unit tests, fmt, clippy, markdownlint)
checkall: shellcheck check fmt clippy markdownlint
# Build chunkah container image
buildimg chunk="":
#!/bin/bash
set -euo pipefail
buildah="${BUILDAH:-buildah}"
args=(-t chunkah --layers=true {{ if chunk == "nochunk" { "--build-arg=FINAL_FROM=rootfs" } else { "--skip-unused-stages=false" } }})
# drop this once we can assume 1.43
version=$(${buildah} version --json | jq -r '.version')
if [[ $(echo -e "${version}\n1.43" | sort -V | head -n1) != "1.43" ]]; then
args+=(-v "$PWD:/run/src" --security-opt=label=disable)
fi
echo ${buildah} build "${args[@]}" .
${buildah} build "${args[@]}" .
# Run end-to-end tests with built chunkah image
test *ARGS:
./tests/e2e/run.sh {{ ARGS }}

202
LICENSE-APACHE Normal file
View file

@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

5
LICENSE-MIT Normal file
View file

@ -0,0 +1,5 @@
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

225
README.md Normal file
View file

@ -0,0 +1,225 @@
# chunkah
An OCI building tool that takes a flat rootfs and outputs a layered OCI image
with content-based layers.
> [!NOTE]
> chunkah is currently under heavy development and not yet ready for production.
> Experimental usage and feedback is much appreciated!
## Motivation
Traditionally, images built using a `Dockerfile` result in a multi-layered image
which model how the `Dockerfile` was written. For example, a separate layer
is created for each `RUN` and `COPY` instructions. This can cause poor layer
caching on clients pulling these images. A single package change may invalidate
a layer much larger than the package itself, requiring re-pulling.
When splitting an image into content-based layers, it doesn't matter how the
final contents of the image were derived. The image is "postprocessed" so that
layers are created in a way that tries to maximize layer reuse. Commonly, this
means grouping together related packages. This has benefits at various levels:
at the network level (common layers do not need to be re-pulled), at the storage
level (common layers are stored once), and at the runtime level (e.g. libraries
are only mapped once).
chunkah allows you to keep building your image as you currently do, and then
perform this content-based layer splitting.
## Highlights
- **Content agnostic** — Compatible with RPM-based images, but not only. Other
package ecosystems can be supported, as well as fully custom content.
- **Container-native** — Best used as a container image, either as part of a
multi-staged build, or standalone.
- **Zero diff** — Apart from modification time, content is never modified.
- **Reproducible** — Supports `SOURCE_DATE_EPOCH` for reproducible layers.
It is a non-goal to support initial building of the root filesystem itself.
Lots of tools for that exist already. It is also currently a non-goal to
preprocess the rootfs to remove common sources of non-reproducibility (such as
[add-determinism]). This can be done by the image build process itself.
## Installation
While it's possible to install chunkah as a native CLI tool using `cargo
install`, it's primarily intended to be used as a container image:
```shell
podman run -ti --rm quay.io/jlebon/chunkah --help
```
## Usage
There are two main ways to use chunkah:
- splitting an existing image
- splitting an image at build time
### Splitting an existing image
#### Using Podman/Buildah
When using Podman/Buildah, the most natural way to split an existing image is to
use the `Containerfile.splitter`, passing the target image as the `--from`:
```shell
IMG=quay.io/fedora/fedora-minimal:latest
buildah build --skip-unused-stages=false --from $IMG \
--build-arg CHUNKAH_CONFIG_STR="$(podman inspect $IMG)" \
https://github.com/jlebon/chunkah/releases/download/latest/Containerfile.splitter
```
Additional arguments can be passed to chunkah using the CHUNKAH_ARGS build
argument.
> [!NOTE]
> You must add the `--skip-unused-stages=false` option (see also [this buildah
> RFE][buildah-rfe]).
>
> For Buildah versions before v1.43, this also requires `-v $(pwd):/run/src
> --security-opt=label=disable`.
Another option is using the chunkah image directly and image mounts:
```shell
IMG=quay.io/fedora/fedora-minimal:latest
podman pull $IMG # image must be available locally
export CHUNKAH_CONFIG_STR=$(podman inspect $IMG)
podman run --rm --mount=type=image,src=$IMG,dest=/chunkah \
-e CHUNKAH_CONFIG_STR quay.io/jlebon/chunkah build | podman load
```
#### Using Docker/Moby
You can use the chunkah image directly using image mounts (requires v28+):
```shell
IMG=quay.io/fedora/fedora-minimal:latest
docker pull $IMG # image must be available locally
export CHUNKAH_CONFIG_STR=$(docker inspect $IMG)
docker run --rm --mount=type=image,src=$IMG,destination=/chunkah \
-e CHUNKAH_CONFIG_STR quay.io/jlebon/chunkah build > out.ociarchive
docker run --rm -ti -v $(pwd):/srv:z -w /srv quay.io/skopeo/stable \
copy oci-archive:out.ociarchive docker-archive:out.dockerarchive
docker load -i out.dockerarchive
```
Note the conversion step using `skopeo`; `chunkah` currently only outputs an OCI
archive, which `docker load` does not natively support.
### Splitting an image at build time (buildah/podman only)
This uses a method called the "`FROM oci-archive:` trick", for lack of a better
term. It has the massive advantage of being compatible with a regular `buildah
build` flow and also makes it more natural to apply configs to the image, but is
specific to the Podman ecosystem. This *will not* work with Docker.
```Dockerfile
FROM quay.io/fedora/fedora-minimal:latest AS builder
RUN dnf install -y git-core && dnf clean all
FROM quay.io/jlebon/chunkah AS chunkah
RUN --mount=from=builder,src=/,target=/chunkah,ro \
--mount=type=bind,target=/run/src,rw \
chunkah build > /run/src/out.ociarchive
FROM oci-archive:out.ociarchive
ENTRYPOINT ["git"]
```
> [!NOTE]
> When building your image, you must also add the `--skip-unused-stages=false`
> option (see also [this buildah RFE][buildah-rfe]), and you cannot use the
> `--jobs` option.
>
> For Buildah versions before v1.43, this also requires `-v $(pwd):/run/src
> --security-opt=label=disable`.
## Advanced Usage
### Understanding components
A component is a logical grouping of files that belong together. For example,
all files from an RPM belong to the same component. Layers are created based on
found components.
A component repo is a source of data from which components can be created. For
example, the rpmdb is a component repo (one can imagine similar component repos
for other distros). There is also an xattr-based component repo (see the section
"Customizing the layers" below). Multiple component repos can be active at once.
### Customizing the layers
It is possible to modify how components are assigned to layers by setting the
`user.component` xattr on files/directories. This can be done using `setfattr`,
e.g.:
```Dockerfile
RUN setfattr -n user.component -v "" /
RUN setfattr -n user.component -v "custom-apps" /usr/bin/my-app
```
Note chunkah requires setting an empty string value on the rootfs itself
to enable xattr scanning. This avoids having to check the xattrs of the
whole rootfs in the common case where this feature is not used. Apart from
this detail, this is compatible with rpm-ostree's support for [the same
feature](https://coreos.github.io/rpm-ostree/build-chunked-oci/#assigning-files-to-specific-layers).
### Limiting the number of layers
By default, the maximum number of layers emitted is 64. This can be increased
(up to 448) or decreased using the `--max-layers` option. If the number of
components exceeds the maximum, chunkah will pack multiple components together.
There is thus a tradeoff in deciding this. Fewer layers means losing the
efficiency gains of content-based layers. Too many layers may mean excessive
processing and overhead when pushing/pulling the image.
### Building from a raw rootfs
For completeness, note it's of course also possible to split any arbitrary
rootfs, regardless of where it comes from.
```shell
podman run --rm -v root:/chunkah:z -e CHUNKAH_CONFIG_STR=$(cat config.json) \
quay.io/jlebon/chunkah build > out.ociarchive
```
> [!NOTE]
> The `:z` option will relabel all files for access by the container, which may
> be expensive for a large rootfs. You can use `--security-opt=label=disable` to
> avoid this, but it disables SELinux separation with the chunkah container.
### Customizing the OCI image config and annotations
The OCI image config can be provided via the `--config` option (as a file) or
`--config-str`/`CHUNKAH_CONFIG_STR` (inline). The primary format is the [OCI
image config] spec as JSON:
```json
{
"Entrypoint": ["/bin/bash"],
"Cmd": ["-c", "echo hi"],
"WorkDir": "/root"
}
```
The output format of `podman inspect` and `docker inspect` are also supported,
mostly for convenience when splitting an existing image, though it does also
have the advantage of capturing annotations. Otherwise, it's also possible to
set annotations directly using `--annotation`. Labels can also be added via
`--label`.
## Origins
chunkah is a generalized successor to rpm-ostree's [build-chunked-oci] command
which does content-based layer splitting on RPM-based [bootable container
images]. Unlike rpm-ostree, chunkah is not tied to bootable containers nor RPMs.
The name is a nod to this ancestry and to buildah, with which it integrates
well.
[add-determinism]: https://github.com/keszybz/add-determinism
[bootable container images]: https://containers.github.io/bootable/
[build-chunked-oci]: https://coreos.github.io/rpm-ostree/build-chunked-oci/
[OCI image config]: https://github.com/opencontainers/image-spec/blob/26647a49f642c7d22a1cd3aa0a48e4650a542269/specs-go/v1/config.go#L24
[buildah-rfe]: https://github.com/containers/buildah/issues/6621

572
src/cmd_build.rs Normal file
View file

@ -0,0 +1,572 @@
use std::collections::HashMap;
use anyhow::{Context, Result};
use camino::Utf8PathBuf;
use cap_std::ambient_authority;
use cap_std::fs::Dir;
use clap::Parser;
use ocidir::oci_spec::image as oci_image;
use serde::Deserialize;
use crate::components::{Component, FileMap};
use crate::ocibuilder::{Builder, Compression};
use crate::packing::{PackItem, calculate_packing};
use crate::utils;
#[derive(Parser, Default)]
pub struct BuildArgs {
/// Path to the rootfs to build from
#[arg(long, env = "CHUNKAH_ROOTFS", hide_env_values = true)]
rootfs: Utf8PathBuf,
/// Output file path (defaults to stdout)
#[arg(short, long, value_name = "PATH")]
output: Option<Utf8PathBuf>,
/// Maximum number of layers to output
#[arg(long, default_value_t = 64)]
max_layers: usize,
/// Read image config from a JSON file
///
/// The file should contain the .Config element from a podman/docker
/// inspect output. This is useful when resplitting an existing image.
#[arg(long = "config", value_name = "PATH", conflicts_with = "config_str")]
config: Option<Utf8PathBuf>,
/// Read image config from a JSON string
///
/// Same as --config but takes a JSON string directly instead of a file path.
#[arg(
long = "config-str",
value_name = "JSON",
env = "CHUNKAH_CONFIG_STR",
hide_env_values = true
)]
config_str: Option<String>,
/// Add a label to the image
///
/// Format: KEY=VALUE. Can be specified multiple times.
#[arg(long = "label", value_name = "KEY=VALUE")]
labels: Vec<String>,
/// Add an annotation to the image manifest
///
/// Format: KEY=VALUE. Can be specified multiple times.
#[arg(long = "annotation", value_name = "KEY=VALUE")]
annotations: Vec<String>,
/// Unix timestamp used as the creation time for the OCI image and as
/// the maximum mtime for files without a known build time.
#[arg(
long,
value_name = "EPOCH",
env = "SOURCE_DATE_EPOCH",
hide_env_values = true
)]
source_date_epoch: Option<u64>,
/// Compress layers and the OCI archive with gzip
///
/// By default, layers and the OCI archive are uncompressed. This flag
/// enables gzip compression for both.
#[arg(long)]
compressed: bool,
/// Gzip compression level (0-9, default: 6)
///
/// Level 0 is no compression (fastest), 9 is maximum compression (slowest).
/// Only applies when --compressed is specified.
#[arg(long, value_name = "LEVEL", default_value_t = 6, value_parser = clap::value_parser!(u32).range(0..=9))]
compression_level: u32,
/// Target architecture for the output image
///
/// If not provided, the architecture from the config is used if found, or
/// the current system architecture otherwise.
#[arg(long, value_name = "ARCH")]
arch: Option<String>,
}
impl BuildArgs {
/// Apply CLI overrides to an OCI config, returning a new config.
fn apply_to_config(&self, config: oci_image::Config) -> Result<oci_image::Config> {
let mut builder = oci_image::ConfigBuilder::default();
// Copy over all fields from base config. Would be nice if we could
// instantiate a ConfigBuilder from a starting Config instead...
macro_rules! copy_if_present {
($($field:ident),+) => {
$(
if let Some(v) = config.$field() {
builder = builder.$field(v.clone());
}
)+
};
}
copy_if_present!(
user,
working_dir,
stop_signal,
entrypoint,
cmd,
env,
exposed_ports,
volumes
);
// labels; CLI args override config
let labels =
parse_key_value_pairs(&self.labels, config.labels().clone().unwrap_or_default())
.context("parsing labels")?;
if !labels.is_empty() {
builder = builder.labels(labels);
}
builder.build().context("building config")
}
}
pub fn run(args: &BuildArgs) -> Result<()> {
let created_epoch = args
.source_date_epoch
.map_or_else(utils::get_current_epoch, Ok)?;
// load base config from file, string, or use empty default
let parsed = if let Some(path) = &args.config {
let content = std::fs::read_to_string(path)
.with_context(|| format!("failed to read config file: {}", path))?;
parse_config(&content).with_context(|| format!("failed to parse config file: {}", path))?
} else if let Some(config_str) = &args.config_str {
parse_config(config_str).context("failed to parse config string")?
} else {
ParsedConfig {
config: oci_image::Config::default(),
annotations: HashMap::new(),
architecture: None,
}
};
let architecture = args.arch.as_deref().or(parsed.architecture.as_deref());
// get the current arch if not provided, but even if provided, this
// normalizes the arch so that `--arch x86_64` also works
let architecture = utils::get_goarch(architecture);
// merge config and CLI annotations
let annotations = parse_key_value_pairs(&args.annotations, parsed.annotations)
.context("parsing annotations")?;
let image_config = build_image_config(args, parsed.config, created_epoch, architecture)
.context("building image config")?;
let rootfs = Dir::open_ambient_dir(args.rootfs.as_std_path(), ambient_authority())
.with_context(|| format!("opening rootfs {}", args.rootfs))?;
let components = crate::scan::scan_for_components(&rootfs, created_epoch)
.with_context(|| format!("scanning {} for components", args.rootfs))?;
// pack components down to max layers
let components = pack_components(args, components).context("packing components")?;
// build the OCI image
let compression = if args.compressed {
Compression::Gzip(args.compression_level)
} else {
Compression::None
};
let builder = Builder::new(&rootfs, components)
.context("creating builder")?
.compression(compression)
.annotations(annotations)
.config(image_config);
if let Some(output_path) = &args.output {
let mut file = std::fs::File::create(output_path)
.with_context(|| format!("creating output file {}", output_path))?;
builder.build(&mut file)
} else {
builder.build(&mut std::io::stdout().lock())
}
}
/// Parse config from a JSON string.
///
/// Supports three formats:
/// 1. Direct OCI config (e.g., `{"Entrypoint": [...]}`)
/// 2. podman/docker inspect output array (e.g., `[{"Config": {...}}]`)
/// 3. Single inspect output object (e.g., `{"Config": {...}}`)
fn parse_config(json_str: &str) -> Result<ParsedConfig> {
let input: ConfigInput =
serde_json::from_str(json_str).context("failed to parse config JSON")?;
match input {
ConfigInput::Inspect(mut vec) => vec
.pop()
.ok_or_else(|| anyhow::anyhow!("inspect output is an empty array")),
ConfigInput::InspectOne(parsed) => Ok(parsed),
ConfigInput::Direct(config) => Ok(ParsedConfig {
config,
annotations: HashMap::new(),
architecture: None,
}),
}
}
/// Parsed config data from either OCI config or podman/docker inspect format.
/// The serde renames allow this to deserialize from inspect format (with "Config" key).
#[derive(Deserialize)]
struct ParsedConfig {
#[serde(rename = "Config")]
config: oci_image::Config,
#[serde(rename = "Annotations", default)]
annotations: HashMap<String, String>,
#[serde(rename = "Architecture")]
architecture: Option<String>,
}
/// Config input format - either direct OCI config or podman/docker inspect output.
#[derive(Deserialize)]
#[serde(untagged)]
enum ConfigInput {
/// podman/docker inspect output (array with Config field)
Inspect(Vec<ParsedConfig>),
/// Single inspect output object (e.g., first element extracted from array)
InspectOne(ParsedConfig),
/// Direct OCI config (e.g., `{"Entrypoint": [...]}`)
Direct(oci_image::Config),
}
/// Build the OCI image configuration from CLI args and a parsed config.
fn build_image_config(
args: &BuildArgs,
config: oci_image::Config,
created: u64,
architecture: &str,
) -> Result<oci_image::ImageConfiguration> {
// apply CLI configs to base OCI config
let config = args
.apply_to_config(config)
.context("applying CLI configs")?;
// this is empty for now; it gets populated as we add components
let rootfs = oci_image::RootFsBuilder::default()
.typ("layers")
.diff_ids(Vec::<String>::new())
.build()?;
// format the created timestamp as RFC 3339
let epoch_i64 = i64::try_from(created)
.with_context(|| format!("created timestamp overflows i64: {created}"))?;
let created = chrono::DateTime::from_timestamp(epoch_i64, 0)
.with_context(|| format!("invalid created timestamp: {}", created))?
.to_rfc3339_opts(chrono::SecondsFormat::Secs, true);
let image_config = oci_image::ImageConfigurationBuilder::default()
.os("linux")
.architecture(architecture)
.config(config)
.rootfs(rootfs)
.created(created)
.build()?;
Ok(image_config)
}
/// Parse KEY=VALUE pairs and merge into an existing map.
fn parse_key_value_pairs(
pairs: &[String],
mut map: HashMap<String, String>,
) -> Result<HashMap<String, String>> {
for pair in pairs {
let (k, v) = pair
.split_once('=')
.with_context(|| format!("must be in KEY=VALUE format: {pair}"))?;
anyhow::ensure!(!k.is_empty(), "key cannot be empty: {pair}");
map.insert(k.to_string(), v.to_string());
}
Ok(map)
}
/// Packs components into layers according to max_layers constraint.
fn pack_components(
args: &BuildArgs,
components: HashMap<String, Component>,
) -> Result<Vec<(String, Component)>> {
let max_layers = args.max_layers;
let mut entries: Vec<Option<(String, Component)>> = components.into_iter().map(Some).collect();
let items: Vec<PackItem> = entries
.iter()
.map(|entry| {
let (_, comp) = entry.as_ref().unwrap();
PackItem {
size: comp.files.values().map(|f| f.size).sum(),
stability: comp.stability,
}
})
.collect();
let packed_groups = calculate_packing(&items, max_layers);
let mut result = Vec::with_capacity(packed_groups.len());
for group in packed_groups {
if group.indices.len() == 1 {
// single component group
let idx = group.indices[0];
let (name, component) = entries[idx].take().expect("packing returned invalid index");
result.push((name, component));
} else {
// merged group - combine components
let mut names = Vec::with_capacity(group.indices.len());
let mut merged_files = FileMap::new();
let mut max_mtime_clamp = 0u64;
for &idx in &group.indices {
let (name, comp) = entries[idx].take().expect("packing returned invalid index");
names.push(name);
// Move "up" the clamp. We're still guaranteed that it's (1)
// a reproducible timestamp for this particular group, and
// (2) it's very unlikely to be after $now so we'll clamp
// scriptlet-created files.
max_mtime_clamp = max_mtime_clamp.max(comp.mtime_clamp);
merged_files.extend(comp.files);
}
let merged_name = names.join(" ");
result.push((
merged_name,
Component {
mtime_clamp: max_mtime_clamp,
stability: group.stability,
files: merged_files,
},
));
}
}
Ok(result)
}
#[cfg(test)]
mod tests {
use super::*;
const CONFIG_FIXTURE: &str = include_str!("../tests/fixtures/empty.image-config.json");
#[test]
fn test_emptydir_roundtrip() {
// Create an OCI archive from an empty rootfs. Then re-open it with
// ocidir and check it's what we expect.
// create a temp directory for the empty rootfs
let rootfs_dir = tempfile::tempdir().unwrap();
let args = BuildArgs {
rootfs: Utf8PathBuf::try_from(rootfs_dir.path().to_path_buf()).unwrap(),
source_date_epoch: Some(1),
..Default::default()
};
// parse config from fixture file
let parsed = parse_config(CONFIG_FIXTURE).unwrap();
let image_config = build_image_config(&args, parsed.config, 1, "amd64").unwrap();
let rootfs =
cap_std::fs::Dir::open_ambient_dir(rootfs_dir.path(), cap_std::ambient_authority())
.unwrap();
// create a single empty component for testing
let components = vec![(
"test".to_string(),
Component {
mtime_clamp: 1,
stability: 0.0,
files: Default::default(),
},
)];
// and build the OCI image
let builder = Builder::new(&rootfs, components)
.unwrap()
.compression(Compression::None)
.annotations(parsed.annotations)
.config(image_config);
let mut output = Vec::new();
builder.build(&mut output).unwrap();
// now extract it back out and open it as an ocidir
let oci_tempdir = tempfile::tempdir().unwrap();
let mut archive = tar::Archive::new(output.as_slice());
archive.unpack(oci_tempdir.path()).unwrap();
let oci_dir_cap =
cap_std::fs::Dir::open_ambient_dir(oci_tempdir.path(), cap_std::ambient_authority())
.unwrap();
let oci_dir = ocidir::OciDir::open(oci_dir_cap).unwrap();
// get the image manifest (we don't set a tag, so use the index)
let index = oci_dir.read_index().unwrap();
let manifest_desc = index.manifests().first().unwrap();
let manifest: oci_image::ImageManifest = oci_dir.read_json_blob(manifest_desc).unwrap();
// get the image config and verify it matches the fixture
let image_config: oci_image::ImageConfiguration =
oci_dir.read_json_blob(manifest.config()).unwrap();
let config = image_config.config().clone().unwrap();
let expected: oci_image::Config = serde_json::from_str(CONFIG_FIXTURE).unwrap();
assert_eq!(config, expected);
// verify the created timestamp is set correctly (epoch 1 = 1970-01-01T00:00:01Z)
assert_eq!(
image_config.created().as_deref(),
Some("1970-01-01T00:00:01Z")
);
// verify there's no history
assert!(
image_config.history().as_ref().is_none_or(|h| h.is_empty()),
"image should have no history entries"
);
// verify there are no layers (empty components are filtered out)
assert!(
manifest.layers().is_empty(),
"empty rootfs should have no layers"
);
}
#[test]
fn test_parse_config_direct_format() {
// Test parsing direct OCI config format
let json = r#"{"Entrypoint": ["/bin/sh"], "Cmd": ["-c", "echo hi"]}"#;
let parsed = parse_config(json).unwrap();
assert_eq!(
parsed.config.entrypoint(),
&Some(vec!["/bin/sh".to_string()])
);
assert_eq!(
parsed.config.cmd(),
&Some(vec!["-c".to_string(), "echo hi".to_string()])
);
// Direct OCI config format has no Architecture field
assert_eq!(parsed.architecture, None);
}
#[test]
fn test_parse_config_inspect_format() {
// Test parsing podman/docker inspect format
let json = r#"[{
"Config": {
"Entrypoint": ["/usr/bin/app"],
"Env": ["PATH=/usr/bin"]
},
"Annotations": {
"org.example.key": "value"
},
"Architecture": "arm64"
}]"#;
let parsed = parse_config(json).unwrap();
assert_eq!(
parsed.config.entrypoint(),
&Some(vec!["/usr/bin/app".to_string()])
);
assert_eq!(
parsed.config.env(),
&Some(vec!["PATH=/usr/bin".to_string()])
);
assert_eq!(
parsed.annotations.get("org.example.key"),
Some(&"value".to_string())
);
assert_eq!(parsed.architecture, Some("arm64".to_string()));
}
#[test]
fn test_parse_config_inspect_single_object() {
// Test parsing a single inspect object (not wrapped in array)
let json = r#"{"Config": {"Entrypoint": ["/bin/app"], "WorkingDir": "/data"}, "Architecture": "amd64"}"#;
let parsed = parse_config(json).unwrap();
assert_eq!(
parsed.config.entrypoint(),
&Some(vec!["/bin/app".to_string()])
);
assert_eq!(parsed.config.working_dir(), &Some("/data".to_string()));
assert_eq!(parsed.architecture, Some("amd64".to_string()));
}
#[test]
fn test_parse_key_value_pairs_invalid() {
let invalid_pairs = ["", "no-equals", "=", "=value"];
for pair in invalid_pairs {
let pairs = vec![pair.into()];
let result = parse_key_value_pairs(&pairs, HashMap::new());
assert!(result.is_err(), "pair {:?} should be rejected", pair);
}
}
#[test]
fn test_parse_key_value_pairs_valid() {
use maplit::hashmap;
let test_cases: &[(&str, HashMap<String, String>)] = &[
("key=value", hashmap! { "key".into() => "value".into() }),
("key=", hashmap! { "key".into() => "".into() }),
(
"key=value=with=equals",
hashmap! { "key".into() => "value=with=equals".into() },
),
];
for (pair, expected) in test_cases {
let pairs = vec![(*pair).into()];
let result = parse_key_value_pairs(&pairs, HashMap::new()).unwrap();
assert_eq!(
&result, expected,
"pair {:?} should produce {:?}",
pair, expected
);
}
}
#[test]
fn test_build_image_config_labels_override() {
// Base config with pre-existing labels
let json = r#"{
"Labels": {
"existing": "from-config",
"override-me": "old-value"
}
}"#;
let parsed = parse_config(json).unwrap();
// CLI labels that override one, add a new one, and later CLI label overrides earlier
let args = BuildArgs {
labels: vec![
"override-me=new-value".into(),
"new-label=first".into(),
"new-label=second".into(), // later CLI label overrides earlier
],
..Default::default()
};
let image_config = build_image_config(&args, parsed.config, 1, "amd64").unwrap();
let labels = image_config
.config()
.as_ref()
.unwrap()
.labels()
.as_ref()
.unwrap();
assert_eq!(labels.get("existing"), Some(&"from-config".to_string()));
assert_eq!(labels.get("override-me"), Some(&"new-value".to_string()));
assert_eq!(labels.get("new-label"), Some(&"second".to_string()));
}
}

328
src/components/mod.rs Normal file
View file

@ -0,0 +1,328 @@
mod rpm;
mod xattr;
use std::collections::{BTreeMap, HashMap};
/// The name of the component for files not claimed by any repo.
pub const UNCLAIMED_COMPONENT: &str = "chunkah/unclaimed";
use anyhow::{Context, Result};
use camino::{Utf8Path, Utf8PathBuf};
use cap_std::fs::Dir;
/// Seconds per day.
pub const SECS_PER_DAY: u64 = 60 * 60 * 24;
/// Period in days for calculating stability probability.
/// TODO: make this configurable via CLI
pub const STABILITY_PERIOD_DAYS: f64 = 7.0;
/// Maximum lookback period in days for changelog analysis.
pub const STABILITY_LOOKBACK_DAYS: u64 = 365;
/// Loaded component repos along with the default mtime to use.
pub struct ComponentsRepos {
repos: Vec<Box<dyn ComponentsRepo>>,
default_mtime_clamp: u64,
}
/// Files belonging to a component.
#[derive(Debug, Clone)]
pub struct Component {
/// The maximum mtime for files in this component during the build phase.
/// File mtimes will be clamped to this value.
pub mtime_clamp: u64,
/// Probability that the component doesn't change over STABILITY_PERIOD_DAYS.
/// Used by the packing algorithm.
pub stability: f64,
/// The files belonging to this component, with their metadata.
pub files: FileMap,
}
/// A map from file paths to their metadata.
pub type FileMap = BTreeMap<Utf8PathBuf, FileInfo>;
/// Cached file metadata from the scan.
#[derive(Debug, Clone)]
pub struct FileInfo {
pub file_type: FileType,
pub mode: u32,
#[allow(dead_code)]
pub size: u64,
pub uid: u32,
pub gid: u32,
pub mtime: u64,
pub ino: u64,
pub nlink: u64,
pub xattrs: Vec<(String, Vec<u8>)>,
}
/// File type for entries in the rootfs.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum FileType {
Directory,
File,
Symlink,
}
impl FileInfo {
/// Create FileInfo from metadata and xattrs.
pub fn from_metadata(
metadata: &cap_std::fs::Metadata,
xattrs: Vec<(String, Vec<u8>)>,
) -> Result<Self> {
use cap_std::fs::MetadataExt;
let file_type = if metadata.file_type().is_dir() {
FileType::Directory
} else if metadata.file_type().is_file() {
FileType::File
} else if metadata.file_type().is_symlink() {
FileType::Symlink
} else {
anyhow::bail!("unsupported file type");
};
Ok(Self {
file_type,
mode: metadata.mode(),
size: metadata.len(),
uid: metadata.uid(),
gid: metadata.gid(),
mtime: metadata.mtime() as u64,
ino: metadata.ino(),
nlink: metadata.nlink(),
xattrs,
})
}
}
impl ComponentsRepos {
/// Detect and load all component repos present in the given rootfs.
///
/// The `files` map is the set of paths in the rootfs. This avoids the xattr
/// repo having to walk the rootfs again. The `default_mtime_clamp` will be
/// used as the mtime clamp for components that don't have a reproducible
/// clamp (e.g. xattr-claimed files, unclaimed files).
pub fn load(rootfs: &Dir, files: &FileMap, default_mtime_clamp: u64) -> Result<Self> {
let mut repos: Vec<Box<dyn ComponentsRepo>> = Vec::new();
if let Some(repo) =
xattr::XattrRepo::load(files, default_mtime_clamp).context("loading xattrs")?
{
repos.push(Box::new(repo));
}
if let Some(repo) = rpm::RpmRepo::load(rootfs).context("loading rpmdb")? {
repos.push(Box::new(repo));
}
// Other backends (e.g. deb, apk, pip, etc.) would go here...
Ok(Self {
repos,
default_mtime_clamp,
})
}
/// Returns true if no repos were loaded.
pub fn is_empty(&self) -> bool {
self.repos.is_empty()
}
/// Claim files from repos and return the mapping of component names to files.
///
/// Repos are sorted by priority (lower values first) before processing.
/// Higher priority repos "win" - if they claim a path, lower priority repos
/// are not consulted for that path. All unclaimed paths go into a catch-all.
pub fn into_components(mut self, files: FileMap) -> HashMap<String, Component> {
let mut claims: HashMap<(usize, ComponentId), FileMap> = HashMap::new();
// make sure they're in priority order
self.repos.sort_by_key(|r| r.default_priority());
// check for claims!
let unclaimed: FileMap = files
.into_iter()
.filter_map(|(path, file_info)| {
// This is O(files x repos), though really the number of active
// repos at any time is incredibly small; in the common case, 1.
for (repo_idx, repo) in self.repos.iter().enumerate() {
let component_ids = repo.claims_for_path(&path, file_info.file_type);
if !component_ids.is_empty() {
for id in component_ids {
claims
.entry((repo_idx, id))
.or_default()
.insert(path.clone(), file_info.clone());
}
return None; // claimed
}
}
Some((path, file_info)) // not claimed
})
.collect();
// build final components map
let mut components = HashMap::new();
for ((repo_idx, comp_id), files) in claims {
let repo = &self.repos[repo_idx];
let info = repo.component_info(comp_id);
let full_name = format!("{}/{}", repo.name(), info.name);
components.insert(
full_name,
Component {
mtime_clamp: info.mtime_clamp,
stability: info.stability,
files,
},
);
}
// and the catch-all component for anything unclaimed
if !unclaimed.is_empty() {
components.insert(
UNCLAIMED_COMPONENT.into(),
Component {
mtime_clamp: self.default_mtime_clamp,
stability: 0.0, // unclaimed files assumed unstable
files: unclaimed,
},
);
}
components
}
}
/// Opaque identifier for a component within a repo.
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
struct ComponentId(usize);
/// Information about a component.
struct ComponentInfo<'a> {
pub name: &'a str,
pub mtime_clamp: u64,
pub stability: f64,
}
/// A trait for any type of "repo" of components (e.g., rpm, dpkg, etc.)
///
/// Components repos are query objects that answer which components claim a path.
trait ComponentsRepo {
/// Returns the name of this repo type (e.g., "rpm", "xattr").
fn name(&self) -> &'static str;
/// Returns the priority of this repo for ordering purposes.
///
/// Lower values indicate higher priority. Used to determine the order in
/// which repos are queried. Higher priority repos "win" - if they claim a
/// path, lower priority repos are not consulted. We could make the default
/// overridable on the CLI in the future.
fn default_priority(&self) -> usize;
/// Query which components claim this path.
///
/// Returns a list of component IDs that claim this path. For most paths,
/// this returns 0 or 1 ID. Directories shared by multiple packages may
/// return multiple IDs.
fn claims_for_path(&self, path: &Utf8Path, file_type: FileType) -> Vec<ComponentId>;
/// Get info about a component by ID.
fn component_info(&self, id: ComponentId) -> ComponentInfo<'_>;
}
#[cfg(test)]
mod tests {
use camino::Utf8Path;
use cap_std::ambient_authority;
use cap_std::fs::Dir;
use cap_std_ext::dirext::CapStdExtDirExt;
use super::*;
const RPM_FIXTURE: &str = include_str!("../../tests/fixtures/fedora.json");
const XATTR_NAME: &str = "user.component";
#[test]
fn test_into_components() {
let tmp = tempfile::tempdir().unwrap();
let rootfs = Dir::open_ambient_dir(tmp.path(), ambient_authority()).unwrap();
rootfs.create_dir_all("usr/bin").unwrap();
rootfs.create_dir_all("usr/lib64").unwrap();
rootfs.create_dir_all("usr/lib/sysimage/rpm").unwrap();
rootfs.create_dir_all("opt/myapp").unwrap();
rootfs.write("usr/bin/bash", "fake bash").unwrap();
rootfs.write("usr/lib64/libc.so.6", "fake libc").unwrap();
rootfs
.write("usr/lib/sysimage/rpm/rpmdb.sqlite", "fake rpmdb")
.unwrap();
rootfs.write("opt/myapp/config", "config").unwrap();
rootfs.write("opt/myapp/data", "data").unwrap();
// set xattr on /usr/bin/bash to claim it for "xattr-component"
rootfs
.setxattr("usr/bin/bash", XATTR_NAME, b"xattr-component")
.unwrap();
// set xattr on /opt/myapp/data to claim it for "myapp"
rootfs
.setxattr("opt/myapp/data", XATTR_NAME, b"myapp")
.unwrap();
let files = crate::scan::scan_rootfs(&rootfs).unwrap();
let xattr_repo = xattr::XattrRepo::load(&files, 0).unwrap().unwrap();
let packages = rpm_qa::load_from_str(RPM_FIXTURE).unwrap();
let rpm_repo = rpm::RpmRepo::load_from_packages(packages).unwrap();
let repos: Vec<Box<dyn ComponentsRepo>> = vec![Box::new(rpm_repo), Box::new(xattr_repo)];
let loaded = ComponentsRepos {
repos,
default_mtime_clamp: 0,
};
let components = loaded.into_components(files);
// example xattr overrides rpm entry
assert!(
components["xattr/xattr-component"]
.files
.contains_key(Utf8Path::new("/usr/bin/bash")),
"/usr/bin/bash should belong to xattr/xattr-component"
);
// example rpm entry
assert!(
components["rpm/glibc"]
.files
.contains_key(Utf8Path::new("/usr/lib64/libc.so.6")),
"/usr/lib64/libc.so.6 should belong to rpm/glibc"
);
// example xattr entry
assert!(
components["xattr/myapp"]
.files
.contains_key(Utf8Path::new("/opt/myapp/data")),
"/opt/myapp/data should belong to xattr/myapp"
);
// example unclaimed entry
assert!(
components[UNCLAIMED_COMPONENT]
.files
.contains_key(Utf8Path::new("/opt/myapp/config")),
"/opt/myapp/config should be unclaimed"
);
// rpmdb paths should be unclaimed
assert!(
components[UNCLAIMED_COMPONENT]
.files
.contains_key(Utf8Path::new("/usr/lib/sysimage/rpm/rpmdb.sqlite")),
"/usr/lib/sysimage/rpm/rpmdb.sqlite should be unclaimed"
);
}
}

470
src/components/rpm.rs Normal file
View file

@ -0,0 +1,470 @@
use std::collections::HashMap;
use anyhow::{Context, Result};
use camino::{Utf8Path, Utf8PathBuf};
use cap_std::fs::Dir;
use indexmap::IndexMap;
use rpm_qa::FileInfo;
use super::{ComponentId, ComponentInfo, ComponentsRepo, FileType};
const REPO_NAME: &str = "rpm";
const RPMDB_PATHS: &[&str] = &["usr/lib/sysimage/rpm", "usr/share/rpm", "var/lib/rpm"];
/// RPM-based components repo implementation.
///
/// Uses the RPM database to determine file ownership and groups files
/// by their SRPM.
pub struct RpmRepo {
/// Unique component (SRPM) names mapped to (buildtime, stability), indexed by ComponentId.
components: IndexMap<String, (u64, f64)>,
/// Mapping from path to list of (ComponentId, FileInfo).
///
/// It's common for directories to be owned by more than one component (i.e.
/// from _different_ SRPMs). It's much more uncommon for files/symlinks
/// though we do handle it to ensure reproducible layers.
path_to_components: HashMap<Utf8PathBuf, Vec<(ComponentId, FileInfo)>>,
}
impl RpmRepo {
/// Load the RPM database from the given rootfs.
///
/// Returns `Ok(None)` if no RPM database is detected.
pub fn load(rootfs: &Dir) -> Result<Option<Self>> {
if !has_rpmdb(rootfs)? {
return Ok(None);
}
let packages = rpm_qa::load_from_rootfs_dir(rootfs).context("loading rpmdb from rootfs")?;
Self::load_from_packages(packages).map(Some)
}
pub fn load_from_packages(packages: rpm_qa::Packages) -> Result<Self> {
let mut components: IndexMap<String, (u64, f64)> = IndexMap::new();
let mut path_to_components: HashMap<Utf8PathBuf, Vec<(ComponentId, FileInfo)>> =
HashMap::new();
for pkg in packages.into_values() {
// Use the source RPM as the component name, falling back to package name
let component_name: &str = pkg
.sourcerpm
.as_deref()
.map(parse_srpm_name)
.unwrap_or(&pkg.name);
let stability = calculate_stability(&pkg.changelog_times, pkg.buildtime)?;
let entry = components.entry(component_name.to_string());
let component_id = ComponentId(entry.index());
entry.or_insert((pkg.buildtime, stability));
for (path, file_info) in pkg.files.into_iter() {
// Accumulate entries for all file types. Skip if this component
// already owns this path (can happen when multiple subpackages
// from the same SRPM own the same path).
let entries = path_to_components.entry(path).or_default();
if !entries.iter().any(|(id, _)| *id == component_id) {
entries.push((component_id, file_info));
}
}
}
Ok(Self {
components,
path_to_components,
})
}
}
impl ComponentsRepo for RpmRepo {
fn name(&self) -> &'static str {
REPO_NAME
}
fn default_priority(&self) -> usize {
10
}
fn claims_for_path(&self, path: &Utf8Path, file_type: FileType) -> Vec<ComponentId> {
// Don't claim RPM database paths - let them fall into chunkah/unclaimed
if let Ok(rel_path) = path.strip_prefix("/")
&& RPMDB_PATHS.iter().any(|p| rel_path.starts_with(p))
{
return Vec::new();
}
self.path_to_components
.get(path)
.map(|entries| {
entries
.iter()
.filter(|(_, fi)| file_info_to_file_type(fi) == Some(file_type))
.map(|(id, _)| *id)
.collect()
})
.unwrap_or_default()
}
fn component_info(&self, id: ComponentId) -> ComponentInfo<'_> {
let (name, (mtime, stability)) = self
.components
.get_index(id.0)
// SAFETY: the ids we're given come from the IndexMap itself when we
// inserted the element, so it must be valid.
.expect("invalid ComponentId");
ComponentInfo {
name,
mtime_clamp: *mtime,
stability: *stability,
}
}
}
/// Check if any known RPM database path exists in the rootfs.
//
// This probably should live in rpm-qa-rs instead.
fn has_rpmdb(rootfs: &Dir) -> anyhow::Result<bool> {
for path in RPMDB_PATHS {
if rootfs
.try_exists(path)
.with_context(|| format!("checking for {path}"))?
{
return Ok(true);
}
}
Ok(false)
}
/// Parse the SRPM name from a full SRPM filename.
///
/// e.g., "bash-5.2.15-5.fc40.src.rpm" -> "bash"
fn parse_srpm_name(srpm: &str) -> &str {
// Remove .src.rpm suffix
let without_suffix = srpm.strip_suffix(".src.rpm").unwrap_or(srpm);
// Find the last two dashes (version-release)
// The name is everything before the second-to-last dash
let parts: Vec<&str> = without_suffix.rsplitn(3, '-').collect();
if parts.len() >= 3 {
parts[2]
} else {
without_suffix
}
}
/// Calculate stability from changelog timestamps and build time.
///
/// Uses a Poisson model. I used Gemini Pro 3 to analyzing RPM changelogs from
/// Fedora and found that once you filter out high-activity event-driven periods
/// (mass rebuilds, Fedora branching events), package updates over a large
/// enough period generally follow a Poisson distribution.
///
/// The lookback period is limited to STABILITY_LOOKBACK_DAYS (1 year).
/// If there are no changelog entries, the build time is used as a fallback.
fn calculate_stability(changelog_times: &[u64], buildtime: u64) -> Result<f64> {
use super::{SECS_PER_DAY, STABILITY_LOOKBACK_DAYS, STABILITY_PERIOD_DAYS};
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.context("getting current time")?
.as_secs();
let lookback_start = now.saturating_sub(STABILITY_LOOKBACK_DAYS * SECS_PER_DAY);
// If there are no changelog entries, use the buildtime as a single data point
let mut relevant_times: Vec<u64> = if changelog_times.is_empty() {
vec![buildtime]
} else {
changelog_times.to_vec()
};
// Filter to entries within the lookback window
relevant_times.retain(|&t| t >= lookback_start);
if relevant_times.is_empty() {
// All changelog entries are older than lookback period.
// No changes in the past year = very stable.
return Ok(0.99);
}
// Find the oldest timestamp in the window
let oldest = relevant_times.iter().min().copied().unwrap();
let span_days = (now.saturating_sub(oldest)) as f64 / SECS_PER_DAY as f64;
if span_days < 1.0 {
// Very recent package, assume unstable
return Ok(0.0);
}
let num_changes = relevant_times.len() as f64;
// lambda in our case is changes per day
let lambda = num_changes / span_days;
Ok((-lambda * STABILITY_PERIOD_DAYS).exp())
}
fn file_info_to_file_type(fi: &FileInfo) -> Option<FileType> {
let file_type = (fi.mode as libc::mode_t) & libc::S_IFMT;
match file_type {
libc::S_IFDIR => Some(FileType::Directory),
libc::S_IFREG => Some(FileType::File),
libc::S_IFLNK => Some(FileType::Symlink),
_ => None,
}
}
#[cfg(test)]
mod tests {
use camino::Utf8Path;
use super::*;
const FIXTURE: &str = include_str!("../../tests/fixtures/fedora.json");
#[test]
fn test_parse_srpm_name() {
// Package names with no dashes in them
assert_eq!(parse_srpm_name("bash-5.2.15-5.fc40.src.rpm"), "bash");
assert_eq!(parse_srpm_name("systemd-256.4-1.fc41.src.rpm"), "systemd");
assert_eq!(parse_srpm_name("python3-3.12.0-1.fc40.src.rpm"), "python3");
assert_eq!(parse_srpm_name("glibc-2.39-5.fc40.src.rpm"), "glibc");
// Package names with dashes in them
assert_eq!(
parse_srpm_name("python-dateutil-2.8.2-1.fc40.src.rpm"),
"python-dateutil"
);
assert_eq!(
parse_srpm_name("cairo-dock-plugins-3.4.1-1.fc40.src.rpm"),
"cairo-dock-plugins"
);
assert_eq!(
parse_srpm_name("xorg-x11-server-1.20.14-1.fc40.src.rpm"),
"xorg-x11-server"
);
// Edge cases with malformed input
// Only one dash (not enough for N-V-R pattern)
assert_eq!(parse_srpm_name("name-version"), "name-version");
// Missing .src.rpm suffix but valid N-V-R pattern
assert_eq!(parse_srpm_name("bash-5.2.15-5.fc40"), "bash");
// No dashes at all
assert_eq!(parse_srpm_name("nodash"), "nodash");
}
#[test]
fn test_claims_for_path() {
let packages = rpm_qa::load_from_str(FIXTURE).unwrap();
let repo = RpmRepo::load_from_packages(packages).unwrap();
// /usr/bin/bash is a file owned by bash
let claims = repo.claims_for_path(Utf8Path::new("/usr/bin/bash"), FileType::File);
assert_eq!(claims.len(), 1);
let info = repo.component_info(claims[0]);
assert_eq!(info.name, "bash");
assert_eq!(info.mtime_clamp, 1753299195);
// /usr/bin/sh is a symlink owned by bash
let claims = repo.claims_for_path(Utf8Path::new("/usr/bin/sh"), FileType::Symlink);
assert_eq!(claims.len(), 1);
let info = repo.component_info(claims[0]);
assert_eq!(info.name, "bash");
// /usr/lib64/libc.so.6 is a file owned by glibc
let claims = repo.claims_for_path(Utf8Path::new("/usr/lib64/libc.so.6"), FileType::File);
assert_eq!(claims.len(), 1);
let info = repo.component_info(claims[0]);
assert_eq!(info.name, "glibc");
assert_eq!(info.mtime_clamp, 1765791404);
// Unowned file should not be claimed
let claims = repo.claims_for_path(Utf8Path::new("/some/unowned/file"), FileType::File);
assert!(claims.is_empty());
// RPMDB paths should not be claimed even if technically owned by rpm package
for rpmdb_path in [
"/usr/lib/sysimage/rpm/rpmdb.sqlite",
"/usr/share/rpm/macros",
"/var/lib/rpm/Packages",
] {
let claims = repo.claims_for_path(Utf8Path::new(rpmdb_path), FileType::File);
assert!(
claims.is_empty(),
"RPMDB path {} should not be claimed",
rpmdb_path
);
}
}
#[test]
fn test_claims_for_path_wrong_type() {
let packages = rpm_qa::load_from_str(FIXTURE).unwrap();
let repo = RpmRepo::load_from_packages(packages).unwrap();
// /usr/bin/bash is a file in RPM, but we query as symlink
let claims = repo.claims_for_path(Utf8Path::new("/usr/bin/bash"), FileType::Symlink);
assert!(claims.is_empty());
// /usr/bin/sh is a symlink in RPM, but we query as file
let claims = repo.claims_for_path(Utf8Path::new("/usr/bin/sh"), FileType::File);
assert!(claims.is_empty());
}
#[test]
fn test_shared_directories_claimed_by_multiple_components() {
let packages = rpm_qa::load_from_str(FIXTURE).unwrap();
let repo = RpmRepo::load_from_packages(packages).unwrap();
// /usr/lib/.build-id is a well-known directory shared by many packages
let claims = repo.claims_for_path(Utf8Path::new("/usr/lib/.build-id"), FileType::Directory);
assert!(
claims.len() >= 2,
"shared dir should be claimed by multiple components"
);
// Verify well-known packages from the fixture are among the claims
let names: std::collections::HashSet<_> = claims
.iter()
.map(|id| repo.component_info(*id).name)
.collect();
for pkg in ["bash", "glibc", "coreutils"] {
assert!(names.contains(pkg), "{pkg} should claim /usr/lib/.build-id");
}
}
#[test]
fn test_load_from_rpmdb_sqlite() {
use std::process::Command;
// skip if rpm command is not available
let rpm_available = Command::new("rpm").arg("--version").output().is_ok();
if !rpm_available {
eprintln!("skipping test: rpm command not available");
return;
}
// create a temp rootfs with the rpmdb.sqlite fixture
let tmp = tempfile::tempdir().unwrap();
let rpmdb_dir = tmp.path().join("usr/lib/sysimage/rpm");
std::fs::create_dir_all(&rpmdb_dir).unwrap();
let fixture_path =
std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("tests/fixtures/rpmdb.sqlite");
std::fs::copy(&fixture_path, rpmdb_dir.join("rpmdb.sqlite")).unwrap();
let rootfs =
cap_std::fs::Dir::open_ambient_dir(tmp.path(), cap_std::ambient_authority()).unwrap();
let repo = RpmRepo::load(&rootfs).unwrap().unwrap();
// Test that paths we know are in filesystem and setup are claimed
let claims = repo.claims_for_path(Utf8Path::new("/"), FileType::Directory);
assert!(!claims.is_empty(), "/ should be claimed");
assert_eq!(repo.component_info(claims[0]).name, "filesystem");
let claims = repo.claims_for_path(Utf8Path::new("/etc"), FileType::Directory);
assert!(!claims.is_empty(), "/etc should be claimed");
// /etc is owned by filesystem
assert_eq!(repo.component_info(claims[0]).name, "filesystem");
let claims = repo.claims_for_path(Utf8Path::new("/etc/passwd"), FileType::File);
assert!(!claims.is_empty(), "/etc/passwd should be claimed");
assert_eq!(repo.component_info(claims[0]).name, "setup");
}
fn now_secs() -> u64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_secs()
}
fn assert_stability_in_range(stability: f64, min: f64, max: f64) {
assert!(
stability >= min && stability <= max,
"stability {stability} not in range [{min}, {max}]"
);
}
#[test]
fn test_calculate_stability_all_old_entries() {
use crate::components::SECS_PER_DAY;
// All entries older than 1 year should return 0.99
let now = now_secs();
let old_time = now - (400 * SECS_PER_DAY); // 400 days ago
let changelog_times = vec![old_time, old_time - SECS_PER_DAY];
let buildtime = old_time;
let stability = calculate_stability(&changelog_times, buildtime).unwrap();
assert_eq!(stability, 0.99);
}
#[test]
fn test_calculate_stability_very_recent() {
// Package built within 1 day should return 0.0
let now = now_secs();
let recent_time = now - 3600; // 1 hour ago
let changelog_times = vec![recent_time];
let buildtime = recent_time;
let stability = calculate_stability(&changelog_times, buildtime).unwrap();
assert_eq!(stability, 0.0);
}
#[test]
fn test_calculate_stability_no_changelog_uses_buildtime() {
use crate::components::SECS_PER_DAY;
// No changelog entries should use buildtime as fallback
let now = now_secs();
let buildtime = now - (30 * SECS_PER_DAY); // 30 days ago
let changelog_times: Vec<u64> = vec![];
let stability = calculate_stability(&changelog_times, buildtime).unwrap();
// 1 change over 30 days = lambda of 1/30
// stability = e^(-lambda * 7) = e^(-7/30) ≈ 0.79
assert_stability_in_range(stability, 0.75, 0.85);
}
#[test]
fn test_calculate_stability_normal_case() {
use crate::components::SECS_PER_DAY;
// Multiple changelog entries within lookback window
let now = now_secs();
// 4 changes over 100 days = lambda of 0.04
// stability = e^(-0.04 * 7) = e^(-0.28) ≈ 0.76
let changelog_times = vec![
now - (10 * SECS_PER_DAY),
now - (30 * SECS_PER_DAY),
now - (60 * SECS_PER_DAY),
now - (100 * SECS_PER_DAY),
];
let buildtime = now - (100 * SECS_PER_DAY);
let stability = calculate_stability(&changelog_times, buildtime).unwrap();
assert_stability_in_range(stability, 0.70, 0.80);
}
#[test]
fn test_calculate_stability_high_frequency() {
use crate::components::SECS_PER_DAY;
// Many changes in a short period = low stability
let now = now_secs();
// 10 changes over 20 days = lambda of 0.5
// stability = e^(-0.5 * 7) = e^(-3.5) ≈ 0.03
let changelog_times: Vec<u64> = (0..10)
.map(|i| now - ((2 + i * 2) * SECS_PER_DAY))
.collect();
let buildtime = now - (20 * SECS_PER_DAY);
let stability = calculate_stability(&changelog_times, buildtime).unwrap();
assert_stability_in_range(stability, 0.0, 0.10);
}
}

247
src/components/xattr.rs Normal file
View file

@ -0,0 +1,247 @@
use std::collections::HashMap;
use anyhow::{Context, Result};
use camino::{Utf8Path, Utf8PathBuf};
use indexmap::IndexSet;
use super::{ComponentId, ComponentInfo, ComponentsRepo, FileInfo, FileMap, FileType};
const XATTR_NAME: &str = "user.component";
const REPO_NAME: &str = "xattr";
/// Xattr-based components repo implementation.
///
/// Uses the `user.component` extended attribute to determine file ownership.
/// Directories with this xattr apply to all files underneath unless overridden.
/// Directory inheritance is pre-computed during load.
pub struct XattrRepo {
/// Component names, indexed by ComponentId.
components: IndexSet<String>,
/// Mapping from path to ComponentId (pre-computed with inheritance).
path_to_component: HashMap<Utf8PathBuf, ComponentId>,
/// Currently, the on-disk mtime is canonical and we clamp it, but it would
/// make sense in the future to support another user xattr to specify a
/// canonical mtime for easier layer reproducibility.
default_mtime_clamp: u64,
}
impl XattrRepo {
/// Load xattr repo by scanning rootfs for user.component xattrs.
/// Pre-computes directory inheritance for all paths in `files`.
/// Uses cached xattrs from FileInfo rather than reading from disk.
pub fn load(files: &FileMap, default_mtime_clamp: u64) -> Result<Option<Self>> {
let mut components: IndexSet<String> = IndexSet::new();
let mut path_to_component: HashMap<Utf8PathBuf, ComponentId> = HashMap::new();
// Track active directory components: (path, ComponentId)
// Directories with xattrs apply their component to descendants
let mut dir_stack: Vec<(&Utf8Path, ComponentId)> = Vec::new();
for (path, file_info) in files {
// Pop directories that are not ancestors of current path
while let Some((dir_path, _)) = dir_stack.last() {
if path.starts_with(dir_path) && path.as_path() != *dir_path {
break;
}
dir_stack.pop();
}
let own_xattr = get_component_xattr(file_info)
.with_context(|| format!("reading xattr for {}", path))?;
// If this path has an xattr, get or create its ComponentId
let own_component_id = own_xattr.as_ref().map(|name| {
// simplify this when we have either
// https://github.com/indexmap-rs/indexmap/issues/355 or
// https://github.com/indexmap-rs/indexmap/issues/388
let idx = components
.get_index_of(name)
.unwrap_or_else(|| components.insert_full(name.clone()).0);
ComponentId(idx)
});
// If this directory has an xattr, push to stack for children to inherit
if file_info.file_type == FileType::Directory
&& let Some(id) = own_component_id
{
dir_stack.push((path.as_path(), id));
}
// Determine effective component: own xattr or inherited from parent dir
let effective_id = own_component_id.or_else(|| dir_stack.last().map(|(_, id)| *id));
if let Some(id) = effective_id {
path_to_component.insert(path.clone(), id);
}
}
if components.is_empty() {
return Ok(None);
}
Ok(Some(Self {
components,
path_to_component,
default_mtime_clamp,
}))
}
}
/// Extract the user.component xattr value from cached xattrs.
fn get_component_xattr(file_info: &FileInfo) -> Result<Option<String>> {
file_info
.xattrs
.iter()
.find(|(k, _)| k == XATTR_NAME)
.map(|(_, v)| {
String::from_utf8(v.clone())
.map_err(|e| anyhow::anyhow!("invalid UTF-8 in {XATTR_NAME} xattr: {e}"))
})
.transpose()
}
impl ComponentsRepo for XattrRepo {
fn name(&self) -> &'static str {
REPO_NAME
}
fn default_priority(&self) -> usize {
0
}
fn claims_for_path(&self, path: &Utf8Path, _file_type: FileType) -> Vec<ComponentId> {
self.path_to_component
.get(path)
.map(|id| vec![*id])
.unwrap_or_default()
}
fn component_info(&self, id: ComponentId) -> ComponentInfo<'_> {
ComponentInfo {
name: self
.components
.get_index(id.0)
// SAFETY: the ids we're given come from the IndexSet itself
// when we inserted the element, so it must be valid.
.expect("invalid ComponentId"),
mtime_clamp: self.default_mtime_clamp,
// TODO: make this configurable via xattr or CLI
stability: 0.0,
}
}
}
#[cfg(test)]
mod tests {
use cap_std::ambient_authority;
use cap_std::fs::Dir;
use cap_std_ext::dirext::CapStdExtDirExt;
use super::*;
/// Helper to set up a rootfs, run setup, and scan files.
/// Returns (tempdir, files) - caller must keep tempdir alive.
fn setup_rootfs<F>(setup: F) -> (tempfile::TempDir, FileMap)
where
F: FnOnce(&Dir),
{
let tmp = tempfile::tempdir().unwrap();
let rootfs = Dir::open_ambient_dir(tmp.path(), ambient_authority()).unwrap();
setup(&rootfs);
let files = crate::scan::scan_rootfs(&rootfs).unwrap();
(tmp, files)
}
/// Helper to set the component xattr on a path.
fn set_component(rootfs: &Dir, path: &str, component: &str) {
rootfs
.setxattr(path, XATTR_NAME, component.as_bytes())
.unwrap();
}
/// Helper to assert a path is claimed by a specific component.
fn assert_component(repo: &XattrRepo, path: &str, file_type: FileType, expected: &str) {
let claims = repo.claims_for_path(Utf8Path::new(path), file_type);
assert_eq!(claims.len(), 1, "{path} should have exactly one claim");
assert_eq!(
repo.component_info(claims[0]).name,
expected,
"{path} should be claimed by {expected}"
);
}
#[test]
fn test_xattr_file_overrides_directory() {
let (_tmp, files) = setup_rootfs(|rootfs| {
// Create a directory with xattr
rootfs.create_dir("mydir").unwrap();
set_component(rootfs, "mydir", "dircomponent");
// File with its own xattr overrides directory
rootfs.write("mydir/special", "content").unwrap();
set_component(rootfs, "mydir/special", "filecomponent");
// File without xattr inherits from directory
rootfs.write("mydir/normal", "content").unwrap();
// File without xattr outside of directory - should not be claimed
rootfs.write("noattr", "content").unwrap();
});
let repo = XattrRepo::load(&files, 0).unwrap().unwrap();
// /mydir and /mydir/normal should be dircomponent
assert_component(&repo, "/mydir", FileType::Directory, "dircomponent");
assert_component(&repo, "/mydir/normal", FileType::File, "dircomponent");
// /mydir/special should be filecomponent
assert_component(&repo, "/mydir/special", FileType::File, "filecomponent");
// /noattr should not be claimed
let claims = repo.claims_for_path(Utf8Path::new("/noattr"), FileType::File);
assert!(claims.is_empty());
}
#[test]
fn test_xattr_inheritance() {
// Tests nested overrides and sibling isolation:
// /a has xattr A, /a/b has xattr B (overrides A), /a/b/c/d has xattr D
// /x has xattr X (sibling of /a, should not interfere)
let (_tmp, files) = setup_rootfs(|rootfs| {
rootfs.create_dir_all("a/b/c/d").unwrap();
rootfs.write("a/other", "content").unwrap();
rootfs.create_dir("x").unwrap();
rootfs.write("x/file", "content").unwrap();
set_component(rootfs, "a", "compA");
set_component(rootfs, "a/b", "compB");
set_component(rootfs, "a/b/c/d", "compD");
set_component(rootfs, "x", "compX");
});
let repo = XattrRepo::load(&files, 0).unwrap().unwrap();
assert_component(&repo, "/a", FileType::Directory, "compA");
assert_component(&repo, "/a/other", FileType::File, "compA"); // inherits from /a
assert_component(&repo, "/a/b", FileType::Directory, "compB");
assert_component(&repo, "/a/b/c", FileType::Directory, "compB"); // inherits from /a/b
assert_component(&repo, "/a/b/c/d", FileType::Directory, "compD"); // own xattr
assert_component(&repo, "/x", FileType::Directory, "compX"); // sibling of /a
assert_component(&repo, "/x/file", FileType::File, "compX"); // inherits from /x
}
#[test]
fn test_xattr_symlink_inherits_from_parent() {
// Symlinks don't support user xattrs, but they should inherit from parent directory
let (_tmp, files) = setup_rootfs(|rootfs| {
rootfs.create_dir("mydir").unwrap();
set_component(rootfs, "mydir", "mycomp");
// Create a symlink inside the directory - it should inherit from parent
rootfs.symlink("../somewhere", "mydir/link").unwrap();
});
let repo = XattrRepo::load(&files, 0).unwrap().unwrap();
// Both should be claimed by mycomp
assert_component(&repo, "/mydir", FileType::Directory, "mycomp");
assert_component(&repo, "/mydir/link", FileType::Symlink, "mycomp");
}
}

41
src/main.rs Normal file
View file

@ -0,0 +1,41 @@
mod cmd_build;
mod components;
mod ocibuilder;
#[allow(dead_code)]
mod packing;
mod scan;
mod tar;
mod utils;
use anyhow::{Context, Result};
use clap::{Parser, Subcommand};
#[derive(Parser)]
#[command(name = "chunkah")]
#[command(about = "A generalized container image rechunker")]
struct Cli {
#[command(subcommand)]
command: Command,
}
#[derive(Subcommand)]
enum Command {
/// Build an OCI archive from a rootfs
Build(Box<cmd_build::BuildArgs>),
}
fn main() -> Result<()> {
// Set up a SIGINT handler that terminates the process. This is needed
// because chunkah may run as PID 1 in a container, which can only receive
// signals it has explicit handlers for. This avoids users having to add
// e.g. --init to get Ctrl-C to behave as expected.
ctrlc::set_handler(|| std::process::exit(130)).context("setting up signal handler")?;
let cli = Cli::parse();
match cli.command {
Command::Build(args) => cmd_build::run(&args)?,
}
Ok(())
}

555
src/ocibuilder.rs Normal file
View file

@ -0,0 +1,555 @@
use std::collections::HashMap;
use std::io::Write;
use anyhow::{Context, Result};
use cap_std::fs::Dir;
use ocidir::oci_spec::image as oci_image;
use crate::components::Component;
/// Compression settings for the OCI image.
#[derive(Clone, Copy, Default)]
pub enum Compression {
/// No compression.
#[default]
None,
/// Gzip compression with the specified level (0-9).
Gzip(u32),
}
/// Builder for creating OCI images from components.
pub struct Builder {
/// The rootfs to build from.
rootfs: Dir,
/// The OCI directory to build into.
oci_dir: cap_std_ext::cap_tempfile::TempDir,
/// The components to include in the image, ordered by stability descending.
components: Vec<(String, Component)>,
/// Compression settings for layers and archive.
compression: Compression,
/// Annotations to add to the image manifest.
annotations: Option<HashMap<String, String>>,
/// The image configuration.
config: Option<oci_image::ImageConfiguration>,
}
impl Builder {
/// Create a new Builder with required parameters.
pub fn new(rootfs: &Dir, components: Vec<(String, Component)>) -> Result<Self> {
let oci_dir = cap_std_ext::cap_tempfile::tempdir(cap_std::ambient_authority())
.context("creating temp directory")?;
Ok(Self {
rootfs: rootfs.try_clone().context("cloning rootfs")?,
oci_dir,
components,
compression: Compression::default(),
annotations: None,
config: None,
})
}
/// Set the compression settings.
pub fn compression(mut self, compression: Compression) -> Self {
self.compression = compression;
self
}
/// Set annotations to add to the image manifest.
pub fn annotations(mut self, annotations: HashMap<String, String>) -> Self {
self.annotations = Some(annotations);
self
}
/// Set the image configuration.
pub fn config(mut self, config: oci_image::ImageConfiguration) -> Self {
self.config = Some(config);
self
}
/// Build the OCI image and write it to the given output.
pub fn build<W: Write>(self, output: &mut W) -> Result<()> {
self.build_oci_dir().context("building OCI directory")?;
let compression = match self.compression {
Compression::None => crate::tar::ArchiveCompression::None,
Compression::Gzip(level) => {
crate::tar::ArchiveCompression::Gzip(flate2::Compression::new(level))
}
};
crate::tar::write_oci_archive(&self.oci_dir, &mut *output, compression)
.context("writing OCI archive")?;
output.flush().context("flushing output")
}
fn build_oci_dir(&self) -> Result<()> {
let oci_dir =
ocidir::OciDir::ensure(self.oci_dir.try_clone().context("cloning temp directory")?)
.context("creating OCI directory")?;
// first, let's create an empty manifest
let mut manifest = oci_dir
.new_empty_manifest()
.context("creating empty manifest")?
.build()
.context("building manifest")?;
let mut config = self.config.clone().unwrap_or_default();
// this is the important bit: we add all the layers
self.add_components(&mut manifest, &mut config)
.context("adding layers to OCI directory")?;
if let Some(annotations) = &self.annotations {
manifest.set_annotations(Some(annotations.clone()));
}
let arch = config.architecture().to_string();
let platform = oci_image::PlatformBuilder::default()
.os("linux")
.architecture(arch.as_str())
.build()
.context("building platform")?;
oci_dir
.insert_manifest_and_config(manifest, config, None, platform)
.context("inserting manifest and config")?;
Ok(())
}
/// Add layers to the OCI directory and update the manifest and config.
fn add_components(
&self,
manifest: &mut oci_image::ImageManifest,
config: &mut oci_image::ImageConfiguration,
) -> Result<()> {
for (name, component) in &self.components {
if component.files.is_empty() {
continue;
}
self.add_component(manifest, config, name, component)
.with_context(|| format!("adding component {}", name))?;
}
// Clear history - we don't want to emit it in the output image
// XXX: add a e.g. push_layer_without_history_annotated to ocidir
config.history_mut().take();
Ok(())
}
/// Add a single component as a layer to the OCI directory.
fn add_component(
&self,
manifest: &mut oci_image::ImageManifest,
config: &mut oci_image::ImageConfiguration,
name: &str,
component: &Component,
) -> Result<()> {
let oci_dir = ocidir::OciDir::open(self.oci_dir.try_clone().context("cloning oci_dir")?)
.context("opening OCI directory")?;
let mut tar_builder =
crate::tar::create_layer(&oci_dir, self.compression).context("creating layer")?;
crate::tar::write_files_to_tar(
&mut tar_builder,
&self.rootfs,
&component.files,
component.mtime_clamp,
)
.context("building tar layer")?;
tar_builder.finish().context("finishing layer tar")?;
let layer = tar_builder
.into_inner()
.context("getting layer writer")?
.complete()
.context("completing layer")?;
let annotations = {
let mut hm = HashMap::new();
hm.insert("org.chunkah.component".to_string(), name.to_string());
hm.insert(
"org.chunkah.stability".to_string(),
format!("{:.3}", component.stability),
);
hm
};
oci_dir.push_layer_with_history_annotated(manifest, config, layer, Some(annotations), None);
Ok(())
}
}
#[cfg(test)]
mod tests {
use super::*;
use camino::Utf8PathBuf;
use cap_std::ambient_authority;
use cap_std::fs::PermissionsExt;
use cap_std_ext::dirext::CapStdExtDirExt;
use maplit::btreeset;
use std::collections::BTreeSet;
use crate::components::FileMap;
/// Helper struct for test results
struct TestOciResult {
oci_dir: ocidir::OciDir,
manifest: oci_image::ImageManifest,
image_config: oci_image::ImageConfiguration,
/// Keep tempdir alive for the lifetime of the test
_oci_tempdir: tempfile::TempDir,
}
impl TestOciResult {
fn first_layer(&self) -> &oci_image::Descriptor {
&self.manifest.layers()[0]
}
fn read_layer_tar(
&self,
layer: &oci_image::Descriptor,
) -> tar::Archive<impl std::io::Read + '_> {
tar::Archive::new(self.oci_dir.read_blob(layer).unwrap())
}
fn get_layer_tar_entries(
&self,
layer: &oci_image::Descriptor,
) -> Vec<(String, tar::EntryType, u64)> {
let mut layer_tar = self.read_layer_tar(layer);
layer_tar
.entries()
.unwrap()
.map(|e| {
let entry = e.unwrap();
let path = entry.path().unwrap().to_string_lossy().to_string();
let entry_type = entry.header().entry_type();
let size = entry.header().size().unwrap();
(path, entry_type, size)
})
.collect()
}
}
/// Specification for a test component: (name, paths, mtime_clamp)
type ComponentSpec = (&'static str, BTreeSet<Utf8PathBuf>, u64);
/// Helper to build an OCI archive and extract it for inspection.
fn build_and_extract<F>(rootfs_setup: F, specs: Vec<ComponentSpec>) -> TestOciResult
where
F: FnOnce(&Dir),
{
// Create temp rootfs and run setup
let rootfs_dir = tempfile::tempdir().unwrap();
let rootfs = Dir::open_ambient_dir(rootfs_dir.path(), ambient_authority()).unwrap();
rootfs_setup(&rootfs);
// Scan the rootfs once
let all_files = crate::scan::scan_rootfs(&rootfs).unwrap();
// Build components by selecting paths from scanned files
let components: Vec<(String, Component)> = specs
.into_iter()
.map(|(name, paths, mtime_clamp)| {
let files: FileMap = paths
.iter()
.filter_map(|p| all_files.get(p).map(|info| (p.clone(), info.clone())))
.collect();
(
name.to_string(),
Component {
mtime_clamp,
stability: 0.0,
files,
},
)
})
.collect();
// Create minimal config
let config = oci_image::ImageConfigurationBuilder::default()
.os("linux")
.architecture("amd64")
.rootfs(
oci_image::RootFsBuilder::default()
.typ("layers")
.diff_ids(Vec::<String>::new())
.build()
.unwrap(),
)
.build()
.unwrap();
// Build OCI archive
let builder = Builder::new(&rootfs, components)
.unwrap()
.compression(Compression::None)
.config(config);
let mut output = Vec::new();
builder.build(&mut output).unwrap();
// Extract to tempdir
let oci_tempdir = tempfile::tempdir().unwrap();
let mut archive = tar::Archive::new(output.as_slice());
archive.unpack(oci_tempdir.path()).unwrap();
// Open with ocidir
let oci_dir_cap = Dir::open_ambient_dir(oci_tempdir.path(), ambient_authority()).unwrap();
let oci_dir = ocidir::OciDir::open(oci_dir_cap).unwrap();
// Get manifest
let index = oci_dir.read_index().unwrap();
let manifest_desc = index.manifests().first().unwrap();
let manifest: oci_image::ImageManifest = oci_dir.read_json_blob(manifest_desc).unwrap();
// Get image config
let image_config: oci_image::ImageConfiguration =
oci_dir.read_json_blob(manifest.config()).unwrap();
TestOciResult {
oci_dir,
manifest,
image_config,
_oci_tempdir: oci_tempdir,
}
}
#[test]
fn test_components_to_layers() {
let result = build_and_extract(
|rootfs| {
rootfs.write("file_a", "content a").unwrap();
rootfs.write("file_b", "content b").unwrap();
},
vec![
(
"component_a",
btreeset! { Utf8PathBuf::from("/file_a") },
1000,
),
(
"component_b",
btreeset! { Utf8PathBuf::from("/file_b") },
1000,
),
],
);
assert_eq!(result.manifest.layers().len(), 2);
// Collect layer info (order is not guaranteed with HashMap)
let layer_info: Vec<_> = result
.manifest
.layers()
.iter()
.map(|layer| {
let component = layer
.annotations()
.as_ref()
.and_then(|a| a.get("org.chunkah.component"))
.cloned();
let entries = result.get_layer_tar_entries(layer);
(component, entries)
})
.collect();
// Find component_a layer
let layer_a = layer_info
.iter()
.find(|(c, _)| c.as_deref() == Some("component_a"))
.expect("should have component_a layer");
assert_eq!(layer_a.1.len(), 1);
assert_eq!(layer_a.1[0].0, "file_a");
assert_eq!(layer_a.1[0].1, tar::EntryType::Regular);
assert_eq!(layer_a.1[0].2, 9); // "content a".len()
// Find component_b layer
let layer_b = layer_info
.iter()
.find(|(c, _)| c.as_deref() == Some("component_b"))
.expect("should have component_b layer");
assert_eq!(layer_b.1.len(), 1);
assert_eq!(layer_b.1[0].0, "file_b");
assert_eq!(layer_b.1[0].1, tar::EntryType::Regular);
assert_eq!(layer_b.1[0].2, 9); // "content b".len()
// Verify no history entries
assert!(
result
.image_config
.history()
.as_ref()
.is_none_or(|h| h.is_empty()),
"image should have no history entries"
);
}
#[test]
fn test_file_metadata() {
let result = build_and_extract(
|rootfs| {
rootfs.write("executable", "#!/bin/sh\necho hi").unwrap();
// Make it executable (0o755)
let meta = rootfs.metadata("executable").unwrap();
let mut perms = meta.permissions();
perms.set_mode(0o755);
rootfs.set_permissions("executable", perms).unwrap();
rootfs.create_dir("mydir").unwrap();
rootfs
.set_permissions("mydir", cap_std::fs::Permissions::from_mode(0o777))
.unwrap();
rootfs.write("xattr_file", "xattr content").unwrap();
rootfs
.setxattr("xattr_file", "user.myattr", b"myvalue")
.unwrap();
rootfs.write("target", "target content").unwrap();
rootfs.symlink("target", "link").unwrap();
},
vec![(
"test",
btreeset! {
Utf8PathBuf::from("/executable"),
Utf8PathBuf::from("/link"),
Utf8PathBuf::from("/mydir"),
Utf8PathBuf::from("/target"),
Utf8PathBuf::from("/xattr_file"),
},
1000,
)],
);
// Get the layer and check metadata
let mut layer_tar = result.read_layer_tar(result.first_layer());
let mut found_xattr = false;
for entry in layer_tar.entries().unwrap() {
let mut entry = entry.unwrap();
let path = entry.path().unwrap().to_string_lossy().to_string();
let header = entry.header();
// Check mtime is clamped
let mtime = header.mtime().unwrap();
assert!(mtime <= 1000, "mtime should be clamped: {}", mtime);
if path == "executable" {
// Check mode preserves executable bit
let mode = header.mode().unwrap();
assert!(
mode & 0o111 != 0,
"executable should have execute bits: {:o}",
mode
);
} else if path == "mydir/" {
assert_eq!(header.entry_type(), tar::EntryType::Directory);
// Verify atypical 777 permissions are preserved
let mode = header.mode().unwrap();
assert_eq!(
mode & 0o777,
0o777,
"directory should have 777 permissions, got {:o}",
mode & 0o777
);
} else if path == "xattr_file" {
// Check xattr survives
let pax = entry.pax_extensions().unwrap().unwrap();
for ext in pax {
let ext = ext.unwrap();
if ext.key().unwrap() == "SCHILY.xattr.user.myattr" {
assert_eq!(ext.value_bytes(), b"myvalue");
found_xattr = true;
}
}
assert!(found_xattr, "xattr should be preserved in layer");
} else if path == "link" {
// Check symlink
assert_eq!(header.entry_type(), tar::EntryType::Symlink);
let link_target = header.link_name().unwrap().unwrap();
assert_eq!(link_target.to_string_lossy(), "target");
}
}
}
#[test]
fn test_nested_directory_structure() {
let result = build_and_extract(
|rootfs| {
rootfs.create_dir_all("usr/bin").unwrap();
rootfs.write("usr/bin/myapp", "app content").unwrap();
},
vec![(
"test",
btreeset! {
Utf8PathBuf::from("/usr"),
Utf8PathBuf::from("/usr/bin"),
Utf8PathBuf::from("/usr/bin/myapp"),
},
1000,
)],
);
// Layer should contain the file and parent directories
let entries = result.get_layer_tar_entries(&result.manifest.layers()[0]);
// Should have /usr, /usr/bin, and /usr/bin/myapp
// Note: directories have trailing slashes in tar
let paths: Vec<_> = entries.iter().map(|(p, _, _)| p.as_str()).collect();
assert!(paths.contains(&"usr/"), "should contain usr/: {:?}", paths);
assert!(
paths.contains(&"usr/bin/"),
"should contain usr/bin/: {:?}",
paths
);
assert!(
paths.contains(&"usr/bin/myapp"),
"should contain usr/bin/myapp: {:?}",
paths
);
}
#[test]
fn test_mtime_clamping() {
use fs_set_times::{SetTimes, SystemTimeSpec};
let result = build_and_extract(
|rootfs| {
// This file has current mtime (>> 500), should be clamped to 500
rootfs.write("clamped_file", "content").unwrap();
// This file has mtime 200 (< 500), should stay at 200
rootfs.write("unclamped_file", "content").unwrap();
let file = rootfs.open("unclamped_file").unwrap();
let mtime = std::time::SystemTime::UNIX_EPOCH + std::time::Duration::from_secs(200);
file.set_times(None, Some(SystemTimeSpec::Absolute(mtime)))
.unwrap();
},
vec![(
"test",
btreeset! {
Utf8PathBuf::from("/clamped_file"),
Utf8PathBuf::from("/unclamped_file"),
},
500,
)],
);
let mut layer_tar = result.read_layer_tar(result.first_layer());
for entry in layer_tar.entries().unwrap() {
let entry = entry.unwrap();
let path = entry.path().unwrap().to_string_lossy().to_string();
let mtime = entry.header().mtime().unwrap();
if path == "clamped_file" {
assert_eq!(mtime, 500, "clamped_file mtime should be clamped to 500");
} else if path == "unclamped_file" {
assert_eq!(mtime, 200, "unclamped_file mtime should stay at 200");
}
}
}
}

421
src/packing.rs Normal file
View file

@ -0,0 +1,421 @@
//! # Packing Algorithm
//!
//! This module implements a clustering algorithm to merge a set of N
//! components into K groups while maximizing the group reuse across updates.
//! The general approach came out of a session with Gemini 3 Pro on an
//! abstracted version of the problem statement.
//!
//! ## Definitions
//!
//! 1. The "stability" of a component is the probability it doesn't change.
//! 2. The "size" of a component is the sum of all its files.
//! 3. The "expected value" of a component is stability x size; basically,
//! long-term how much data do we avoid pulling from this component on
//! average.
//! 4. The expected value of two components combined in the same group is
//! (size1 + size2) x (stability1 x stability2).
//!
//! We want to find the group arrangement which maximize total expected value
//! (TEV). The final groups become OCI layers.
//!
//! ## Algorithm
//!
//! 1. Create a separate group for each component. If N <= K, we're done.
//! 2. Otherwise, we need to merge some groups. Every merge will reduce TEV.
//! We should look for the merge that will result in the smallest TEV loss,
//! so we first need to calculate the TEV loss for all possible merges. For
//! every possible merge of two components, we calculate what the EV of the
//! merged group would be and how much smaller it is from the EV of keeping
//! them separate. Doing this for every component is O(N^2), but meh, N in
//! our case is trivially small for modern computer speeds. Shove all those
//! potential merges and their losses in a BinaryHeap. The top will then
//! always be the merge with the smallest loss.
//! 3. Pop the heap to get the next optimal merge and do that merge. Calculate
//! losses for this new merged group vs all the remaining groups and insert
//! into the heap.
//! 4. Keep doing 3. until we get to K groups.
use std::cmp::Ordering;
use std::collections::BinaryHeap;
/// Input item for packing
#[derive(Debug, Clone)]
pub struct PackItem {
/// Total size in bytes of all files in this component
pub size: u64,
/// Probability the component doesn't change between updates (0.0 to 1.0)
pub stability: f64,
}
/// Output group from packing
#[derive(Debug, Clone)]
pub struct PackGroup {
/// Indices into the original input slice
pub indices: Vec<usize>,
/// Total size in bytes of all files in this group
pub size: u64,
/// Combined stability of the group (product of individual stabilities)
pub stability: f64,
}
impl PackGroup {
fn expected_value(&self) -> f64 {
self.size as f64 * self.stability
}
}
/// A candidate merge operation stored in the heap.
#[derive(Debug)]
struct MergeCandidate {
loss: f64,
group_a_id: usize,
group_b_id: usize,
}
impl Ord for MergeCandidate {
fn cmp(&self, other: &Self) -> Ordering {
// Note here we *reverse* the order of comparison because we want a
// min-heap not a max-heap.
other
.loss
.partial_cmp(&self.loss)
.unwrap_or(Ordering::Equal)
}
}
impl PartialOrd for MergeCandidate {
fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
Some(self.cmp(other))
}
}
impl PartialEq for MergeCandidate {
fn eq(&self, other: &Self) -> bool {
self.loss == other.loss
}
}
impl Eq for MergeCandidate {}
/// Calculates how to pack items into at most `max_groups` groups in a way
/// that attempts to maximize group reuse. See module docstring for algorithm
/// details.
///
/// Returns groups sorted by stability descending (most stable first). Each
/// group contains indices into the original input slice.
pub fn calculate_packing(items: &[PackItem], max_groups: usize) -> Vec<PackGroup> {
if items.is_empty() || max_groups == 0 {
return Vec::new();
}
let n = items.len();
// if we already have fewer items than max_groups, no packing is needed
if n <= max_groups {
let mut result: Vec<PackGroup> = items
.iter()
.enumerate()
.map(|(i, item)| PackGroup {
indices: vec![i],
size: item.size,
stability: item.stability,
})
.collect();
sort_by_stability_desc(&mut result);
return result;
}
// use a Vec<Option> to track active groups; merged groups are appended
let mut groups: Vec<Option<PackGroup>> = items
.iter()
.enumerate()
.map(|(i, item)| {
Some(PackGroup {
indices: vec![i],
size: item.size,
stability: item.stability,
})
})
.collect();
let mut active_count = n;
let mut merge_candidates = BinaryHeap::new();
// pre-calculate merge losses for all initial pairs
for i in 0..n {
for j in (i + 1)..n {
// SAFETY: we just created these groups above
let g_a = groups[i].as_ref().unwrap();
let g_b = groups[j].as_ref().unwrap();
let loss = calculate_merge_loss(g_a, g_b);
merge_candidates.push(MergeCandidate {
loss,
group_a_id: i,
group_b_id: j,
});
}
}
// do the next best merge until we're within the constraint
while active_count > max_groups {
let Some(merge_op) = merge_candidates.pop() else {
break;
};
// skip stale candidates (groups already merged)
if groups[merge_op.group_a_id].is_none() || groups[merge_op.group_b_id].is_none() {
continue;
}
// SAFETY: we just verified above that both are Some
let g_a = groups[merge_op.group_a_id].take().unwrap();
let g_b = groups[merge_op.group_b_id].take().unwrap();
let mut new_indices = g_a.indices;
new_indices.extend(g_b.indices);
// append merged group
let new_id = groups.len();
groups.push(Some(PackGroup {
indices: new_indices,
size: g_a.size + g_b.size,
stability: g_a.stability * g_b.stability,
}));
active_count -= 1;
// calculate losses between new group and all remaining groups
let created_group = groups[new_id].as_ref().unwrap();
for (other_id, other_group_opt) in groups.iter().enumerate() {
if other_id == new_id {
continue;
}
// is this still an active group?
if let Some(other_group) = other_group_opt {
let loss = calculate_merge_loss(created_group, other_group);
merge_candidates.push(MergeCandidate {
loss,
group_a_id: new_id,
group_b_id: other_id,
});
}
}
}
// collect and sort results by stability descending
let mut result: Vec<PackGroup> = groups.into_iter().flatten().collect();
sort_by_stability_desc(&mut result);
result
}
fn sort_by_stability_desc(items: &mut [PackGroup]) {
items.sort_by(|a, b| {
b.stability
.partial_cmp(&a.stability)
.unwrap_or(Ordering::Equal)
});
}
fn calculate_merge_loss(a: &PackGroup, b: &PackGroup) -> f64 {
let ev_separate = a.expected_value() + b.expected_value();
let combined_size = (a.size + b.size) as f64;
let combined_prob = a.stability * b.stability;
let ev_merged = combined_size * combined_prob;
// Loss = expected value destroyed by merging
ev_separate - ev_merged
}
#[cfg(test)]
mod tests {
use super::*;
use std::collections::HashSet;
// Note from author: it's tricky to test this algorithm properly because
// since it's a greedy algorithm, it's not guaranteed to always yield
// the truly optimal solution. Here we test some simplified cases. In the
// future, it'd be nice to set up a harness with real test data that we can
// use to evaluate different algorithms or potential improvements. At least
// that way we get a comparative validation of the algorithm.
/// Verifies invariants that must hold for any valid packing result.
fn verify_packing_result(input: &[PackItem], result: &[PackGroup], max_groups: usize) {
// check group count respects max_groups
assert!(
result.len() <= max_groups,
"too many groups: {} > {}",
result.len(),
max_groups
);
// check all indices present exactly once (no loss, no duplication)
let mut output_indices: Vec<usize> =
result.iter().flat_map(|g| &g.indices).copied().collect();
output_indices.sort();
let expected_indices: Vec<usize> = (0..input.len()).collect();
assert_eq!(output_indices, expected_indices, "indices mismatch");
// check no empty groups
assert!(
result.iter().all(|g| !g.indices.is_empty()),
"found empty group"
);
// check sorted by stability descending
for i in 1..result.len() {
assert!(
result[i - 1].stability >= result[i].stability,
"groups not sorted by stability: {:?}",
result.iter().map(|g| g.stability).collect::<Vec<_>>()
);
}
// check total size preserved
let input_total: u64 = input.iter().map(|c| c.size).sum();
let output_total: u64 = result
.iter()
.flat_map(|g| &g.indices)
.map(|&idx| input[idx].size)
.sum();
assert_eq!(input_total, output_total, "total size mismatch");
}
#[test]
fn test_edge_cases() {
// empty input
assert!(calculate_packing(&[], 5).is_empty());
// max_groups = 0
let items = vec![PackItem {
size: 100,
stability: 0.5,
}];
assert!(calculate_packing(&items, 0).is_empty());
// single item
let items = vec![PackItem {
size: 100,
stability: 0.5,
}];
let result = calculate_packing(&items, 5);
assert_eq!(result.len(), 1);
assert_eq!(result[0].indices, vec![0]);
verify_packing_result(&items, &result, 5);
}
#[test]
fn test_no_packing_needed() {
// Items with different stabilities
let items = vec![
PackItem {
size: 100,
stability: 0.9,
},
PackItem {
size: 200,
stability: 0.8,
},
PackItem {
size: 300,
stability: 0.7,
},
];
let result = calculate_packing(&items, 5);
assert_eq!(result.len(), 3);
// Should be sorted by stability descending
assert_eq!(result[0].indices, vec![0]); // 0.9
assert_eq!(result[1].indices, vec![1]); // 0.8
assert_eq!(result[2].indices, vec![2]); // 0.7
verify_packing_result(&items, &result, 5);
}
#[test]
fn test_pack_to_one_group() {
let items = vec![
PackItem {
size: 100,
stability: 0.5,
},
PackItem {
size: 200,
stability: 0.5,
},
PackItem {
size: 300,
stability: 0.5,
},
];
let result = calculate_packing(&items, 1);
assert_eq!(result.len(), 1);
assert_eq!(result[0].indices.len(), 3);
// All items should be in the single group
let indices: HashSet<usize> = result[0].indices.iter().copied().collect();
assert_eq!(indices, HashSet::from([0, 1, 2]));
verify_packing_result(&items, &result, 1);
}
#[test]
fn test_size_constant_stability_changes() {
// merging two high-stability items has less loss than merging
// a high-stability with a low-stability item
// index 0: stable_1, index 1: stable_2, index 2: unstable
let items = vec![
PackItem {
size: 1000,
stability: 0.99,
},
PackItem {
size: 1000,
stability: 0.99,
},
PackItem {
size: 1000,
stability: 0.3,
},
];
let result = calculate_packing(&items, 2);
assert_eq!(result.len(), 2);
// the two stable items (indices 0 and 1) should be merged together
let merged_group = result
.iter()
.find(|g| g.indices.len() == 2)
.expect("one group should have 2 items");
let merged_indices: HashSet<usize> = merged_group.indices.iter().copied().collect();
assert_eq!(merged_indices, HashSet::from([0, 1]));
verify_packing_result(&items, &result, 2);
}
#[test]
fn test_stability_constant_size_changes() {
// with uniform stability, merging smaller items has less loss
// index 0: huge, index 1: small1, index 2: small2
let items = vec![
PackItem {
size: 10000,
stability: 0.5,
},
PackItem {
size: 10,
stability: 0.5,
},
PackItem {
size: 10,
stability: 0.5,
},
];
let result = calculate_packing(&items, 2);
assert_eq!(result.len(), 2);
// The two small items (indices 1 and 2) should be merged together (least loss)
// The huge item (index 0) should stay separate
let huge_group = result.iter().find(|g| g.indices.contains(&0));
assert!(huge_group.is_some());
assert_eq!(huge_group.unwrap().indices.len(), 1);
let small_group = result.iter().find(|g| g.indices.contains(&1));
assert!(small_group.is_some());
assert!(small_group.unwrap().indices.contains(&2));
verify_packing_result(&items, &result, 2);
}
}

174
src/scan.rs Normal file
View file

@ -0,0 +1,174 @@
use std::collections::{BTreeMap, HashMap};
use std::ops::ControlFlow;
use std::path::Path;
use anyhow::{Context, Result};
use camino::Utf8Path;
use cap_std::fs::Dir;
use cap_std_ext::dirext::{CapStdExtDirExt, WalkConfiguration};
use crate::components::{self, Component, FileInfo, FileMap};
/// Scan the rootfs for components and return a mapping of component names to components.
pub fn scan_for_components(
rootfs: &Dir,
default_mtime_clamp: u64,
) -> Result<HashMap<String, Component>> {
let files = scan_rootfs(rootfs).context("scanning rootfs")?;
let repos = components::ComponentsRepos::load(rootfs, &files, default_mtime_clamp)
.context("loading components")?;
if repos.is_empty() {
// error for now but i guess we could just warn and have a single layer
anyhow::bail!("no supported component repo found in rootfs");
}
Ok(repos.into_components(files))
}
/// Scan the rootfs and return a map of file paths to their metadata.
/// We use cap-std-ext's walk here, which doesn't follow symlinks.
pub fn scan_rootfs(rootfs: &Dir) -> Result<FileMap> {
let mut files = BTreeMap::new();
let config = WalkConfiguration::default().path_base(Path::new("/"));
rootfs
.walk(&config, |component| {
let path: &Utf8Path = component
.path
.try_into()
.map_err(|_| anyhow::anyhow!("path is not valid UTF-8"))?;
let rel_path = path.strip_prefix("/").unwrap_or(path);
let fs_path = if rel_path.as_str().is_empty() {
"."
} else {
rel_path.as_str()
};
let metadata = rootfs
.symlink_metadata(fs_path)
.with_context(|| format!("getting metadata for {}", path))?;
let xattrs = read_xattrs(rootfs, fs_path)
.with_context(|| format!("reading xattrs for {}", path))?;
let file_info = FileInfo::from_metadata(&metadata, xattrs)
.with_context(|| format!("processing metadata for {}", path))?;
files.insert(path.to_owned(), file_info);
Ok::<_, anyhow::Error>(ControlFlow::Continue(()))
})
.context("failed to walk rootfs")?;
Ok(files)
}
/// Read all xattrs for a path.
pub fn read_xattrs(rootfs: &Dir, fs_path: &str) -> anyhow::Result<Vec<(String, Vec<u8>)>> {
use std::ffi::OsStr;
let xattr_list = rootfs
.listxattrs(fs_path)
.with_context(|| format!("listing xattrs for {}", fs_path))?;
let mut xattrs = Vec::new();
for key in xattr_list.iter() {
// Skip selinux attributes for now. It would only bloat images since
// _every_ file has SELinux attributes but they come from the container
// runtime, not the tar layer, which is ignored. Bootable containers
// could use them, but don't currently. We can make it opt in once it's
// desirable.
if key == OsStr::new("security.selinux") {
continue;
}
if let Some(value) = rootfs
.getxattr(fs_path, key)
.with_context(|| format!("reading xattr {} for {}", key.display(), fs_path))?
{
// Technically, keeping the key as OsStr would be more correct,
// but we'll need UTF-8 to shove it in a PAX header anyway so might
// as well error now. Note libarchive and GNU tar differ here.
// libarchive does urlencoding, GNU tar just writes the key as is
// anyway. We'll cross that bridge when/if we get to it.
let key_str = key
.to_str()
.with_context(|| format!("non-UTF8 xattr key {} on {}", key.display(), fs_path))?;
xattrs.push((key_str.to_string(), value));
}
}
Ok(xattrs)
}
#[cfg(test)]
mod tests {
use camino::Utf8Path;
use cap_std::ambient_authority;
use super::*;
use crate::components::FileType;
/// Helper to get the file type for a path.
fn get_file_type(files: &FileMap, path: &str) -> Option<FileType> {
files.get(Utf8Path::new(path)).map(|f| f.file_type)
}
#[test]
fn test_scan_rootfs_does_not_follow_symlinks() {
let tmp = tempfile::tempdir().unwrap();
let rootfs = Dir::open_ambient_dir(tmp.path(), ambient_authority()).unwrap();
rootfs.create_dir("realdir").unwrap();
rootfs.write("realdir/file.txt", "content").unwrap();
rootfs.symlink("realdir", "linkdir").unwrap();
rootfs.symlink("enoent", "broken").unwrap();
rootfs.symlink("../../../etc/passwd", "escape").unwrap();
let files = scan_rootfs(&rootfs).unwrap();
assert_eq!(get_file_type(&files, "/realdir"), Some(FileType::Directory));
assert_eq!(
get_file_type(&files, "/realdir/file.txt"),
Some(FileType::File)
);
assert_eq!(get_file_type(&files, "/linkdir"), Some(FileType::Symlink));
assert_eq!(get_file_type(&files, "/broken"), Some(FileType::Symlink));
assert_eq!(get_file_type(&files, "/escape"), Some(FileType::Symlink));
}
#[test]
fn test_scan_rootfs_empty() {
let tmp = tempfile::tempdir().unwrap();
let rootfs = Dir::open_ambient_dir(tmp.path(), ambient_authority()).unwrap();
let files = scan_rootfs(&rootfs).unwrap();
// Should be empty. Note even the root directory is not included.
// Root entries are not commonly in the tar stream. Container
// runtimes ignore them so we may not even have read the real perms,
// nor what we emit will be read. Bootable containers and other
// OCI-but-not-container-runtime users could make use of them, but we'll
// probably want to make it opt in if the use case shows up.
assert_eq!(files.len(), 0);
}
#[test]
fn test_scan_rootfs_nested_structure() {
let tmp = tempfile::tempdir().unwrap();
let rootfs = Dir::open_ambient_dir(tmp.path(), ambient_authority()).unwrap();
rootfs.create_dir_all("a/b/c").unwrap();
rootfs.write("a/b/c/file", "content").unwrap();
let files = scan_rootfs(&rootfs).unwrap();
assert_eq!(get_file_type(&files, "/a"), Some(FileType::Directory));
assert_eq!(get_file_type(&files, "/a/b"), Some(FileType::Directory));
assert_eq!(get_file_type(&files, "/a/b/c"), Some(FileType::Directory));
assert_eq!(get_file_type(&files, "/a/b/c/file"), Some(FileType::File));
}
}

670
src/tar.rs Normal file
View file

@ -0,0 +1,670 @@
use std::collections::HashMap;
use std::io::Write;
use anyhow::{Context, Result};
use camino::{Utf8Path, Utf8PathBuf};
use ocidir::oci_spec::image as oci_image;
use ocidir::{BlobWriter, WriteComplete};
use crate::components::{FileInfo, FileMap, FileType};
/// Compression options for OCI archives.
pub enum ArchiveCompression {
/// No compression.
None,
/// Gzip compression with the specified level.
Gzip(flate2::Compression),
}
/// A passthrough writer that performs no compression.
// XXX: upstream this to ocidir as e.g. create_uncompressed_layer() ?
pub(crate) struct NoCompression<'a>(BlobWriter<'a>);
impl std::io::Write for NoCompression<'_> {
fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
self.0.write(buf)
}
fn flush(&mut self) -> std::io::Result<()> {
self.0.flush()
}
}
impl<'a> WriteComplete<BlobWriter<'a>> for NoCompression<'a> {
fn complete(self) -> std::io::Result<BlobWriter<'a>> {
Ok(self.0)
}
}
/// Layer writer that can be either compressed or uncompressed.
pub enum LayerWriter<'a> {
Uncompressed(ocidir::LayerWriter<'a, NoCompression<'a>>),
Gzip(ocidir::LayerWriter<'a, flate2::write::GzEncoder<BlobWriter<'a>>>),
}
impl<'a> Write for LayerWriter<'a> {
fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
match self {
LayerWriter::Uncompressed(w) => w.write(buf),
LayerWriter::Gzip(w) => w.write(buf),
}
}
fn flush(&mut self) -> std::io::Result<()> {
match self {
LayerWriter::Uncompressed(w) => w.flush(),
LayerWriter::Gzip(w) => w.flush(),
}
}
}
impl<'a> LayerWriter<'a> {
/// Complete the layer and return the layer.
pub fn complete(self) -> Result<ocidir::Layer> {
match self {
LayerWriter::Uncompressed(w) => w.complete().context("completing uncompressed layer"),
LayerWriter::Gzip(w) => w.complete().context("completing gzip layer"),
}
}
}
/// Create a tar builder for a new layer in an OCI directory.
pub fn create_layer(
oci_dir: &ocidir::OciDir,
compression: crate::ocibuilder::Compression,
) -> Result<tar::Builder<LayerWriter<'_>>> {
let layer_writer = match compression {
crate::ocibuilder::Compression::None => {
let layer_writer = oci_dir
.create_custom_layer(|bw| Ok(NoCompression(bw)), oci_image::MediaType::ImageLayer)
.context("creating uncompressed layer writer")?;
LayerWriter::Uncompressed(layer_writer)
}
crate::ocibuilder::Compression::Gzip(level) => {
let level = flate2::Compression::new(level);
let layer_writer = oci_dir
.create_custom_layer(
|bw| Ok(flate2::write::GzEncoder::new(bw, level)),
oci_image::MediaType::ImageLayerGzip,
)
.context("creating gzip layer writer")?;
LayerWriter::Gzip(layer_writer)
}
};
Ok(tar::Builder::new(layer_writer))
}
/// Build a tar layer from a list of files and return the completed layer.
///
/// Parent directories are automatically created as needed using metadata from
/// the files map. This uses a stack-based approach that leverages the sorted order
/// of the input BTreeMap for efficiency.
pub fn write_files_to_tar<W: Write>(
tar_builder: &mut tar::Builder<W>,
rootfs: &cap_std::fs::Dir,
files: &FileMap,
mtime_clamp: u64,
) -> Result<()> {
// Stack of written directory paths - leverages sorted iteration order
let mut dir_stack: Vec<&Utf8Path> = Vec::new();
// Track inode -> first path written for hardlink detection.
let mut inode_to_path: HashMap<u64, Utf8PathBuf> = HashMap::new();
for (path, file_info) in files {
// Pop directories that are not ancestors of current path
while let Some(top) = dir_stack.last() {
if path.starts_with(top) && path.as_path() != *top {
break;
}
dir_stack.pop();
}
// Collect ancestors that need to be written (between stack top and
// current path's parent)
let ancestors: Vec<_> = path
.ancestors()
.skip(1) // skip self
.filter(|p| !p.as_str().is_empty() && *p != "/")
.take_while(|p| dir_stack.last().is_none_or(|top| *p != *top))
.collect();
// Write ancestors in reverse order (shallowest first) and push to stack
for ancestor in ancestors.into_iter().rev() {
let ancestor_path = Utf8PathBuf::from(ancestor);
// XXX: somehow reuse existing FileInfos for that dir, which may
// live in other components
let ancestor_info = if let Some(info) = files.get(&ancestor_path) {
info.clone()
} else {
let rel_path = ancestor.strip_prefix("/").unwrap_or(ancestor);
let metadata = rootfs
.symlink_metadata(rel_path)
.with_context(|| format!("getting metadata for {}", ancestor))?;
let xattrs = crate::scan::read_xattrs(rootfs, rel_path.as_str())
.with_context(|| format!("reading xattrs for {}", ancestor))?;
FileInfo::from_metadata(&metadata, xattrs)
.with_context(|| format!("processing metadata for {}", ancestor))?
};
write_dir_entry(tar_builder, ancestor, mtime_clamp, &ancestor_info)
.with_context(|| format!("writing parent directory {}", ancestor))?;
dir_stack.push(ancestor);
}
match file_info.file_type {
FileType::Directory => {
write_dir_entry(tar_builder, path, mtime_clamp, file_info)?;
// We might enter this directory in the next iteration; push it
dir_stack.push(path.as_path());
}
FileType::File => {
// Handle hardlinks
if file_info.nlink > 1 {
if let Some(first_path) = inode_to_path.get(&file_info.ino) {
write_hardlink_entry(
tar_builder,
path,
first_path,
mtime_clamp,
file_info,
)?;
continue;
}
// First occurrence of this hardlinked file
inode_to_path.insert(file_info.ino, path.clone());
}
write_file_entry(tar_builder, rootfs, path, mtime_clamp, file_info)?;
}
FileType::Symlink => {
write_symlink_entry(tar_builder, rootfs, path, mtime_clamp, file_info)?;
}
}
}
Ok(())
}
/// Write the OCI directory as a tar archive to a writer.
// XXX: Consider upstreaming this to ocidir-rs.
pub fn write_oci_archive<W: Write>(
oci_dir: &cap_std::fs::Dir,
writer: W,
compression: ArchiveCompression,
) -> Result<()> {
match compression {
ArchiveCompression::None => write_oci_archive_to(oci_dir, writer),
ArchiveCompression::Gzip(level) => {
let gzip_writer = flate2::write::GzEncoder::new(writer, level);
write_oci_archive_to(oci_dir, gzip_writer)
}
}
}
/// Strip leading "/" from a path, returning the path unchanged if no prefix.
fn strip_root_prefix(path: &Utf8Path) -> &Utf8Path {
path.strip_prefix("/").unwrap_or(path)
}
/// Prepare a tar header with common metadata from FileInfo.
fn write_header_from_file_info(header: &mut tar::Header, file_info: &FileInfo, mtime_clamp: u64) {
let mtime = std::cmp::min(file_info.mtime, mtime_clamp);
header.set_mtime(mtime);
header.set_uid(file_info.uid as u64);
header.set_gid(file_info.gid as u64);
header.set_mode(file_info.mode);
}
/// Append xattrs as PAX extensions to the tar stream.
///
/// This must be called before appending the actual file entry.
/// Uses the SCHILY.xattr.{key} format that tools like tar understand.
fn append_xattrs<W: Write>(
tar_builder: &mut tar::Builder<W>,
xattrs: &[(String, Vec<u8>)],
path: &str,
) -> Result<()> {
if xattrs.is_empty() {
return Ok(());
}
let pax_extensions: Vec<_> = xattrs
.iter()
.map(|(k, v)| (format!("SCHILY.xattr.{k}"), v.clone()))
.collect();
tar_builder
.append_pax_extensions(
pax_extensions
.iter()
.map(|(k, v)| (k.as_str(), v.as_slice())),
)
.with_context(|| format!("appending xattrs for {}", path))?;
Ok(())
}
/// Write a directory entry to the tar archive.
fn write_dir_entry<W: Write>(
tar_builder: &mut tar::Builder<W>,
path: &Utf8Path,
mtime_clamp: u64,
file_info: &FileInfo,
) -> Result<()> {
let rel_path = strip_root_prefix(path);
let mut header = tar::Header::new_gnu();
header.set_entry_type(tar::EntryType::Directory);
header.set_size(0);
write_header_from_file_info(&mut header, file_info, mtime_clamp);
append_xattrs(tar_builder, &file_info.xattrs, path.as_str())
.with_context(|| format!("appending xattrs for {}", path))?;
let tar_dir_path = if rel_path.as_str().is_empty() {
"./".to_string()
} else {
format!("{}/", rel_path)
};
tar_builder
.append_data(&mut header, &tar_dir_path, std::io::empty())
.with_context(|| format!("appending directory {}", path))?;
Ok(())
}
/// Write a hardlink entry to the tar archive.
fn write_hardlink_entry<W: Write>(
tar_builder: &mut tar::Builder<W>,
path: &Utf8Path,
link_target: &Utf8Path,
mtime_clamp: u64,
file_info: &FileInfo,
) -> Result<()> {
let rel_path = strip_root_prefix(path);
let rel_target = strip_root_prefix(link_target);
let mut header = tar::Header::new_gnu();
header.set_entry_type(tar::EntryType::Link);
header.set_size(0);
write_header_from_file_info(&mut header, file_info, mtime_clamp);
// Mask out file type bits; it's harmless but it matches what GNU tar
// and Python's tarfile do as well. Not doing this does though result in
// libarchive's strmode not showing the file as 'h' which shows up in diffs
// pre vs post-chunkah.
header.set_mode(file_info.mode & 0o7777);
tar_builder
.append_link(&mut header, rel_path.as_str(), rel_target.as_str())
.with_context(|| format!("appending hardlink {} -> {}", path, link_target))?;
Ok(())
}
/// Write a regular file entry to the tar archive.
fn write_file_entry<W: Write>(
tar_builder: &mut tar::Builder<W>,
rootfs: &cap_std::fs::Dir,
path: &Utf8Path,
mtime_clamp: u64,
file_info: &FileInfo,
) -> Result<()> {
let rel_path = strip_root_prefix(path);
let content = rootfs
.read(rel_path)
.with_context(|| format!("reading {}", path))?;
let mut header = tar::Header::new_gnu();
header.set_entry_type(tar::EntryType::Regular);
header.set_size(content.len() as u64);
write_header_from_file_info(&mut header, file_info, mtime_clamp);
append_xattrs(tar_builder, &file_info.xattrs, path.as_str())
.with_context(|| format!("appending xattrs for {}", path))?;
tar_builder
.append_data(&mut header, rel_path.as_str(), content.as_slice())
.with_context(|| format!("appending file {}", path))?;
Ok(())
}
/// Write a symlink entry to the tar archive.
fn write_symlink_entry<W: Write>(
tar_builder: &mut tar::Builder<W>,
rootfs: &cap_std::fs::Dir,
path: &Utf8Path,
mtime_clamp: u64,
file_info: &FileInfo,
) -> Result<()> {
let rel_path = strip_root_prefix(path);
let target = rootfs
.read_link_contents(rel_path)
.with_context(|| format!("reading symlink {}", path))?;
let mut header = tar::Header::new_gnu();
header.set_entry_type(tar::EntryType::Symlink);
header.set_size(0);
write_header_from_file_info(&mut header, file_info, mtime_clamp);
append_xattrs(tar_builder, &file_info.xattrs, path.as_str())
.with_context(|| format!("appending xattrs for {}", path))?;
tar_builder
.append_link(&mut header, rel_path.as_str(), target)
.with_context(|| format!("appending symlink {}", path))?;
Ok(())
}
fn write_oci_archive_to<W: Write>(oci_dir: &cap_std::fs::Dir, writer: W) -> Result<()> {
use cap_std_ext::dirext::CapStdExtDirExt;
use std::ops::ControlFlow;
// Template headers for directories and files - cloned and modified as needed
let dir_header = {
let mut header = tar::Header::new_gnu();
header.set_entry_type(tar::EntryType::Directory);
header.set_size(0);
header.set_mode(0o755);
header.set_mtime(0);
header.set_uid(0);
header.set_gid(0);
header
};
let file_header = {
let mut header = tar::Header::new_gnu();
header.set_entry_type(tar::EntryType::Regular);
header.set_mode(0o644);
header.set_mtime(0);
header.set_uid(0);
header.set_gid(0);
header
};
let mut tar = tar::Builder::new(writer);
let config = cap_std_ext::dirext::WalkConfiguration::default().sort_by_file_name();
oci_dir
.walk(&config, |component| {
let path = component.path;
if component.file_type.is_dir() {
let mut header = dir_header.clone();
// Tar directories need a trailing slash
let path_str = format!("{}/", path.display());
tar.append_data(&mut header, &path_str, std::io::empty())
.with_context(|| format!("appending directory {}", path.display()))?;
} else if component.file_type.is_file() {
let content = component
.dir
.read(component.filename)
.with_context(|| format!("reading {}", path.display()))?;
let mut header = file_header.clone();
header.set_size(content.len() as u64);
tar.append_data(&mut header, path, content.as_slice())
.with_context(|| format!("appending {}", path.display()))?;
} else {
anyhow::bail!("unsupported file type for {}", path.display());
}
Ok::<_, anyhow::Error>(ControlFlow::Continue(()))
})
.context("walking OCI directory")?;
tar.finish().context("finishing tar archive")?;
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use cap_std::ambient_authority;
use cap_std::fs::Dir;
use cap_std_ext::dirext::CapStdExtDirExt;
/// Helper to create a rootfs in a tempdir, run setup, write files to tar, and return raw bytes.
/// Optionally accepts a closure to modify the scanned files before writing to tar.
fn write_tar_bytes<F, M>(setup: F, modify_files: Option<M>, mtime_clamp: u64) -> Vec<u8>
where
F: FnOnce(&Dir),
M: FnOnce(&mut FileMap),
{
let tmp = tempfile::tempdir().unwrap();
let rootfs = Dir::open_ambient_dir(tmp.path(), ambient_authority()).unwrap();
setup(&rootfs);
let mut files = crate::scan::scan_rootfs(&rootfs).unwrap();
if let Some(modify) = modify_files {
modify(&mut files);
}
let mut output = Vec::new();
{
let mut tar_builder = tar::Builder::new(&mut output);
write_files_to_tar(&mut tar_builder, &rootfs, &files, mtime_clamp).unwrap();
tar_builder.finish().unwrap();
}
output
}
/// Helper to create a minimal OCI directory structure.
fn create_minimal_oci_dir() -> (tempfile::TempDir, Dir) {
let tmp = tempfile::tempdir().unwrap();
let oci_dir = Dir::open_ambient_dir(tmp.path(), ambient_authority()).unwrap();
oci_dir
.write("oci-layout", r#"{"imageLayoutVersion":"1.0.0"}"#)
.unwrap();
oci_dir.create_dir("blobs").unwrap();
(tmp, oci_dir)
}
#[test]
fn test_write_files_to_tar_preserves_xattrs() {
let output = write_tar_bytes(
|rootfs| {
rootfs.write("file", "content").unwrap();
rootfs
.setxattr("file", "user.testattr", b"testvalue")
.unwrap();
},
None::<fn(&mut FileMap)>,
1000,
);
// Extract using tar command (which properly restores xattrs)
let extract_dir = tempfile::tempdir().unwrap();
let mut child = std::process::Command::new("tar")
.args(["xf", "-", "--xattrs"])
.current_dir(extract_dir.path())
.stdin(std::process::Stdio::piped())
.spawn()
.unwrap();
child.stdin.take().unwrap().write_all(&output).unwrap();
let status = child.wait().unwrap();
assert!(status.success(), "tar extraction failed");
let extracted = Dir::open_ambient_dir(extract_dir.path(), ambient_authority()).unwrap();
let xattr_value = extracted.getxattr("file", "user.testattr").unwrap();
assert_eq!(
xattr_value.as_deref(),
Some(b"testvalue".as_slice()),
"file xattr should be preserved after extraction"
);
}
#[test]
fn test_write_files_to_tar_symlink() {
let output = write_tar_bytes(
|rootfs| {
rootfs.write("target", "content").unwrap();
rootfs.symlink("target", "link").unwrap();
},
None::<fn(&mut FileMap)>,
1000,
);
let mut archive = tar::Archive::new(output.as_slice());
let mut found_link = false;
for entry in archive.entries().unwrap() {
let entry = entry.unwrap();
if entry.header().entry_type() == tar::EntryType::Symlink {
let link_name = entry.header().link_name().unwrap().unwrap();
assert_eq!(link_name.to_string_lossy(), "target");
found_link = true;
}
}
assert!(found_link, "symlink should be in tar");
}
#[test]
fn test_write_files_to_tar_creates_parent_dirs() {
// Parent directories not in files are created via symlink_metadata() fallback
let output = write_tar_bytes(
|rootfs| {
rootfs.create_dir_all("a/b/c").unwrap();
rootfs.write("a/b/c/file", "content").unwrap();
},
Some(|files: &mut FileMap| {
// Remove parent dirs to test symlink_metadata() fallback
files.remove(&Utf8PathBuf::from("/a"));
files.remove(&Utf8PathBuf::from("/a/b"));
}),
1000,
);
let mut archive = tar::Archive::new(output.as_slice());
let paths: Vec<_> = archive
.entries()
.unwrap()
.map(|e| e.unwrap().path().unwrap().to_string_lossy().to_string())
.collect();
// Should have all directories and file
assert!(paths.contains(&"a/".to_string()), "missing a/: {:?}", paths);
assert!(
paths.contains(&"a/b/".to_string()),
"missing a/b/: {:?}",
paths
);
assert!(
paths.contains(&"a/b/c/".to_string()),
"missing a/b/c/: {:?}",
paths
);
assert!(
paths.contains(&"a/b/c/file".to_string()),
"missing a/b/c/file: {:?}",
paths
);
// Directories should come before the file (sorted order)
let a_pos = paths.iter().position(|p| p == "a/").unwrap();
let ab_pos = paths.iter().position(|p| p == "a/b/").unwrap();
let abc_pos = paths.iter().position(|p| p == "a/b/c/").unwrap();
let file_pos = paths.iter().position(|p| p == "a/b/c/file").unwrap();
assert!(a_pos < ab_pos, "a/ should come before a/b/");
assert!(ab_pos < abc_pos, "a/b/ should come before a/b/c/");
assert!(abc_pos < file_pos, "a/b/c/ should come before a/b/c/file");
}
#[test]
fn test_write_oci_archive_uncompressed() {
let (_tmp, oci_dir) = create_minimal_oci_dir();
let mut output = Vec::new();
write_oci_archive(&oci_dir, &mut output, ArchiveCompression::None).unwrap();
// Verify it's a valid tar
let mut archive = tar::Archive::new(output.as_slice());
let entries: Vec<_> = archive.entries().unwrap().map(|e| e.unwrap()).collect();
assert!(!entries.is_empty());
}
#[test]
fn test_write_oci_archive_gzip() {
let (_tmp, oci_dir) = create_minimal_oci_dir();
let mut output = Vec::new();
write_oci_archive(
&oci_dir,
&mut output,
ArchiveCompression::Gzip(flate2::Compression::fast()),
)
.unwrap();
// Verify it's gzip compressed (magic bytes)
assert_eq!(output[0], 0x1f);
assert_eq!(output[1], 0x8b);
// Decompress and verify it's a valid tar
let decoder = flate2::read::GzDecoder::new(output.as_slice());
let mut archive = tar::Archive::new(decoder);
let entries: Vec<_> = archive.entries().unwrap().map(|e| e.unwrap()).collect();
assert!(!entries.is_empty());
}
#[test]
fn test_write_files_to_tar_hardlinks() {
let tmp = tempfile::tempdir().unwrap();
let rootfs = Dir::open_ambient_dir(tmp.path(), ambient_authority()).unwrap();
// Create original file and a hardlink to it
rootfs.write("original", "content").unwrap();
std::fs::hard_link(tmp.path().join("original"), tmp.path().join("hardlink")).unwrap();
let files = crate::scan::scan_rootfs(&rootfs).unwrap();
// Verify the scan picked up the hardlink metadata
let original_info = files.get(&Utf8PathBuf::from("/original")).unwrap();
let hardlink_info = files.get(&Utf8PathBuf::from("/hardlink")).unwrap();
assert_eq!(original_info.ino, hardlink_info.ino, "inodes should match");
assert!(original_info.nlink > 1, "nlink should be > 1");
assert!(hardlink_info.nlink > 1, "nlink should be > 1");
let mut output = Vec::new();
{
let mut tar_builder = tar::Builder::new(&mut output);
write_files_to_tar(&mut tar_builder, &rootfs, &files, 1000).unwrap();
tar_builder.finish().unwrap();
}
// Parse the tar and verify entries
let mut archive = tar::Archive::new(output.as_slice());
for entry in archive.entries().unwrap() {
let entry = entry.unwrap();
let path = entry.path().unwrap().to_string_lossy().to_string();
match entry.header().entry_type() {
tar::EntryType::Regular => assert_eq!(path, "hardlink"),
tar::EntryType::Link => {
assert_eq!(path, "original");
let hardlink_target = entry
.header()
.link_name()
.unwrap()
.unwrap()
.to_string_lossy()
.to_string();
assert_eq!(hardlink_target, "hardlink");
}
_ => {}
}
}
// Sanity-check they extract as hardlinks
let extract_dir = tempfile::tempdir().unwrap();
let mut child = std::process::Command::new("tar")
.args(["xf", "-"])
.current_dir(extract_dir.path())
.stdin(std::process::Stdio::piped())
.spawn()
.unwrap();
child.stdin.take().unwrap().write_all(&output).unwrap();
let status = child.wait().unwrap();
assert!(status.success(), "tar extraction failed");
use std::os::unix::fs::MetadataExt;
let original_meta = std::fs::metadata(extract_dir.path().join("original")).unwrap();
let hardlink_meta = std::fs::metadata(extract_dir.path().join("hardlink")).unwrap();
assert_eq!(
original_meta.ino(),
hardlink_meta.ino(),
"extracted files should have same inode"
);
}
}

37
src/utils.rs Normal file
View file

@ -0,0 +1,37 @@
use std::time::{SystemTime, UNIX_EPOCH};
use anyhow::{Context, Result};
pub fn get_current_epoch() -> Result<u64> {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.context("system time is before UNIX epoch")
.map(|d| d.as_secs())
}
/// Returns the OCI/Go architecture string.
///
/// If `arch` is provided, translates it to OCI format.
/// Otherwise, uses the current system architecture.
pub fn get_goarch(arch: Option<&str>) -> &str {
match arch.unwrap_or(std::env::consts::ARCH) {
"x86_64" => "amd64",
"aarch64" => "arm64",
"powerpc64" => "ppc64le",
arch => arch,
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_get_goarch() {
assert_eq!(get_goarch(Some("x86_64")), "amd64");
assert_eq!(get_goarch(Some("aarch64")), "arm64");
assert_eq!(get_goarch(Some("powerpc64")), "ppc64le");
assert_eq!(get_goarch(Some("amd64")), "amd64"); // passthrough
assert_eq!(get_goarch(Some("unknown")), "unknown"); // passthrough
}
}

120
tests/e2e/run.sh Executable file
View file

@ -0,0 +1,120 @@
#!/bin/bash
set -euo pipefail
shopt -s inherit_errexit
usage() {
cat <<EOF
Usage: $(basename "$0") [OPTIONS] [TEST...]
Run end-to-end tests for chunkah.
Options:
-v, --verbose Show test output in real-time (default: only on failure)
-h, --help Show this help message
Arguments:
TEST Test name(s) to run (without 'test-' prefix and '.sh' suffix).
If none specified, all test-*.sh files are run.
Environment:
CHUNKAH_IMG Container image to test (default: localhost/chunkah:latest)
Examples:
$(basename "$0") # Run all tests
$(basename "$0") fedora-build # Run only test-fedora-build.sh
$(basename "$0") -v fedora-build # Run with verbose output
EOF
}
CHUNKAH_IMG=${CHUNKAH_IMG:-localhost/chunkah:latest}
export CHUNKAH_IMG
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
cd "${SCRIPT_DIR}"
ONLY_TESTS=()
VERBOSE=false
while [[ $# -gt 0 ]]; do
case $1 in
--help|-h)
usage
exit 0
;;
--verbose|-v)
VERBOSE=true
shift
;;
-*)
echo "Error: Unknown option: $1" >&2
echo "Try '$(basename "$0") --help' for more information." >&2
exit 1
;;
*)
ONLY_TESTS+=("$1")
shift
;;
esac
done
# Find and run test scripts
if [[ ${#ONLY_TESTS[@]} -gt 0 ]]; then
tests=()
for name in "${ONLY_TESTS[@]}"; do
testfile="test-${name}.sh"
if [[ ! -f "${testfile}" ]]; then
echo "Error: Test not found: ${testfile}" >&2
exit 1
fi
tests+=("${testfile}")
done
else
tests=( test-*.sh )
fi
total=${#tests[@]}
passed=0
failed=0
failed_tests=()
echo ""
echo "Running ${total} e2e test(s)..."
echo ""
for test in "${tests[@]}"; do
echo "=== Running ${test} ==="
TMPDIR=$(mktemp -d)
export TMPDIR
abstest=$(realpath "${test}")
if [[ ${VERBOSE} == true ]]; then exec 3>&2; else exec 3>"${TMPDIR}/.out"; fi
if (cd "${TMPDIR}" && bash "${abstest}") >&3 2>&3; then
echo "=== PASSED: ${test} ==="
passed=$((passed+1))
else
echo "=== FAILED: ${test} ==="
failed=$((failed+1))
failed_tests+=("${test}")
if [[ ${VERBOSE} == false ]]; then
echo "--- Test output ---"
cat "${TMPDIR}/.out"
echo "--- End output ---"
fi
fi
exec 3>&-
rm -rf "${TMPDIR}"
unset TMPDIR
echo ""
done
echo "========================================"
echo "Results: ${passed} passed, ${failed} failed"
echo "========================================"
if [[ "${failed}" -gt 0 ]]; then
echo "Failed tests:"
for t in "${failed_tests[@]}"; do
echo " - ${t}"
done
exit 1
fi

46
tests/e2e/test-fedora-build.sh Executable file
View file

@ -0,0 +1,46 @@
#!/bin/bash
set -xeuo pipefail
shopt -s inherit_errexit
TARGET_IMAGE="quay.io/fedora/fedora-minimal:latest"
CHUNKED_IMAGE="localhost/fedora-minimal-chunked:test"
# Find repo root from script location
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
cleanup() {
podman rmi -f "${CHUNKED_IMAGE}" 2>/dev/null || true
}
trap cleanup EXIT
tmp_args=()
version=$(${BUILDAH:-buildah} version --json | jq -r '.version')
min_version=$(echo -e "${version}\n1.43" | sort -V | head -n1)
if [[ "${min_version}" != "1.43" ]]; then
tmp_args+=(-v "${PWD}:/run/src" --security-opt=label=disable)
fi
# build split image using Containerfile.splitter API
podman pull "${TARGET_IMAGE}"
config_str=$(podman inspect "${TARGET_IMAGE}")
${BUILDAH:-buildah} build --skip-unused-stages=false \
--from "${TARGET_IMAGE}" --build-arg CHUNKAH="${CHUNKAH_IMG:?}" \
--build-arg CHUNKAH_CONFIG_STR="${config_str}" \
-t "${CHUNKED_IMAGE}" "${tmp_args[@]}" "${REPO_ROOT}/Containerfile.splitter"
# sanity-check it
podman run --rm "${CHUNKED_IMAGE}" cat /etc/os-release | grep Fedora
# use skopeo to inspect the ociarchive and check for layer annotations
layer_annotations=$(skopeo inspect "containers-storage:${CHUNKED_IMAGE}" | \
jq -r '.LayersData[].Annotations["org.chunkah.component"] // empty')
# check for some expected components
grep -q "rpm/filesystem" <<< "${layer_annotations}"
grep -q "rpm/setup" <<< "${layer_annotations}"
grep -q "rpm/glibc" <<< "${layer_annotations}"
# sanity-check we got at least 16 components
n=$(wc -l <<< "${layer_annotations}")
[[ ${n} -gt 16 ]]

50
tests/e2e/test-fedora-run.sh Executable file
View file

@ -0,0 +1,50 @@
#!/bin/bash
set -xeuo pipefail
shopt -s inherit_errexit
SOURCE_IMAGE="localhost/fedora:test"
CHUNKED_IMAGE="localhost/fedora-chunked:test"
cleanup() {
podman rmi -f "${SOURCE_IMAGE}" 2>/dev/null || true
podman rmi -f "${CHUNKED_IMAGE}" 2>/dev/null || true
}
trap cleanup EXIT
# build a derived image so we can test file cap handling
podman build -t "${SOURCE_IMAGE}" -f - <<'EOF'
FROM quay.io/fedora/fedora:latest
# create a test binary and set a capability on it
RUN cp /usr/bin/true /usr/bin/test-caps && setcap cap_net_raw+ep /usr/bin/test-caps
EOF
CHUNKAH_CONFIG_STR=$(podman inspect "${SOURCE_IMAGE}")
# run chunkah!
podman run --rm --mount=type=image,src="${SOURCE_IMAGE}",target=/chunkah \
-e CHUNKAH_CONFIG_STR="${CHUNKAH_CONFIG_STR}" \
"${CHUNKAH_IMG:?}" build > out.ociarchive
# XXX: need to fix 'podman load' to only print image ID on its stdout, like 'podman pull'
iid=$(podman load -i out.ociarchive)
iid=${iid#*sha256:}
podman tag "${iid}" "${CHUNKED_IMAGE}"
# sanity-check it
podman run --rm "${CHUNKED_IMAGE}" cat /etc/os-release | grep Fedora
# use skopeo to inspect the ociarchive and check for layer annotations
layer_annotations=$(skopeo inspect "containers-storage:${CHUNKED_IMAGE}" | \
jq -r '.LayersData[].Annotations["org.chunkah.component"] // empty')
# check for some expected components
grep -q "rpm/filesystem" <<< "${layer_annotations}"
grep -q "rpm/setup" <<< "${layer_annotations}"
grep -q "rpm/glibc" <<< "${layer_annotations}"
# sanity-check we got at least 16 layers
n=$(skopeo inspect "containers-storage:${CHUNKED_IMAGE}" | jq '.LayersData | length')
[[ ${n} -ge 16 ]]
# verify that security.capability xattrs are preserved
caps=$(podman run --rm "${CHUNKED_IMAGE}" getcap /usr/bin/test-caps)
[[ "${caps}" == *"cap_net_raw=ep"* ]]

11
tests/fixtures/empty.Containerfile vendored Normal file
View file

@ -0,0 +1,11 @@
FROM scratch
USER myuser
EXPOSE 8888
ENV MYENV1=MYVAL1
ENV MYENV2=MYVAL2
ENTRYPOINT ["my", "entrypoint"]
CMD ["my", "cmd"]
VOLUME /myvolume
WORKDIR /
LABEL mylabel=myval
STOPSIGNAL SIGQUIT

27
tests/fixtures/empty.image-config.json vendored Normal file
View file

@ -0,0 +1,27 @@
{
"User": "myuser",
"ExposedPorts": {
"8888/tcp": {}
},
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"MYENV1=MYVAL1",
"MYENV2=MYVAL2"
],
"Entrypoint": [
"my",
"entrypoint"
],
"Cmd": [
"my",
"cmd"
],
"Volumes": {
"/myvolume": {}
},
"WorkingDir": "/",
"Labels": {
"mylabel": "myval"
},
"StopSignal": "SIGQUIT"
}

38308
tests/fixtures/fedora.json vendored Normal file
View file

File diff suppressed because one or more lines are too long

13
tests/fixtures/minimal.Containerfile vendored Normal file
View file

@ -0,0 +1,13 @@
# This isn't used by any unit tests yet, but I've used the resulting OCI archive
# a few times to debug basic OCI parsing/chunking stuff.
ARG ADD_PACKAGES=""
FROM quay.io/fedora/fedora-minimal AS builder
RUN dnf install -y \
--installroot /target \
--use-host-config \
--nodocs \
--setopt=install_weak_deps=False \
filesystem setup $ADD_PACKAGES && rm -rf /var/cache/*
FROM scratch
COPY --from=builder /target /

41
tests/fixtures/regenerate.sh vendored Executable file
View file

@ -0,0 +1,41 @@
#!/bin/bash
set -euo pipefail
shopt -s inherit_errexit
cd "$(dirname "$0")"
echo ">>> REGENERATING: fedora.json" >&2
# Packages to cherry-pick from the full rpm -qa output
PACKAGES=(
bash
glibc
coreutils
rpm
shadow-utils
util-linux-core
setup
perl-POSIX
)
# Build jq filter to select packages by name
filter=$(printf '"%s",' "${PACKAGES[@]}")
filter="[${filter%,}]"
podman run --rm quay.io/fedora/fedora-minimal:latest rpm -qa --json | \
jq -s --argjson names "${filter}" '
[.[] | select(.Name as $n | $names | index($n))] | sort_by(.Name) | .[]
' > fedora.json
echo ">>> REGENERATING: empty.image-config.json" >&2
buildah build --omit-history -f empty.Containerfile -t chunkah-empty
podman inspect chunkah-empty | jq '.[0].Config' > empty.image-config.json
podman rmi chunkah-empty
echo ">>> REGENERATING rpmdb.sqlite" >&2
podman rm -f chunkah-test-fixture-tmp
podman run --name chunkah-test-fixture-tmp --rm quay.io/hummingbird-ci/builder bash -c '
dnf install --installroot /mnt -y --use-host-config --nodocs --setopt=install_weak_deps=False filesystem setup &>2
sqlite3 /mnt/usr/lib/sysimage/rpm/rpmdb.sqlite "PRAGMA journal_mode = DELETE;" &>2
cat /mnt/usr/lib/sysimage/rpm/rpmdb.sqlite
' > rpmdb.sqlite

BIN
tests/fixtures/rpmdb.sqlite vendored Normal file
View file

Binary file not shown.

240
tools/split-and-diff.sh Executable file
View file

@ -0,0 +1,240 @@
#!/bin/bash
set -euo pipefail
shopt -s inherit_errexit
# Verify an image split through chunkah has identical data/metadata except for
# mtimes by comparing rootfs tarballs with diffoscope.
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
usage() {
cat <<EOF
Usage: $(basename "$0") [OPTIONS] <source-image> <output-file>
Split an image through chunkah and compare the result with diffoscope.
Options:
--raw Compare raw layer tarballs without flattening (requires
single-layer images)
--no-filter-mtime Don't filter out mtime differences from output
-h, --help Show this help message
Arguments:
source-image Source image to split (e.g. docker://quay.io/fedora/fedora-minimal)
output-file Path to write diffoscope output (filtered for mtime)
Environment:
CHUNKAH_IMG Container image to use for splitting
(default: quay.io/jlebon/chunkah:latest)
Exit codes:
0 No differences found (output file is empty)
1 Differences found (output file contains diff)
Examples:
$(basename "$0") docker://quay.io/fedora/fedora-minimal /tmp/diff.txt
$(basename "$0") --raw oci-archive:image.ociarchive /tmp/diff.txt
CHUNKAH_IMG=localhost/chunkah:latest $(basename "$0") docker://... /tmp/out.txt
EOF
}
# Parse options
raw_mode=false
filter_mtime=true
while [[ $# -gt 0 ]]; do
case "${1}" in
--raw)
raw_mode=true
shift
;;
--no-filter-mtime)
filter_mtime=false
shift
;;
-h|--help)
usage
exit 0
;;
-*)
echo "Error: Unknown option: ${1}" >&2
echo "Try '$(basename "$0") --help' for more information." >&2
exit 1
;;
*)
break
;;
esac
done
# Validate arguments
if [[ $# -lt 2 ]]; then
echo "Error: Missing required arguments" >&2
echo "Try '$(basename "$0") --help' for more information." >&2
exit 1
fi
source_image="${1}"
output_file="${2}"
# Check required tools
for tool in skopeo podman buildah diffoscope jq; do
if ! command -v "${tool}" &> /dev/null; then
echo "Error: Required tool '${tool}' not found" >&2
exit 1
fi
done
# Generate unique run ID for image names to allow concurrent runs
run_id="chunkah-diff-$$-$(date +%s)"
# Setup cleanup trap with safe handling of unset variables
tmpdir=""
original_image=""
split_image=""
# shellcheck disable=SC2329 # invoked via trap
cleanup() {
if [[ -n "${tmpdir}" ]]; then
chmod -R u+w "${tmpdir}"
rm -rf "${tmpdir}"
fi
if [[ -n "${original_image}" ]]; then podman rmi -f "${original_image}" 2>/dev/null || true; fi
if [[ -n "${split_image}" ]]; then podman rmi -f "${split_image}" 2>/dev/null || true; fi
}
trap cleanup EXIT
tmpdir=$(mktemp -d)
# Extract rootfs tarball from an OCI archive by flattening all layers
extract_rootfs_tarball() {
local ociarchive="${1}"
local output_tar="${2}"
local workdir
workdir=$(mktemp -d)
mkdir -p "${workdir}/oci" "${workdir}/rootfs"
tar -xf "${ociarchive}" -C "${workdir}/oci"
local manifest_digest
manifest_digest=$(jq -r '.manifests[0].digest | split(":")[1]' "${workdir}/oci/index.json")
# Extract each layer in order (auto-detect compression)
jq -r '.layers[].digest | split(":")[1]' "${workdir}/oci/blobs/sha256/${manifest_digest}" | \
while read -r layer_digest; do
local layer_blob="${workdir}/oci/blobs/sha256/${layer_digest}"
# Use file magic to detect compression, fallback to gzip
case "$(file -b "${layer_blob}")" in
*gzip*) tar -xzf "${layer_blob}" -C "${workdir}/rootfs" ;;
*zstd*) tar -I zstd -xf "${layer_blob}" -C "${workdir}/rootfs" ;;
*POSIX\ tar*) tar -xf "${layer_blob}" -C "${workdir}/rootfs" ;;
*) tar -xzf "${layer_blob}" -C "${workdir}/rootfs" ;;
esac
done
# Fix files with permissions 000 so they can be read for comparison
find "${workdir}/rootfs" -type f -perm 000 -exec chmod 644 {} +
# Create deterministic tarball (sorted entries)
tar -cf "${output_tar}" -C "${workdir}/rootfs" --sort=name .
chmod -R u+w "${workdir}"
rm -rf "${workdir}"
}
# Get raw layer tarball from an OCI archive (for --raw mode)
get_raw_layer_tarball() {
local ociarchive="${1}"
local output_tar="${2}"
local workdir
workdir=$(mktemp -d)
tar -xf "${ociarchive}" -C "${workdir}"
local manifest_digest
manifest_digest=$(jq -r '.manifests[0].digest | split(":")[1]' "${workdir}/index.json")
# Get the single layer (--raw mode expects single-layer images)
local layer_count
layer_count=$(jq '.layers | length' "${workdir}/blobs/sha256/${manifest_digest}")
if [[ "${layer_count}" -ne 1 ]]; then
echo "Error: --raw mode requires single-layer images, found ${layer_count} layers" >&2
rm -rf "${workdir}"
return 1
fi
local layer_digest
layer_digest=$(jq -r '.layers[0].digest | split(":")[1]' "${workdir}/blobs/sha256/${manifest_digest}")
# Copy the raw layer blob (may be compressed)
cp "${workdir}/blobs/sha256/${layer_digest}" "${output_tar}"
rm -rf "${workdir}"
}
# Prepare original image
echo "Preparing original image..."
original_image="${run_id}-original"
skopeo copy "${source_image}" "oci-archive:${tmpdir}/original.ociarchive"
skopeo copy "oci-archive:${tmpdir}/original.ociarchive" "containers-storage:${original_image}"
# Split image via Containerfile.splitter
echo "Splitting image through chunkah..."
config_str=$(podman inspect "${original_image}")
# Check buildah version for compatibility (< 1.43 needs extra args)
buildah_args=()
buildah_version=$(buildah version --json | jq -r '.version')
min_version=$(echo -e "${buildah_version}\n1.43" | sort -V | head -n1)
if [[ "${min_version}" != "1.43" ]]; then
buildah_args+=(-v "${tmpdir}:/run/src" --security-opt=label=disable)
fi
# Support custom chunkah image via CHUNKAH_IMG env var
chunkah_img="${CHUNKAH_IMG:-quay.io/jlebon/chunkah:latest}"
pushd "${tmpdir}" > /dev/null
buildah build --skip-unused-stages=false \
"${buildah_args[@]}" \
--from "containers-storage:${original_image}" \
--build-arg "CHUNKAH=${chunkah_img}" \
--build-arg CHUNKAH_ARGS="--max-layers=1" \
--build-arg "CHUNKAH_CONFIG_STR=${config_str}" \
"${REPO_ROOT}/Containerfile.splitter"
popd > /dev/null
split_image=$(podman load -i "${tmpdir}/out.ociarchive" | awk '{print $NF}' | sed 's/^sha256://')
skopeo copy "containers-storage:${split_image}" "oci-archive:${tmpdir}/split.ociarchive"
# Extract and compare
echo "Comparing images with diffoscope..."
if [[ "${raw_mode}" == "true" ]]; then
# Raw mode: compare unextracted layer tarballs directly
get_raw_layer_tarball "${tmpdir}/original.ociarchive" "${tmpdir}/original-layer.tar"
get_raw_layer_tarball "${tmpdir}/split.ociarchive" "${tmpdir}/split-layer.tar"
original_tar="${tmpdir}/original-layer.tar"
split_tar="${tmpdir}/split-layer.tar"
else
# Default: flatten layers to rootfs tarballs
extract_rootfs_tarball "${tmpdir}/original.ociarchive" "${tmpdir}/original-rootfs.tar"
extract_rootfs_tarball "${tmpdir}/split.ociarchive" "${tmpdir}/split-rootfs.tar"
original_tar="${tmpdir}/original-rootfs.tar"
split_tar="${tmpdir}/split-rootfs.tar"
fi
diffoscope "${original_tar}" "${split_tar}" \
> "${tmpdir}/diffoscope-raw.txt" 2>&1 || true
if [[ "${filter_mtime}" == "true" ]]; then
# Filter out mtime-related lines
grep -v -E '(mtime|Mtime|modification time)' "${tmpdir}/diffoscope-raw.txt" \
> "${output_file}" || true
else
cp "${tmpdir}/diffoscope-raw.txt" "${output_file}"
fi
# Exit 0 if no differences (empty output), 1 if differences found
if [[ -s "${output_file}" ]]; then
echo "Differences found (see ${output_file})"
exit 1
fi
echo "No differences found"
exit 0