These changes came out of a session with Opus to generalize this script
a bit more so it doesn't have the magic "auto-discovery based on tag"
behaviour anymore, but instead you just pass in all the images you want
in the order of "updates" directly on the CLI.
That makes it more intuitive to use and generally useful.
While we're here, simplify a bunch more things like dropping the
"compare originals" feature. That can just be a separate invocation
essentially.
Also support the fallback to image history for component info similar to
what we did for `inspect-layers.sh`.
And finally, rename `--show-components` to `--show-changed-components`
and add a `--show-unchanged-components`, which is also useful to know
sometimes. Sort those components by layer size.
Assisted-by: Claude Opus 4.6
Bump cap-std-ext from 4 to 5, rpm-qa from 0.1 to 0.2, and ocidir from
0.6 to 0.7. They all need to be bumped together because they all use
cap-std-ext.
No code changes were needed; all APIs remain compatible.
Assisted-by: Claude Opus 4.6
Noticed this while working on the previous commit. We were picking up
legacy XFS SGI_ACL_* xattr aliases for ACLs, which is just yucky. AFAICT
there's no use case for preserving some trusted xattrs, so just skip
them all for now until we find out otherwise.
Assisted-by: Claude Opus 4.6
Yet another source of reproducibility issues. If a file has multiple
xattrs, the order in which we read them in may change, but serializing
them to the tar layer needs to be deterministic.
Sort them at scan time.
Extend the buildtime e2e test to try to cover this by adding more
xattrs (including ACLs which we didn't have any coverage for before).
Though it's not easy to really test this without injecting our own
llistxattr(2).
Assisted-by: Claude Opus 4.6
When building with older buildah, we have to use the `-v $PWD:/run/src`
trick, but then we need to clean up the OCI archive. I don't like using
the `RUN` trick to remove the OCI archive for various reasons. But it's
trivial anyway to just clean it up ourselves.
This will be helpful for future contributors. I also wanted a place
to write down my thoughts on using AI coding agents responsibly and
respectfully.
Assisted-by: Claude Opus 4.6
We need a real filesystem to apply security labels because overlayfs
prevents that unprivileged.
As mentioned in the comment, we probably should rewrite that whole
script to !bash to do the pretty diff we're looking for by inspecting
the objects using ostree-native APIs.
Anyway, this works for now.
Add a convenience target that wraps the Containerfile.splitter workflow
from the README. It takes an image and optional chunkah arguments.
Assisted-by: Claude Opus 4.5
One thing sorely missing right now is any feedback of what chunkah is
doing.
This remedies the situation quite a bit by hooking up the tracing crate
and adding logging throughout the codebase. I might've went overboard
with the trace level, though it's really nice to be able to follow
exactly the data flow for any given path, from scanning, claiming,
packing, merging, and serializing to tar.
Change all our e2e tests to use the debug level.
Assisted-by: Claude Opus 4.5
Closes: https://github.com/coreos/chunkah/issues/17
Fix three sources of non-determinism that prevented chunkah from
producing reproducible OCI images across runs:
1. the way we calculated the clamp time for RPM components wasn't
accounting for subpackages of a same SRPM having different build times
2. when calculating stability, we were using the current time to define
the lookback period; pass through the default clamp time (which is
derived from `SOURCE_DATE_EPOCH` if defined) and use that instead
3. when packing, we weren't providing deterministic input to the
algorithm, so results may vary very slightly
4. when reconstructing merged components, we need to ensure their names
are deterministic as well
Add an e2e test that builds the same image twice with SOURCE_DATE_EPOCH
and verifies the OCI archives are byte-identical.
Closes#18
Assisted-by: Claude Opus 4.6
Set up Packit here and downstream so that everything happens
automagically when we cut a release upstream.
Move canonical spec file from downstream to upstream. Switch to
`%autochangelog`.
A principle of chunkah is that the resulting rootfs should be exactly
equivalent to the input rootfs, except for mtimes. So a huge gap of our
CI was its lack of any verification that this invariant holds.
This patch plugs that hole:
- First, a new tool is added for making it easier to diff two images.
This is as useful for humans/agents as it is for tests.
- Second, e2e tests which split existing images make use of this tool to
assert equivalence.
Assisted-by: Claude Opus 4.6
Add example usage in the docstrings for chunk-image-series.py and
analyze-layer-reuse.py showing how to use them to analyze FCOS images.
Assisted-by: Claude Opus 4.5
This allows us to actually evaluate changes to the packing algorithm in
a methodical way.
Sample usage for FCOS:
```
tools/chunk-image-series.py quay.io/fedora/fedora-coreos \
--tag-filter '*.*.3.*' --limit 5 --keep-originals \
--prefix fedora-coreos-test \
--chunkah-image quay.io/jlebon/chunkah:dev \
-- --compressed --prune /sysroot/
```
And then to do the actual analysis:
```
tools/analyze-layer-reuse.py localhost/fedora-coreos-test
--compare-originals
tools/analyze-layer-reuse.py localhost/fedora-coreos-test
--compare-originals --uncompressed
```
Planning also to analyze Hummingbird images, but images got reset there
recently so we lost a lot of a history to make this analysis practical.
Assisted-by: Claude Opus 4.5
We want to support images with no RPM database where users manually
bridge the package manager -> component gap using xattrs. But right
now this breaks because we always expect at least one component to have
non-zero stability, which is information only provided by the RPM repo
currently.
Ideally, users would be able to also signal stability somehow when using
the xattr model, but for now at least just default them to 0.5. The
important thing is that they not be 0.0 since it's not _guaranteed_ they
all change and so that there is still a real TEV loss to merging items.
Assisted-by: Claude Opus 4.5
When layer annotations are not available (e.g. because of
https://github.com/containers/buildah/issues/6652), fall back to
extracting component names from OCI history entries. This leverages the
history entries added in commit f09d54c which store the component name
in the comment field.
Assisted-by: Claude Opus 4.5
The FCOS stable image was updated and now has a slightly larger
unclaimed layer size (~55MB vs the previous threshold of 50MB). Bump the
threshold to 100MB to give more headroom. I'll be looking at the split
algorithm more closely soon as part of #16.
Assisted-by: Claude Opus 4.5
Add OCI history entries for each layer we emit. AIUI, history is
not mandated by the OCI image spec and while most tools handle
images without it fine (buildah even has a `--omit-history` flag),
some currently break (e.g. bootc). We could fix issues where we
encounter them, but it's much easier to match the rest of the
ecosystem and have it.
As a bonus, this gives us a more reliable way to retain
component information than via layer annotations (see e.g.
https://github.com/containers/buildah/issues/6652). And there's tooling
like `podman history` built to display that info more easily.
Assisted-by: Claude Opus 4.5
Implement buildah-compatible label removal using `--label KEY-` to
remove a specific label and `--label -` to clear all labels. I came
upon this need when playing with "faking" a plain FCOS by taking the
OSTree-based production images and pruning `/sysroot/`: we also in that
case want to drop the `ostree.final-diffid` and `ostree.commit` labels.
(There's also another issue before that transformed FCOS image is fully
accepted by bootc, which is that today the latter mandates OCI history,
which we don't currently emit. Will send a patch to bootc soon.)
Assisted-by: Claude Opus 4.5
Let's just have it always match the latest release.
Generalize the `lockcheck` CI check to become `versioncheck`, which also
covers the README.md. The grep lines are a bit yucky but meh.
Assisted-by: Claude Opus 4.5
Missed this when I bumped to v0.1.1. Check it explicitly in CI before
build (to catch mismatches before cargo modifies Cargo.lock) and in
release.py.
Assisted-by: Claude Opus 4.5
Set core.commentchar to ';' when creating the signed tag so that
release notes containing markdown headers (lines starting with '#')
are preserved in the tag message.
Assisted-by: Claude Opus 4.5
One annoying thing about Rust packaging is that it's very easy to miss
when you're bundling what is normally distributed as a shared library.
We almost always want to avoid that. We avoided it for openssl, but we
need the same treatment for zlib.
Include the splitter Containerfile in GitHub releases. Still thinking
about this, this file is meant to be used directly via HTTP but because
it's an end-user API, I want it to be:
1. accessible through a not-too-ugly URL
2. versioned somehow so users can pin version and we can make changes
Assisted-by: Claude Opus 4.5
Update CI workflow to differentiate between pushes to main and tag
pushes. On main pushes, only push to quay.io/jlebon/chunkah:dev. On
tag pushes, rebuild and run tests, then push to both :tag and :latest.
Also add ignore-versions regex to the cleanup step to ensure tagged
releases are never deleted from ghcr.io.
Assisted-by: Claude Opus 4.5
