For bootable containers let's add a note about minimum versions
of bootc required to use chunkah chunked images. The versions are
v1.1.3, which contains [1] for images <= 128 layers, and v1.9.0,
which contains [2], for images with > 128 layers.
[1] c49ccec505
[2] 207b7f0abf
Zizmor is a widely used static analysis tool for GitHub Actions that can
identify many common security issues in workflows. For reference, see:
https://docs.zizmor.sh/
Also explicitly ignore Zizmor warning for unpinned uses of Fedora
container, as tests are intentionally run against the lateset image.
Template expansions in code contexts can be a source of code injection
vulnerabilities: https://docs.zizmor.sh/audits/#template-injection
Instead, use environment variables to pass the results of template
expansions to shell commands in a way that's aware of shell syntax.
Add top-level empty permissions block in all CI workflows, and provide
explanatory comments for all write permissions. The only actual
permissions change is to `unit-tests`, which was previously given overly
broad default permissions but only needs to be able to read repo
contents.
See https://docs.zizmor.sh/audits/#excessive-permissions for more info
on how the default permissions are overly broad.
Instead of truncating the changed/unchanged component lists to 5
entries, show the full list using textwrap.fill() to wrap at 80
columns with proper indentation.
Assisted-By: <anthropic/claude-opus-4.6>
This is needed to workaround the current podman bug that will
output more matches than we think it should:
https://github.com/containers/podman/issues/28643
Assisted-By: <anthropic/claude-opus-4.6>
I had switch to `--certificate-github-workflow-repository` at
the last minute in #119, but hadn't tested it properly. It's only
an additional filter in cosign v3 which otherwise still requires
`--certificate-identity` or `--certificate-identity-regexp`. That
makes it less useful since it's redundant with those so let's just use
`--certificate-identity-regexp'.
Assisted-by: Pi (Claude Opus 4.6)
Use Sigstore cosign with GitHub Actions OIDC identity tokens to sign
container images pushed to quay.io. This allows users to verify that
images were built by the CI pipeline without requiring any private key
management.
Closes: #117
Assisted-by: Pi (Claude Opus 4.6)
Our README right now is the main documentation for this project. As new
features land, I want to be able to also update the README. But I don't
want users to get confused by unreleased features.
Let's add a dedicated README for development, and then it gets synced
over to the main README at release time.
Assisted-by: Pi (Claude Opus 4.6)
Currently, to split el10 images correctly, one needs to use a chunkah
image built from el10 so that rpm can read PQC signatures in the rpmdb
(because Fedora's rpm-sequoia doesn't support PQC yet).
We could switch the base image to c10s or UBI10 or provide separate
images, but I don't like any of those options. But I do want a good
OOTB experience.
So for now just temporarily hack this up by installing rpm-sequoia
from c10s on top of the Fedora base since it's ABI-compatible. If this
breaks before Fedora gains PQC support, we can re-evaluate the other
options.
Assisted-by: Pi (Claude Opus 4.6)
The old algorithm used a greedy merge approach (BinaryHeap-based,
minimizing total expected value loss per merge) which was fundamentally
unstable: small changes in input caused completely different merge
decisions, resulting in poor layer reuse across updates.
Additionally, because of its greediness, it easily fell into a local
optimum with a giant catch-all bucket with low stability and safer more
stable items in the other layers.
Replace it with a two-phase statistical partitioning approach inspired
by rpm-ostree's chunking algorithm:
- Phase 1 classifies components by size, giving large components
(linux-firmware, kernel, firefox) their own singleton layers.
- Phase 2 assigns all remaining components to bins using stability tiers
(high/mid/low) and deterministic name-based hashing, which ensures
stable bin membership across builds without needing to track prior
build state.
Also remove the stability fallback that assigned min(known)/2 to
components without stability data (xattr, bigfiles, unclaimed). These
now stay at 0.0 and are naturally handled by the stability tiers.
Benchmarked against rpm-ostree's native chunking on FCOS F43 (11
biweekly stable releases) and Silverblue F43 (3 weekly builds):
FCOS: 46.4% reuse, 916 MiB avg download uncompressed (rpm-ostree: 34.0%, 1.2 GiB)
Silverblue: 47.9% reuse, 2.8 GiB avg download uncompressed (rpm-ostree: 38.5%, 3.5 GiB)
Assisted-by: OpenCode (Claude Opus 4.6)
Add an OpenCode skill that guides benchmarking of packing
algorithm changes. Covers the full pipeline from building
chunkah through analyzing layer reuse, including key pitfalls
like duplicate tags and sample size sensitivity.
Assisted-by: OpenCode (Claude Opus 4.6)
Wrap the processing loop in try/finally so that temporary
localhost/tmp-chunk-src images are always removed, even if
chunking fails or is interrupted mid-run. Previously these
would leak and cause duplicate tag issues in subsequent runs.
Assisted-by: OpenCode (Claude Opus 4.6)
Switch test-arch.sh and test-self.sh to use --write-manifest-to for
checking the unclaimed component size. The previous approach checked
the layer size via skopeo inspect, which is inaccurate since the
unclaimed component may share a layer with other components.
For test-self.sh, also drop the is_chunked shortcut so that we always
run chunkah and produce a manifest.
Assisted-by: OpenCode (Claude Opus 4.6)
Add a --remote flag to both bump (default: origin) and cut (default:
upstream) so the git remote is configurable. Switch gh release create
from --notes-from-tag to --notes-file since the former is incompatible
with --repo. In the --no-push flow, the notes file is now preserved
and referenced in the printed hint.
Like 23435c3 ("tools/release.py: hardcode coreos/chunkah in gh API
calls"), this is another tweak I only realized I needed once I reforked
chunkah in my personal github account.
Assisted-by: OpenCode (Claude Opus 4.6)
Allow specifying which commit to tag and archive when cutting a release,
rather than always using HEAD. The commit ref flows through to both
git tag and create-archives.sh's git archive call. When not specified,
both default to HEAD as before.
Assisted-by: OpenCode (Claude Opus 4.6)
Replace the release-notes agent skill with an OpenCode custom command
that can be invoked as /polish-notes. The command takes a version
argument and an optional file path for raw notes, fetches PR details
to determine user-facing impact, and writes polished notes to
.release-notes-<version>.md.
Assisted-by: OpenCode (Claude Opus 4.6)
Add a generate-notes subcommand that fetches release notes from GitHub
and prints them to stdout, allowing the notes to be prepared and edited
separately from the cut workflow. The cut command gains a --notes-file
flag to read pre-prepared notes, skipping the fetch and editor steps.
When --notes-file is not provided, cut falls back to the existing
fetch-and-edit behavior.
Assisted-by: OpenCode (Claude Opus 4.6)
Didn't need this until now because I was doing PRs through branches
pushed directly to coreos/chunkah (because I hadn't yet reforked chunkah
back into my personal repo since transferring ownership).
Use the new -t/--tag switch added in dc805a6 to embed the target image
name directly in the OCI archive. This lets podman load tag the image
automatically, removing the need to parse the image ID from podman load
output and manually call podman tag.
Assisted-by: OpenCode (Claude Opus 4.6)
I.e. chunk images which are already local. This is useful e.g. for
Fedora images manually downloaded from pungi composes (because Fedora
doesn't keep historical tags in its repos).
`skopeo inspect` on container-storage images may or may not return
compressed sizes rather than uncompressed sizes depending on how the
image was imported.
Switch to using the layer sizes in `podman history` if available, which
is always going to be the uncompressed size. Ideally we'd always used a
compressed size here since that's actually the data amount on the wire
and that's what we're minimizing, but comparing compression possibly
done by different compressors is awkward. But most of all though we
should deifnitely not compare images with different size semantics.
In the `podman run chunkah | podman load` flow for splitting an existing
image, it's annoying that the resulting image is unnamed. Add support
for a `-t`/`--tag` switch which sets the right annotation on the
manifest descriptor in the OCI archive that e.g. `podman` and `docker`
pay attention to.
While we're here, update the Docker docs; the containerd backend does
support OCI archives natively and that's the default backend nowadays.
So that simplifies things nicely and makes it almost identical to the
podman instructions.
Assisted-by: OpenCode (Claude Opus 4.6)
Semi-arbitrarily chosen value. But 10 feels like the point where
you're probably doing this mechanically (e.g. mapping from a currently
incompatible repo database) than by hand on specific files.
If those components become a large part of your total component set, you
really want to add stability information for better packing.
This is the second part of #98. If the user didn't provide any interval
via the new xattr API, assume weekly.
Previously, we were using the fallback of (minimum stability)/2, which
(1) is way too sensitive to churny outliers in the package set, and (2)
doesn't really have a basis in what users are expressing.
I'm converting users' intent of bothering to add an xattr here
as them saying that it has _some_ amount of stability, which I'm
semi-arbitrarily ballparking as 'weekly'.
While we're here, tweak the README section on xattrs a bit more to
reflect reality.
With xattrs now supporting stability, it's super easy to have a unit
test which exercises packing more from end-to-end.
Do this.
This at the same time tests the xattr stability feature itself.
Assisted-by: OpenCode (Claude Opus 4.6)
Allow users to specify the stability of xattr components via
`user.update-interval`. The value can be either a positive integer
representing the average update interval in days (e.g. '30') or a
named label (daily, weekly, biweekly, monthly, quarterly, yearly). That
makes it easier to understand than having them plug in the probability
directly.
This plugs a notable gap we've had so far in the xattr repo, which
without this information was useful mostly when there was no to little
packing pressure. As soon as aggressive packing is required, treating a
high stability component as low can lead to suboptimal outcomes.
Closes: #98
Assisted-by: OpenCode (Claude Opus 4.6)
Compression impacts both the OCI archive itself and tar layers. It's
the same setting but the logging right now gives the impression that it
doesn't apply to the latter.
First, the Fedora minimal image is too tiny to get a good signal out of
benchmarking improvements.
Second, bump warmups to 3 since I've seen cases where 1 is not enough
to preload caches.
Third, redirect to a real file rather than `/dev/null` so it's closer to
how chunkah is actually used.
Chunking an ostree object store produces poor results and is almost
certainly a user error. Detect this by checking the scanned file map
for /sysroot/ostree paths (which naturally respects --prune) and warn
with a 7-second sleep so the message is visible in scrolling output.
This is a soft warning rather than a hard error since chunkah should
not strictly enforce this -- no data loss is involved, just suboptimal
layer splitting.
Refs: https://github.com/coreos/chunkah/issues/97
Assisted-by: OpenCode (Claude Opus 4)
The ALPM component repository can parse local package databases
managed by Arch Linux' pacman package manager.
This allows chunkah to chunk container images based on Arch Linux.
