Update the Highlights and Understanding components sections to reflect
that pacman (ALPM) is now a supported component repo alongside RPM.
Assisted-by: Pi (Claude Opus 4.6)
ocidir 0.8 moved read_blob, read_index, read_json_blob, and other
read-only methods from inherent methods on OciDir to a new OciRead
trait. Import OciRead in test modules that call these methods.
Assisted-by: Pi (Claude Opus 4.6)
If a package has multiple changelog entries within a day, those are
likely either duplicates or reflect some sort of short-term packaging
problem, rather than indicating a higher frequency of releases.
Therefore, it more accurately reflects package stability to treat these
as a single release for the purpose of stability calculation.
Such duplicate changelog entries can cause stability to be significantly
underestimated for some packages; for example, the `systemd` package
currently has stability score around 0.260 without this change, and
0.416 with this change.
Ironically, we weren't actually verifying signatures on the rpm-sequoia
rpm we got from the CentOS repo...
Let's make sure to do that.
Assisted-by: Pi (Claude Opus 4.6)
It's silly right now how chunkah tars up an ocidir, but the primary
way this is consumed is by the containers stack, which will immediately
untar it back to consume it.
We can't avoid this in the `podman run | podman load` flow, but we can
in the Containerfile flow by simply leaving the output as an ocidir, and
then using `FROM oci:` instead of `FROM oci-archive:`.
To expose this, add support for transport prefixes on the argument
passed to `--output`; `oci:PATH` writes the OCI directory layout
directly to disk, while `oci-archive:PATH` or `PATH` preserves the
existing behavior.
Update `Containerfile.splitter` and docs to use the new format. And of
course, update the chunkah image build itself to use this!
Anyone using chunkah from a Containerfile is encouraged to switch over
to this. The efficiency gains are non-trivial. E.g. for the fedora-bootc
image, this roughly cuts in half the "handover time" from chunkah to
buildah.
Closes: #103
Assisted-by: Pi (Claude Opus 4.6)
For bootable containers let's add a note about minimum versions
of bootc required to use chunkah chunked images. The versions are
v1.1.3, which contains [1] for images <= 128 layers, and v1.9.0,
which contains [2], for images with > 128 layers.
[1] c49ccec505
[2] 207b7f0abf
Zizmor is a widely used static analysis tool for GitHub Actions that can
identify many common security issues in workflows. For reference, see:
https://docs.zizmor.sh/
Also explicitly ignore Zizmor warning for unpinned uses of Fedora
container, as tests are intentionally run against the lateset image.
Template expansions in code contexts can be a source of code injection
vulnerabilities: https://docs.zizmor.sh/audits/#template-injection
Instead, use environment variables to pass the results of template
expansions to shell commands in a way that's aware of shell syntax.
Add top-level empty permissions block in all CI workflows, and provide
explanatory comments for all write permissions. The only actual
permissions change is to `unit-tests`, which was previously given overly
broad default permissions but only needs to be able to read repo
contents.
See https://docs.zizmor.sh/audits/#excessive-permissions for more info
on how the default permissions are overly broad.
Instead of truncating the changed/unchanged component lists to 5
entries, show the full list using textwrap.fill() to wrap at 80
columns with proper indentation.
Assisted-By: <anthropic/claude-opus-4.6>
This is needed to workaround the current podman bug that will
output more matches than we think it should:
https://github.com/containers/podman/issues/28643
Assisted-By: <anthropic/claude-opus-4.6>
I had switch to `--certificate-github-workflow-repository` at
the last minute in #119, but hadn't tested it properly. It's only
an additional filter in cosign v3 which otherwise still requires
`--certificate-identity` or `--certificate-identity-regexp`. That
makes it less useful since it's redundant with those so let's just use
`--certificate-identity-regexp'.
Assisted-by: Pi (Claude Opus 4.6)
Use Sigstore cosign with GitHub Actions OIDC identity tokens to sign
container images pushed to quay.io. This allows users to verify that
images were built by the CI pipeline without requiring any private key
management.
Closes: #117
Assisted-by: Pi (Claude Opus 4.6)
Our README right now is the main documentation for this project. As new
features land, I want to be able to also update the README. But I don't
want users to get confused by unreleased features.
Let's add a dedicated README for development, and then it gets synced
over to the main README at release time.
Assisted-by: Pi (Claude Opus 4.6)
Currently, to split el10 images correctly, one needs to use a chunkah
image built from el10 so that rpm can read PQC signatures in the rpmdb
(because Fedora's rpm-sequoia doesn't support PQC yet).
We could switch the base image to c10s or UBI10 or provide separate
images, but I don't like any of those options. But I do want a good
OOTB experience.
So for now just temporarily hack this up by installing rpm-sequoia
from c10s on top of the Fedora base since it's ABI-compatible. If this
breaks before Fedora gains PQC support, we can re-evaluate the other
options.
Assisted-by: Pi (Claude Opus 4.6)
The old algorithm used a greedy merge approach (BinaryHeap-based,
minimizing total expected value loss per merge) which was fundamentally
unstable: small changes in input caused completely different merge
decisions, resulting in poor layer reuse across updates.
Additionally, because of its greediness, it easily fell into a local
optimum with a giant catch-all bucket with low stability and safer more
stable items in the other layers.
Replace it with a two-phase statistical partitioning approach inspired
by rpm-ostree's chunking algorithm:
- Phase 1 classifies components by size, giving large components
(linux-firmware, kernel, firefox) their own singleton layers.
- Phase 2 assigns all remaining components to bins using stability tiers
(high/mid/low) and deterministic name-based hashing, which ensures
stable bin membership across builds without needing to track prior
build state.
Also remove the stability fallback that assigned min(known)/2 to
components without stability data (xattr, bigfiles, unclaimed). These
now stay at 0.0 and are naturally handled by the stability tiers.
Benchmarked against rpm-ostree's native chunking on FCOS F43 (11
biweekly stable releases) and Silverblue F43 (3 weekly builds):
FCOS: 46.4% reuse, 916 MiB avg download uncompressed (rpm-ostree: 34.0%, 1.2 GiB)
Silverblue: 47.9% reuse, 2.8 GiB avg download uncompressed (rpm-ostree: 38.5%, 3.5 GiB)
Assisted-by: OpenCode (Claude Opus 4.6)
Add an OpenCode skill that guides benchmarking of packing
algorithm changes. Covers the full pipeline from building
chunkah through analyzing layer reuse, including key pitfalls
like duplicate tags and sample size sensitivity.
Assisted-by: OpenCode (Claude Opus 4.6)
Wrap the processing loop in try/finally so that temporary
localhost/tmp-chunk-src images are always removed, even if
chunking fails or is interrupted mid-run. Previously these
would leak and cause duplicate tag issues in subsequent runs.
Assisted-by: OpenCode (Claude Opus 4.6)
Switch test-arch.sh and test-self.sh to use --write-manifest-to for
checking the unclaimed component size. The previous approach checked
the layer size via skopeo inspect, which is inaccurate since the
unclaimed component may share a layer with other components.
For test-self.sh, also drop the is_chunked shortcut so that we always
run chunkah and produce a manifest.
Assisted-by: OpenCode (Claude Opus 4.6)
Add a --remote flag to both bump (default: origin) and cut (default:
upstream) so the git remote is configurable. Switch gh release create
from --notes-from-tag to --notes-file since the former is incompatible
with --repo. In the --no-push flow, the notes file is now preserved
and referenced in the printed hint.
Like 23435c3 ("tools/release.py: hardcode coreos/chunkah in gh API
calls"), this is another tweak I only realized I needed once I reforked
chunkah in my personal github account.
Assisted-by: OpenCode (Claude Opus 4.6)
Allow specifying which commit to tag and archive when cutting a release,
rather than always using HEAD. The commit ref flows through to both
git tag and create-archives.sh's git archive call. When not specified,
both default to HEAD as before.
Assisted-by: OpenCode (Claude Opus 4.6)
