Chunkah is actively maintained and the algorithm seems to be more stable
for the layer plan which results in smaller sizes for weekly updates
compared to rpm-ostree. Although it looks like daily updates are
slightly bigger for the time being.
In the future there may be a way to consume previous build manifests as
well to further shrink down image sizes.
I am not deleting the old build-chunked-oci code yet, to make this PR
easier to revert in case we have to.
Rpm-ostree is basically the only blocker for rootless CI, which is
generally what we should be doing, PolP and all that. Now we only need
root privileges for bootc install. Althought to keep this PR more simple
I will implement this later.
It seems like bootc install needs more space now, not sure exactly why
that happens but seems to be related due to pruned /sysroot.
xref https://github.com/ublue-os/aurora/issues/2324
Co-authored-by: inffy <10782843+inffy@users.noreply.github.com>
This label was previously created by the hhd/legacy-rechunker instead of
on the image build process. We are now using the build-chunked-oci and
in the near future chunkah which do not mess with the labels.
Our changelogs script expects this label to be present in order for the
commits since last release to show up.
fixes: https://github.com/ublue-os/aurora/issues/2338
Currently we are tagging these:
```
latest
latest-44.20260531
latest-20260531
latest-44
44-44.20260531
44-20260531
```
and this PR will add:
```
latest-44.20260531.2
latest-20260531.2
44-44.20260531.2
44-20260531.2
```
and for stable:
```
stable
stable-44.20260531
stable-20260531
stable-daily
44.20260531
stable-daily-44.20260531
stable-daily-20260531
```
```
stable-44.20260531.1
stable-20260531.1
stable-daily-44.20260531.1
stable-daily-20260531.1
```
Today there has been no stable image released so it starts out with .1
but there has been a single latest image, so it bumps it to .2
Very much inspired by ublue-os/main's solution [1], but this
implementation has a bug where where it will find the first "missing"
POINT (not the highest!) in the tags.
This would falsely choose the wrong POINT in the case we delete a bad
image and there is a "gap".
For example, we have the images .1-3 already on the registry, then
delete .2 and this implementation would correctly choose the next image
as .4 (instead of .2) as it's not just a dumb while loop.
This is very much an edge case and makes the jq command quite cursed but
I think this should always be reliable.
This feature has been broken previously because we didn't tag the image
with an index point number so no matter how many images we have built
that day, it was always .1 in the various VESRION fields in os-release.
fixes: https://github.com/ublue-os/aurora/issues/1833
[1]: ef33e7ca96/Justfile (L239-L251)
The ubuntu 26.04 provided version is still too old, recently they
stabilized the --fmt flag [1]. This is consistent with the other
invokations and also allows us to have some whitespace between variable
definitions for example as they seemed to have changed that.
[1]: 21b3833bf0
* fix(ci): push production tags after signing
If the signing step fails, causes for this can be ghcr jank or a
sigstore outage and not testing our changes properly beforehand. Then we
have already pushed images to the registry with production tags that
users can update to.
The clients won't actually update to these images as we enforce
verification on the client side to pull images from ghcr.io/ublue-os/
only with a valid signature attached. Otherwise this leads to an "update
failed" on the users POV, which is very confusing. This has been a
longstanding issue across the org [1] [2] [3].
To workaround this, we push a tag to the registry first that we don't
care about and clients don't check against for updates. I chose the job
id because it always changes.
This also happens to get rid of the action we use for retries to
workaround ghcr jank, we were forced to do this in the past because the
old podman version we used at the time didn't have those features.
Using podman to push signatures and images at the same time is not
something that is viable at the moment unfortunately but is definitely
the approach we should be using once it works well.
This effort can be seen at [4]. Although we are likely hitting [5] here
where all layers get invalidated because of this bug. We are effectively
pushing full images/no shared layers every time we do podman push. Which
causes the overall push step in CI to take upwards of 20 minutes.
fixes: https://github.com/ublue-os/aurora/issues/2135
[1]: https://github.com/ublue-os/bazzite/pull/4902
[2]: https://github.com/ublue-os/main/issues/643
[3]: https://github.com/ublue-os/bazzite/issues/3044
[4]: https://github.com/containers/podman/issues/27796
[5]: https://github.com/ublue-os/aurora/pull/2199
Signed-off-by: renner <renner0@posteo.de>
* fix: cosign verify
It seems like cosign tries to verify all .sig files on the registry for
that digest, one of which is generated by cosign sign with a static key
and one is generated by actions/attest using an OIDC.
This exactly broke, which is what we have been using for the longest
time:
```
cosign verify --key cosign.pub ghcr.io/ublue-os/aurora:latest
Error: no matching attestations: expected key signature, not certificate
error during command execution: no matching attestations: expected key
signature, not certificate
```
And this currently works to verify the image, although it is of course
not the same thing as singing with the ublue cosign key.
This will not work going forward after this commit:
```
cosign verify \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
--certificate-identity-regexp="github.com/ublue-os/aurora/.github/workflows/*" \
ghcr.io/ublue-os/aurora:latest
```
Signed-off-by: renner <renner0@posteo.de>
* feat: use staging for the temporary tag
This is just a minor cosmetic thing and recudes the amount of tags we
have because it's static and not changing unlike the run id.
Signed-off-by: renner <renner0@posteo.de>
* fix: use empty string by default for temporary tag
Signed-off-by: renner <renner0@posteo.de>
* fix: don't use signing config
Or else we are getting this error with new cosign 3.X.X
```
Error: must provide --new-bundle-format or --bundle where applicable
with --signing-config or --use-signing-config error during command
```
Signed-off-by: renner <renner0@posteo.de>
* chore: bump cosign-installer to v4.1.2
* chore: move oras install to beginning as well
if this fails at the beginning because of jank then it's not so bad
* fix(ci): make sbom signing work
```
Error: error getting signing config from TUF: error setting TUF options:
error setting TUF mirror: error reading remote.json: open
/home/runner/.sigstore/root/remote.json: permission denied
```
---------
Signed-off-by: renner <renner0@posteo.de>
* feat(ci): stop building stable-daily images
These do not make super much sense, this is in preparation for the
testing branches so we don't build too many images.
We will still build stable images in PRs to find out if things are
broken on those images only.
To not break existing users, point this to `:stable` instead,
effectively lying about the cadence now. We have to do it this way
because stable-daily has ZFS and `:latest` doesn't to avoid breaking
that use case. In the future when we have removed ZFS we can point this
to `:latest`.
* fix: only publish :stable on schedule
We do not want to publish stable on every PR merge
---------
Co-authored-by: inffy <10782843+inffy@users.noreply.github.com>
This makes it easier to test changes that ideally need to be tested on a
fresh system. This is basically copy pasted from Zirconium.
Not uploading the image here as an artifact, I don't think this is too
useful and a bunch of work to resize to make it not suck to download.
But we should still get alerted when we change things that break bootc
install for whatever reason.
We can easily experiment with this now and implement the rest when it
makes sense. This also unblocks rootless CI.
We need /ostree (which is a symlink to /sysroot/ostree) to exist or else
things break.
Because we are generating our SBOMs before rechunking, we still have
leftover ostree objects which are getting picked up by syft and causes
changelogs to report wrong versions of packages.
fixes: https://github.com/ublue-os/aurora/issues/2077
* feat(ci): introduce temporary compatibility tag for f44
Instead of having 2 workflows for building beta and latest, which will
produce functionally the same image, we have one workflow instead.
Latest workflow will tag beta images.
* trigger latest workflow on beta
We are dealing with less long-lived keys now and we are supposed to use
this more modern way of signing our artifacts anyway where we can.
Sister PR to: https://github.com/get-aurora-dev/common/pull/158
We are dealing with less long-lived keys now and we are supposed to use
this more modern way of signing our artifacts anyway where we can.
Sister PR to: https://github.com/get-aurora-dev/common/pull/158
Due to the nature of these caches being immutable (hihi) we can only
initially create caches with a certain key and can't update them with
fresh packages, this means they would get stale after a certain time. To
workaround this we restore from cache x and write and publish to cache
y, this has the drawback that this growth will never stop as we don't
delete old and unused packages residing inside the cache. This is not a
super big issue as the vast majority of packages don't change too often
but is still an annoyance.
As per the eviction policy [1], caches not accessed for 7 days will be
deleted automatically. This should result in a cycle where a scheduled
stable (or other manually triggered) workflows will create new cache
that will be used by all workflows until the next one is created again.
With this change, manual intervention should never be needed while
always having a fresh cache.
This also addresses that the aurora-dx-nvidia-open image got falsely
recognized as aurora-dx and attempted to write cache previously.
[1]: https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching#usage-limits-and-eviction-policy
We need to check if the previous build was actually made with
rpm-ostree, otherwise this isn't going to do anything, implemented like
how upstream checks for this, see the comment.
Over time the layer plan due to reuse of the previous build's rechunking
plan might become undesirable/inefficient or due to bugs in rpm-ostree
itself that would trigger issues in the rechunker during the build might
force us to reset the layer plan where we do not consume our previous
build.
Currently we compose a fresh rechunked image every build. This is not
ideal for image layer difference reduction and thereforce update sizes
as we do not take the rechunking plan/package layout from the previous
build (that users will likely have downloaded) into account.
When we have a new release of rpm-ostree we can replace most of the
pulling logic here with the new `--previous-build` flag.
* chore(ci): remove akmods check for fedora version (#1966)
This is unnecessary and doesn't work all that well, caching is not super
well implemented anyway. Setting the fedora version per image tag
statically works just fine and if we were to set some version that is
not yet available then it will simply fail a couple seconds later on the
cosign verification or akmods image pull.
* chore(beta): adjust fedora version logic for beta in rechunker
This is unnecessary and doesn't work all that well, caching is not super
well implemented anyway. Setting the fedora version per image tag
statically works just fine and if we were to set some version that is
not yet available then it will simply fail a couple seconds later on the
cosign verification or akmods image pull.
We are writing cache to the proper location but not adjusting the
restore keys properly, i.e. aurora wants to look for cache at aurora
even tho we are purposefully only allowing aurora-dx cache to be
created.
Currently we are very inefficient with our cache usage as every image
has it's own cache where most of the packages are the same across
variants anyway. See 973333f.
This should fix a couple problems around this as we are now also
considering the Fedora version in the cache key and only allowing one
blessed image to create cache. Which is why I decided to implement this
into the Justfile even tho this is only needed when running from github
as it just makes things easier to troubleshoot.
aurora-dx makes the most sense here as that is the heaviest image. We
don't need a separate nvidia image cache as the biggest RPMs exclusive
to that variant are cached in akmods anyway and don't have to be
downloaded.
Only allowing cache to be created at all on workflow_dispatch and on
schedule happens to go hand in hand with the limitation/design of the
cache action as we only build images from the main branch in those two
cases, rest is merge queue, which also happens to address the
"infighting" across stable and latest builds when they kick off at the
same time.
This should make it easier going forward especially when we take future
possible ARM, testing images or any other reason for our matrix to get
bigger into consideration.
We are writing cache to the proper location but not adjusting the
restore keys properly, i.e. aurora wants to look for cache at aurora
even tho we are purposefully only allowing aurora-dx cache to be
created.
Currently we are very inefficient with our cache usage as every image
has it's own cache where most of the packages are the same across
variants anyway. See 973333f.
This should fix a couple problems around this as we are now also
considering the Fedora version in the cache key and only allowing one
blessed image to create cache. Which is why I decided to implement this
into the Justfile even tho this is only needed when running from github
as it just makes things easier to troubleshoot.
aurora-dx makes the most sense here as that is the heaviest image. We
don't need a separate nvidia image cache as the biggest RPMs exclusive
to that variant are cached in akmods anyway and don't have to be
downloaded.
Only allowing cache to be created at all on workflow_dispatch and on
schedule happens to go hand in hand with the limitation/design of the
cache action as we only build images from the main branch in those two
cases, rest is merge queue, which also happens to address the
"infighting" across stable and latest builds when they kick off at the
same time.
This should make it easier going forward especially when we take future
possible ARM, testing images or any other reason for our matrix to get
bigger into consideration.
* feat(ci): enable zstd:chunked compression
https://github.com/projectbluefin/common/issues/239
* fix: only use zstd compression for artifacts
The old podman version we use on the 24.04 runners seems to be really
slow with pushing zstd:chunked compressed images (to tarballs), we are
only using the newer podman version from brew when we actually push
images to the registry. For testing PRs this is good enough.
This makes the Exporting to OCI Archive step take 1m30s again and 4m for
DX, with :chunked it took 4m and 7m respectively.
* feat: rechunk with rpm-ostree
The legacy rechunker is basically unmaintained, this is a much safer
option to reduce our (delta) image size.
https://coreos.github.io/rpm-ostree/build-chunked-oci/
We can further reduce the layer diff by utilizing --previous-build in
the next rpm-ostree release. It still has a couple problems in the
current release in combination with non-rpm layers.
https://github.com/coreos/chunkah is very early in development, don't
bother for now.
* fix: handle dance regarding stable, stable-daily
* chore: add horrible workaround because of hhd-dev/rechunk
* fix: for layer annotations not being pushed
We need a newer podman version than ubuntu 24.04 provides,
homebrew is the most elegant solution.
We need the layer annotations for the rpm-ostree rechunker so that
updates are more efficient.
workaround for:
https://github.com/containers/podman/issues/27796
* feat: assign layers to files we download
* chore: autostart rechunker-group-fix
* fix: don't do beta tag logic
* chore: only run when deployed with ostree
so this doesn't run on the ISO etc will need to be changed on composefs
native backend.
Previously we were fetching the public keys over the network, this method
was not resilient against bad actors who take over one of our
dependencies and publish malicious images with those keys.
* feat: enable KDE 6.6 Beta builds
latest kernel is nasty, so just going for the coreos-stable kernel now
on beta
The panel task manager widget config is not present in the beta packages
* fix: use latest kernel
ostree hotfix and sysext is fixed in 6.18.5
* fix firewall installation
* feat: FROM quay.io/fedora-ostree-desktops/kinoite
Instead of using the ublue-os/kinoite-main base image we would be using
the ones built by Timothée over at https://gitlab.com/fedora/ostree/ci-test
This is a good path forward for Aurora because:
- we no longer have to undo things we inherit from ublue-os/main
- less overall dependencies
- we can make more conscious decisions about what goes into the image
- it's easier now to build images on top of upcoming Fedora versions
(even KDE nightly ones) or to base on top of potential future images
that are more aligned with our goals and needs
This would come at the cost of maintaining most of what main did
previously ourselves.
* chore: remove ublue-os-akmods-addons
* chore: codecs from negativo
* feat: add flathub
* fix: fedora flatpak be gone
* feat: miscellaneous workarounds
* feat: CoreOS rescue boot generator
* feat: allow brew programs with sudo
* chore: re-add packages from main
* chore: remove packages we removed in main
* chore: renovate
there is no latest tag
* fix: remove explicit ublue akmods copr cleanup
already included with all the others
* chore: remove added fedora flatpak service
We don't need this
* fix(renovate): set proper tag
---------
Co-authored-by: inffy <10782843+inffy@users.noreply.github.com>
