* feat(ci): add config for backporting PRs to stable branch
This will automatically create PRs to the stable branch when something
has been merged to main if the cherry-pick label is present.
Co-authored-by: inffy <10782843+inffy@users.noreply.github.com>
* configure git author for the bot
---------
Co-authored-by: inffy <10782843+inffy@users.noreply.github.com>
* feat(ci): testing stream
* feat(ci): workaround for scheduled stable build on non-default branch
* feat(renovate): initial config for testing stream
We want to get builds that are triggered by kinoite updates from
renovate but promote changes like get-aurora-dev/common and
ublue-os/brew from main to the stable branch, while still receiving
updates for things like github actions.
This is probably the right way to handle the differentiation of PR and
production runs instead of doing it via the not a pull request
condition.
Co-authored-by: inffy <10782843+inffy@users.noreply.github.com>
This takes 1min to run but is more reliable than the
container-storage-action in some cases on runners with only 20GB of
available disk space on /. We correctly mount /mnt to
/var/lib/containers via the container-storage-action when we have low
space available but we are running out of space on /var/tmp, which this
action does not address, this usually happens on the COMMIT step of the
image build.
Observed here: [1]
[1]:
https://github.com/ublue-os/aurora/actions/runs/26897102965/job/79339038302
The ubuntu 26.04 provided version is still too old, recently they
stabilized the --fmt flag [1]. This is consistent with the other
invokations and also allows us to have some whitespace between variable
definitions for example as they seemed to have changed that.
[1]: 21b3833bf0
* fix(ci): push production tags after signing
If the signing step fails, causes for this can be ghcr jank or a
sigstore outage and not testing our changes properly beforehand. Then we
have already pushed images to the registry with production tags that
users can update to.
The clients won't actually update to these images as we enforce
verification on the client side to pull images from ghcr.io/ublue-os/
only with a valid signature attached. Otherwise this leads to an "update
failed" on the users POV, which is very confusing. This has been a
longstanding issue across the org [1] [2] [3].
To workaround this, we push a tag to the registry first that we don't
care about and clients don't check against for updates. I chose the job
id because it always changes.
This also happens to get rid of the action we use for retries to
workaround ghcr jank, we were forced to do this in the past because the
old podman version we used at the time didn't have those features.
Using podman to push signatures and images at the same time is not
something that is viable at the moment unfortunately but is definitely
the approach we should be using once it works well.
This effort can be seen at [4]. Although we are likely hitting [5] here
where all layers get invalidated because of this bug. We are effectively
pushing full images/no shared layers every time we do podman push. Which
causes the overall push step in CI to take upwards of 20 minutes.
fixes: https://github.com/ublue-os/aurora/issues/2135
[1]: https://github.com/ublue-os/bazzite/pull/4902
[2]: https://github.com/ublue-os/main/issues/643
[3]: https://github.com/ublue-os/bazzite/issues/3044
[4]: https://github.com/containers/podman/issues/27796
[5]: https://github.com/ublue-os/aurora/pull/2199
Signed-off-by: renner <renner0@posteo.de>
* fix: cosign verify
It seems like cosign tries to verify all .sig files on the registry for
that digest, one of which is generated by cosign sign with a static key
and one is generated by actions/attest using an OIDC.
This exactly broke, which is what we have been using for the longest
time:
```
cosign verify --key cosign.pub ghcr.io/ublue-os/aurora:latest
Error: no matching attestations: expected key signature, not certificate
error during command execution: no matching attestations: expected key
signature, not certificate
```
And this currently works to verify the image, although it is of course
not the same thing as singing with the ublue cosign key.
This will not work going forward after this commit:
```
cosign verify \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
--certificate-identity-regexp="github.com/ublue-os/aurora/.github/workflows/*" \
ghcr.io/ublue-os/aurora:latest
```
Signed-off-by: renner <renner0@posteo.de>
* feat: use staging for the temporary tag
This is just a minor cosmetic thing and recudes the amount of tags we
have because it's static and not changing unlike the run id.
Signed-off-by: renner <renner0@posteo.de>
* fix: use empty string by default for temporary tag
Signed-off-by: renner <renner0@posteo.de>
* fix: don't use signing config
Or else we are getting this error with new cosign 3.X.X
```
Error: must provide --new-bundle-format or --bundle where applicable
with --signing-config or --use-signing-config error during command
```
Signed-off-by: renner <renner0@posteo.de>
* chore: bump cosign-installer to v4.1.2
* chore: move oras install to beginning as well
if this fails at the beginning because of jank then it's not so bad
* fix(ci): make sbom signing work
```
Error: error getting signing config from TUF: error setting TUF options:
error setting TUF mirror: error reading remote.json: open
/home/runner/.sigstore/root/remote.json: permission denied
```
---------
Signed-off-by: renner <renner0@posteo.de>
* feat(ci): stop building stable-daily images
These do not make super much sense, this is in preparation for the
testing branches so we don't build too many images.
We will still build stable images in PRs to find out if things are
broken on those images only.
To not break existing users, point this to `:stable` instead,
effectively lying about the cadence now. We have to do it this way
because stable-daily has ZFS and `:latest` doesn't to avoid breaking
that use case. In the future when we have removed ZFS we can point this
to `:latest`.
* fix: only publish :stable on schedule
We do not want to publish stable on every PR merge
---------
Co-authored-by: inffy <10782843+inffy@users.noreply.github.com>
This makes it easier to test changes that ideally need to be tested on a
fresh system. This is basically copy pasted from Zirconium.
Not uploading the image here as an artifact, I don't think this is too
useful and a bunch of work to resize to make it not suck to download.
But we should still get alerted when we change things that break bootc
install for whatever reason.
Kinda unnecessary to do this, kinda expensive operation for the time
being as it's an image pull, in the future this will be easier or we are
gonna use chunkah anyway.
Brew is a little slow to install things, this version is good enough for
our needs until we can upgrade our runners to 26.04.
I took this from bootc and this also works with this newer ubuntu
release.
Because we are generating our SBOMs before rechunking, we still have
leftover ostree objects which are getting picked up by syft and causes
changelogs to report wrong versions of packages.
fixes: https://github.com/ublue-os/aurora/issues/2077
* feat(ci): introduce temporary compatibility tag for f44
Instead of having 2 workflows for building beta and latest, which will
produce functionally the same image, we have one workflow instead.
Latest workflow will tag beta images.
* trigger latest workflow on beta
While it's nice that this doesn't break anything on modern systems, it
also has no advantages *for now* as it's just falling back to zstd.
Let's be more careful with this and only do zstd compression for the
time being so we don't break older systems that may not have updated to
a modern version of rpm-ostree that supports zstd:chunked because they
haven't been turned on for quite a while for example.
* chore(deps): update actions/upload-artifact action to v7.0.1
* fix: set exact version for upload-artifact
so we prevent two PRs being opened by renovate for the same action
---------
Co-authored-by: ubot-7274[bot] <217212047+ubot-7274[bot]@users.noreply.github.com>
Co-authored-by: renner <renner0@posteo.de>
Due to the nature of these caches being immutable (hihi) we can only
initially create caches with a certain key and can't update them with
fresh packages, this means they would get stale after a certain time. To
workaround this we restore from cache x and write and publish to cache
y, this has the drawback that this growth will never stop as we don't
delete old and unused packages residing inside the cache. This is not a
super big issue as the vast majority of packages don't change too often
but is still an annoyance.
As per the eviction policy [1], caches not accessed for 7 days will be
deleted automatically. This should result in a cycle where a scheduled
stable (or other manually triggered) workflows will create new cache
that will be used by all workflows until the next one is created again.
With this change, manual intervention should never be needed while
always having a fresh cache.
This also addresses that the aurora-dx-nvidia-open image got falsely
recognized as aurora-dx and attempted to write cache previously.
[1]: https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching#usage-limits-and-eviction-policy
* feat(nvidia): add service to sync Flatpak driver version (#1983)
Co-authored-by: inffy <10782843+inffy@users.noreply.github.com>
* chore(ci): replace retry action (#1995)
* ci: replace extractions/setup-just with brew install (#2000)
* chore(deps): update ghcr.io/ublue-os/brew:latest docker digest to 1a058eb (#2002)
Co-authored-by: ubot-7274[bot] <217212047+ubot-7274[bot]@users.noreply.github.com>
* fix: move dracut configs to files instead of flags (#2003)
While this doesn't make a difference to our image build itself, it will
make the life easier for derived images as they will inherit all the
modules from us that we set anyway. This undoes some of the changes made
in 32a66fc719. These are specific to the things we install here so it
doesn't make sense to go into common.
---------
Co-authored-by: Adam Fidel <ledif@users.noreply.github.com>
Co-authored-by: inffy <10782843+inffy@users.noreply.github.com>
Co-authored-by: Dylan M. Taylor <dylan@dylanmtaylor.com>
Co-authored-by: ubot-7274[bot] <217212047+ubot-7274[bot]@users.noreply.github.com>
Over time the layer plan due to reuse of the previous build's rechunking
plan might become undesirable/inefficient or due to bugs in rpm-ostree
itself that would trigger issues in the rechunker during the build might
force us to reset the layer plan where we do not consume our previous
build.
Currently we are very inefficient with our cache usage as every image
has it's own cache where most of the packages are the same across
variants anyway. See 973333f.
This should fix a couple problems around this as we are now also
considering the Fedora version in the cache key and only allowing one
blessed image to create cache. Which is why I decided to implement this
into the Justfile even tho this is only needed when running from github
as it just makes things easier to troubleshoot.
aurora-dx makes the most sense here as that is the heaviest image. We
don't need a separate nvidia image cache as the biggest RPMs exclusive
to that variant are cached in akmods anyway and don't have to be
downloaded.
Only allowing cache to be created at all on workflow_dispatch and on
schedule happens to go hand in hand with the limitation/design of the
cache action as we only build images from the main branch in those two
cases, rest is merge queue, which also happens to address the
"infighting" across stable and latest builds when they kick off at the
same time.
This should make it easier going forward especially when we take future
possible ARM, testing images or any other reason for our matrix to get
bigger into consideration.
Currently we are very inefficient with our cache usage as every image
has it's own cache where most of the packages are the same across
variants anyway. See 973333f.
This should fix a couple problems around this as we are now also
considering the Fedora version in the cache key and only allowing one
blessed image to create cache. Which is why I decided to implement this
into the Justfile even tho this is only needed when running from github
as it just makes things easier to troubleshoot.
aurora-dx makes the most sense here as that is the heaviest image. We
don't need a separate nvidia image cache as the biggest RPMs exclusive
to that variant are cached in akmods anyway and don't have to be
downloaded.
Only allowing cache to be created at all on workflow_dispatch and on
schedule happens to go hand in hand with the limitation/design of the
cache action as we only build images from the main branch in those two
cases, rest is merge queue, which also happens to address the
"infighting" across stable and latest builds when they kick off at the
same time.
This should make it easier going forward especially when we take future
possible ARM, testing images or any other reason for our matrix to get
bigger into consideration.
* feat(ci): enable zstd:chunked compression
https://github.com/projectbluefin/common/issues/239
* fix: only use zstd compression for artifacts
The old podman version we use on the 24.04 runners seems to be really
slow with pushing zstd:chunked compressed images (to tarballs), we are
only using the newer podman version from brew when we actually push
images to the registry. For testing PRs this is good enough.
This makes the Exporting to OCI Archive step take 1m30s again and 4m for
DX, with :chunked it took 4m and 7m respectively.
* feat: rechunk with rpm-ostree
The legacy rechunker is basically unmaintained, this is a much safer
option to reduce our (delta) image size.
https://coreos.github.io/rpm-ostree/build-chunked-oci/
We can further reduce the layer diff by utilizing --previous-build in
the next rpm-ostree release. It still has a couple problems in the
current release in combination with non-rpm layers.
https://github.com/coreos/chunkah is very early in development, don't
bother for now.
* fix: handle dance regarding stable, stable-daily
* chore: add horrible workaround because of hhd-dev/rechunk
* fix: for layer annotations not being pushed
We need a newer podman version than ubuntu 24.04 provides,
homebrew is the most elegant solution.
We need the layer annotations for the rpm-ostree rechunker so that
updates are more efficient.
workaround for:
https://github.com/containers/podman/issues/27796
* feat: assign layers to files we download
* chore: autostart rechunker-group-fix
* fix: don't do beta tag logic
* chore: only run when deployed with ostree
so this doesn't run on the ISO etc will need to be changed on composefs
native backend.
* fix(ci): run scorecard once per week with stable
This wasn't supposed to run at every 15 minutes
* chore(ci): do not build images when changing scorecard
* chore: remove OpenSSF regression check
* remove the mentions of the regression check
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This uses https://github.com/actions/cache, this should avoid repulling
packages from the Fedora repositories every time we rebuild, downloading
them from Github is minimally faster from my testing.
This pulls in the package cache before every build and uploads the new
cache again (with new packages) so it can be reused on later runs. It's
kinda janky as the first image i.e. aurora-dx-nvidia-open :stable and
:latest are competing with each other basically as they have the same
image name. Only one job run can write to the same cache (we have four
of those). So there will be some inefficiencies as some runs won't
upload packages. But the only difference should be nvidia specific
things and rocm which we only install on our regular images. So it's
not perfect in any way. But this is easy to revert so I say we just try
it.
All this is possible because we set
--mount=type=cache,dst=/var/cache/libdnf5 and keepcache=1 for dnf.
docs: https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching
Lowers the barrier to testing PRs a little bit as you don't have to
build an image locally anymore and is a good compromise in my opinion as
we would avoid using production signing keys for PR builds that should
probably be cleaned up more aggressively than the usual 90 days.
* chore(ci): store SBOMs as OCI artifact
Our current approach by uploading our rather large SBOMs to rekor
doesn't work anymore for unknown reasons to me and is breaking our
changelog generation in turn.
The compressed SBOMs are now a couple MiB big and stored on ghcr like
everything else which should make things easier for us.
* chore: don't compress SBOMs
More straightforward to consume them this way. Uncompressed
SBOMs are around 40MiB big instead of 5MiB.
* chore: sign sbom artifact
* chore: update type to application/vnd.spdx+json
* chore: use lightweight ubuntu-slim for moderator
See: https://github.com/ublue-os/aurora/issues/1728
* chore: do not build when moderator.yml changed
remove files that are in get-aurora-dev/{common,iso}
Github removed /mnt from their runners and /dev/root is now
145G, before it was 72G so the btrfs action was needed.
This action is broken now and always used the fallback path and has no
purpose anymore.
* feat: enable KDE 6.6 Beta builds
latest kernel is nasty, so just going for the coreos-stable kernel now
on beta
The panel task manager widget config is not present in the beta packages
* fix: use latest kernel
ostree hotfix and sysext is fixed in 6.18.5
* fix firewall installation
* feat: FROM quay.io/fedora-ostree-desktops/kinoite
Instead of using the ublue-os/kinoite-main base image we would be using
the ones built by Timothée over at https://gitlab.com/fedora/ostree/ci-test
This is a good path forward for Aurora because:
- we no longer have to undo things we inherit from ublue-os/main
- less overall dependencies
- we can make more conscious decisions about what goes into the image
- it's easier now to build images on top of upcoming Fedora versions
(even KDE nightly ones) or to base on top of potential future images
that are more aligned with our goals and needs
This would come at the cost of maintaining most of what main did
previously ourselves.
* chore: remove ublue-os-akmods-addons
* chore: codecs from negativo
* feat: add flathub
* fix: fedora flatpak be gone
* feat: miscellaneous workarounds
* feat: CoreOS rescue boot generator
* feat: allow brew programs with sudo
* chore: re-add packages from main
* chore: remove packages we removed in main
* chore: renovate
there is no latest tag
* fix: remove explicit ublue akmods copr cleanup
already included with all the others
* chore: remove added fedora flatpak service
We don't need this
* fix(renovate): set proper tag
---------
Co-authored-by: inffy <10782843+inffy@users.noreply.github.com>
* feat(ci): update changelog to use SBOMs
* reword some comments
* update sbom parsing based on new format
* Fix duplicate package handling
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* address review comment about exceptions
* add line for ROCm
* we need cosign in the workflow
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
This extra information doesn't seem to crash runners and includes way
more useful information we need for the changelogs.
Follow up of: https://github.com/ublue-os/aurora/pull/1548
* feat(ci): re-enable SBOM generation
Other methods of generating it causes the runner to run out of memory.
* fix: don't run in PRs
* Update .github/workflows/reusable-build.yml
Co-authored-by: Adam Fidel <ledif@users.noreply.github.com>
* Update .github/workflows/reusable-build.yml
Co-authored-by: Adam Fidel <ledif@users.noreply.github.com>
---------
Co-authored-by: Adam Fidel <ledif@users.noreply.github.com>
