Upstream NetworkManager now autogenerates a lo.nmconnection connection
to manage the local loopback device. Let's ignore that connection for
the purposes of our comparison and propagation here.
Closes https://github.com/coreos/fedora-coreos-tracker/issues/1385
When testing out if these rules applied correctly in the initramfs
I wasn't observing the device would properly get NM_UNMANAGED=1
until I added change/move.
This is similar to what is done by NM over in
38d3834e2c/data/85-nm-unmanaged.rules (L5)
We previously had the 68-azure-sriov-nm-unmanaged.rules but we
figured out we also want those rules in the initramfs. This
commit makes a new overlay.d entry and groups together a new
dracut module along with the udev rules in that overlay.d directory.
Fixes https://github.com/coreos/fedora-coreos-tracker/issues/1383
I noticed recently that Azure is bringing up networking in the initramfs
on every boot [1]. This behavior was originally introduced in c1cf903. After
some discussion with @jlebon and @bgilbert we are going to convert the
coreos-enable-network.service to only be pulled in and enalbed on the
Ignition boot so that it doesn't run on subsequent boots.
[1] https://github.com/coreos/fedora-coreos-tracker/issues/1383#issuecomment-1384526420
It's possible that some other process has a file descriptor open
for `/proc/cmdline`; if so our unmount will fail. Do the unmount
lazily - it was designed for exactly this situation.
This was seen in an OCP installation on vSphere at least once.
Starting with FCOS 36.20220906.1.0, 36.20220906.2.0, and 36.20220820.3.0,
coreos-assembler inadvertently failed to configure ostree to set
`grub_users=""` in non-default BLS configs, allowing old deployments to
be booted without a GRUB password. Add a service that fixes this setting
on the first boot after upgrade, if the aleph version corresponds to an
affected release.
This can be reverted after the next update barrier in all streams.
For https://github.com/coreos/fedora-coreos-tracker/issues/1333.
Since Fedora 36, the ifcfg-rh plugin implementing support for reading
network configuration files in the legacy ifcfg format has been split
out into its own RPM subpackage.
We can now remove our configuration workaround and instead ensure that
we do not ship this plugin at all.
See: https://github.com/coreos/fedora-coreos-config/pull/1991
This is a bigger hammer for the same issue as
170df10014
While I haven't been able to reproduce the problem in synthetic
testing with qemu, I am increasingly confident that there *are*
other udev rules running concurrently with our mounting attempt.
This was motivated by https://issues.redhat.com/browse/OCPBUGS-1505
where I specifically noticed that the ISO appears as a USB
device (which is really itself virtual and provided by the iLO),
and appears much later system bootup than I see with qemu, even
with emulated USB CD-ROM.
A possibly concrete culprit is `cdrom_id`, which on RHEL8 does
`open(O_EXCL)` on the CD-ROM device. For more information, see:
https://github.com/coreos/fedora-coreos-config/pull/1986#pullrequestreview-1120840529
Co-authored-by: Jonathan Lebon <jonathan@jlebon.com>
We've found the system can stall waiting for run-media-iso.mount
and apparently any operation seems to be effective at reviving
the system. Let's add a workaround that will kick in after 10 seconds
and try to revive the boot.
A lot more context over in
https://github.com/coreos/fedora-coreos-tracker/issues/1233#issuecomment-1238814171
Here's an example of this working:
```
[ 2.749303] systemd[1]: ignition-ostree-transposefs-restore.service - Ignition OSTree: Restore Partitions was skipped because of a failed condition check (ConditionPathIsDire
ctory=/run/ignition-ostree-transposefs).
[ 2.750547] ignition[591]: disks: disks passed
[ 2.750882] ignition[591]: Ignition finished successfully
[ 12.645801] kauditd_printk_skb: 19 callbacks suppressed
[ 12.645804] audit: type=1131 audit(1662520856.021:30): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=workaround-stalled-media-iso-mount comm="systemd" exe=
"/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 12.647774] bash[564]: warn: tracker issue workaround engaged for https://github.com/coreos/fedora-coreos-tracker/issues/1233
[ 12.648646] systemd[1]: workaround-stalled-media-iso-mount.service: Deactivated successfully.
[ 12.649281] systemd[1]: Mounting run-media-iso.mount - /run/media/iso...
```
The `tss` user/group entries have been fixed in `tpm2-tss-3.2.0-3`,
so this drops all existing workarounds.
Both are static entries and are already covered by kola tests.
Now that we no longer use LUKS, `cryptsetup.target` is no longer
activated. We do still need some ordering here though so that we don't
run too early when e.g. `virtio_blk` isn't even loaded yet.
Run after `coreos-gpt-setup.service` which also normally needs block
device access.
Instead of shipping the rootfs and bootfs using LUKS, we're pivoting to
verifying integrity using dm-verity instead.
Add `veritysetup` to the OS. Teach `transposefs` to transfer the
filesystem over dm-verity. Delete the root and boot hash partitions.
Delete some of the bits we previously added for LUKS enablement.
For more information, see:
https://github.com/coreos/coreos-assembler/pull/3039
Conceptually, I find it clearer to follow the Ignition config if the
`luks` section comes before the `filesystems` section since the latter
refers to things from the former and that's the order in which the
operations are carried out in practice (and in turn the block device
hierarchy).
Using `BOOTDEV` like this here will not be valid once we're able to
switch to relying on the boot link and drop
`ignition-ostree-secex-config.sh`. It's cleaner to just use the
`by-label/` symlink instead.
Unlike the default `\;` variant, this variant will actually cause `find`
to exit nonzero if the executed command failed. (Its main purpose is
actually to pass multiple items to the command rather than one at a
time, but we're more interested in the cleaner error-handling semantic.)
Just like we have `mount_and_restore_filesystem_by_label`, add a
`mount_and_save_filesystem_by_label` that can be used to save both the
rootfs and bootfs.
This translates all Fedora CoreOS system groups (i.e. those coming
from group manifest) into equivalent sysusers.d fragments.
We currently ship four kinds of system groups:
* basic groups coming from the `/etc/group` file in the `setup`
package (10-groups-basic.conf)
* the legacy GID for the `nobody` group (00-group-nobody.conf)
* extra groups with static GIDs that adhere to the Fedora
allocation table (10-groups-static-extra.conf)
* extra groups with static GIDs that are specific to CoreOS
(00-groups-coreos-static.conf)
These sysusers.d fragments are meant as a transitory measure to
help migrate the existing group entries from the current nss-altfiles
setup to a plain sysusers.d world.
If initrd services don't sequence themselves Before anything, it's
possible that they won't be started before initrd-cleanup.service isolates
to initrd-switch-root.target and kills them off. Add at least
Before=initrd.target to avoid this.
sudo wants them to be 440, not 600.
Also update the kola test to run "visudo -c" rather than checking the
perms ourselves. This not only checks perms, but also ensures that the
files are syntactically valid.
Fixes https://github.com/coreos/fedora-coreos-config/issues/1872.
Follow-up to #1819.
Tweak the description of `ignition-ostree-secex-config.service` for
consistency and drop an unnecessary conditional in the associated
script.
While looking at the boot space issue[1], I tested whether stripping
binaries saved anything. Currently, dracut tries to strip binaries if
`strip` (or `eu-strip`) are available, which isn't the case in FCOS.
Adding it as a test, it turns out stripping barely saves anything
because we've already split out debug symbols into separate RPMs, and
the remaining symbols don't take much space.
So let's just tell dracut to stop trying to opportunistically strip
anything to be consistent. This then squashes a message emitted about
`strip` not being found.
[1] https://github.com/coreos/fedora-coreos-tracker/issues/1247
BZ: https://bugzilla.redhat.com/show_bug.cgi?id=2075085
RHCOS may have (and has now) kernel with a known issue, which makes not
possible to use 'coreos-boot-disk' symlink in the config. Once we have
kernel >= 4.18.0-387.el8.s390x we should revert this commit
Signed-off-by: Nikita Dubrovskii <nikita@linux.ibm.com>
Only for systems running with Secure Execution. During firstboot this reencryptes filesystems
labeled 'boot' and 'root'.
Co-authored-by: Jonathan Lebon <jonathan@jlebon.com>
We don't ship Bluetooth support. Should squash:
dracut: 62bluetooth: Could not find any command of
'/usr/lib/bluetooth/bluetoothd /usr/libexec/bluetooth/bluetoothd'!
Follow-up to #1838.
We're not actually shipping any of these today; the way dracut
works is it looks for them and emits a loud warning if they're not
found.
This is purely to quiet build time warnings.
Closes: https://github.com/coreos/fedora-coreos-tracker/issues/1244
A lot going on for this simple service. See the tracker issue above
for more info, but briefly:
Anaconda has historically injected `quiet` into the kernel command
line in many cases, and this suppresses *both* kernel and systemd
output. On computers in general, but particularly many bare metal
servers, the serial console can be slow. This can cause reliability
issues.
However for servers, we usually *do* want to see informational
output when they boot. For example, today the kernel "mitigations"
information for hardware vulnerabilities is output.
This change is a compromise; we boot up at the kernel's default
verbosity level (which for Fedora and derivatives is the upstream 7),
but switch to 4 very early on in the real root. At that point, we've
gotten most of the boot time output, and our initramfs is not
extremely performance sensitive right now.
Also, we explicitly only *lower* the output log level, and only if
there isn't explicitly `debug` on the kernel command line.
We didn't have a strong ordering between the propagation service and the
Ignition `files` stage. We need to run after it so that we only
propagate the multipath config if the Ignition config didn't create one.
In practice currently the propagation service runs before, which didn't
cause any issues because Ignition can still just clobber what we
propagated. But it could become racier in the future, and it's confusing
because the logic we run and messages we log assume the `files` stage
ran.
The path used there is incorrect. The dropins dir is
`/etc/multipath/conf.d`:
686797b7d0/Makefile.inc (L100)
But anyway, we don't actually need to create that directory; it's
already in the base image. So just drop it.
We were saying "no initramfs automatic multipath configuration to
propagate" even when there was an initramfs config but the file already
existed in the real root. Let's split out those no op conditions more
and log clearer explicit messages.
This should be a purely cosmetic change. Prompted by looking at logging
from this service to debug a multipath-related issue.
After ignition's stages partition table could be modified, and therefore
kernel may have an old picture of disks and partitions. During verification
we force kernel to reread the partition table for each disk and call udevsettle,
which in turn leads to modification of "/dev/disk/by-*/*" symlinks and systemd
may send SIGTERM to services with "Require=dev-disk-by\x2dlabel-boot.device".
To avoid that this checks for the unique fs labeled 'boot' after ignition's
disks stage was finished but before mounting stage.
Issue: https://github.com/coreos/fedora-coreos-tracker/issues/1105
Signed-off-by: Nikita Dubrovskii <nikita@linux.ibm.com>
It's a stub service that only parses a config file, and we never
adequately documented the config file. We have no current plans to
finish implementing the service, so let's drop it.
Fixes https://github.com/coreos/fedora-coreos-tracker/issues/770.
A generator can be run multiple times. This symlink is a bit of an abuse
of systemd generators because it writes things outside of the expected
output directories. So we need to make sure that we're idempotent.
Upstream NM started adding a readme-ifcfg-rh.txt file to explain
and encourage people to use keyfiles instead. This file doesn't
do anything, but breaks our logic that tries to determine if the
user provided networking config to the machine.
This file was added upstream in
https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/96d7362
Ignition 2.14.0 adds ignition-delete-config.service, which deletes
Ignition configs from VMware and VirtualBox on first boot. Add
coreos-ignition-delete-config.service to do the same thing on existing
machines on upgrade, using a stamp file in /var/lib to avoid multiple
runs.
Add a drop-in for ignition-delete-config.service that creates a stamp
file in /run, and then chain from that stamp to the long-term stamp in
/var/lib, ensuring that we don't delete configs twice on newly-
provisioned machines. The upstream service can't create the stamp
directly in /var/lib because it runs before /var is mounted.
Prevent coreos-ignition-delete-config.service from running if
ignition-delete-config.service is masked, ensuring that the mask operation
documented upstream prevents the config from ever being deleted, as
intended.
We can remove this after the next barrier release in FCOS and barrier
equivalent in RHCOS.
This service was used for people who wanted to stay on
iptables-legacy. Now that there has been a barrier release
it is no longer needed.
Also move the iptables-nft enablement into fedora-coreos-base
and delete related tests.
The WALinuxAgent-udev rpm now installs the udev rules into the
initramfs [1] so we can drop this glue code.
This reverts commit 8c07b73914.
[1] 521b67bc85
The reboot and consequently the timeout masked valuable debug
information. The reboot also caused some cascading errors due to the
fact that the system would try and run as if all required dependencies
were satisfied during the first boot.
Closes https://github.com/coreos/fedora-coreos-tracker/issues/928.
When a user tries to boot a live system with an appended rootfs without
enough RAM, the kernel will unpack the initramfs and rootfs until it
runs out of memory, and then continue booting with a partial OS image.
This causes random confusing boot errors, such as a service failing
because its binary is missing, or the rootfs squashfs failing to mount
with some error that looks like filesystem corruption.
Make this failure less confusing by having timeout.sh try to write a small
file, and if that fails, tell the user that they need more RAM. Suppress
reporting of unit failures in this case, since the unit logs will just
muddy the issue.
This works reliably when the rootfs image can't be unpacked, i.e. when
the system has a moderate amount of RAM. It doesn't work reliably when
we're so low on RAM that the initramfs image can't be unpacked (because
the compressed initramfs and rootfs images have used up our RAM); in that
case we might be missing important binaries/libraries and boot can fail
at any point. It would be possible to construct a more robust detection
mechanism, e.g. by running a statically-linked program that's unpacked
first. But this can't be completely robust (/sbin/init might not have
been unpacked, so we might not boot at all) and doesn't seem worth it for
now.
This code will also trigger if we run out of RAM in the
coreos.live.rootfs_url case, which is nice but less important, since that
case produces a clear error during rootfs fetch.
Closes https://github.com/coreos/fedora-coreos-tracker/issues/1055.
In Ignition, we've two config validation checks, the one after
fetching a config and the second after merging configs. Sometimes,
a warning goes away after merging, however, it's possible that a
warning appears in case merging creates a contradiction between two
fields. So this workflow eventually sends duplicate warnings in journal
entries. Hence, we need to avoid displaying duplicate Ignition warnings
on the console. Also, Ignition warnings are highlighted with yelllow.
Fixes https://github.com/coreos/fedora-coreos-tracker/issues/1125
This change displays all Ignition warnings on the console. Also, renamed
the existing Ignition service to make it more generalized for handling
status information about the Ignition run.
The associated script only exists in the initrd. But also, we only need
the symlink in the initrd anyway.
Fixes 6d7b9ad5 ("overlay.d: add udev rule for creating stable symlink to
boot disk").
Fixes https://github.com/openshift/os/issues/755.
It appears that on debug kernels the "/dev/disk/by-label/$label" device *can* not
appear at all (some sort of race condition) and thus calling `realpath` on it will
fail. Let's just make the call to `realpath` not be fatal so we can workaround this
issue that has been around in the kernel for some time.
Closes https://github.com/coreos/fedora-coreos-tracker/issues/1092
`nm-initrd-generator` will take into account the `BOOTIF=` kernel
argument. Let's account for that in our `propagate_initramfs_networking()`
code so don't propagate networking config if `BOOTIF` is the only
networking config.
Fixes https://github.com/coreos/fedora-coreos-tracker/issues/1048
As discussed in
https://github.com/coreos/fedora-coreos-tracker/issues/1055, the
rootfs_url karg is preferred over initrd concatenation due to its lower
memory footprint and generally better performance. We originally chose to
disallow its use with TFTP to reduce the feature matrix, but that forces
users without HTTP servers to fall back to concatenation. Enable TFTP
so rootfs_url can be used in all environments.
In #829, we changed multipath config propagation to happen on subsequent
boots to handle upgrading nodes. Back then, we didn't yet support first
boot multipath, so not propagating on first boot didn't matter anyway.
But with #1011 we now support multipath on first boot. So we need to
adapt this so we always propagate, including on first boot.
Creates '/dev/disk/coreos-boot-disk' symlink to boot disk for
use by Ignition config, therefore 'storage.disks' section
could be defined without hardcoding /dev/sda, /dev/vda, etc.
Issue: https://github.com/coreos/fedora-coreos-tracker/issues/759
Signed-off-by: Nikita Dubrovskii <nikita@linux.ibm.com>
Tell coreos-assembler to not automatically add the overlays to the final
manifest. Add references to the auto-generated OSTree layers in our
lower-level manifests where most appropriate.
Prep for using this to make an overlay only active on the `next-devel`
stream.
Requires: https://github.com/coreos/coreos-assembler/pull/2641
On re-provisioning a FCOS instance, Ignition will append notices like
"wrote ssh authorized keys file for user: %s"
to "/etc/issue.d/30_ssh_authorized_keys.issue".
The problem is that the journal from the previous install remains available
and multiple entries are displayed when the journal is searched.
The problem is fixed by limiting the journal output to the most recent boot.
Fixes https://github.com/coreos/fedora-coreos-tracker/issues/1033
This adds support for copying NetworkManager keyfiles from
`/etc/coreos-firstboot-network`. It's similar in functionality to the
existing support for `/boot/coreos-firstboot-network`, but is intended
for the live path (ISO or PXE boot) where there is no boot partition.
Users won't interact directly with this; instead, it's expected that
`coreos-installer` will provide high-level commands to customize the ISO
or PXE artifacts to embed keyfiles via the existing layered initrd
mechanisms.
Closes: https://github.com/coreos/fedora-coreos-tracker/issues/841
Mostly consists of swapping out `ignition-diskful.target` for
`ignition-complete.target` and making the boot partition logic
conditional on `! is-live-image`.
Prep for supporting an alternate source directory in the live case.
We already run after `coreos-gpt-setup.service` which has those same
requirements. It was good to be explicit and specify it again, but in
the future this unit will also support running in live images, where we
can't require `/dev/disk/by-label/boot` to exist.
From the last paragraph in:
https://docs.microsoft.com/en-us/azure/virtual-machines/linux/time-sync#check-for-ptp-clock-source
> In Linux VMs with Accelerated Networking enabled, you may see multiple
> PTP devices listed because the Mellanox mlx5 driver also creates a
> /dev/ptp device. Because the initialization order can be different
> each time Linux boots, the PTP device corresponding to the Azure host
> might be /dev/ptp0 or it might be /dev/ptp1, which makes it difficult
> to configure chronyd with the correct clock source. To solve this
> problem, the most recent Linux images have a udev rule that creates
> the symlink /dev/ptp_hyperv to whichever /dev/ptp entry corresponds to
> the Azure host. Chrony should be configured to use this symlink
> instead of /dev/ptp0 or /dev/ptp1.
See: https://github.com/coreos/fedora-coreos-tracker/issues/1031
In the multipath case, we're mounting
`/dev/disk/by-label/dm-mpath-root`, not `by-label/root`.
Also order against `sysroot.mount` because in the general case, there's
actually nothing that prevents the restamping to run at the same time as
the mount AFAICT if the `root` karg points to some other device path we
didn't predict.
See: https://github.com/openshift/os/issues/683 (which hopefully this
fixes).
In a live system, we can't set kargs in an Ignition config because we
don't control the bootloader settings. If the config specifies kargs
anyway, we currently fail with "/dev/disk/by-label/boot: not a block
device", which isn't very clear.
In the live case, use the new rdcore kargs --current option to perform a
dry run on /proc/cmdline, and if it doesn't create the stamp file, we
don't need to do anything and can succeed anyway. Otherwise, fail with a
clearer error message.
Fixes https://github.com/coreos/fedora-coreos-tracker/issues/917.
Upstream NM changed the behavior of `ip=dhcp,dhcp6` to also wait
20 seconds for DHCPv6. We don't want to require all users by default
to wait for DHCPv6 especially when it's doubtful most users even have
a ipv6 DHCP server. We can stick with our current behavior by utilizing
`ip=auto` instead.
Fixes https://github.com/coreos/fedora-coreos-tracker/issues/1000
The new `boot` karg will only show up on the second boot and onwards. So
for the first boot, let's persist it in `/run` to tell
`coreos-boot-mount-generator` the UUID to use.
This is analogous to `systemd-fstab-generator` parsing `root=` kargs.
There is a precedent for this in the FIPS module. In fact, we lift code
from there to make sure we're API compatible.
The goal of supporting a `boot` karg is to eliminate possible races for
the `by-label/boot` symlink in the real root if multiple exist.
Part of: https://github.com/coreos/fedora-coreos-tracker/issues/976
kdump now writes the temporary initrd in `/var/lib/kdump` and the
SELinux policy changes have landed in Fedora 34. Thus remove the
workaround keeping `/boot` read write when kdump support is requested.
This reverts commit 3167d834e1.
Fixescoreos/fedora-coreos-tracker#977
This change helps to address the problem of displaying a warning
about running Ignition twice while reprovisioning a bare metal
CoreOS instance with a persistent /var partition.
This will allow us to conditionalize it based on the major Fedora
version when the fix starts landing in Fedora Linux (starting with
Rawhide). This also moves the workaround into a new dracut module
(36coreos-multipath-fix) and out of the old module (35coreos-multipath)
to make it easier to add from postprocess.
This will allow us to conditionalize it based on the major Fedora
version. The fix has currently landed in rawhide but not F35 and
we want to continue to execute the check everywhere.
We do `udevadm settle` in a few places to e.g. wait for symlinks to
update based on whatever operation we just did. But if `udevadm settle`
fails, we shouldn't fail the boot for it. It could be failing on some
completely unrelated thing (since it waits for all events).
Ideally in those scripts, we'd wait only for the specific events that we
care about. But even so, we should just opportunistically keep booting.
As a plus, even if we end up failing further down, we'll get a clearer
error.
Related: https://bugzilla.redhat.com/show_bug.cgi?id=2009662
This adds a dropin for 'multipathd.socket' adding the same start
conditions that are present on the service unit. It is a temporary
workaround that can be removed once the packaged one is fixed.
Ref: https://bugzilla.redhat.com/show_bug.cgi?id=2008098
This is a temporary workaround to remove a problematic stop command
from `multipathd.service`, until the already-merged proper fix gets
released in dracut.
RHEL 8 has systemd 239, which has
https://github.com/systemd/systemd/issues/9374. As a result,
"journalctl -f ... | head -1" hangs because journalctl doesn't properly
exit. Rewrite the check to call loginctl once per second until it finds
a session. This approach is uglier, but the service only runs in the live
ISO, on QEMU, when there's no Ignition config, and when kola's virtio port
is open, so the minimal extra overhead shouldn't affect production
systems.
Fixes the kola testiso iso-live-login scenario in RHCOS.
Without --force, the boot process races with service shutdown, leaving
enough time for the disks stage to start. With --force, systemd
immediately kills processes, unmounts filesystems, and reboots.
Fixes https://github.com/coreos/fedora-coreos-tracker/issues/896.
Some users sometimes may not realize that they're using a pre-booted
version of a CoreOS image. This makes things confusing because they then
don't understand why the Ignition config wasn't applied.
There's no way to consistently detect this, but at least we can print an
informational message about (1) when Ignition ran, and (2) how many
boots ago that was.
This enhances the Ignition issue we already write for whether a
user config was provided rather than creating a separate one.
Related: https://bugzilla.redhat.com/show_bug.cgi?id=1977949
Related: https://github.com/coreos/ignition/issues/1214
For some reason in rawhide this is no longer getting pulled in to the
initramfs. Since we require the use of it here then let's be explicit
and pull it in.
Otherwise, we'll end up racing with `initrd-cleanup.service` which wants
to kill everything. It has `After=initrd.target` and we do have
`Before=initrd.target`, but that's not being respected, we think because
`initrd-parse-etc.service` does an explicit `systemctl start` on it.
Anyway, we need to dig more into this, but for now this will unblock us.
We've seen races with ignition-kargs.service, which accesses /boot rw.
Let's introduce some ordering here. Need to use `Before` because otherwise
we get a systemd ordering cycle.
Fixes: https://github.com/coreos/fedora-coreos-tracker/issues/883
Upstream dracut updated NM to run as a systemd service
(with full dbus support) in the initrd in [1]. Adapt our
systemd units to handle this case.
This should still work fine for RHCOS because we still have
`Before=dracut-initqueue.service`, which can be dropped when
everyone is on dracut 0.54+.
Fixes: https://github.com/coreos/fedora-coreos-tracker/issues/842
Upstream Ignition can't assume is-live-image exists, because the distro
may not have provided it. But we know the coreos-live module is providing
it, so drop the check.
We originally did this in #326 because we wanted to support booting the
live ISO without networking. This was solved on the initramfs side by
the conditional networking work (#426). But for the real root, this was
still useful because if booting the ISO interactively on a system
without any network, or a non-DHCP network, we didn't want the user to
have to wait until the service timed out before getting a shell.
The core issue however is that we're requesting `network-online.target`
at all. It's an "active unit" which means that it's only pulled in the
transaction, possibly delaying boot, if another systemd unit needs it.
And ideally, no service would need it as per:
https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/
In our case, this unit was fedora-coreos-pinger. We drop that
requirement here:
https://github.com/coreos/fedora-coreos-pinger/pull/41
With that, we no longer pull in `network-online.target` and so no longer
delay reaching the console even if NetworkManager isn't able to get an
active connection for whatever reason. This matches how it works on
traditional Fedora as well.
Having a short timeout actually also had a counterproductive effect in
the automated install case. There, `coreos-installer.service` does pull
in `network-online.target` (which with
https://github.com/coreos/coreos-installer/pull/565 we could consider
dropping as advised by systemd, though we probably should bump the
number of retries some more in that case), but because of the short
timeout, we genuinely may not yet have the network fully up before we
run (see https://bugzilla.redhat.com/show_bug.cgi?id=1967483).
I personally don't like including the .sh extension on installed
executable scripts because it "leaks" that it's a shell script, whereas
all we should care about is that it's an executable.
This is mostly a mechanical move of ignition-firstboot-complete.service
to this repo. The only differences are tweaking the service unit name
and the Description and Documentation keys.
Prompted by tweaks we'd like to do here, and possibly eventually moving
it to the initramfs fully (merging it with coreos-boot-edit.service).
The renaming also allows us to loosen the order of operation in which
this is done, so that we can get this in to enable the new unit, while
keeping the old unit disabled, without necessarily waiting for the
original unit file itself to drop out of the `ignition` package.
For background, see: https://github.com/coreos/ignition/issues/1125
We were relying on the zram kmod already being in the initramfs so far,
though that assumption for whatever reason is now incorrect in rawhide,
causing rootfs reprovisioning related tests to fail.
Anyway, we should be more explicit here about what our requirements are.
We can't `mv` out of the module directory since `/usr` is mounted
read-only in the dracut container when rpm-ostree runs it.
Follow up to #938.
Reported-by: Luca BRUNO <luca.bruno@coreos.com>
We're hitting issues in some cases where users are seeing the
`tune2fs -U random` on the /boot filesystem fail even though it's
ran directly after an `e2fsck`. To prevent needing a filesystem
check at all let's store the metadata checksum seed in the superblock
which will allow us to change the UUID without needing any filesystem
checks.
This was not possible before but we recently got a grub fix so that
having this filesystem feature enabled wouldn't cause grub2 to throw
an error.
- https://lists.gnu.org/archive/html/grub-devel/2021-06/msg00031.html
Fixes: https://github.com/coreos/fedora-coreos-tracker/issues/735
This karg will override the check to see if Networking Configuration is
in the real root (i.e. written by Ignition most likely) and force
propagation of initramfs networking anyway. It should only be used in
cases where you know exactly what you are doing. For example, in the
problem described in https://bugzilla.redhat.com/show_bug.cgi?id=1958930
the networking configs didn't conflict at all, thus this could be used
safely. This option will help users who get stuck like they were in that
bug to get unstuck easily and then change their strategy later.
Fixes: https://github.com/coreos/fedora-coreos-tracker/issues/853
As we've learned in https://bugzilla.redhat.com/show_bug.cgi?id=1954025,
we can't assume that we can use any individual path before multipathd
unifies them. This means that it's not correct to turn it on from the
second boot onwards only, which is the current approach.
So we need to support multipath at first boot, and further, we need to
delay all I/O to the boot disk to *after* multipathd takes ownership.
This patch does this by introducing a generator and a target. We need to
use a target because `After=multipathd.service` is not enough to ensure
that multipathd finished setting up the devices. The target explicitly
waits for the multipathed boot device to appear. (Note though that we
*can't* assume that root is also directly on top of the multipath device
because of LUKS-on-root.)
This creates an awkward UX, which is that multipath is the only root
setup option which is *not* driven by Ignition but instead set up behind
its back.
This will be somewhat better once Ignition supports kernel arguments,
because the `rd.multipath` karg can be fed that way. But long-term, we
should consider teaching Ignition to configure multipath devices. This
would fix the configuration problem (right now, we only support
first-booting with default configs), and would free users from adding
the necessary kargs, which would be done by the rootmap code as usual.
But still, even in that world, there's a gap where the Ignition config
could be on the boot device, in which case multipathd would need to be
turned on beforehand.
Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1954025
This will check if a system is still using cgroupsv1 and generate a
message to be printed as part of CLHM.
Co-authored-by: Dusty Mabe <dusty@dustymabe.com>
If root is on multipath (which is today for CoreOS always `rd.multipath=default`)
then we *know* we must use it for `/boot`. We're not going to
support "tearing" where `/boot` is on a non-mpath device but
`/` is on mpath.
The current code is I believe racy because at the time the generator
runs (and systemd generators run *early*), we're querying the
"current" properties of the device at
`/dev/disk/by-label/boot`. But multipathd could still be in
the process of setting up and replacing the target of that
symlink. This can cause systemd to tear down and reinitialize
the mount, causing races.
https://bugzilla.redhat.com/show_bug.cgi?id=1944660
We're currently gating on `ENV{DM_SUSPENDED}=="Active"` but
`10-dm.rules` does:
```
ENV{DM_SUSPENDED}=="Active", ENV{DM_SUSPENDED}="0"
ENV{DM_SUSPENDED}=="Suspended", ENV{DM_SUSPENDED}="1"
```
So what I think is happening here is that our rule happens to
run before that kicks in, so we make the links once, but not
thereafter.
Change the condition to match what `13-dm-disk.rules` from
LVM is doing.
Also slightly reorder the code and add some comments for extra clarity.
Even though it's on a comment line, because it's in a heredoc, bash does
try to execute this. This fails on FCOS thankfully because there is no
`man` on FCOS, but it still logs an error message. (And... any derived
system which does ship `man`, I think this actually would dump the
manpage into the unit.)
We want it to run in the multipath case, in which users may provide a
`root=` karg. So drop that condition.
This service is already queued against `ignition-diskful.target` so it
only runs on first boot. It also already correctly does nothing if
somehow the UUID was already changed.
In the case of multipath,
`ignition-ostree-mount-firstboot-sysroot.service` won't kick in because
the `root=` karg is already present on first boot. But we still need to
grow the partition and filesystem in that case.
What we really mean here is that the sysroot should already be mounted
before we growfs. Drop the `Requires=` and add `sysroot.mount` to the
list of `After=` so that we cover the case where the sysroot is mounted
via the mount unit generated via systemd-fstab-generator from the
`root=` karg.
Anaconda is long gone now. We do want this service to always run on
first boot, regardless of whether there's a `root=` karg or not. In the
case of multipath, a `root=` karg is expected on first boot.
Since I'm going through all of the TODO f34 items now
let's update this one to remind us in the f35 cycle if
we haven't already removed it before then.
In f34+, we're hitting an SELinux denial from c-l-h-m trying to do
`agetty --reload` and causing `/run/agetty.reload` to be created with
the wrong label. This then prevents agetty from adding an inotify watch
to know when to reload the prompt to display new information. For more
details, see:
https://github.com/coreos/fedora-coreos-config/pull/859#issuecomment-783713383
This is tracked at https://bugzilla.redhat.com/show_bug.cgi?id=1932053.
With this workaround, we create the file up front in the initrd so that
it gets relabeled by systemd on switchroot and thus will already exists
with the right label well before c-l-h-m or anything else tries to
`agetty --reload`.
Comments in systemd services at the end of lines cause unnecessary
warning messges:
Failed to add dependency on on, ignoring: Invalid argument
Failed to add dependency on dracut, ignoring: Invalid argument
Failed to add dependency on 053+, ignoring: Invalid argument
The services would still start so this is a cosmetic change only.
We don't know how long it will take for networking to come up, so let's
match Ignition's semantic of retrying to fetch remote resources forever.
`curl` doesn't natively have support for this, so wrap it in a `while`
loop. This also works around the fact that el8's `curl` doesn't support
`--retry-all-errors`.
This was previously added in #929.
In dracut 053+ dracut starting running NetworkManager from a systemd
unit if systemd was used to start the system [1]. Let's adapt for that
by ordering our units before `nm-run.service`. We'll also keep the
ordering on dracut-initqueue.service for now for backwards compatibility
and it's harmless to keep around.
[1] https://github.com/dracutdevs/dracut/commit/c17c5b7
NTP servers provided by AWS and GCP implement a "leap smear" to
hide leap seconds from clients. When chronyd is configured with
leap-smearing servers, the leapsectz directive must be disabled to
avoid the kernel making a wrong correction of the clock at the time of
the leap second.
Disable any leapsectz directives coming from the default configuration
and enable it only in the Azure case (the PTP clock doesn't provide
information about leap seconds).
Right now, `kdump` fails to generate its initramfs because
it cannot place it next to our default initramfs in
`/boot/ostree`, which is read-only.
This is a workaround to get `kdump` working again until
it gains the ability to place its generated initramfs
in a different location (e.g. `/var/lib/kdump`).
xref: https://bugzilla.redhat.com/show_bug.cgi?id=1918493
This should help us get around some race conditions on startup
where we've seen curl exit with "No route to host" errors. This
most likely happens because the Networking in the kernel is still
being brought up (seen more in complex networking scenarios) and
curl hits that error the first time it tries. Since "No route to
host" isn't considered retryable, curl exits.
Instead let's just verify we can get to the remote at all in an
initial `curl --head` call. In this one we'll use --retry-all-errors
so that we will retry all errors. Once the connectivity to the remote
is verified then it should be safe to start downloading.
Right now, specifying `mountOptions` for the rootfs in the Ignition
config is ignored. To fix that, we simply need to have
`coreos-rootflags` inspect the Ignition config if the rootfs was
reprovisioned and extract the field.
This then takes effect on first boot (via
`ignition-ostree-mount-sysroot.sh` calling out to it) and subsequent
boots (via `rdcore rootmap` calling out to `coreos-rootflags`).
Closes: https://github.com/coreos/fedora-coreos-config/issues/805
By default, we use `prjquota` for the rootfs for container orchestrators
to monitor and set drive space limits.
However, with the added support for rootfs reprovisioning, we made this
conditional on the rootfs not being reprovisioned, with the assumption
that you can just set whatever options you'd like instead. Though
actually doing that is really awkward right now, and it requires
surfacing prjquota in user documentations (see
https://bugzilla.redhat.com/show_bug.cgi?id=1940704#c2).
Since AFAICT `prjquota` doesn't actually have any significant overhead,
let's just simplify the messaging to: we *always* enable prjquota on XFS
root filesystems. Users who want to override this can fallback to
`rpm-ostree kargs` (and eventually once we have
https://github.com/coreos/fedora-coreos-config/issues/805, to
`mountOptions: []`).
Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1940704
Since we've created a new filesystem anyway.
This implicitly also works around the service racing against the
`by-label/root` symlink not being up to date after reprovisioning (which
`ignition-ostree-transposefs-restore.service` now intercepts and fixes).
We need to support in-place LUKS root reprovisioning. This is pretty
straightforward, modulo the terrible clevis hack used to resize the
container. We'll want to circle back and clean this up once there's a
better interface to do this (but at least until then, we have tests
which cover this).
Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1934174
The semantic we want to capture for auto-growing the rootfs is that we
want it to kick in as long as either no rootfs reprovisioning happened,
or it was in-place (i.e. reused the same partition).
We detected this by checking that the block device (i.e. partition)
backing the rootfs and its size didn't change. But this breaks with
in-place LUKS reprovisioning because the immediate block device does
change (to the LUKS container), even though it's ultimately still backed
by the same untouched partition.
Fix this by (1) verifying that the rootfs is still ultimately backed by
the same partition, and (2) that partition has not been resized.
This doesn't actually add support for growing LUKS containers yet, but
it's prep for that.
Now that coreos-growpart is only in the initramfs, its scope is
essentially reduced to handling the `/sysroot`. Let's formalize this by
renaming the script to match the service which calls it and dropping
support for passing the target path as an argument.
Also freshen up the comments at the top of the file.
Previously we pinned clevis to v15 in order to work around
https://github.com/latchset/clevis/pull/295.
But overrides are not set up in the tagger for mechanical streams, so we
can't easily pin in `branched` and `rawhide`. Let's just unpin and
temporarily ship `seq` ourselves to work around this. Put that temporary
hack in `40ignition-ostree` since that's the closest module conceptually
to clevis.
I commonly do `cosa run` to test something related to rootfs
reprovisioning and forget to allocate more RAM. The failure mode is a
massive spam of "Input/output error" messages from `cp` unable to copy
more files into RAM (and there doesn't seem to be a way to have it fail
on first error; yet another reason we should rewrite this into !shell).
Instead, just sanity-check that we have at least 1G of RAM and error out
with a nice and clear message if not.
Let's keep 40-coreos.preset to be only `enable` stanzas for
"our stuff" overall.
Create a separate `40-coreos-systemd.preset` which only
relates to systemd things because it really is a distinct
problem domain.
It requires swap, which we don't turn on by default. (It also requires
cgroups v2, which we currently don't turn on by default but are aiming
to in the near future.)
Because multipath support now works on the second boot, the propagation
code in `coreos-teardown-network.service` didn't work. Split that code
out into a separate dracut module and have it run on subsequent boots.
Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1920571
We can assume now that we always have coreos-relabel and drop the
fallback to relabeling via tmpfiles. This dates back from when this code
lived in ignition-dracut and it felt odd to rely on coreos-relabel.
I'm trying to hack on systemd using `cosa build-fast`, which
doesn't run through the `remove-from-packages` path which we
use to disable systemd-firstboot.
Let's also add this cheap mechanism to disable it even if it's
installed.
Noticed that `ignition-ostree-transposefs-save.service` didn't break the
boot immediately after an error while debugging logs with @mike-nguyen.
Let's cargo-cult the `OnFailure` bits into those services too.
I think we may still be suffering from
https://github.com/systemd/systemd/issues/14142 in RHEL systemd.
The model we're moving towards is one where multipath enablement is
configured via Ignition and so only takes effect on the next boot:
https://github.com/openshift/os/pull/484#issuecomment-760454177
Therefore, we don't need to worry here about support for growing a
multipathed root partition. This simplifies a big chunk of the script.
ostree has had support for leaving `/boot` mounted read-only
for a long time: https://github.com/ostreedev/ostree/pull/1767
(And then later extended to `/sysroot`)
Particularly for CoreOS, only a few things should be touching
`/boot`, and we control all of them. Those projects should
create a new mount namespace and remount these partitions
writable just while they need it.
The main thing we're accomplishing here is making the system
more resilient against accidental damage from a sysadmin
root shell as well as configuration management tools like
Puppet/Ansible. None of those should be directly manipulating
files on these partitions, they should go through the API
of one of our projects (e.g. `rpm-ostree kargs`, `bootupctl`) etc.
While we're here, also add `nodev,nosuid` because some
OS hardening scanners like to see this. IMO it's of minimal
value, but hey, might as well.
We want the Azure udev rules present in the initrd, so that Ignition
is able to detect the disks and act on them.
If the udev rules end up being installed into the initramfs as part of
the WALinuxAgent-udev package, the dracut module should be removed.
See https://bugzilla.redhat.com/show_bug.cgi?id=1909287
BZ#1748432 was filed while working on #159 and the udev rules were
split out into a separate package as part of F32. Let's stop carrying
the rules in the overlay and use the provided package now.
Because dracut directly sources `module-setup.sh` scripts and isn't
compatible with `set -e`, we can't just turn it on in our modules.
Instead, let's just manually add `|| exit 1` in all the calls to
`systemctl add-{requires,wants}` so that we catch regressions like
https://github.com/coreos/fedora-coreos-config/issues/799 at build time
in the future.
Unlike `ln -sf`, `systemctl add-{requires,wants}` wants to verify that
the target and member units all exist. Because we had some dracut
modules which ran earlier than the `30ignition` module, we would lose
the service enablements which hook into e.g. `ignition-complete.target`
and related units.
Let's simply rename our modules so that they're ordered after
`30ignition`. While we're here, let's consistently prefix them by
`coreos-`. So overall, the changes are:
- `30ignition-coreos` -> `35coreos-ignition`
- `20live` -> `35coreos-live`
- `15coreos-network` -> `35coreos-network`
I've verified that there are now no "Failed to add dependency on unit"
error messages at build-time. (The next patch enforces this more
strictly.)
Fixes: #778Closes: #799
If the firmware writes to an individual replica, the RAID will
desynchronize. Linux md will return reads from either replica, and then
any dependent writes could corrupt the filesystem. To prevent this,
fcct will no longer put the ESP on a RAID; instead we create multiple
independent filesystems and copy the contents to each. This is okay
because bootupd and fwupd should be the only things that care about the
contents of the ESP.
Don't worry too much about backward compatibility because we're making
this change soon after the functionality landed, and before it was
documented. For the record, old configs will fail on new systems
(because the partitions will be RAID members) but new configs will
skip copying /boot on old systems (because there's no filesystem labeled
"EFI-SYSTEM").
Drop issue file snippets directly into `/etc/issue.d`, which
is ready by `agetty`.
Following https://github.com/coreos/console-login-helper-messages/pull/91,
new versions of console-login-helper-messages will no longer
support users dropping files into its private directories
(e.g. `/run/console-login-helper-messages/issue.d`).
By default, upon inserting the zram module, it'll automatically create
one zram device. Because we don't know if we're the first ones or not,
it's not safe to make use of that device.
Instead, tell it to not do that so that if we *are* the first ones to
load the module, we don't end up with an extra zram device. If we
aren't, it's a no-op.
When we do `mv -t $mountpoint`, everything we move *into* the
mountpoints gets its SELinux label preserved, but the root of the
filesystem itself (i.e. `$mountpoint`) doesn't get relabeled.
This is something that `rsync` would've done for us if we had used that
instead of `mv`. (Though I recall using `rsync` was also slower than
`mv` in my tests.) Anyway, I think eventually we want to rewrite some of
this stuff into rdcore.
Sometimes, the `/dev/disk/by-label/root` symlink still points to the
original block device rather than the latest one. This then leads to
e.g. us mounting the wrong device and get errors like:
```
[ 54.814061] ignition-ostree-transposefs[938]: mount: /sysroot: unknown filesystem type 'crypto_LUKS'.
```
It's not that we're not waiting long enough for udev to update them.
It's more like udev misses those events entirely. Filed:
https://bugzilla.redhat.com/show_bug.cgi?id=1908780
Since we know what the underlying device should be (from the Ignition
config), let's just detect those cases and do a `udevadm trigger`.
Now that the upstream patch for chrony change has merged (https://src.fedoraproject.org/rpms/chrony/pull-request/3)
and available in F-33, we can safely remove this code, which was a temporary fix to allow chrony propagate NTP servers
via DHCP.
Use `systemctl`'s built-in way of adding `Wants=` and `Requires=`
dependencies, since this is the recommended way of adding dependencies
and we can do so with a single command.
If the config we are going to copy into the real root is just the
defaults let's just not copy any configuration in because
NetworkManager's "unconfigured" behavior is good enough.
This has side benefits of making the coreos-installer --copy-network
workflow slightly more intuitive.
Fixes: https://github.com/coreos/fedora-coreos-tracker/issues/696
Serialize transposefs-restore and firstboot-uuid, since they both want
to mutate the root filesystem. Also have firstboot-uuid wait for udev to
settle, since closing a device node for write can cause udevd to recreate
/dev/disk links.
We're sometimes hitting a flake in the RHCOS CI for LUKS-related tests:
ignition-ostree-dracut-rootfs[909]: mount: /sysroot: unknown filesystem type 'crypto_LUKS'.
systemd[1]: Failed to start Ignition OSTree: restore rootfs.
I think what may be happening here is that the `/dev/disk/by-label/root`
symlink is still pointing at the original partition which now hosts the
LUKS device and didn't actually get updated to point to the newly
created filesystem on top of it.
The `disks` stage does do a `udevadm trigger` on exit, though I'm not
sure how foolproof that is.
What we could do eventually to work around this is get the `device` from
the Ignition config and just mount that instead. Though to confirm first
this is what's happening, let's make the mount operations explicitly
call out what devices they're mounting.
timeout.sh was getting confused by systemctl output that looked like this:
● ignition-disks.service loaded failed failed Ignition (disks)
The result was output like this:
Displaying logs from failed units: ●
Invalid unit name "●" escaped as "\xe2\x97\x8f" (maybe you should use systemd-escape?).
-- Logs begin at Fri 2020-12-04 04:09:20 UTC, end at Fri 2020-12-04 04:09:33 UTC. --
-- No entries --
Press Enter for emergency shell or wait 5 minutes for reboot.
If the Ignition config creates any PowerPC PReP partitions, save the
existing PReP contents before ignition-disks (in case the partition is
overwritten) and copy them to the new partitions afterward. We don't
require the config to create a PReP RAID array because the OS doesn't
use or modify the PReP partition at runtime.
If the Ignition config creates any BIOS Boot partitions, save the
existing BIOS-BOOT contents and the corresponding boot sector before
ignition-disks (in case the partition is overwritten) and copy them to
the new partitions (and corresponding disks) afterward. Also verify that
the offset of the new BIOS-BOOT partitions matches the old one, since
otherwise GRUB will fail when it tries to use them. We don't require the
config to create a BIOS-BOOT RAID array because the OS doesn't use or
modify the BIOS-BOOT partition at runtime.
We'll be generalizing the rootfs save/restore code to support saving
and restoring other partitions. Generalize the name to "transposefs"
and move the saved rootfs data to /run/ignition-ostree-transposefs/root.
This enables networking in initramfs on Azure, via a platform
condition.
While the Ignition configuration may be self-contained locally
on the config-drive, other bits referenced by the provisioning
flow (e.g. Afterburn hostname and boot check-in) always require
reaching remote endpoints.
This reverts commit 56b0ceba8b.
The intention was to remove this service once the streams all had it and
barriers had been created in the update graph. It should be safe to
remove now.
For simplicity, stop dropping stamp file since it is very rare that
we would benefit from it. coreos-boot-edit.service will not
use the existence of the stamp file as a triggering condition and
will always be started.
