Drop our dependency on setuptools and write our own PEP 517 build
backend. This turns out to be relatively straightforward — only ~100
lines, and avoids the workarounds we had to do to prevent setuptools
from trying to write to the source tree. It's also dramatically
faster.
While we're at it, we can now fix a long-standing nag: we now have a
proper place to make sure that `modules/checkout` gets called before
building an sdist or a wheel. That means that you can run `tox` or `pip
install .` in a freshly checked-out tree and expect correct results.
We have to add a command-line interface to support wheel creation on
RHEL8, where pip is not yet PEP517-aware.
We don't implement any optional features of PEP 517 and we also don't
support editable installs.
Drop our attempts to cache ferny and systemd_ctypes — it's just not
worth it. Both of them are tiny.
Instead, establish that we expect submodules in modules/ to be checked
out, generally. Add a (trivial) script to do that and call it from a
bunch of "entry point" type places.
Note: we don't dist modules/checkout, and even if we did, it wouldn't
work, since it needs to be run in a git checkout. We therefore also
carefully avoid adding calls to this script from anything that lands in
the tarball.
Building the dist tarball in the current Fedora 37 cockpit/tasks
container now fails when running as user podman container, when using
systemd-homed:
cp: failed to preserve ownership for 'cockpit-279.39.gb4b41375f/tools/compile': Value too large for defined data type
This was due to a change in systemd 251 with mapped user IDs[1][2].
Work around this by copying the files instead of symlinking them. That
is the default behaviour anyway, and does not hurt much: git trees are
regularly cleaned by developers anyway, and dist tarballs contain a copy
either way.
[1] 1d679b208d/NEWS (L789)
[2] https://bugzilla.redhat.com/show_bug.cgi?id=2144558
Keep `Version: 0` in the spec file. This works fine with `autogen rpm`, `dnf
builddep, and Packit. It was the last macro, so now it's a real working .spec.
This avoids having to invoke this script all the time.
This simplifies our packit config a bit and stops swimming against the stream.
The only place where we want a real version is `make dist`, so that built RPMs
have a correct version. Rework the rule to modify the file in-place in distdir.
Unfortunately this uncovers another packit bug: If the .spec file is not
in the same directory as the tarball, `packit srpm` gets upset:
<https://github.com/packit/packit/issues/1621>. Work around that by
copying the spec to the root directory, and pointing to it.
Currently, we choose the version number that we use at the time that
./autogen.sh is first run. That means that if you configure your tree
and hack on it for a while, making commits as you go, the
overly-specific version number will soon be out of date. This version
number is the number used when building tarballs for testing. Even
switching to old branches on previous versions can result in the version
number not being updated.
Let's move to automake's suggested approach[1] to dynamic versioning by
introducing a `version.m4` file. We write that file from two places:
- `./autogen.sh` writes an approximate version number (the last release)
appended by +git (eg. "270+git")
- from `make dist` we write the value we find in `$(VERSION)` and call
autoreconf to update the configure script accordingly
The expectation is now that anyone who calls `make dist` should specify
the VERSION explicitly. That's two places:
- from `tools/make-dist` we specify the full `git describe` style
version
- from the release workflow we specify the tag name
Since the only thing that uses `make dump-dist` anymore is
`tools/make-dist`, and since it now provides its own version number, we
can drop the rule and print the filename directly.
Since we move to `version.m4`-style versioning, our tarballs will no
longer contain the `.tarball` file.
Finally, since we're going to end up having more tarballs created, with
various version numbers, add `cockpit-*.tar.xz` to `CLEANFILES` to help
keep things under control.
[1] https://www.gnu.org/software/automake/manual/html_node/Rebuilding.html
Many of our scripts follow the pattern of using something like
cd "$(realpath -m $0/../../..)"
in order to change to the top source directory. The idea of applying
`..` to a regular file (`$0`, the path to the script itself) is a bit
weird, but the `realpath -m` (`--canonicalize-missing`) flag allows that
by ignoring the content of the filesystem. In effect, the first `..`
removes the basename, and then the remaining `../..` express the
location of the top directory relative to the directory containing the
script.
Experiments building Cockpit in container environments have shown that
`realpath -m` is non-POSIX.
There's an alternative, however, and it has the advantage of not using
any external tools at all: we can do the first step of removing the
basename by using shell parameter expansion: `"${0%/*}"`. From there,
we can just append the correct number of `../..` to get the desired
path:
cd "${0%/*}/../.."
Update this pattern (and similar) in all the scripts which use it.
When running `./autogen.sh mock` we call back into `./autogen.sh` again,
by mistake, instead of invoking the just-created `./configure` script.
That causes autoreconf to run a second time.
Fix that to call `./configure` directly.
AC_HELP_STRING has been obsolete and replaced with AS_HELP_STRING since
2003. Since autoconf 2.70 obsolete construct warnings are enabled by
default, as autoconf < 2.70 is still widely used obsolete
warnings are now explicitly enabled in `./autogen.sh`.
https://www.gnu.org/software/autoconf/manual/autoconf-2.67/html_node/Pretty-Help-Strings.html
AC_ISC_POSIX and AC_HEADER_STDC macros are both obsolete and no longer
required. AC_ISC_POSIX checks for "Interactive Corporation Unix" which
is no longer sold and Sun dropped support for it in 2006.
AC_HEADER_STDC checks if the system has C header files conforming to
ANSI C89, which all current systems should have now.
The obsolete AC_PROG_CC_C99 macro checks if the C compiler is in C99
mode and if not tried to make it so. Since at least GCC 5.4.0 is
-std=gnu11 if no options are provided.
I tried to move over to standard upstream gettext, but I noticed that
our gettext stuff is sufficiently "custom" that we don't actually use
this in any meaningful way. We even define our own "update-po" rule,
for example.
Just drop it.
Move the call to `test/common/pixel-tests pull` from `autogen.sh` to
`testlib.py`, just before the pixels are compared.
Only do the checkout if there isn't already a checkout there.
This is a first step only: we probably also want to do more complicated
checks to see if we need to update the submodule to a different version,
even if the directory already exists.
Most python things that need to use the bots first import parent.py to
set up the path to enable importing things from the bots directory.
Use this as a nice place for a hook to check out the bots, if they're
not already there.
With this in place, we no longer need the blanket call in autogen.
Turn node_modules into a git submodule that we can cache locally and
check out quickly. Add a tools/node-modules script to support checking
out existing versions and creating and pushing new versions.
This depends on an already-existing cockpit-project/node-cache
repository on github.
The biggest benefit here is that normal users won't need to run npm
anymore. Additionally, there will be no "node modules out of date"
messages anymore: they'll be updated automatically, as-needed.
Drop tools/npm-install, as we no longer need it.
Commit d3f51710dd moved from node_modules/.stamp to package-lock.json
as our canonical stamp file, which regressed make_dist. This fixes
running autogen/configure/make dist with pre-built dist again, as it
does not pointlessly rebuild everything again.
In autogen.sh, don't run npm install if package-lock.json already
exists.
This provided a nice workaround for the pain of running `npm install`,
but we are about to solve this problem in a different way.
Let's start by removing this workaround.
Move our retry logic for `npm install` out of ./autogen.sh into
tools/npm-install. Call it from ./autogen.sh but only conditionally, if
NO_NPM is not set to "1".
Store this file in tools/ and copy it over at 'make dist'. This makes
it a lot easier to clean up the node_modules directory: we just have to
delete it. It also makes it easier to check for the directory being
missing: then we can skip `npm prune`.
Projects should be able to prepare bots on their own without running
`./configure` first. Therefore introduce this tool which can do exactly
that.
Closes#12920
* If `$COCKPIT_BOTS_REF` is set, check out that bots version instead of
master.
* Use git cache in $XDG_CACHE_HOME if available. Our CI uses that to
save downloads, and it does not get in the way for local developers.
* Stop duplicating code in autogen.sh, just call `make bots`. The tree
is configured at this point.
Ported from https://github.com/cockpit-project/starter-kit/pull/233Closes#12908
Commit cea63000a was a thinko -- for our CI we *don't* want our test to
clobber a pre-existing bots/ checkout, as we often use this to run tests
against an updated image or to validate a changes to the bots project.
On developer machines, bots may also be a symlink to an actual bots
directory in development, so don't clobber that.
Closes#12905
There is a category of failure of 'npm install' that leaves
node_modules/ in a corrupt state and won't be fixed by just running 'npm
install' again. Try harder.
Closes#12537
autogen.sh contains an unsafe invocation of find|xargs to iterate over
the node_modules and delete all test directories that it finds (with the
goal of reducing the size of the cache tarball by removing unit tests).
This is a pretty strange thing to do to the node_modules directory,
particularly when you contemplate incremental invocations of 'npm
install'. The same effect can be accomplished by using tar --exclude
when creating the tarball. Do that instead.
Closes#12535
Make sure we have at least npm 3. This will prevent others from wasting
time trying to chase down the dependencies problems reported by older
versions (such as the one found in Debian).
Closes#8987
Running `make dist` in an out-of-tree build that is not underneath
srcdir previously resulted in a broken tarball, containing pkg/* and
node_modules/* under a path to the srcdir.
E. g., diffing file lists between an in-tree and out-of-tree dist
tarball looked like this:
-cockpit-154.x/pkg/lib/journal.js
+cockpit-154.x/upstream/cockpit/pkg/lib/journal.js
Avoid this by putting pkg and node_modules symlinks into the build dir
and chopping off absolute build dir prefixes in webpack-make, instead of
creating path prefixes to the srcdir.
Change the invocation of webpack-make to always pass a canonical
absolute $SRCDIR, so that maybePushInput() does not have to handle
different cases of absolute vs. relative srcdir.
Bower is unreliable, and we have to clear out the whole thing
in autogen.sh. Now that we're using webpack for bundling, we can
just bundle from nodejs modules instead.
This should solve bot problems that timeout when updating the modules
as well as allow automatically updating dependencies.
After this commit developers need to clear out their build and
run autogen.sh again from scratch.
$ git clean -fdx
$ mkdir build
$ cd build
$ ../autogen.sh --prefix=/usr
$ make -j8 all
Closes#6814
A 'dist' directory is used by both the 'webpack' workflow
and the 'make' workflow. When switching between these two
workflows it's confusing to have two dist directories.
Help the user create one dist directory by just doing it in
autogen.sh
Closes#5659
We're not a standard intltool project, so lets just focus on
the stuff we use and need:
* Install mo files in LC_MESSAGES
* Install po.xx.js files
* Have a standard update-po target
* Detect unmarked translations in 'make check'
Closes#5362
Now that we're fully moved to webpack we can have a more
traditional model for our build time dependencies, and solve
several problems in the process.
1. Having a bower_components/ directory makes the project
behave in a more standard way for other javascript developers.
2. Populating bower_components/ at autogen.sh time rather than
committing the changes allows the project to behave as
expected for packagers who want to understand dependencies.
3. A separate bower_components/ directory lets us more simply
build a cockpit-deps-122.tar.gz style tarball ... although
that's for a later commit.
4. Moving our own common code into pkg/lib allows us to run
normal unit tests against it rather than having all such
tests in the playground. Again implemented in a later commit.
For package.json and devel time dependencies, we have problems
with newer versions not being supported by the versions of
Node that are included in distros. So we have to be careful
for updates. We don't mind micro updates. That is the
third digit of the semver number.
However for Bower packages, and particularly those that are
included in Cockpit itself, each point update needs explicit
testing through our integration tests. So we lock these ones
down tightly.
In addition prune the node_modules directory during autogen.sh
Closes#5266
Call `make dist` outside of the test VMs instead of calling autogen.sh
inside. Adjust the debian and rpm packaging to reflect that.
Tests are now running on tarballs that are more similar to release
tarballs and test packages are more similar to packages that
distributions ship.
The system running the tests must now also have build dependencies
installed.
make-source, the script that prepares the tarball, tries not to
interfere too much with existing builds. It configures the package in
tmp-dist/ without setting up the redirect Makefile. However, due to
limitations in autotools, it must use an existing in-tree configuration.
Closes#4178
Reviewed-by: Stef Walter <stefw@redhat.com>
If package.json has changed or no node_modules exists run
npm install during tools/make-source
This lets the verify machines get the right node modules
into the builder vms.
Closes#4150
Reviewed-by: Dominik Perpeet <dperpeet@redhat.com>
Distribute devel dependencies in the tarball, and include
licenses appropriately.
To do this we need to make two changes:
* Use xz to compress the tarball, so that it stays relatively small
* Use the tar-pax format because file names get very long.
Closes#3436
Reviewed-by: Dominik Perpeet <dperpeet@redhat.com>
In cases where the user is using an out of tree build directory
(they should be) we put in a redirect Makefile so they can still
run 'make' commands from the top of $srcdir
Closes#3359
Reviewed-by: Marius Vollmer <marius.vollmer@redhat.com>
This lets us be very clear about where these development
dependencies are coming from, and/or should be.
Certain distros would like to replace these with their
own copies. They should be linked into the same locations
as they would be if they were placed there by 'npm install'
Closes#3257
Reviewed-by: Marius Vollmer <marius.vollmer@redhat.com>
* Fixed typo in comment
And call it cockpit-lvm. The reason for this is that we would have
conflicting bus names, polkit permissions, unit files, and all
sorts of bits with storaged 0.x anyway.
storaged 0.x is in the process of being replaced with storaged 2.x,
and this is already happening in the distros.
Since we were the only consumers of storaged 2.x, rename everything
bring it in and kill the external storaged project.
Closes#2097
Reviewed-by: Peter <petervo@redhat.com>
Builds storaged as cockpit-storaged to avoid
collisions with storaged v2.
Signed-off-by: Stef Walter <stefw@redhat.com>
* Pull in build fixes from storaged
