`nanobind_INCLUDE_DIR` is only populated when MLIR discovers nanobind
through the Python package. When `nanobind_DIR` is supplied explicitly,
the variable remains unset and the configuration output contains an
empty path:
```
Found nanobind v2.13.0:
```
The nanobind CMake package location is already reported by both
discovery paths.
We can only print the detected nanobind version in the final status
message. Or maybe if we want we can find the include dir for the
explicit path too and merge them to `nanobind_INCLUDE_DIR` but I think
this is better.
NumPy stopped being requested as a CMake Python component when its
unused build-time dependency was removed in #108465 . Remove the
remaining status message, which now prints empty version and include
directory values.
`PYTHON_MODULE_PREFIX`, `PYTHON_MODULE_SUFFIX`, and
`PYTHON_MODULE_EXTENSION` are remnants of the previous pybind11-based
Python discovery path and are no longer populated or used by the
nanobind build.
The relevant Python SOABI is already reported through `Python3_SOABI`,
so we can remove the redundant status message, which currently prints
only empty values.
Under Windows, the paths may contain `\` path separators.
`MLIR_TABLEGEN_EXE` is added into MLIRConfig.cmake where backslashes are
interpreted as escape characters. This results in syntax errors during
`find_package(MLIR)`.
Fix by using CMake-normalized paths (`/` separator under all platforms).
LLVM/Clang do not put their tablegen paths in their config files.
Fixes#197175.
`MLIRConfig.cmake` did not propagate `MLIR_LINK_MLIR_DYLIB` to out-of-tree MLIR users. Standalone projects using `find_package(MLIR CONFIG)` and `include(AddMLIR)` therefore could not observe the value used by `mlir_target_link_libraries(...)` when MLIR was configured with `-DMLIR_LINK_MLIR_DYLIB=ON`.
Export `MLIR_LINK_MLIR_DYLIB` through `MLIRConfig.cmake`.
AI tool usage:
I used ChatGPT/Codex to help understand the issue, identify the analogous LLVM CMake pattern, and draft the PR description. I manually reviewed the final patch and verification.
`add_mlir_python_modules()` is used to build the nanobind runtime
library and dialect extension DSOs with a default set of compile and
link options. Projects using these modules may need to augment these
options to handle the warnings of this specific part of the build
differently or to add specific libraries to the shared objects. This
patch introduces four optional CMake variables that allow callers to
provide additional compiler and linker flags:
```
MLIR_BINDINGS_PYTHON_EXTRA_NANOBIND_COMPILE_OPTIONS
MLIR_BINDINGS_PYTHON_EXTRA_NANOBIND_LINK_LIBS
MLIR_BINDINGS_PYTHON_EXTRA_EXTENSION_COMPILE_OPTIONS
MLIR_BINDINGS_PYTHON_EXTRA_EXTENSION_LINK_LIBS
```
NFC if the new cmake variables are unset.
This PR improves native binary generation by avoiding
`llvm::sys::ExecuteAndWait` call for ocloc and instead
leveraging `oclocInvoke()` that consumes an in-memory SPIR-V string.
Co-authored-by: Artem Kroviakov <artem.kroviakov@intel.com>
When multiple dialects share td `#includes` (e.g. `affine` includes
`arith`), each dialect's `*_enum_gen.py` file registers attribute
builders under the same keys, causing "already registered" errors on the
second import; the first commit checks in such a case which currently
fails on main:
```
# | RuntimeError: Attribute builder for 'Arith_CmpFPredicateAttr' is already registered with func: <function _arith_cmpfpredicateattr at 0x78d13cbe9a80>
```
This PR implements a two-pronged fix:
1. Add `allow_existing=True` to `register_attribute_builder` (and the
underlying C++ `registerAttributeBuilder`). When set, silently skips
registration if the key already exists (first-wins semantics). This
handles `EnumInfo`-based builders which have no dialect prefix (e.g.
`AtomicRMWKindAttr`, `Arith_CmpFPredicateAttr`), which may be emitted by
every dialect whose td file includes the defining file;
2. Filter `EnumAttr` builders by `-bind-dialect` in
`EnumPythonBindingGen.cpp` and register them under dialect qualified
keys (`"dialect.AttrName"`). Update `OpPythonBindingGen.cpp` to look up
the same qualified keys for EnumAttr typed op attributes (detected via
`isSubClassOf("EnumAttr")`). Pass `-bind-dialect` from
`AddMLIRPython.cmake`.
This approach incurs no changes to `ir.py` registrations (no "builtin."
prefix), and no manual builder additions to individual dialect Python
files (unlike the previous attempt
https://github.com/llvm/llvm-project/pull/117918).
Note, this PR was "clauded" not "coded".
* Removed an explicit `nb::sig` for `static_typeid`. The inferred type
would
work just fine, and unqualified `TypeID`, which was there previously,
only
really works for core types in the `ir` submodule.
* `DefaultingPyMlir*` helpers also produce qualified types, e.g.
`_mlir.ir.Location` instead of bare `Location`.
* `ir.*.__enter__` now returns a concrete type instead of `object`, e.g.
`ir.Context.__enter__` returns `Context`.
* `loc_tracebacks` uses `Generator` as the return type, since this is
what
`contextmanager` expects in typeshed.
* Changed static methods on subclasses of `DenseElementsAttribute` to
return
that concrete subclass, instead of `DenseElementsAttribute`.
---------
Co-authored-by: Maksim Levental <maksim.levental@gmail.com>
Previously, transitively inherited calls to
`target_include_directories(foo SYSTEM ...)` were being squashed into a
flat list of includes, effectively stripping off `-isystem` and
unintentionally forwarding warnings from such dependencies.
To correctly propagate `SYSTEM` dependencies, use
`target_link_libraries` to forward the parent target's link dependencies
to the OBJECT library (similar to the `_static` flow below). Unlike a
flat `target_include_directories`, this lets CMake resolve transitive
SYSTEM include directories through the proper dependency chain.
Note that `target_link_libraries` on an OBJECT library propagates all
usage requirements, not just includes. This also brings in transitive
`INTERFACE_COMPILE_DEFINITIONS`, `INTERFACE_COMPILE_OPTIONS`, and
`INTERFACE_COMPILE_FEATURES`. This is arguably more correct, as the
OBJECT library compiles the same sources and should see the same flags.
The existing `target_include_directories` call is retained for include
directories set directly on the target (not through link dependencies).
CMake deduplicates include directories that appear through both paths.
Compile definitions and options may technically appear twice (once via
the OBJECT library, once via the consuming target), but duplicate `-D`
and flag entries are harmless in practice.
Also fix `clang_target_link_libraries` and `mlir_target_link_libraries`
to forward the link type (PUBLIC/PRIVATE/INTERFACE) to `obj.*` targets.
Previously the type keyword was silently dropped, resulting in plain-
signature `target_link_libraries` calls. This is now required because
the new keyword-signature call in `llvm_add_library` would otherwise
conflict (CMake requires all calls on a target to use the same
signature).
…83889)"
This reverts commit 2342db00ab.
Revert "[CMake] Propagate dependencies to OBJECT libraries in
`add_llvm_library` (#183541)"
This reverts commit e3c045415a.
Add `MLIR_ENABLE_PYTHON_STABLE_ABI` cmake flag to build bindings against
the Python limited/stable API (abi3 / PEP 384). This allow for
compatibility across different >=3.12 versions with a single .so /
wheel. We also require CMake >=3.26.
The stable ABI restricts usage to a subset of the CPython C API: frame
and code object structs are opaque, so introspection APIs like
`PyCode_Addr2Location`, `PyFrame_GetLasti`, and `PyFrame_GetCode` are
unavailable. The traceback-based auto-location logic is dropped because
we don’t have stable ABI to produce complete locations.
Assisted-by: claude
Previously, transitively inherited calls to
`target_include_directories(foo SYSTEM ...)` were being squashed into a
flat list of includes, effectively stripping off `-isystem` and
unintentionally forwarding warnings from such dependencies.
To correctly propagate `SYSTEM` dependencies, use
`target_link_libraries` to forward the parent target's link dependencies
to the OBJECT library (similar to the `_static` flow below). Unlike a
flat `target_include_directories`, this lets CMake resolve transitive
SYSTEM include directories through the proper dependency chain.
Note that `target_link_libraries` on an OBJECT library propagates all
usage requirements, not just includes. This also brings in transitive
`INTERFACE_COMPILE_DEFINITIONS`, `INTERFACE_COMPILE_OPTIONS`, and
`INTERFACE_COMPILE_FEATURES`. This is arguably more correct, as the
OBJECT library compiles the same sources and should see the same flags.
The existing `target_include_directories` call is retained for include
directories set directly on the target (not through link dependencies).
CMake deduplicates include directories that appear through both paths.
Compile definitions and options may technically appear twice (once via
the OBJECT library, once via the consuming target), but duplicate `-D`
and flag entries should be harmless in practice.
Also fix `clang_target_link_libraries` and `mlir_target_link_libraries`
to forward the link type (PUBLIC/PRIVATE/INTERFACE) to `obj.*` targets.
Previously the type keyword was silently dropped, resulting in
plain-signature `target_link_libraries` calls. This is now required
because the new keyword-signature call in `llvm_add_library` would
otherwise conflict (CMake requires all calls on a target to use the same
signature).
Some MLIR libraries are intended to be dlopen-ed, but currently all MLIR
libraries link against LLVMSupport. After the recent PCH introduction,
this causes these libraries to implicitly use the LLVMSupport PCH, which
results in the definition of llvm::*ABIBreakingChecks, which results in
a ODR violation when loaded with dlopen.
Conceptually, libraries that are designed to be dlopen-ed should not
simply link against LLVM libraries in non-dylib builds for this reason.
(This apparently was a problem before with mlir_apfloat_wrappers.)
To fix builds, remove LLVMSupport from runtime libraries that don't need
it and, as a workaround, disable PCH for libraries that are in a weird
state (use LLVMSupport but happen to not export symbols currently).
This PR completed work from
https://github.com/llvm/llvm-project/pull/178290.
Switched the last few python bindings that still relied on LLVM over to
the C API, and dropped `LLVMsupport` dependency from MLIR cmake.
Currently, UniformQuantizedType only supports built-in MLIR storage
types such as Integer. LLM quantization research introducing feature of
using NF4 as a low precision datatype (see
https://arxiv.org/pdf/2305.14314). There is a growing need to make the
system extensible and maintainable as more types are added. Ensuring
that MLIR can natively support NF4 through a clean, extensible interface
is essential for both current and future quantization workflows.
**Current Approach and Its Limitations:**
- The present implementation relies on dynamic checks (e.g., type
switches or if-else chains) to determine the storage type and retrieve
type-specific information for legality checks.
- This approach works for a small, fixed set of types, but as the number
of supported types grows, the code becomes harder to read, maintain, and
extend.
**Proposed Interface-Based Approach:**
- Define a StorageTypeInterface that specifies the required methods any
storage type must implement to be used in UniformQuantizedType.
- Each storage type (Integer, Float8E5M2, Float8E4M3FN, and new types
like NF4) would implement this interface, encapsulating their
type-specific logic.
- When UniformQuantizedType needs to check legality or retrieve
information, it can use MLIR’s dyn_cast mechanism to check if the type
implements the interface and then call the required methods.
- This design decouples UniformQuantizedType from the specifics of each
storage type, making it easy to add new types (such as NF4) without
modifying the core logic or introducing more type checks.
**Benefits:**
- Extensibility: New storage types can be added by simply implementing
the interface, without touching the core UniformQuantizedType logic.
- Readability: The code is cleaner, as it avoids large switch statements
or if-else chains.
- Maintainability: Type-specific logic is encapsulated within each type,
reducing the risk of errors and making the codebase easier to understand
and update.
Like IREE's
[`IREECompilerBuildPythonModules`](47fe908217/compiler/bindings/python/CMakeLists.txt (L300-L305)).
Note, setting `MLIR_BINDINGS_PYTHON_NB_DOMAIN` on
`add_mlir_python_modules` explicitly would also workaround issues
related to this (and is generally recommended for projects building
multiple bindings packages).
---------
Co-authored-by: Rahul Kayaith <rkayaith@amd.com>
Remove problematic `return()` statements from
mlir_configure_python_dev_packages macro and convert nanobind detection
to FATAL_ERROR.
Using `return()` in a CMake macro is problematic because it returns from
the calling scope, not the macro itself, causing the parent
CMakeLists.txt to silently skip subsequent configuration steps. This
leads to confusing build failures when Python bindings are enabled but
nanobind is missing.
With this change, users get an immediate error with clear instructions
to install nanobind or set nanobind_DIR, rather than silent
configuration failures.
# What
This PR adds a shared library `MLIRPythonSupport` which contains all of
the CRTP classes ike `PyConcreteValue`, `PyConcreteType`,
`PyConcreteAttribute`, as well as other useful code like `Defaulting*`
and etc enabling their reuse in downstream projects. Downstream projects
can now do
```c++
struct PyTestType : mlir::python::MLIR_BINDINGS_PYTHON_DOMAIN::PyConcreteType<PyTestType> {
...
};
class PyTestAttr : public mlir::python::MLIR_BINDINGS_PYTHON_DOMAIN::PyConcreteAttribute<PyTestAttr> {
...
}
NB_MODULE(_mlirPythonTestNanobind, m) {
PyTestType::bind(m);
PyTestAttr::bind(m);
}
```
instead of using the discordant alternative
`mlir_type_subclass`/`mlir_attr_subclass` (same goes for
`PyConcreteValue`/`mlir_value_subclass`).
# Why
This PR is mostly code motion (along with CMake) but before I describe
the changes I want to state the goals/benefits:
1. Currently upstream "core" extensions and "dialect" extensions ([all
of the `Dialect*` extensions
here](d7c734b5a1/mlir/lib/Bindings/Python))
are a two-tier system;
**a**. [core
extensions](https://github.com/llvm/llvm-project/blob/main/mlir/lib/Bindings/Python/IRTypes.cpp#L361)
enjoy first class support as far as type inference[^3], type stub
generation, and ease of implementation, while dialect extensions [have
poorer support](https://reviews.llvm.org/D150927), incorrect type stub
generation much more tedious (boilerplate) implementation;
**b**. Crucially, this two-tiered system is reflected in the fact that
**the two sets of types/attributes are not in the same Python object
hierarchy**. To wit: `isinstance(..., Type)` and `isinstance(...,
Attribute)` are not supported for the dialect extensions[^2];
**c**. Since these types are not exposed in public headers, downstream
users (dialect extensions or not) cannot write functions that overload
on e.g. `PyFloat8*Type` - that's quite a [useful
feature](fdbee98df8/cpp_ext/TorchOps.cpp (L29-L69))!
2. The dialect extensions incur a sizeable performance penalty relative
to the core extensions in that every single trip across the wire (either
`python->cpp` or `cpp->python`) requires work in addition to nanobind's
own casting/construction pipeline;
**a**. When going from `python->cpp`, [we extract the capsule object
from the Python
object](https://github.com/llvm/llvm-project/blob/main/mlir/include/mlir/Bindings/Python/NanobindAdaptors.h#L219C24-L219C46)
and then extract from the capsule the `Mlir*` opaque struct/ptr. This
side isn't so onerous;
**b**. When going from `cpp->python` we call long-hand call Python
`import` APIs and construct the Python object using `_CAPICreate`. Note,
there at least 2 `attr` calls incurred in addition to `_CAPICreate`;
this is already much more [efficiently handled by nanobind
itself](4ba51fcf79/src/nb_internals.h (L381-L382))!
3. This division blocks various features: in some configurations[^1] we
trigger a circular import bug because "dialect" types and attributes
perform an [import of the root `_mlir`
module](bd9651bf78/mlir/include/mlir/Bindings/Python/NanobindAdaptors.h (L585))
when they are created (the types themselves, not even instances of those
types). This blocks type stub generation for dialect extensions (i.e.,
the reason we currently only generate type stubs for `_mlir`).
# How
Prior this was not done/possible because of "ODR" issues but I have
resolved those issues; the basic idea for how we solve this is "move
things we want to share into shared libraries":
1. Move IRCore (stuff like `PyConcreteValue`, `PyConcreteType`,
`PyConcreteAttribute`) into `MLIRPythonSupport`;
- Note, we move the rest of the things in `IRModule.h` (renamed to
`IRCore.h`) because `PyConcreteValue`, `PyConcreteType`,
`PyConcreteAttribute` depend on them. This makes for a bigger PR than
one would hope for but ultimately I think we should give people access
to these classes to use as they see fit (specifically inherit from, but
also liberally use in bindings signatures instead of the opaque `Mlir*`
struct wrappers).
2. Put all of this code into a nested namespace
`MLIR_BINDINGS_PYTHON_DOMAIN` which is determined by a compile time
define (and tied to `MLIR_BINDINGS_PYTHON_NB_DOMAIN`). This is necessary
in order to prevent conflicts on both symbol name **and** typeid
(necessary for nanobind to not double register binded types) between
multiple bindings libraries (e.g., `torch-mlir`, and `jax`). Note
[nanobind doesn't support `module_local` like
pybind11](https://nanobind.readthedocs.io/en/latest/porting.html#removed-features).
It does support `NB_DOMAIN` but that is not sufficient for
disambiguating typeids across projects (to wit: we currently define
`NB_DOMAIN` and it was still necessary to move everything to a nested
namespace);
3. Build the [nanobind library itself as a shared
object](https://github.com/wjakob/nanobind/blob/master/cmake/nanobind-config.cmake#L127)
(and link it to both the extensions and `MLIRPythonSupport`).
4. CMake to make this work, in-tree, out-of-tree, downstream, upstream,
etc.
# Testing
Three tests are added here
1. `PythonTestModuleNanobind` is ported to use
`PyConcreteType<PyTestType>` instead of `mlir_type_subclass` and
`PyConcreteAttribute<PyTestAttr>` instead of `mlir_atrr_subclass`,
verifying this works for non-core extensions in-tree;
2. `StandaloneExtensionNanobind` is ported to use `struct PyCustomType :
mlir::python::MLIR_BINDINGS_PYTHON_DOMAIN::PyConcreteType<PyCustomType>`
instead of `mlir_type_subclass` verifying this works for non-core
extensions out-of-tree;
3. `StandaloneExtensionNanobind`'s `smoketest` is extended to also load
another bindings package (namely `mlir`) verifying
`MLIR_BINDINGS_PYTHON_DOMAIN` successfully disambiguates symbols and
typeids.
I have also tested this downstream:
https://github.com/llvm/eudsl/pull/287 as well run the following builder
bots:
mlir-nvidia-gcc7:
https://lab.llvm.org/buildbot/#/buildrequests/6654424?redirect_to_build=true
I have also tested against IREE:
https://github.com/iree-org/iree/pull/21916
# Integration
It is highly recommended to set the CMake var
`MLIR_BINDINGS_PYTHON_NB_DOMAIN` (which will also determine
`MLIR_BINDINGS_PYTHON_DOMAIN`) to something unique for each downstream.
This can also be passed explicitly to `add_mlir_python_modules` if your
project builds multiple bindings packages. I added a `WARNING` to this
effect in `AddMLIRPython.cmake`.
[^3]: Python values being typed correctly when exiting from cpp;
[^1]: Specifically when the modules are imported using `importlib`,
which occurs with nanobind's
[stubgen](https://github.com/wjakob/nanobind/blob/master/src/stubgen.py#L965);
[^2]: The workaround we implemented was a class method for the dialect
bindings called `Class.isinstance(...)`;
# Problem:
There are two build system bugs on MacOS in the case where one intends
to use multiple bindings packages simultaneously (same Python
interpreter session):
1. The nanobind modules are built with
[`-Wl,-flat_namespace`](8518d2c405/llvm/cmake/modules/HandleLLVMOptions.cmake (L268))
thereby leading to ambiguous symbols across multiple whatever dylibs;
2. Intra-library symbol resolution (within the C API aggregate dylib)
fails to resolve symbols correctly unless things are built with
`-DCMAKE_C_VISIBILITY_PRESET=hidden -DCMAKE_CXX_VISIBILITY_PRESET=hidden
-DCMAKE_VISIBILITY_INLINES_HIDDEN=ON`.
# Repro:
On a Mac (with this patch applied):
1. Build without `twolevel_namespace` and without hidden vis properties
and run `LIT_FILTER=test.toy ninja check-mlir` (assuming you have
`-DLLVM_BUILD_EXAMPLES=ON -DLLVM_INCLUDE_EXAMPLES=ON`) and you will see:
```
LLVM ERROR: can't create Attribute 'mlir::StringAttr' because storage
uniquer isn't initialized: the dialect was likely not loaded, or the
attribute wasn't added with addAttributes<...>() in the
Dialect::initialize() method.
```
2. Build with `twolevel_namespace` but not hidden vis and run the same
lit test and you will see:
```
LLVM ERROR: Attempting to attach an interface to an unregistered
operation builtin.unrealized_conversion_cast.
```
# Fix
We only do a partial fix here (adding `twolevel_namespace` to Python
bindings modules) because a full fix requires adding visibility
attributes to all object files. I added docs discussing this.
# Why is this not happening on Linux
Using `DYLD_PRINT_BINDINGS=1` I observe that for the checked-in/updated
test (without the fix) `libMLIRPythonCAPI` resolves many of its symbols
to `libStandalonePythonCAPI`:
```
dyld[98449]: looking for weak-def symbol '__ZN4mlir6TypeID3getINS_13AffineMapAttrEEES0_v':
dyld[98449]: found __ZN4mlir6TypeID3getINS_13AffineMapAttrEEES0_v in map, using impl from /Users/maksimlevental/dev_projects/llvm-project/cmake-build-debug/tools/mlir/test/Examples/standalone/python_packages/standalone/mlir_standalone/_mlir_libs/libStandalonePythonCAPI.dylib
dyld[98449]: <libMLIRPythonCAPI.dylib/bind#22> -> 0x11348fa9c <libStandalonePythonCAPI.dylib/__ZN4mlir6TypeID3getINS_13AffineMapAttrEEES0_v>)
dyld[98449]: looking for weak-def symbol '__ZN4mlir6TypeID3getINS_9ArrayAttrEEES0_v':
dyld[98449]: found __ZN4mlir6TypeID3getINS_9ArrayAttrEEES0_v in map, using impl from /Users/maksimlevental/dev_projects/llvm-project/cmake-build-debug/tools/mlir/test/Examples/standalone/python_packages/standalone/mlir_standalone/_mlir_libs/libStandalonePythonCAPI.dylib
dyld[98449]: <libMLIRPythonCAPI.dylib/bind#23> -> 0x11348f990 <libStandalonePythonCAPI.dylib/__ZN4mlir6TypeID3getINS_9ArrayAttrEEES0_v>)
dyld[98449]: looking for weak-def symbol '__ZN4mlir6TypeID3getINS_14DictionaryAttrEEES0_v':
dyld[98449]: found __ZN4mlir6TypeID3getINS_14DictionaryAttrEEES0_v in map, using impl from /Users/maksimlevental/dev_projects/llvm-project/cmake-build-debug/tools/mlir/test/Examples/standalone/python_packages/standalone/mlir_standalone/_mlir_libs/libStandalonePythonCAPI.dylib
dyld[98449]: <libMLIRPythonCAPI.dylib/bind#24> -> 0x11348eec0 <libStandalonePythonCAPI.dylib/__ZN4mlir6TypeID3getINS_14DictionaryAttrEEES0_v>)
```
Turns out this is "expected" behavior:
> It appears on macOS, when a static library is compiled without
-fvisibility=hidden, its C++ template instantiations could lead to
leftover weak symbols that are resolved and bound at runtime
https://joyeecheung.github.io/blog/2025/01/11/executable-loading-and-startup-performance-on-macos/🤷
We recently moved over to compiling with clang-cl on Windows. This ended
up causing a large increase in warnings, particularly due to how
warnings are handled in nanobind. cd91d0fff9
initially set -Wall -Wextra and -Wpedantic while fixing another issue,
which is probably not what we want to do on third-party code. We also
need to disable -Wmissing-field-initializers to get things clean in this
configuration.
Reviewers: makslevental, jpienaar, rkayaith
Reviewed By: makslevental
Pull Request: https://github.com/llvm/llvm-project/pull/166828
If `PYTHONPATH` is set and points to the build location of the python
bindings package then when stubgen runs, `_mlir` will get imported twice
and bad things will happen (e.g., `Assertion !instance && “PyGlobals
already constructed”’`). This happens because `mlir` is a namespace
package and the importer/loader can't distinguish between
`mlir._mlir_libs._mlir` and `_mlir_libs._mlir` imported from `CWD`. Or
something like that. The fix is to filter out any entries in
`PYTHONPATH` that end in `MLIR_BINDINGS_PYTHON_INSTALL_PREFIX/..` (e.g.,
`python_packages/mlir_core/`).
This a reland of https://github.com/llvm/llvm-project/pull/155741 which
was reverted at https://github.com/llvm/llvm-project/pull/157831. This
version is narrower in scope - it only turns on automatic stub
generation for `MLIRPythonExtension.Core._mlir` and **does not do
anything automatically**. Specifically, the only CMake code added to
`AddMLIRPython.cmake` is the `mlir_generate_type_stubs` function which
is then used only in a manual way. The API for
`mlir_generate_type_stubs` is:
```
Arguments:
MODULE_NAME: The fully-qualified name of the extension module (used for importing in python).
DEPENDS_TARGETS: List of targets these type stubs depend on being built; usually corresponding to the
specific extension module (e.g., something like StandalonePythonModules.extension._standaloneDialectsNanobind.dso)
and the core bindings extension module (e.g., something like StandalonePythonModules.extension._mlir.dso).
OUTPUT_DIR: The root output directory to emit the type stubs into.
OUTPUTS: List of expected outputs.
DEPENDS_TARGET_SRC_DEPS: List of cpp sources for extension library (for generating a DEPFILE).
IMPORT_PATHS: List of paths to add to PYTHONPATH for stubgen.
PATTERN_FILE: (Optional) Pattern file (see https://nanobind.readthedocs.io/en/latest/typing.html#pattern-files).
Outputs:
NB_STUBGEN_CUSTOM_TARGET: The target corresponding to generation which other targets can depend on.
```
Downstream users should use `mlir_generate_type_stubs` in coordination
with `declare_mlir_python_sources` to turn on stub generation for their
own downstream dialect extensions and upstream dialect extensions if
they so choose. Standalone example shows an example.
Note, downstream will also need to set
`-DMLIR_PYTHON_PACKAGE_PREFIX=...` correctly for their bindings.
In https://github.com/llvm/llvm-project/pull/155741 I broke the cardinal
rule of CMake: nothing happens when you think it happens 🤷. Meaning:
`declare_mlir_python_sources(SOURCES_GLOB
"_mlir_libs/${_module_name}/**/*.pyi")` wasn't picking up any sources
_because they aren't generated yet_. This of course makes sense in
retrospect (the stubs are generated as part of the build process, post
extension compile, rather than the configure process).
Thus, the API needs to be:
```
GENERATE_TYPE_STUBS: List of generated type stubs expected from stubgen, relative to _mlir_libs.
```
Partially as a result of this omission, the stubs weren't being
installed into either the build dir nor the install dir. That is also
fixed now:
**Source dir (for easy reference):**
<img width="300" height="674" alt="image"
src="https://github.com/user-attachments/assets/a569f066-c2bd-4361-91f3-1c75181e51da"
/>
**Build dir (for forthcoming typechecker tests):**
<img width="398" height="551" alt="image"
src="https://github.com/user-attachments/assets/017859f9-fddb-49ee-85e5-915f5b5f7377"
/>
**Install dir:**
<img width="456" height="884" alt="image"
src="https://github.com/user-attachments/assets/8051be7e-898c-4ec8-a11e-e2408b241a56"
/>
This PR turns on automatic type stub generation (rather than relying on
hand-written/updated stubs). It uses nanobind's [stubgen
facility](https://nanobind.readthedocs.io/en/latest/typing.html#stub-generation).
If you would like to enable this functionality you can add
`GENERATE_TYPE_STUBS` to `declare_mlir_python_extension` .
Tool targets like `mlir-opt` rely on the `mlir-headers` or
`mlir-generic-headers` targets to run first to generate headers.
However, many of the `IncGen` targets are not specified as dependencies
of the header targets in CMake, which causes spurious build failures when
using a high number of parallel build jobs.
Thus, this commit introduces a pair of new CMake macros
`add_mlir_dialect_tablegen_target` and
`add_mlir_generic_tablegen_target` to
`AddMLIR.cmake`, which can be used in place of
`add_public_tablegen_target` to
ensure (by convention) that `IncGen` targets are added to the
`mlir-headers`
(resp. `mlir-generic-headers`) target dependencies.
Most uses of `add_public_tablegen_target` in the dialects have been
refactored to use the new macros.
A CMake change included in CMake 4.0 makes `AIX` into a variable
(similar to `APPLE`, etc.)
ff03db6657
However, `${CMAKE_SYSTEM_NAME}` unfortunately also expands exactly to
`AIX` and `if` auto-expands variable names in CMake. That means you get
a double expansion if you write:
`if (${CMAKE_SYSTEM_NAME} MATCHES "AIX")`
which becomes:
`if (AIX MATCHES "AIX")`
which is as if you wrote:
`if (ON MATCHES "AIX")`
You can prevent this by quoting the expansion of "${CMAKE_SYSTEM_NAME}",
due to policy
[CMP0054](https://cmake.org/cmake/help/latest/policy/CMP0054.html#policy:CMP0054)
which is on by default in 4.0+. Most of the LLVM CMake already does
this, but this PR fixes the remaining cases where we do not.
When `LLVM_INSTALL_TOOLCHAIN_ONLY=ON`, the MLIR shared library
(`libMLIR*`) is not installed even though it is built with the
`INSTALL_WITH_TOOLCHAIN` argument to the `add_mlir_library` cmake
function. This patch ensures that `libMLIR*` is installed when
`LLVM_INSTALL_TOOLCHAIN_ONLY=ON`.
Patch verified
[here](https://github.com/llvm/llvm-project/issues/151247#issuecomment-3156387055).
fixes#151247
Depending on the order of CMake includes the
`tablegen_compile_commands.yml` was previously missing entries due to
being deleted after a `tablegen` commands.
This PR fixes this by 1) making sure `AddMLIR` includes `TableGen` such
that new compile commands are added to a fresh YML file and 2) using an
include guard to ensure the file is cleared exactly once
This PR introduces a new tool, mlir-irdl-to-cpp, that converts IRDL to
C++ definitions.
The C++ definitions allow use of the IRDL-defined dialect in MLIR C++
infrastructure, enabling the use of conversion patterns with IRDL
dialects for example. This PR also adds CMake utilities to easily
integrate the IRDL dialects into MLIR projects.
Note that most IRDL features are not supported. In general, we are only
able to define simple types and operations.
- The only type constraint supported is irdl.any.
- Variadic operands and results are not supported.
- Verifiers for the IRDL constraints are not generated.
- Attributes are not supported.
---------
Co-authored-by: Théo Degioanni <theo.degioanni.llvm.deluge062@simplelogin.fr>
Co-authored-by: Fehr Mathieu <mathieu.fehr@gmail.com>
This PR introduces a new tool, mlir-irdl-to-cpp, that converts IRDL to
C++ definitions.
The C++ definitions allow use of the IRDL-defined dialect in MLIR C++
infrastructure, enabling the use of conversion patterns with IRDL
dialects for example. This PR also adds CMake utilities to easily
integrate the IRDL dialects into MLIR projects.
Note that most IRDL features are not supported. In general, we are only
able to define simple types and operations.
- The only type constraint supported is `irdl.any`.
- Variadic operands and results are not supported.
- Verifiers for the IRDL constraints are not generated.
- Attributes are not supported.
---------
Co-authored-by: Théo Degioanni <theo.degioanni.llvm.deluge062@simplelogin.fr>
Co-authored-by: Fehr Mathieu <mathieu.fehr@gmail.com>
This reverts commit fa65a228f4.
This relands commit ab405fb6e9.
There was an issue where CMake versions <3.23.0 would not properly parse
dep files, causing the build to file. This patch fixes that by just
making CMake versions <3.23.0 use the fallback behavior.
CMP0116 was originally set to old to get rid of warnings. However, this
behavior is now set to new by default with the minimum CMake version
that LLVM requires so does not produce any warnings, and setting it
explicitly to old does produce a warning in newer CMake versions. Due to
these reasons, remove this check for now.
Splitting off from removing the CMP0114 check just in case something
breaks.
Partially fixes#83727.
