Cold-Air-Networks/mirror-llvm-project
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 7
mirror-llvm-project/mlir/python/CMakeLists.txt

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

1032 lines
32 KiB
Text
Raw Normal View History

Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
include(AddMLIRPython)
[MLIR][Python] reland (narrower) type stub generation (#157930) This a reland of https://github.com/llvm/llvm-project/pull/155741 which was reverted at https://github.com/llvm/llvm-project/pull/157831. This version is narrower in scope - it only turns on automatic stub generation for `MLIRPythonExtension.Core._mlir` and **does not do anything automatically**. Specifically, the only CMake code added to `AddMLIRPython.cmake` is the `mlir_generate_type_stubs` function which is then used only in a manual way. The API for `mlir_generate_type_stubs` is: ``` Arguments: MODULE_NAME: The fully-qualified name of the extension module (used for importing in python). DEPENDS_TARGETS: List of targets these type stubs depend on being built; usually corresponding to the specific extension module (e.g., something like StandalonePythonModules.extension._standaloneDialectsNanobind.dso) and the core bindings extension module (e.g., something like StandalonePythonModules.extension._mlir.dso). OUTPUT_DIR: The root output directory to emit the type stubs into. OUTPUTS: List of expected outputs. DEPENDS_TARGET_SRC_DEPS: List of cpp sources for extension library (for generating a DEPFILE). IMPORT_PATHS: List of paths to add to PYTHONPATH for stubgen. PATTERN_FILE: (Optional) Pattern file (see https://nanobind.readthedocs.io/en/latest/typing.html#pattern-files). Outputs: NB_STUBGEN_CUSTOM_TARGET: The target corresponding to generation which other targets can depend on. ``` Downstream users should use `mlir_generate_type_stubs` in coordination with `declare_mlir_python_sources` to turn on stub generation for their own downstream dialect extensions and upstream dialect extensions if they so choose. Standalone example shows an example. Note, downstream will also need to set `-DMLIR_PYTHON_PACKAGE_PREFIX=...` correctly for their bindings.
2025-09-20 10:47:32 -08:00
# Specifies that all MLIR packages are co-located under the `MLIR_PYTHON_PACKAGE_PREFIX.`
# top level package (the API has been embedded in a relocatable way).
add_compile_definitions("MLIR_PYTHON_PACKAGE_PREFIX=${MLIR_PYTHON_PACKAGE_PREFIX}.")
[mlir][Python] create MLIRPythonSupport (#171775) # What This PR adds a shared library `MLIRPythonSupport` which contains all of the CRTP classes ike `PyConcreteValue`, `PyConcreteType`, `PyConcreteAttribute`, as well as other useful code like `Defaulting*` and etc enabling their reuse in downstream projects. Downstream projects can now do ```c++ struct PyTestType : mlir::python::MLIR_BINDINGS_PYTHON_DOMAIN::PyConcreteType<PyTestType> { ... }; class PyTestAttr : public mlir::python::MLIR_BINDINGS_PYTHON_DOMAIN::PyConcreteAttribute<PyTestAttr> { ... } NB_MODULE(_mlirPythonTestNanobind, m) { PyTestType::bind(m); PyTestAttr::bind(m); } ``` instead of using the discordant alternative `mlir_type_subclass`/`mlir_attr_subclass` (same goes for `PyConcreteValue`/`mlir_value_subclass`). # Why This PR is mostly code motion (along with CMake) but before I describe the changes I want to state the goals/benefits: 1. Currently upstream "core" extensions and "dialect" extensions ([all of the `Dialect*` extensions here](https://github.com/llvm/llvm-project/tree/d7c734b5a14bd91e1c76e2ce0014c19f9deef487/mlir/lib/Bindings/Python)) are a two-tier system; **a**. [core extensions](https://github.com/llvm/llvm-project/blob/main/mlir/lib/Bindings/Python/IRTypes.cpp#L361) enjoy first class support as far as type inference[^3], type stub generation, and ease of implementation, while dialect extensions [have poorer support](https://reviews.llvm.org/D150927), incorrect type stub generation much more tedious (boilerplate) implementation; **b**. Crucially, this two-tiered system is reflected in the fact that **the two sets of types/attributes are not in the same Python object hierarchy**. To wit: `isinstance(..., Type)` and `isinstance(..., Attribute)` are not supported for the dialect extensions[^2]; **c**. Since these types are not exposed in public headers, downstream users (dialect extensions or not) cannot write functions that overload on e.g. `PyFloat8*Type` - that's quite a [useful feature](https://github.com/nod-ai/PI/blob/fdbee98df8376f47818e6b47e1cf089528c9d48d/cpp_ext/TorchOps.cpp#L29-L69)! 2. The dialect extensions incur a sizeable performance penalty relative to the core extensions in that every single trip across the wire (either `python->cpp` or `cpp->python`) requires work in addition to nanobind's own casting/construction pipeline; **a**. When going from `python->cpp`, [we extract the capsule object from the Python object](https://github.com/llvm/llvm-project/blob/main/mlir/include/mlir/Bindings/Python/NanobindAdaptors.h#L219C24-L219C46) and then extract from the capsule the `Mlir*` opaque struct/ptr. This side isn't so onerous; **b**. When going from `cpp->python` we call long-hand call Python `import` APIs and construct the Python object using `_CAPICreate`. Note, there at least 2 `attr` calls incurred in addition to `_CAPICreate`; this is already much more [efficiently handled by nanobind itself](https://github.com/wjakob/nanobind/blob/4ba51fcf795971c5d603d875ae4184bc0c9bd8e6/src/nb_internals.h#L381-L382)! 3. This division blocks various features: in some configurations[^1] we trigger a circular import bug because "dialect" types and attributes perform an [import of the root `_mlir` module](https://github.com/llvm/llvm-project/blob/bd9651bf78f2b1713a8203e0bd5b97f7ff199924/mlir/include/mlir/Bindings/Python/NanobindAdaptors.h#L585) when they are created (the types themselves, not even instances of those types). This blocks type stub generation for dialect extensions (i.e., the reason we currently only generate type stubs for `_mlir`). # How Prior this was not done/possible because of "ODR" issues but I have resolved those issues; the basic idea for how we solve this is "move things we want to share into shared libraries": 1. Move IRCore (stuff like `PyConcreteValue`, `PyConcreteType`, `PyConcreteAttribute`) into `MLIRPythonSupport`; - Note, we move the rest of the things in `IRModule.h` (renamed to `IRCore.h`) because `PyConcreteValue`, `PyConcreteType`, `PyConcreteAttribute` depend on them. This makes for a bigger PR than one would hope for but ultimately I think we should give people access to these classes to use as they see fit (specifically inherit from, but also liberally use in bindings signatures instead of the opaque `Mlir*` struct wrappers). 2. Put all of this code into a nested namespace `MLIR_BINDINGS_PYTHON_DOMAIN` which is determined by a compile time define (and tied to `MLIR_BINDINGS_PYTHON_NB_DOMAIN`). This is necessary in order to prevent conflicts on both symbol name **and** typeid (necessary for nanobind to not double register binded types) between multiple bindings libraries (e.g., `torch-mlir`, and `jax`). Note [nanobind doesn't support `module_local` like pybind11](https://nanobind.readthedocs.io/en/latest/porting.html#removed-features). It does support `NB_DOMAIN` but that is not sufficient for disambiguating typeids across projects (to wit: we currently define `NB_DOMAIN` and it was still necessary to move everything to a nested namespace); 3. Build the [nanobind library itself as a shared object](https://github.com/wjakob/nanobind/blob/master/cmake/nanobind-config.cmake#L127) (and link it to both the extensions and `MLIRPythonSupport`). 4. CMake to make this work, in-tree, out-of-tree, downstream, upstream, etc. # Testing Three tests are added here 1. `PythonTestModuleNanobind` is ported to use `PyConcreteType<PyTestType>` instead of `mlir_type_subclass` and `PyConcreteAttribute<PyTestAttr>` instead of `mlir_atrr_subclass`, verifying this works for non-core extensions in-tree; 2. `StandaloneExtensionNanobind` is ported to use `struct PyCustomType : mlir::python::MLIR_BINDINGS_PYTHON_DOMAIN::PyConcreteType<PyCustomType>` instead of `mlir_type_subclass` verifying this works for non-core extensions out-of-tree; 3. `StandaloneExtensionNanobind`'s `smoketest` is extended to also load another bindings package (namely `mlir`) verifying `MLIR_BINDINGS_PYTHON_DOMAIN` successfully disambiguates symbols and typeids. I have also tested this downstream: https://github.com/llvm/eudsl/pull/287 as well run the following builder bots: mlir-nvidia-gcc7: https://lab.llvm.org/buildbot/#/buildrequests/6654424?redirect_to_build=true I have also tested against IREE: https://github.com/iree-org/iree/pull/21916 # Integration It is highly recommended to set the CMake var `MLIR_BINDINGS_PYTHON_NB_DOMAIN` (which will also determine `MLIR_BINDINGS_PYTHON_DOMAIN`) to something unique for each downstream. This can also be passed explicitly to `add_mlir_python_modules` if your project builds multiple bindings packages. I added a `WARNING` to this effect in `AddMLIRPython.cmake`. [^3]: Python values being typed correctly when exiting from cpp; [^1]: Specifically when the modules are imported using `importlib`, which occurs with nanobind's [stubgen](https://github.com/wjakob/nanobind/blob/master/src/stubgen.py#L965); [^2]: The workaround we implemented was a class method for the dialect bindings called `Class.isinstance(...)`;
2026-01-05 08:08:13 -09:00
set(MLIRPythonModules_ROOT_PREFIX "${MLIR_BINARY_DIR}/${MLIR_BINDINGS_PYTHON_INSTALL_PREFIX}")
set(PYTHON_SOURCE_DIR "${MLIR_SOURCE_DIR}/lib/Bindings/Python")
[MLIR][Python] reland (narrower) type stub generation (#157930) This a reland of https://github.com/llvm/llvm-project/pull/155741 which was reverted at https://github.com/llvm/llvm-project/pull/157831. This version is narrower in scope - it only turns on automatic stub generation for `MLIRPythonExtension.Core._mlir` and **does not do anything automatically**. Specifically, the only CMake code added to `AddMLIRPython.cmake` is the `mlir_generate_type_stubs` function which is then used only in a manual way. The API for `mlir_generate_type_stubs` is: ``` Arguments: MODULE_NAME: The fully-qualified name of the extension module (used for importing in python). DEPENDS_TARGETS: List of targets these type stubs depend on being built; usually corresponding to the specific extension module (e.g., something like StandalonePythonModules.extension._standaloneDialectsNanobind.dso) and the core bindings extension module (e.g., something like StandalonePythonModules.extension._mlir.dso). OUTPUT_DIR: The root output directory to emit the type stubs into. OUTPUTS: List of expected outputs. DEPENDS_TARGET_SRC_DEPS: List of cpp sources for extension library (for generating a DEPFILE). IMPORT_PATHS: List of paths to add to PYTHONPATH for stubgen. PATTERN_FILE: (Optional) Pattern file (see https://nanobind.readthedocs.io/en/latest/typing.html#pattern-files). Outputs: NB_STUBGEN_CUSTOM_TARGET: The target corresponding to generation which other targets can depend on. ``` Downstream users should use `mlir_generate_type_stubs` in coordination with `declare_mlir_python_sources` to turn on stub generation for their own downstream dialect extensions and upstream dialect extensions if they so choose. Standalone example shows an example. Note, downstream will also need to set `-DMLIR_PYTHON_PACKAGE_PREFIX=...` correctly for their bindings.
2025-09-20 10:47:32 -08:00
Move MLIR python sources to mlir/python. * NFC but has some fixes for CMake glitches discovered along the way (things not cleaning properly, co-mingled depends). * Includes previously unsubmitted fix in D98681 and a TODO to fix it more appropriately in a smaller followup. Differential Revision: https://reviews.llvm.org/D101493
2021-04-28 12:04:17 -08:00
################################################################################
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
# Structural groupings.
Move MLIR python sources to mlir/python. * NFC but has some fixes for CMake glitches discovered along the way (things not cleaning properly, co-mingled depends). * Includes previously unsubmitted fix in D98681 and a TODO to fix it more appropriately in a smaller followup. Differential Revision: https://reviews.llvm.org/D101493
2021-04-28 12:04:17 -08:00
################################################################################
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
declare_mlir_python_sources(MLIRPythonSources)
declare_mlir_python_sources(MLIRPythonSources.Dialects
ADD_TO_PARENT MLIRPythonSources)
[mlir][python] factor out pure python core sources (#71592) I'd like to be able to install just the Python core sources (without building/including the pybind sources).
2023-11-07 16:52:43 -09:00
declare_mlir_python_sources(MLIRPythonSources.Core
ADD_TO_PARENT MLIRPythonSources)
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
################################################################################
# Pure python sources and generated code
################################################################################
[mlir][python] factor out pure python core sources (#71592) I'd like to be able to install just the Python core sources (without building/including the pybind sources).
2023-11-07 16:52:43 -09:00
declare_mlir_python_sources(MLIRPythonSources.Core.Python
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
[mlir][python] factor out pure python core sources (#71592) I'd like to be able to install just the Python core sources (without building/including the pybind sources).
2023-11-07 16:52:43 -09:00
ADD_TO_PARENT MLIRPythonSources.Core
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
SOURCES
_mlir_libs/__init__.py
[MLIR][Python] reland (narrower) type stub generation (#157930) This a reland of https://github.com/llvm/llvm-project/pull/155741 which was reverted at https://github.com/llvm/llvm-project/pull/157831. This version is narrower in scope - it only turns on automatic stub generation for `MLIRPythonExtension.Core._mlir` and **does not do anything automatically**. Specifically, the only CMake code added to `AddMLIRPython.cmake` is the `mlir_generate_type_stubs` function which is then used only in a manual way. The API for `mlir_generate_type_stubs` is: ``` Arguments: MODULE_NAME: The fully-qualified name of the extension module (used for importing in python). DEPENDS_TARGETS: List of targets these type stubs depend on being built; usually corresponding to the specific extension module (e.g., something like StandalonePythonModules.extension._standaloneDialectsNanobind.dso) and the core bindings extension module (e.g., something like StandalonePythonModules.extension._mlir.dso). OUTPUT_DIR: The root output directory to emit the type stubs into. OUTPUTS: List of expected outputs. DEPENDS_TARGET_SRC_DEPS: List of cpp sources for extension library (for generating a DEPFILE). IMPORT_PATHS: List of paths to add to PYTHONPATH for stubgen. PATTERN_FILE: (Optional) Pattern file (see https://nanobind.readthedocs.io/en/latest/typing.html#pattern-files). Outputs: NB_STUBGEN_CUSTOM_TARGET: The target corresponding to generation which other targets can depend on. ``` Downstream users should use `mlir_generate_type_stubs` in coordination with `declare_mlir_python_sources` to turn on stub generation for their own downstream dialect extensions and upstream dialect extensions if they so choose. Standalone example shows an example. Note, downstream will also need to set `-DMLIR_PYTHON_PACKAGE_PREFIX=...` correctly for their bindings.
2025-09-20 10:47:32 -08:00
_mlir_libs/_mlir/py.typed
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
ir.py
passmanager.py
[mlir] Add PDL C & Python usage (#94714) Following a rather direct approach to expose PDL usage from C and then Python. This doesn't yes plumb through adding support for custom matchers through this interface, so constrained to basics initially. This also exposes greedy rewrite driver. Only way currently to define patterns is via PDL (just to keep small). The creation of the PDL pattern module could be improved to avoid folks potentially accessing the module used to construct it post construction. No ergonomic work done yet. --------- Signed-off-by: Jacques Pienaar <jpienaar@google.com>
2024-06-11 06:45:12 -08:00
rewrite.py
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
dialects/_ods_common.py
[MLIR][Python] Add a DSL for defining dialects in Python bindings (#169045) Python bindings for the IRDL dialect were introduced in #158488. They are currently usable—for constructing IR and dynamically loading modules that contain `irdl.dialect` into MLIR. However, there are still several pain points when working with them: * The IRDL IR-building interface is not very intuitive and tends to be quite verbose. * We do not yet have the corresponding `OpView` classes for IRDL-defined operations. To address these issues, I propose creating a wrapper (effectively a small “DSL”) on top of the existing IRDL Python bindings. This wrapper aims to simplify IR construction and automatically generate the corresponding `OpView` types. A simple example is shown below. Currently, using the IRDL bindings looks like this: ```python m = Module.create() with InsertionPoint(m.body): myint = irdl.dialect("myint") with InsertionPoint(myint.body): constant = irdl.operation_("constant") with InsertionPoint(constant.body): iattr = irdl.base(base_name="#builtin.integer") i32 = irdl.is_(TypeAttr.get(IntegerType.get_signless(32))) irdl.attributes_([iattr], ["value"]) irdl.results_([i32], ["cst"], [irdl.Variadicity.single]) add = irdl.operation_("add") with InsertionPoint(add.body): i32 = irdl.is_(TypeAttr.get(IntegerType.get_signless(32))) irdl.operands_( [i32, i32], ["lhs", "rhs"], [irdl.Variadicity.single, irdl.Variadicity.single], ) irdl.results_([i32], ["res"], [irdl.Variadicity.single]) irdl.load_dialects(m) ``` With the proposed DSL (module name `mlir.dialects.ext`), the equivalent implementation becomes: ```python class MyInt(Dialect, name="myint"): pass i32 = IntegerType[32] class ConstantOp(MyInt.Operation, name="constant"): value: IntegerAttr cst: Result[i32] class AddOp(MyInt.Operation, name="add"): lhs: Operand[i32] rhs: Operand[i32] res: Result[i32] MyInt.load() ``` Compared with the current IRDL Python bindings, this DSL mainly adds the following: * **A more intuitive interface** for constructing IRDL definitions (as shown in the example). * **Automatic generation of the corresponding `OpView` classes**—including `__init__` methods and property getters for each defined operation. Similar to TableGen’s `ins`, operands and attributes can be interleaved in arbitrary order. Special handling is also implemented for optional and variadic operands/results (such as computing segment sizes) so that they feel as natural to use as native operations. * **Lazy insertion of ops**: all ops are created and inserted only when `Dialect.load()` is called, which makes it unnecessary to specify an MLIR context immediately when defining an IRDL dialect. * **Basic type inference** in operation builders (i.e. `OpViewCls.__init__`) for trivial result types. The current DSL does not yet cover all IRDL operations. Several features are not supported at the moment: - Defining new types or attributes - Parametric constraints - Adding regions to operations --------- Co-authored-by: Rolf Morel <rolfmorel@gmail.com>
2026-01-25 06:08:45 -09:00
dialects/ext.py
Move MLIR python sources to mlir/python. * NFC but has some fixes for CMake glitches discovered along the way (things not cleaning properly, co-mingled depends). * Includes previously unsubmitted fix in D98681 and a TODO to fix it more appropriately in a smaller followup. Differential Revision: https://reviews.llvm.org/D101493
2021-04-28 12:04:17 -08:00
)
[mlir][python] meta region_op (#75673)
2023-12-21 08:20:29 -09:00
declare_mlir_python_sources(MLIRPythonSources.Core.Python.Extras
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
ADD_TO_PARENT MLIRPythonSources.Core.Python
SOURCES
extras/types.py
extras/meta.py
)
[MLIR][Python] Do not install ExecutionEngine wrapper when disabled (#211250) The `_mlirExecutionEngine` extension is declared only when `MLIR_ENABLE_EXECUTION_ENGINE` is enabled. However, `execution_engine.py` and its type stub are currently added to the Python package unconditionally. This leaves an installed `mlir.execution_engine` module that fails immediately when imported because its native extension is absent. It also exposes a type stub for an unavailable API. Making sure they're added only if `MLIR_ENABLE_EXECUTION_ENGINE` is enabled.
2026-07-22 23:15:12 -08:00
# Keep this source group unconditional because runtime/*.py contains generic
# memref helpers that do not depend on the ExecutionEngine extension. Only the
# wrapper and type stub require _mlirExecutionEngine.
set(_execution_engine_python_sources)
if(MLIR_ENABLE_EXECUTION_ENGINE)
set(_execution_engine_python_sources
execution_engine.py
_mlir_libs/_mlirExecutionEngine.pyi
)
endif()
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
declare_mlir_python_sources(MLIRPythonSources.ExecutionEngine
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
ADD_TO_PARENT MLIRPythonSources
SOURCES
[MLIR][Python] Do not install ExecutionEngine wrapper when disabled (#211250) The `_mlirExecutionEngine` extension is declared only when `MLIR_ENABLE_EXECUTION_ENGINE` is enabled. However, `execution_engine.py` and its type stub are currently added to the Python package unconditionally. This leaves an installed `mlir.execution_engine` module that fails immediately when imported because its native extension is absent. It also exposes a type stub for an unavailable API. Making sure they're added only if `MLIR_ENABLE_EXECUTION_ENGINE` is enabled.
2026-07-22 23:15:12 -08:00
${_execution_engine_python_sources}
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
SOURCES_GLOB
runtime/*.py
)
[MLIR][Python] Do not install ExecutionEngine wrapper when disabled (#211250) The `_mlirExecutionEngine` extension is declared only when `MLIR_ENABLE_EXECUTION_ENGINE` is enabled. However, `execution_engine.py` and its type stub are currently added to the Python package unconditionally. This leaves an installed `mlir.execution_engine` module that fails immediately when imported because its native extension is absent. It also exposes a type stub for an unavailable API. Making sure they're added only if `MLIR_ENABLE_EXECUTION_ENGINE` is enabled.
2026-07-22 23:15:12 -08:00
unset(_execution_engine_python_sources)
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
[mlir] Leverage CMake interface libraries for mlir python This is already partially the case, but we can rely more heavily on interface libraries and how they are imported/exported in other to simplify the implementation of the mlir python functions in Cmake. This change also makes a couple of other changes: 1) Add a new CMake function which handles "pure" sources. This was done inline previously 2) Moves the headers associated with CAPI libraries to the libraries themselves. These were previously managed in a separate source target. They can now be added directly to the CAPI libraries using DECLARED_HEADERS. 3) Cleanup some dependencies that showed up as an issue during the refactor This is a big CMake change that should produce no impact on the build of mlir and on the produced *build tree*. However, this change fixes an issue with the *install tree* of mlir which was previously unusable for projects like torch-mlir because both the "pure" and "extension" targets were pointing to either the build or source trees. Reviewed By: stellaraccident Differential Revision: https://reviews.llvm.org/D128230
2022-06-28 09:39:13 -08:00
declare_mlir_python_sources(MLIRPythonCAPI.HeaderSources
[mlir] Overhaul C/Python registration APIs to properly scope registration/loading activities. Since the very first commits, the Python and C MLIR APIs have had mis-placed registration/load functionality for dialects, extensions, etc. This was done pragmatically in order to get bootstrapped and then just grew in. Downstreams largely bypass and do their own thing by providing various APIs to register things they need. Meanwhile, the C++ APIs have stabilized around this and it would make sense to follow suit. The thing we have observed in canonical usage by downstreams is that each downstream tends to have native entry points that configure its installation to its preferences with one-stop APIs. This patch leans in to this approach with `RegisterEverything.h` and `mlir._mlir_libs._mlirRegisterEverything` being the one-stop entry points for the "upstream packages". The `_mlir_libs.__init__.py` now allows customization of the environment and Context by adding "initialization modules" to the `_mlir_libs` package. If present, `_mlirRegisterEverything` is treated as such a module. Others can be added by downstreams by adding a `_site_initialize_{i}.py` module, where '{i}' is a number starting with zero. The number will be incremented and corresponding module loaded until one is not found. Initialization modules can: * Perform load time customization to the global environment (i.e. registering passes, hooks, etc). * Define a `register_dialects(registry: DialectRegistry)` function that can extend the `DialectRegistry` that will be used to bootstrap the `Context`. * Define a `context_init_hook(context: Context)` function that will be added to a list of callbacks which will be invoked after dialect registration during `Context` initialization. Note that the `MLIRPythonExtension.RegisterEverything` is not included by default when building a downstream (its corresponding behavior was prior). For downstreams which need the default MLIR initialization to take place, they must add this back in to their Python CMake build just like they add their own components (i.e. to `add_mlir_python_common_capi_library` and `add_mlir_python_modules`). It is perfectly valid to not do this, in which case, only the things explicitly depended on and initialized by downstreams will be built/packaged. If the downstream has not been set up for this, it is recommended to simply add this back for the time being and pay the build time/package size cost. CMake changes: * `MLIRCAPIRegistration` -> `MLIRCAPIRegisterEverything` (renamed to signify what it does and force an evaluation: a number of places were incidentally linking this very expensive target) * `MLIRPythonSoure.Passes` removed (without replacement: just drop) * `MLIRPythonExtension.AllPassesRegistration` removed (without replacement: just drop) * `MLIRPythonExtension.Conversions` removed (without replacement: just drop) * `MLIRPythonExtension.Transforms` removed (without replacement: just drop) Header changes: * `mlir-c/Registration.h` is deleted. Dialect registration functionality is now in `IR.h`. Registration of upstream features are in `mlir-c/RegisterEverything.h`. When updating MLIR and a couple of downstreams, I found that proper usage was commingled so required making a choice vs just blind S&R. Python APIs removed: * mlir.transforms and mlir.conversions (previously only had an __init__.py which indirectly triggered `mlirRegisterTransformsPasses()` and `mlirRegisterConversionPasses()` respectively). Downstream impact: Remove these imports if present (they now happen as part of default initialization). * mlir._mlir_libs._all_passes_registration, mlir._mlir_libs._mlirTransforms, mlir._mlir_libs._mlirConversions. Downstream impact: None expected (these were internally used). C-APIs changed: * mlirRegisterAllDialects(MlirContext) now takes an MlirDialectRegistry instead. It also used to trigger loading of all dialects, which was already marked with a TODO to remove -- it no longer does, and for direct use, dialects must be explicitly loaded. Downstream impact: Direct C-API users must ensure that needed dialects are loaded or call `mlirContextLoadAllAvailableDialects(MlirContext)` to emulate the prior behavior. Also see the `ir.c` test case (e.g. ` mlirContextGetOrLoadDialect(ctx, mlirStringRefCreateFromCString("func"));`). * mlirDialectHandle* APIs were moved from Registration.h (which now is restricted to just global/upstream registration) to IR.h, arguably where it should have been. Downstream impact: include correct header (likely already doing so). C-APIs added: * mlirContextLoadAllAvailableDialects(MlirContext): Corresponds to C++ API with the same purpose. Python APIs added: * mlir.ir.DialectRegistry: Mapping for an MlirDialectRegistry. * mlir.ir.Context.append_dialect_registry(MlirDialectRegistry) * mlir.ir.Context.load_all_available_dialects() * mlir._mlir_libs._mlirAllRegistration: New native extension that exposes a `register_dialects(MlirDialectRegistry)` entry point and performs all upstream pass/conversion/transforms registration on init. In this first step, we eagerly load this as part of the __init__.py and use it to monkey patch the Context to emulate prior behavior. * Type caster and capsule support for MlirDialectRegistry This should make it possible to build downstream Python dialects that only depend on a subset of MLIR. See: https://github.com/llvm/llvm-project/issues/56037 Here is an example PR, minimally adapting IREE to these changes: https://github.com/iree-org/iree/pull/9638/files In this situation, IREE is opting to not link everything, since it is already configuring the Context to its liking. For projects that would just like to not think about it and pull in everything, add `MLIRPythonExtension.RegisterEverything` to the list of Python sources getting built, and the old behavior will continue. Reviewed By: mehdi_amini, ftynse Differential Revision: https://reviews.llvm.org/D128593
2022-07-16 15:09:03 -08:00
ROOT_DIR "${MLIR_SOURCE_DIR}/include"
[MLIR][python] Export CAPI headers. * Adds source targets (not included in the full set that downstreams use by default) to bundle mlir-c/ headers into the mlir/_mlir_libs/include directory. * Adds a minimal entry point to get include and library directories. * Used by npcomp to export a full CAPI (which is then used by the Torch extension to link npcomp). Reviewed By: mikeurbach Differential Revision: https://reviews.llvm.org/D107090
2021-07-29 11:06:22 -08:00
SOURCES_GLOB "mlir-c/*.h"
)
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
################################################################################
# Dialect bindings
################################################################################
[mlir][python] python binding for the affine.store op (#68816) This PR creates the necessary files to support bindings for operations in the affine dialect. This is the first of many PRs which will progressively introduce affine.load, affine.for, etc operations. I would like to acknowledge the work by Nelli's author @makslevental : https://github.com/makslevental/nelli/blob/main/nelli/mlir/affine/affine.py which jump-starts the work.
2023-10-11 12:37:11 -08:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/AffineOps.td
SOURCES
dialects/affine.py
DIALECT_NAME affine
GEN_ENUM_BINDINGS)
[MLIR][python bindings] add vendor gpu dialects Differential Revision: https://reviews.llvm.org/D157820
2023-08-13 10:30:28 -08:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/AMDGPUOps.td
SOURCES
dialects/amdgpu.py
[mlir][python bindings] generate all the enums This PR implements python enum bindings for *all* the enums - this includes `I*Attrs` (including positional/bit) and `Dialect/EnumAttr`. There are a few parts to this: 1. CMake: a small addition to `declare_mlir_dialect_python_bindings` and `declare_mlir_dialect_extension_python_bindings` to generate the enum, a boolean arg `GEN_ENUM_BINDINGS` to make it opt-in (even though it works for basically all of the dialects), and an optional `GEN_ENUM_BINDINGS_TD_FILE` for handling corner cases. 2. EnumPythonBindingGen.cpp: there are two weedy aspects here that took investigation: 1. If an enum attribute is not a `Dialect/EnumAttr` then the `EnumAttrInfo` record is canonical, as far as both the cases of the enum **and the `AttrDefName`**. On the otherhand, if an enum is a `Dialect/EnumAttr` then the `EnumAttr` record has the correct `AttrDefName` ("load bearing", i.e., populates `ods.ir.AttributeBuilder('<NAME>')`) but its `enum` field contains the cases, which is an instance of `EnumAttrInfo`. The solution is to generate an one enum class for both `Dialect/EnumAttr` and "independent" `EnumAttrInfo` but to make that class interopable with two builder registrations that both do the right thing (see next sub-bullet). 2. Because we don't have a good connection to cpp `EnumAttr`, i.e., only the `enum class` getters are exposed (like `DimensionAttr::get(Dimension value)`), we have to resort to parsing e.g., `Attribute.parse(f'#gpu<dim {x}>')`. This means that the set of supported `assemblyFormat`s (for the enum) is fixed at compile of MLIR (currently 2, the only 2 I saw). There might be some things that could be done here but they would require quite a bit more C API work to support generically (e.g., casting ints to enum cases and binding all the getters or going generically through the `symbolize*` methods, like `symbolizeDimension(uint32_t)` or `symbolizeDimension(StringRef)`). A few small changes: 1. In addition, since this patch registers default builders for attributes where people might've had their own builders already written, I added a `replace` param to `AttributeBuilder.insert` (`False` by default). 2. `makePythonEnumCaseName` can't handle all the different ways in which people write their enum cases, e.g., `llvm.CConv.Intel_OCL_BI`, which gets turned into `INTEL_O_C_L_B_I` (because `llvm::convertToSnakeFromCamelCase` doesn't look for runs of caps). So I dropped it. On the otherhand regularization does need to done because some enums have `None` as a case (and others might have other python keywords). 3. I turned on `llvm` dialect generation here in order to test `nvvm.WGMMAScaleIn`, which is an enum with [[ https://github.com/llvm/llvm-project/blob/d7e26b56207cbd8995296c5bb7c11ce676b649da/mlir/include/mlir/IR/EnumAttr.td#L22-L25 | no explicit discriminator ]] for the `neg` case. Note, dialects that didn't get a `GEN_ENUM_BINDINGS` don't have any enums to generate. Let me know if I should add more tests (the three trivial ones I added exercise both the supported `assemblyFormat`s and `replace=True`). Reviewed By: stellaraccident Differential Revision: https://reviews.llvm.org/D157934
2023-08-23 10:27:08 -08:00
DIALECT_NAME amdgpu
GEN_ENUM_BINDINGS)
[MLIR][python bindings] add vendor gpu dialects Differential Revision: https://reviews.llvm.org/D157820
2023-08-13 10:30:28 -08:00
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/AsyncOps.td
SOURCES_GLOB dialects/async_dialect/*.py
[mlir][python] Fix generation of Python bindings for `async` dialect (#75960) The Python bindings generated for "async" dialect didn't include any of the "async" dialect ops. This PR fixes issues with generation of Python bindings for "async" dialect and adds a test case to use them.
2024-04-20 17:49:39 -08:00
DIALECT_NAME async)
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
[mlir][bufferization] Fix Python bindings Differential Revision: https://reviews.llvm.org/D126179
2022-05-23 08:10:12 -08:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/BufferizationOps.td
SOURCES
dialects/bufferization.py
[mlir][python bindings] generate all the enums This PR implements python enum bindings for *all* the enums - this includes `I*Attrs` (including positional/bit) and `Dialect/EnumAttr`. There are a few parts to this: 1. CMake: a small addition to `declare_mlir_dialect_python_bindings` and `declare_mlir_dialect_extension_python_bindings` to generate the enum, a boolean arg `GEN_ENUM_BINDINGS` to make it opt-in (even though it works for basically all of the dialects), and an optional `GEN_ENUM_BINDINGS_TD_FILE` for handling corner cases. 2. EnumPythonBindingGen.cpp: there are two weedy aspects here that took investigation: 1. If an enum attribute is not a `Dialect/EnumAttr` then the `EnumAttrInfo` record is canonical, as far as both the cases of the enum **and the `AttrDefName`**. On the otherhand, if an enum is a `Dialect/EnumAttr` then the `EnumAttr` record has the correct `AttrDefName` ("load bearing", i.e., populates `ods.ir.AttributeBuilder('<NAME>')`) but its `enum` field contains the cases, which is an instance of `EnumAttrInfo`. The solution is to generate an one enum class for both `Dialect/EnumAttr` and "independent" `EnumAttrInfo` but to make that class interopable with two builder registrations that both do the right thing (see next sub-bullet). 2. Because we don't have a good connection to cpp `EnumAttr`, i.e., only the `enum class` getters are exposed (like `DimensionAttr::get(Dimension value)`), we have to resort to parsing e.g., `Attribute.parse(f'#gpu<dim {x}>')`. This means that the set of supported `assemblyFormat`s (for the enum) is fixed at compile of MLIR (currently 2, the only 2 I saw). There might be some things that could be done here but they would require quite a bit more C API work to support generically (e.g., casting ints to enum cases and binding all the getters or going generically through the `symbolize*` methods, like `symbolizeDimension(uint32_t)` or `symbolizeDimension(StringRef)`). A few small changes: 1. In addition, since this patch registers default builders for attributes where people might've had their own builders already written, I added a `replace` param to `AttributeBuilder.insert` (`False` by default). 2. `makePythonEnumCaseName` can't handle all the different ways in which people write their enum cases, e.g., `llvm.CConv.Intel_OCL_BI`, which gets turned into `INTEL_O_C_L_B_I` (because `llvm::convertToSnakeFromCamelCase` doesn't look for runs of caps). So I dropped it. On the otherhand regularization does need to done because some enums have `None` as a case (and others might have other python keywords). 3. I turned on `llvm` dialect generation here in order to test `nvvm.WGMMAScaleIn`, which is an enum with [[ https://github.com/llvm/llvm-project/blob/d7e26b56207cbd8995296c5bb7c11ce676b649da/mlir/include/mlir/IR/EnumAttr.td#L22-L25 | no explicit discriminator ]] for the `neg` case. Note, dialects that didn't get a `GEN_ENUM_BINDINGS` don't have any enums to generate. Let me know if I should add more tests (the three trivial ones I added exercise both the supported `assemblyFormat`s and `replace=True`). Reviewed By: stellaraccident Differential Revision: https://reviews.llvm.org/D157934
2023-08-23 10:27:08 -08:00
DIALECT_NAME bufferization
GEN_ENUM_BINDINGS_TD_FILE
"../../include/mlir/Dialect/Bufferization/IR/BufferizationEnums.td"
)
[mlir][bufferization] Fix Python bindings Differential Revision: https://reviews.llvm.org/D126179
2022-05-23 08:10:12 -08:00
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/BuiltinOps.td
SOURCES
dialects/builtin.py
DIALECT_NAME builtin)
[mlir][complex] Add Python bindings for complex ops. Reviewed By: aartbik Differential Revision: https://reviews.llvm.org/D127916
2022-06-15 15:00:51 -08:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/ComplexOps.td
SOURCES
dialects/complex.py
DIALECT_NAME complex)
[mlir][python] Enable python bindings for Index dialect (#85827) This small patch enables python bindings for the index dialect. --------- Co-authored-by: Steven Varoumas <steven.varoumas1@huawei.com>
2024-03-20 07:56:22 -08:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/IndexOps.td
SOURCES
dialects/index.py
DIALECT_NAME index
GEN_ENUM_BINDINGS)
[mlir] Fixup python bindings after splitting cf ops from std.
2022-02-05 20:50:27 -09:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/ControlFlowOps.td
SOURCES
dialects/cf.py
DIALECT_NAME cf)
[mlir] Rename the Standard dialect to the Func dialect The last remaining operations in the standard dialect all revolve around FuncOp/function related constructs. This patch simply handles the initial renaming (which by itself is already huge), but there are a large number of cleanups unlocked/necessary afterwards: * Removing a bunch of unnecessary dependencies on Func * Cleaning up the From/ToStandard conversion passes * Preparing for the move of FuncOp to the Func dialect See the discussion at https://discourse.llvm.org/t/standard-dialect-the-final-chapter/6061 Differential Revision: https://reviews.llvm.org/D120624
2022-02-26 13:49:54 -09:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/FuncOps.td
SOURCES
dialects/func.py
DIALECT_NAME func)
[mlir][python] Add bindings for OpenACC dialect (#163620) Adds initial support for Python bindings to the OpenACC dialect. * The bindings do not provide any niceties yet, just the barebones exposure of the dialect to Python. Construction of OpenACC ops is therefore verbose and somewhat inconvenient, as evidenced by the test. * The test only constructs one module, but I attempted to use enough operations to be meaningful. It does not test all the ops exposed, but does contain a realistic example of a memcpy idiom.
2025-10-16 06:30:19 -08:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/OpenACCOps.td
SOURCES
dialects/openacc.py
DIALECT_NAME acc
DEPENDS acc_common_td
)
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/GPUOps.td
SOURCES_GLOB dialects/gpu/*.py
[mlir][python bindings] generate all the enums This PR implements python enum bindings for *all* the enums - this includes `I*Attrs` (including positional/bit) and `Dialect/EnumAttr`. There are a few parts to this: 1. CMake: a small addition to `declare_mlir_dialect_python_bindings` and `declare_mlir_dialect_extension_python_bindings` to generate the enum, a boolean arg `GEN_ENUM_BINDINGS` to make it opt-in (even though it works for basically all of the dialects), and an optional `GEN_ENUM_BINDINGS_TD_FILE` for handling corner cases. 2. EnumPythonBindingGen.cpp: there are two weedy aspects here that took investigation: 1. If an enum attribute is not a `Dialect/EnumAttr` then the `EnumAttrInfo` record is canonical, as far as both the cases of the enum **and the `AttrDefName`**. On the otherhand, if an enum is a `Dialect/EnumAttr` then the `EnumAttr` record has the correct `AttrDefName` ("load bearing", i.e., populates `ods.ir.AttributeBuilder('<NAME>')`) but its `enum` field contains the cases, which is an instance of `EnumAttrInfo`. The solution is to generate an one enum class for both `Dialect/EnumAttr` and "independent" `EnumAttrInfo` but to make that class interopable with two builder registrations that both do the right thing (see next sub-bullet). 2. Because we don't have a good connection to cpp `EnumAttr`, i.e., only the `enum class` getters are exposed (like `DimensionAttr::get(Dimension value)`), we have to resort to parsing e.g., `Attribute.parse(f'#gpu<dim {x}>')`. This means that the set of supported `assemblyFormat`s (for the enum) is fixed at compile of MLIR (currently 2, the only 2 I saw). There might be some things that could be done here but they would require quite a bit more C API work to support generically (e.g., casting ints to enum cases and binding all the getters or going generically through the `symbolize*` methods, like `symbolizeDimension(uint32_t)` or `symbolizeDimension(StringRef)`). A few small changes: 1. In addition, since this patch registers default builders for attributes where people might've had their own builders already written, I added a `replace` param to `AttributeBuilder.insert` (`False` by default). 2. `makePythonEnumCaseName` can't handle all the different ways in which people write their enum cases, e.g., `llvm.CConv.Intel_OCL_BI`, which gets turned into `INTEL_O_C_L_B_I` (because `llvm::convertToSnakeFromCamelCase` doesn't look for runs of caps). So I dropped it. On the otherhand regularization does need to done because some enums have `None` as a case (and others might have other python keywords). 3. I turned on `llvm` dialect generation here in order to test `nvvm.WGMMAScaleIn`, which is an enum with [[ https://github.com/llvm/llvm-project/blob/d7e26b56207cbd8995296c5bb7c11ce676b649da/mlir/include/mlir/IR/EnumAttr.td#L22-L25 | no explicit discriminator ]] for the `neg` case. Note, dialects that didn't get a `GEN_ENUM_BINDINGS` don't have any enums to generate. Let me know if I should add more tests (the three trivial ones I added exercise both the supported `assemblyFormat`s and `replace=True`). Reviewed By: stellaraccident Differential Revision: https://reviews.llvm.org/D157934
2023-08-23 10:27:08 -08:00
DIALECT_NAME gpu
GEN_ENUM_BINDINGS)
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/LinalgOps.td
SOURCES
SOURCES_GLOB
dialects/linalg/*.py
DIALECT_NAME linalg
[mlir][python bindings] generate all the enums This PR implements python enum bindings for *all* the enums - this includes `I*Attrs` (including positional/bit) and `Dialect/EnumAttr`. There are a few parts to this: 1. CMake: a small addition to `declare_mlir_dialect_python_bindings` and `declare_mlir_dialect_extension_python_bindings` to generate the enum, a boolean arg `GEN_ENUM_BINDINGS` to make it opt-in (even though it works for basically all of the dialects), and an optional `GEN_ENUM_BINDINGS_TD_FILE` for handling corner cases. 2. EnumPythonBindingGen.cpp: there are two weedy aspects here that took investigation: 1. If an enum attribute is not a `Dialect/EnumAttr` then the `EnumAttrInfo` record is canonical, as far as both the cases of the enum **and the `AttrDefName`**. On the otherhand, if an enum is a `Dialect/EnumAttr` then the `EnumAttr` record has the correct `AttrDefName` ("load bearing", i.e., populates `ods.ir.AttributeBuilder('<NAME>')`) but its `enum` field contains the cases, which is an instance of `EnumAttrInfo`. The solution is to generate an one enum class for both `Dialect/EnumAttr` and "independent" `EnumAttrInfo` but to make that class interopable with two builder registrations that both do the right thing (see next sub-bullet). 2. Because we don't have a good connection to cpp `EnumAttr`, i.e., only the `enum class` getters are exposed (like `DimensionAttr::get(Dimension value)`), we have to resort to parsing e.g., `Attribute.parse(f'#gpu<dim {x}>')`. This means that the set of supported `assemblyFormat`s (for the enum) is fixed at compile of MLIR (currently 2, the only 2 I saw). There might be some things that could be done here but they would require quite a bit more C API work to support generically (e.g., casting ints to enum cases and binding all the getters or going generically through the `symbolize*` methods, like `symbolizeDimension(uint32_t)` or `symbolizeDimension(StringRef)`). A few small changes: 1. In addition, since this patch registers default builders for attributes where people might've had their own builders already written, I added a `replace` param to `AttributeBuilder.insert` (`False` by default). 2. `makePythonEnumCaseName` can't handle all the different ways in which people write their enum cases, e.g., `llvm.CConv.Intel_OCL_BI`, which gets turned into `INTEL_O_C_L_B_I` (because `llvm::convertToSnakeFromCamelCase` doesn't look for runs of caps). So I dropped it. On the otherhand regularization does need to done because some enums have `None` as a case (and others might have other python keywords). 3. I turned on `llvm` dialect generation here in order to test `nvvm.WGMMAScaleIn`, which is an enum with [[ https://github.com/llvm/llvm-project/blob/d7e26b56207cbd8995296c5bb7c11ce676b649da/mlir/include/mlir/IR/EnumAttr.td#L22-L25 | no explicit discriminator ]] for the `neg` case. Note, dialects that didn't get a `GEN_ENUM_BINDINGS` don't have any enums to generate. Let me know if I should add more tests (the three trivial ones I added exercise both the supported `assemblyFormat`s and `replace=True`). Reviewed By: stellaraccident Differential Revision: https://reviews.llvm.org/D157934
2023-08-23 10:27:08 -08:00
DEPENDS LinalgOdsGen
GEN_ENUM_BINDINGS)
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/LLVMOps.td
SOURCES
dialects/llvm.py
DIALECT_NAME llvm
GEN_ENUM_BINDINGS)
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
[mlir] move PDL-related transform ops into an extension The initial bring-up of the Transform dialect relied on PDL to provide the default handle type (`!pdl.operation`) and the matching capability. Both are now provided natively by the Transform dialect removing the reason to have a hard dependency on the PDL dialect and its interpreter. Move PDL-related transform operations into a separate extension. This requires us to introduce a dialect state extension mechanism into the Transform dialect so it no longer needs to know about PDL constraint functions that may be injected by extensions similarly to operations and types. This mechanism will be reused to connect pattern application drivers and the Transform dialect. This completes the restructuring of the Transform dialect to remove overrilance on PDL. Note to downstreams: flow that are using `!pdl.operation` with Transform dialect operations will now require `transform::PDLExtension` to be applied to the transform dialect in order to provide the transform handle type interface for `!pdl.operation`. Reviewed By: springerm Differential Revision: https://reviews.llvm.org/D151104
2023-05-22 06:36:58 -08:00
declare_mlir_dialect_extension_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/TransformPDLExtensionOps.td
SOURCES
dialects/transform/pdl.py
DIALECT_NAME transform
EXTENSION_NAME transform_pdl_extension)
[MLIR][Transform] expose transform.debug extension in Python (#145550) Removes the Debug... prefix on the ops in tablegen, in line with pretty much all other Transform-dialect extension ops. This means that the ops in Python look like `debug.EmitParamAsRemarkOp`/`debug.emit_param_as_remark` instead of `debug.DebugEmitParamAsRemarkOp`/`debug.debug_emit_param_as_remark`.
2025-06-25 07:39:01 -08:00
declare_mlir_dialect_extension_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
[MLIR][Transform][SMT] Introduce transform.smt.constrain_params (#159450) Introduces a Transform-dialect SMT-extension so that we can have an op to express constrains on Transform-dialect params, in particular when these params are knobs -- see transform.tune.knob -- and can hence be seen as symbolic variables. This op allows expressing joint constraints over multiple params/knobs together. While the op's semantics are clearly defined, per SMTLIB, the interpreted semantics -- i.e. the `apply()` method -- for now just defaults to failure. In the future we should support attaching an implementation so that users can Bring Your Own Solver and thereby control performance of interpreting the op. For now the main usage is to walk schedule IR and collect these constraints so that knobs can be rewritten to constants that satisfy the constraints.
2025-09-21 12:32:45 -08:00
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/TransformSMTExtensionOps.td
SOURCES
dialects/transform/smt.py
DIALECT_NAME transform
EXTENSION_NAME transform_smt_extension)
declare_mlir_dialect_extension_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
[MLIR][Transform] expose transform.debug extension in Python (#145550) Removes the Debug... prefix on the ops in tablegen, in line with pretty much all other Transform-dialect extension ops. This means that the ops in Python look like `debug.EmitParamAsRemarkOp`/`debug.emit_param_as_remark` instead of `debug.DebugEmitParamAsRemarkOp`/`debug.debug_emit_param_as_remark`.
2025-06-25 07:39:01 -08:00
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/TransformDebugExtensionOps.td
SOURCES
dialects/transform/debug.py
DIALECT_NAME transform
EXTENSION_NAME transform_debug_extension)
[MLIR][Transform] Introduce `transform.tune.knob` op (#146732) A new transform op to represent that an attribute is to be chosen from a set of alternatives and that this choice is made available as a `!transform.param`. When a `selected` argument is provided, the op's `apply()` semantics is that of just making this selected attribute available as the result. When `selected` is not provided, `apply()` complains that nothing has resolved the non-determinism that the op is representing.
2025-07-08 02:00:34 -08:00
declare_mlir_dialect_extension_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/TransformTuneExtensionOps.td
SOURCES
dialects/transform/tune.py
DIALECT_NAME transform
EXTENSION_NAME transform_tune_extension)
[mlir] provide Python bindings for the Transform dialect Python bindings for extensions of the Transform dialect are defined in separate Python source files that can be imported on-demand, i.e., that are not imported with the "main" transform dialect. This requires a minor addition to the ODS-based bindings generator. This approach is consistent with the current model for downstream projects that are expected to bundle MLIR Python bindings: such projects can include their custom extensions into the bundle similarly to how they include their dialects. Reviewed By: nicolasvasilache Differential Revision: https://reviews.llvm.org/D126208
2022-05-30 05:14:02 -08:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/TransformOps.td
SOURCES
dialects/transform/__init__.py
Revert stubgen-related patches (#157831) Despite several hotfixes, things remain broken, in particular: - installation/distribution (`ninja install / install-distribution`); - downstream projects with bindings exposed. See https://github.com/llvm/llvm-project/pull/157583#issuecomment-3274164357 for more details. Reverts #155741, #157583, #157697. Let's make sure things are fixed and re-land as a unit.
2025-09-10 03:35:38 -08:00
_mlir_libs/_mlir/dialects/transform/__init__.pyi
[mlir][python bindings] generate all the enums This PR implements python enum bindings for *all* the enums - this includes `I*Attrs` (including positional/bit) and `Dialect/EnumAttr`. There are a few parts to this: 1. CMake: a small addition to `declare_mlir_dialect_python_bindings` and `declare_mlir_dialect_extension_python_bindings` to generate the enum, a boolean arg `GEN_ENUM_BINDINGS` to make it opt-in (even though it works for basically all of the dialects), and an optional `GEN_ENUM_BINDINGS_TD_FILE` for handling corner cases. 2. EnumPythonBindingGen.cpp: there are two weedy aspects here that took investigation: 1. If an enum attribute is not a `Dialect/EnumAttr` then the `EnumAttrInfo` record is canonical, as far as both the cases of the enum **and the `AttrDefName`**. On the otherhand, if an enum is a `Dialect/EnumAttr` then the `EnumAttr` record has the correct `AttrDefName` ("load bearing", i.e., populates `ods.ir.AttributeBuilder('<NAME>')`) but its `enum` field contains the cases, which is an instance of `EnumAttrInfo`. The solution is to generate an one enum class for both `Dialect/EnumAttr` and "independent" `EnumAttrInfo` but to make that class interopable with two builder registrations that both do the right thing (see next sub-bullet). 2. Because we don't have a good connection to cpp `EnumAttr`, i.e., only the `enum class` getters are exposed (like `DimensionAttr::get(Dimension value)`), we have to resort to parsing e.g., `Attribute.parse(f'#gpu<dim {x}>')`. This means that the set of supported `assemblyFormat`s (for the enum) is fixed at compile of MLIR (currently 2, the only 2 I saw). There might be some things that could be done here but they would require quite a bit more C API work to support generically (e.g., casting ints to enum cases and binding all the getters or going generically through the `symbolize*` methods, like `symbolizeDimension(uint32_t)` or `symbolizeDimension(StringRef)`). A few small changes: 1. In addition, since this patch registers default builders for attributes where people might've had their own builders already written, I added a `replace` param to `AttributeBuilder.insert` (`False` by default). 2. `makePythonEnumCaseName` can't handle all the different ways in which people write their enum cases, e.g., `llvm.CConv.Intel_OCL_BI`, which gets turned into `INTEL_O_C_L_B_I` (because `llvm::convertToSnakeFromCamelCase` doesn't look for runs of caps). So I dropped it. On the otherhand regularization does need to done because some enums have `None` as a case (and others might have other python keywords). 3. I turned on `llvm` dialect generation here in order to test `nvvm.WGMMAScaleIn`, which is an enum with [[ https://github.com/llvm/llvm-project/blob/d7e26b56207cbd8995296c5bb7c11ce676b649da/mlir/include/mlir/IR/EnumAttr.td#L22-L25 | no explicit discriminator ]] for the `neg` case. Note, dialects that didn't get a `GEN_ENUM_BINDINGS` don't have any enums to generate. Let me know if I should add more tests (the three trivial ones I added exercise both the supported `assemblyFormat`s and `replace=True`). Reviewed By: stellaraccident Differential Revision: https://reviews.llvm.org/D157934
2023-08-23 10:27:08 -08:00
DIALECT_NAME transform
GEN_ENUM_BINDINGS_TD_FILE
"../../include/mlir/Dialect/Transform/IR/TransformAttrs.td"
)
[mlir] python enum bindings generator Add an ODS (tablegen) backend to generate Python enum classes and attribute builders for enum attributes defined in ODS. This will allow us to keep the enum attribute definitions in sync between C++ and Python, as opposed to handwritten enum classes in Python that may end up using mismatching values. This also makes autogenerated bindings more convenient even in absence of mixins. Use this backend for the transform dialect failure propagation mode enum attribute as demonstration. Reviewed By: ingomueller-net Differential Revision: https://reviews.llvm.org/D156553
2023-07-28 08:03:33 -08:00
[MLIR][transform][python] add sugared python abstractions for transform dialect (#75073) This adds Python abstractions for the different handle types of the transform dialect The abstractions allow for straightforward chaining of transforms by calling their member functions. As an initial PR for this infrastructure, only a single transform is included: `transform.structured.match`. With a future `tile` transform abstraction an example of the usage is: ```Python def script(module: OpHandle): module.match_ops(MatchInterfaceEnum.TilingInterface).tile(tile_sizes=[32,32]) ``` to generate the following IR: ```mlir %0 = transform.structured.match interface{TilingInterface} in %arg0 %tiled_op, %loops = transform.structured.tile_using_for %0 [32, 32] ``` These abstractions are intended to enhance the usability and flexibility of the transform dialect by providing an accessible interface that allows for easy assembly of complex transformation chains.
2023-12-15 03:04:43 -09:00
declare_mlir_python_sources(
MLIRPythonSources.Dialects.transform.extras
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
GEN_ENUM_BINDINGS
SOURCES
[mlir][python] move transform extras (#76102)
2023-12-20 14:29:11 -09:00
dialects/transform/extras/__init__.py)
[MLIR][transform][python] add sugared python abstractions for transform dialect (#75073) This adds Python abstractions for the different handle types of the transform dialect The abstractions allow for straightforward chaining of transforms by calling their member functions. As an initial PR for this infrastructure, only a single transform is included: `transform.structured.match`. With a future `tile` transform abstraction an example of the usage is: ```Python def script(module: OpHandle): module.match_ops(MatchInterfaceEnum.TilingInterface).tile(tile_sizes=[32,32]) ``` to generate the following IR: ```mlir %0 = transform.structured.match interface{TilingInterface} in %arg0 %tiled_op, %loops = transform.structured.tile_using_for %0 [32, 32] ``` These abstractions are intended to enhance the usability and flexibility of the transform dialect by providing an accessible interface that allows for easy assembly of complex transformation chains.
2023-12-15 03:04:43 -09:00
[mlir] expose transform interpreter to Python (#82365) Transform interpreter functionality can be used standalone without going through the interpreter pass, make it available in Python.
2024-02-21 01:01:00 -09:00
declare_mlir_python_sources(
MLIRPythonSources.Dialects.transform.interpreter
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
SOURCES
dialects/transform/interpreter/__init__.py)
[mlir][transform][bufferization][python] Add .td file for bindings. Reviewed By: springerm, ftynse Differential Revision: https://reviews.llvm.org/D155564
2023-07-18 00:15:53 -08:00
declare_mlir_dialect_extension_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/BufferizationTransformOps.td
SOURCES
dialects/transform/bufferization.py
DIALECT_NAME transform
EXTENSION_NAME bufferization_transform)
[mlir][transform][gpu][python] Add .td file for bindings. Reviewed By: ftynse Differential Revision: https://reviews.llvm.org/D155602
2023-07-18 07:02:32 -08:00
declare_mlir_dialect_extension_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/GPUTransformOps.td
SOURCES
dialects/transform/gpu.py
DIALECT_NAME transform
EXTENSION_NAME gpu_transform)
[mlir] Introduce Transform ops for loops Introduce transform ops for "for" loops, in particular for peeling, software pipelining and unrolling, along with a couple of "IR navigation" ops. These ops are intended to be generalized to different kinds of loops when possible and therefore use the "loop" prefix. They currently live in the SCF dialect as there is no clear place to put transform ops that may span across several dialects, this decision is postponed until the ops actually need to handle non-SCF loops. Additionally refactor some common utilities for transform ops into trait or interface methods, and change the loop pipelining to be a returning pattern. Reviewed By: springerm Differential Revision: https://reviews.llvm.org/D127300
2022-06-09 01:10:32 -08:00
declare_mlir_dialect_extension_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/SCFLoopTransformOps.td
SOURCES
dialects/transform/loop.py
DIALECT_NAME transform
EXTENSION_NAME loop_transform)
[mlir][memref][transform][python] Create .td file for bindings. This patch creates the .td files for the Python bindings of the transform ops of the MemRef dialect and integrates them into the build systems (CMake and Bazel). Reviewed By: ftynse Differential Revision: https://reviews.llvm.org/D156536
2023-07-28 05:47:23 -08:00
declare_mlir_dialect_extension_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/MemRefTransformOps.td
SOURCES
dialects/transform/memref.py
DIALECT_NAME transform
EXTENSION_NAME memref_transform)
[mlir] enable python bindings for nvgpu transforms (#68088) Expose the autogenerated bindings. Co-authored-by: Martin Lücke <mluecke@google.com>
2023-10-03 04:52:52 -08:00
declare_mlir_dialect_extension_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/NVGPUTransformOps.td
SOURCES
dialects/transform/nvgpu.py
DIALECT_NAME transform
EXTENSION_NAME nvgpu_transform)
[mlir] provide Python bindings for the Transform dialect Python bindings for extensions of the Transform dialect are defined in separate Python source files that can be imported on-demand, i.e., that are not imported with the "main" transform dialect. This requires a minor addition to the ODS-based bindings generator. This approach is consistent with the current model for downstream projects that are expected to bundle MLIR Python bindings: such projects can include their custom extensions into the bundle similarly to how they include their dialects. Reviewed By: nicolasvasilache Differential Revision: https://reviews.llvm.org/D126208
2022-05-30 05:14:02 -08:00
declare_mlir_dialect_extension_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/LinalgStructuredTransformOps.td
SOURCES
dialects/transform/structured.py
DIALECT_NAME transform
[mlir][python bindings] generate all the enums This PR implements python enum bindings for *all* the enums - this includes `I*Attrs` (including positional/bit) and `Dialect/EnumAttr`. There are a few parts to this: 1. CMake: a small addition to `declare_mlir_dialect_python_bindings` and `declare_mlir_dialect_extension_python_bindings` to generate the enum, a boolean arg `GEN_ENUM_BINDINGS` to make it opt-in (even though it works for basically all of the dialects), and an optional `GEN_ENUM_BINDINGS_TD_FILE` for handling corner cases. 2. EnumPythonBindingGen.cpp: there are two weedy aspects here that took investigation: 1. If an enum attribute is not a `Dialect/EnumAttr` then the `EnumAttrInfo` record is canonical, as far as both the cases of the enum **and the `AttrDefName`**. On the otherhand, if an enum is a `Dialect/EnumAttr` then the `EnumAttr` record has the correct `AttrDefName` ("load bearing", i.e., populates `ods.ir.AttributeBuilder('<NAME>')`) but its `enum` field contains the cases, which is an instance of `EnumAttrInfo`. The solution is to generate an one enum class for both `Dialect/EnumAttr` and "independent" `EnumAttrInfo` but to make that class interopable with two builder registrations that both do the right thing (see next sub-bullet). 2. Because we don't have a good connection to cpp `EnumAttr`, i.e., only the `enum class` getters are exposed (like `DimensionAttr::get(Dimension value)`), we have to resort to parsing e.g., `Attribute.parse(f'#gpu<dim {x}>')`. This means that the set of supported `assemblyFormat`s (for the enum) is fixed at compile of MLIR (currently 2, the only 2 I saw). There might be some things that could be done here but they would require quite a bit more C API work to support generically (e.g., casting ints to enum cases and binding all the getters or going generically through the `symbolize*` methods, like `symbolizeDimension(uint32_t)` or `symbolizeDimension(StringRef)`). A few small changes: 1. In addition, since this patch registers default builders for attributes where people might've had their own builders already written, I added a `replace` param to `AttributeBuilder.insert` (`False` by default). 2. `makePythonEnumCaseName` can't handle all the different ways in which people write their enum cases, e.g., `llvm.CConv.Intel_OCL_BI`, which gets turned into `INTEL_O_C_L_B_I` (because `llvm::convertToSnakeFromCamelCase` doesn't look for runs of caps). So I dropped it. On the otherhand regularization does need to done because some enums have `None` as a case (and others might have other python keywords). 3. I turned on `llvm` dialect generation here in order to test `nvvm.WGMMAScaleIn`, which is an enum with [[ https://github.com/llvm/llvm-project/blob/d7e26b56207cbd8995296c5bb7c11ce676b649da/mlir/include/mlir/IR/EnumAttr.td#L22-L25 | no explicit discriminator ]] for the `neg` case. Note, dialects that didn't get a `GEN_ENUM_BINDINGS` don't have any enums to generate. Let me know if I should add more tests (the three trivial ones I added exercise both the supported `assemblyFormat`s and `replace=True`). Reviewed By: stellaraccident Differential Revision: https://reviews.llvm.org/D157934
2023-08-23 10:27:08 -08:00
EXTENSION_NAME structured_transform
GEN_ENUM_BINDINGS_TD_FILE
"../../include/mlir/Dialect/Linalg/TransformOps/LinalgTransformEnums.td"
)
[mlir] provide Python bindings for the Transform dialect Python bindings for extensions of the Transform dialect are defined in separate Python source files that can be imported on-demand, i.e., that are not imported with the "main" transform dialect. This requires a minor addition to the ODS-based bindings generator. This approach is consistent with the current model for downstream projects that are expected to bundle MLIR Python bindings: such projects can include their custom extensions into the bundle similarly to how they include their dialects. Reviewed By: nicolasvasilache Differential Revision: https://reviews.llvm.org/D126208
2022-05-30 05:14:02 -08:00
[mlir][sparse] Generates python bindings for SparseTensorTransformOps. (#66937)
2023-09-20 14:35:50 -08:00
declare_mlir_dialect_extension_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/SparseTensorTransformOps.td
SOURCES
dialects/transform/sparse_tensor.py
DIALECT_NAME transform
EXTENSION_NAME sparse_tensor_transform)
[mlir][transform][tensor][python] Add .td files for bindings. Reviewed By: ftynse Differential Revision: https://reviews.llvm.org/D156914
2023-08-02 08:58:44 -08:00
declare_mlir_dialect_extension_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/TensorTransformOps.td
SOURCES
dialects/transform/tensor.py
DIALECT_NAME transform
EXTENSION_NAME tensor_transform)
[mlir] python bindings for vector transform ops Provide Python bindings for transform ops defined in the vector dialect. All of these ops are sufficiently simple that no mixins are necessary for them to be nicely usable. Reviewed By: ingomueller-net Differential Revision: https://reviews.llvm.org/D156554
2023-07-28 08:06:31 -08:00
declare_mlir_dialect_extension_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/VectorTransformOps.td
SOURCES
dialects/transform/vector.py
DIALECT_NAME transform
[mlir][python bindings] generate all the enums This PR implements python enum bindings for *all* the enums - this includes `I*Attrs` (including positional/bit) and `Dialect/EnumAttr`. There are a few parts to this: 1. CMake: a small addition to `declare_mlir_dialect_python_bindings` and `declare_mlir_dialect_extension_python_bindings` to generate the enum, a boolean arg `GEN_ENUM_BINDINGS` to make it opt-in (even though it works for basically all of the dialects), and an optional `GEN_ENUM_BINDINGS_TD_FILE` for handling corner cases. 2. EnumPythonBindingGen.cpp: there are two weedy aspects here that took investigation: 1. If an enum attribute is not a `Dialect/EnumAttr` then the `EnumAttrInfo` record is canonical, as far as both the cases of the enum **and the `AttrDefName`**. On the otherhand, if an enum is a `Dialect/EnumAttr` then the `EnumAttr` record has the correct `AttrDefName` ("load bearing", i.e., populates `ods.ir.AttributeBuilder('<NAME>')`) but its `enum` field contains the cases, which is an instance of `EnumAttrInfo`. The solution is to generate an one enum class for both `Dialect/EnumAttr` and "independent" `EnumAttrInfo` but to make that class interopable with two builder registrations that both do the right thing (see next sub-bullet). 2. Because we don't have a good connection to cpp `EnumAttr`, i.e., only the `enum class` getters are exposed (like `DimensionAttr::get(Dimension value)`), we have to resort to parsing e.g., `Attribute.parse(f'#gpu<dim {x}>')`. This means that the set of supported `assemblyFormat`s (for the enum) is fixed at compile of MLIR (currently 2, the only 2 I saw). There might be some things that could be done here but they would require quite a bit more C API work to support generically (e.g., casting ints to enum cases and binding all the getters or going generically through the `symbolize*` methods, like `symbolizeDimension(uint32_t)` or `symbolizeDimension(StringRef)`). A few small changes: 1. In addition, since this patch registers default builders for attributes where people might've had their own builders already written, I added a `replace` param to `AttributeBuilder.insert` (`False` by default). 2. `makePythonEnumCaseName` can't handle all the different ways in which people write their enum cases, e.g., `llvm.CConv.Intel_OCL_BI`, which gets turned into `INTEL_O_C_L_B_I` (because `llvm::convertToSnakeFromCamelCase` doesn't look for runs of caps). So I dropped it. On the otherhand regularization does need to done because some enums have `None` as a case (and others might have other python keywords). 3. I turned on `llvm` dialect generation here in order to test `nvvm.WGMMAScaleIn`, which is an enum with [[ https://github.com/llvm/llvm-project/blob/d7e26b56207cbd8995296c5bb7c11ce676b649da/mlir/include/mlir/IR/EnumAttr.td#L22-L25 | no explicit discriminator ]] for the `neg` case. Note, dialects that didn't get a `GEN_ENUM_BINDINGS` don't have any enums to generate. Let me know if I should add more tests (the three trivial ones I added exercise both the supported `assemblyFormat`s and `replace=True`). Reviewed By: stellaraccident Differential Revision: https://reviews.llvm.org/D157934
2023-08-23 10:27:08 -08:00
EXTENSION_NAME vector_transform
GEN_ENUM_BINDINGS_TD_FILE
"../../include/mlir/Dialect/Vector/Transforms/VectorTransformsBase.td"
)
[mlir] python bindings for vector transform ops Provide Python bindings for transform ops defined in the vector dialect. All of these ops are sufficiently simple that no mixins are necessary for them to be nicely usable. Reviewed By: ingomueller-net Differential Revision: https://reviews.llvm.org/D156554
2023-07-28 08:06:31 -08:00
[mlir][x86vector] Python bindings for x86vector dialect (#179958) Registers python bindings for x86vector dialect and transform ops.
2026-02-05 10:05:17 -09:00
declare_mlir_dialect_extension_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
[mlir][x86] Rename x86vector to x86 (#183311) Renames 'x86vector' dialect to 'x86'. This is the first PR in series of cleanups around dialects targeting x86 platforms. The new naming scheme is shorter, cleaner, and opens possibility of integrating other x86-specific operations not strictly fitting pure vector representation. For example, the generalization will allow for future merger of AMX dialect into the x86 dialect to create one-stop x86 operations collection and boost discoverability.
2026-02-26 01:21:58 -09:00
TD_FILE dialects/X86TransformOps.td
[mlir][x86vector] Python bindings for x86vector dialect (#179958) Registers python bindings for x86vector dialect and transform ops.
2026-02-05 10:05:17 -09:00
SOURCES
[mlir][x86] Rename x86vector to x86 (#183311) Renames 'x86vector' dialect to 'x86'. This is the first PR in series of cleanups around dialects targeting x86 platforms. The new naming scheme is shorter, cleaner, and opens possibility of integrating other x86-specific operations not strictly fitting pure vector representation. For example, the generalization will allow for future merger of AMX dialect into the x86 dialect to create one-stop x86 operations collection and boost discoverability.
2026-02-26 01:21:58 -09:00
dialects/transform/x86.py
[mlir][x86vector] Python bindings for x86vector dialect (#179958) Registers python bindings for x86vector dialect and transform ops.
2026-02-05 10:05:17 -09:00
DIALECT_NAME transform
[mlir][x86] Rename x86vector to x86 (#183311) Renames 'x86vector' dialect to 'x86'. This is the first PR in series of cleanups around dialects targeting x86 platforms. The new naming scheme is shorter, cleaner, and opens possibility of integrating other x86-specific operations not strictly fitting pure vector representation. For example, the generalization will allow for future merger of AMX dialect into the x86 dialect to create one-stop x86 operations collection and boost discoverability.
2026-02-26 01:21:58 -09:00
EXTENSION_NAME x86_transform)
[mlir][x86vector] Python bindings for x86vector dialect (#179958) Registers python bindings for x86vector dialect and transform ops.
2026-02-05 10:05:17 -09:00
[MLIR][XeGPU][Transform] add xegpu.set_desc_layout transform op (#165615) Adds the first XeGPU transform op, `xegpu.set_desc_layout`, which attachs a `xegpu.layout` attribute to the descriptor that a `xegpu.create_nd_tdesc` op returns.
2025-11-06 05:25:34 -09:00
declare_mlir_dialect_extension_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/XeGPUTransformOps.td
SOURCES
dialects/transform/xegpu.py
DIALECT_NAME transform
EXTENSION_NAME xegpu_transform)
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/MathOps.td
SOURCES dialects/math.py
DIALECT_NAME math)
[MLIR] Replace std ops with arith dialect ops Precursor: https://reviews.llvm.org/D110200 Removed redundant ops from the standard dialect that were moved to the `arith` or `math` dialects. Renamed all instances of operations in the codebase and in tests. Reviewed By: rriddle, jpienaar Differential Revision: https://reviews.llvm.org/D110797
2021-10-12 15:14:57 -08:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
[mlir][arith] Change dialect name from Arithmetic to Arith Suggested by @lattner in https://discourse.llvm.org/t/rfc-define-precise-arith-semantics/65507/22. Tested with: `ninja check-mlir check-mlir-integration check-mlir-mlir-spirv-cpu-runner check-mlir-mlir-vulkan-runner check-mlir-examples` and `bazel build --config=generic_clang @llvm-project//mlir:all`. Reviewed By: lattner, Mogball, rriddle, jpienaar, mehdi_amini Differential Revision: https://reviews.llvm.org/D134762
2022-09-29 07:14:47 -08:00
TD_FILE dialects/ArithOps.td
[MLIR] Replace std ops with arith dialect ops Precursor: https://reviews.llvm.org/D110200 Removed redundant ops from the standard dialect that were moved to the `arith` or `math` dialects. Renamed all instances of operations in the codebase and in tests. Reviewed By: rriddle, jpienaar Differential Revision: https://reviews.llvm.org/D110797
2021-10-12 15:14:57 -08:00
SOURCES
dialects/arith.py
[mlir][python bindings] generate all the enums This PR implements python enum bindings for *all* the enums - this includes `I*Attrs` (including positional/bit) and `Dialect/EnumAttr`. There are a few parts to this: 1. CMake: a small addition to `declare_mlir_dialect_python_bindings` and `declare_mlir_dialect_extension_python_bindings` to generate the enum, a boolean arg `GEN_ENUM_BINDINGS` to make it opt-in (even though it works for basically all of the dialects), and an optional `GEN_ENUM_BINDINGS_TD_FILE` for handling corner cases. 2. EnumPythonBindingGen.cpp: there are two weedy aspects here that took investigation: 1. If an enum attribute is not a `Dialect/EnumAttr` then the `EnumAttrInfo` record is canonical, as far as both the cases of the enum **and the `AttrDefName`**. On the otherhand, if an enum is a `Dialect/EnumAttr` then the `EnumAttr` record has the correct `AttrDefName` ("load bearing", i.e., populates `ods.ir.AttributeBuilder('<NAME>')`) but its `enum` field contains the cases, which is an instance of `EnumAttrInfo`. The solution is to generate an one enum class for both `Dialect/EnumAttr` and "independent" `EnumAttrInfo` but to make that class interopable with two builder registrations that both do the right thing (see next sub-bullet). 2. Because we don't have a good connection to cpp `EnumAttr`, i.e., only the `enum class` getters are exposed (like `DimensionAttr::get(Dimension value)`), we have to resort to parsing e.g., `Attribute.parse(f'#gpu<dim {x}>')`. This means that the set of supported `assemblyFormat`s (for the enum) is fixed at compile of MLIR (currently 2, the only 2 I saw). There might be some things that could be done here but they would require quite a bit more C API work to support generically (e.g., casting ints to enum cases and binding all the getters or going generically through the `symbolize*` methods, like `symbolizeDimension(uint32_t)` or `symbolizeDimension(StringRef)`). A few small changes: 1. In addition, since this patch registers default builders for attributes where people might've had their own builders already written, I added a `replace` param to `AttributeBuilder.insert` (`False` by default). 2. `makePythonEnumCaseName` can't handle all the different ways in which people write their enum cases, e.g., `llvm.CConv.Intel_OCL_BI`, which gets turned into `INTEL_O_C_L_B_I` (because `llvm::convertToSnakeFromCamelCase` doesn't look for runs of caps). So I dropped it. On the otherhand regularization does need to done because some enums have `None` as a case (and others might have other python keywords). 3. I turned on `llvm` dialect generation here in order to test `nvvm.WGMMAScaleIn`, which is an enum with [[ https://github.com/llvm/llvm-project/blob/d7e26b56207cbd8995296c5bb7c11ce676b649da/mlir/include/mlir/IR/EnumAttr.td#L22-L25 | no explicit discriminator ]] for the `neg` case. Note, dialects that didn't get a `GEN_ENUM_BINDINGS` don't have any enums to generate. Let me know if I should add more tests (the three trivial ones I added exercise both the supported `assemblyFormat`s and `replace=True`). Reviewed By: stellaraccident Differential Revision: https://reviews.llvm.org/D157934
2023-08-23 10:27:08 -08:00
DIALECT_NAME arith
GEN_ENUM_BINDINGS)
[MLIR] Replace std ops with arith dialect ops Precursor: https://reviews.llvm.org/D110200 Removed redundant ops from the standard dialect that were moved to the `arith` or `math` dialects. Renamed all instances of operations in the codebase and in tests. Reviewed By: rriddle, jpienaar Differential Revision: https://reviews.llvm.org/D110797
2021-10-12 15:14:57 -08:00
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/MemRefOps.td
[mlir] fix python bindings cmake
2021-10-13 07:29:19 -08:00
SOURCES
dialects/memref.py
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
DIALECT_NAME memref)
[MLIR][Python] Add shard Dialect Python Bindings (#162578) Add Python bindings for `shard` dialect. Provide means for creating constructs in this dialect in Python.
2025-10-21 13:22:40 -08:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/ShardOps.td
SOURCES
dialects/shard.py
DIALECT_NAME shard
GEN_ENUM_BINDINGS)
[mlir][python] Add Python bindings for ml_program dialect. Differential Revision: https://reviews.llvm.org/D125852
2022-05-17 21:42:39 -08:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/MLProgramOps.td
SOURCES
dialects/ml_program.py
DIALECT_NAME ml_program)
[MLIR][python bindings] add vendor gpu dialects Differential Revision: https://reviews.llvm.org/D157820
2023-08-13 10:30:28 -08:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/NVGPUOps.td
SOURCES
dialects/nvgpu.py
[mlir][python bindings] generate all the enums This PR implements python enum bindings for *all* the enums - this includes `I*Attrs` (including positional/bit) and `Dialect/EnumAttr`. There are a few parts to this: 1. CMake: a small addition to `declare_mlir_dialect_python_bindings` and `declare_mlir_dialect_extension_python_bindings` to generate the enum, a boolean arg `GEN_ENUM_BINDINGS` to make it opt-in (even though it works for basically all of the dialects), and an optional `GEN_ENUM_BINDINGS_TD_FILE` for handling corner cases. 2. EnumPythonBindingGen.cpp: there are two weedy aspects here that took investigation: 1. If an enum attribute is not a `Dialect/EnumAttr` then the `EnumAttrInfo` record is canonical, as far as both the cases of the enum **and the `AttrDefName`**. On the otherhand, if an enum is a `Dialect/EnumAttr` then the `EnumAttr` record has the correct `AttrDefName` ("load bearing", i.e., populates `ods.ir.AttributeBuilder('<NAME>')`) but its `enum` field contains the cases, which is an instance of `EnumAttrInfo`. The solution is to generate an one enum class for both `Dialect/EnumAttr` and "independent" `EnumAttrInfo` but to make that class interopable with two builder registrations that both do the right thing (see next sub-bullet). 2. Because we don't have a good connection to cpp `EnumAttr`, i.e., only the `enum class` getters are exposed (like `DimensionAttr::get(Dimension value)`), we have to resort to parsing e.g., `Attribute.parse(f'#gpu<dim {x}>')`. This means that the set of supported `assemblyFormat`s (for the enum) is fixed at compile of MLIR (currently 2, the only 2 I saw). There might be some things that could be done here but they would require quite a bit more C API work to support generically (e.g., casting ints to enum cases and binding all the getters or going generically through the `symbolize*` methods, like `symbolizeDimension(uint32_t)` or `symbolizeDimension(StringRef)`). A few small changes: 1. In addition, since this patch registers default builders for attributes where people might've had their own builders already written, I added a `replace` param to `AttributeBuilder.insert` (`False` by default). 2. `makePythonEnumCaseName` can't handle all the different ways in which people write their enum cases, e.g., `llvm.CConv.Intel_OCL_BI`, which gets turned into `INTEL_O_C_L_B_I` (because `llvm::convertToSnakeFromCamelCase` doesn't look for runs of caps). So I dropped it. On the otherhand regularization does need to done because some enums have `None` as a case (and others might have other python keywords). 3. I turned on `llvm` dialect generation here in order to test `nvvm.WGMMAScaleIn`, which is an enum with [[ https://github.com/llvm/llvm-project/blob/d7e26b56207cbd8995296c5bb7c11ce676b649da/mlir/include/mlir/IR/EnumAttr.td#L22-L25 | no explicit discriminator ]] for the `neg` case. Note, dialects that didn't get a `GEN_ENUM_BINDINGS` don't have any enums to generate. Let me know if I should add more tests (the three trivial ones I added exercise both the supported `assemblyFormat`s and `replace=True`). Reviewed By: stellaraccident Differential Revision: https://reviews.llvm.org/D157934
2023-08-23 10:27:08 -08:00
DIALECT_NAME nvgpu
GEN_ENUM_BINDINGS)
[MLIR][python bindings] add vendor gpu dialects Differential Revision: https://reviews.llvm.org/D157820
2023-08-13 10:30:28 -08:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/NVVMOps.td
SOURCES
dialects/nvvm.py
[mlir][python bindings] generate all the enums This PR implements python enum bindings for *all* the enums - this includes `I*Attrs` (including positional/bit) and `Dialect/EnumAttr`. There are a few parts to this: 1. CMake: a small addition to `declare_mlir_dialect_python_bindings` and `declare_mlir_dialect_extension_python_bindings` to generate the enum, a boolean arg `GEN_ENUM_BINDINGS` to make it opt-in (even though it works for basically all of the dialects), and an optional `GEN_ENUM_BINDINGS_TD_FILE` for handling corner cases. 2. EnumPythonBindingGen.cpp: there are two weedy aspects here that took investigation: 1. If an enum attribute is not a `Dialect/EnumAttr` then the `EnumAttrInfo` record is canonical, as far as both the cases of the enum **and the `AttrDefName`**. On the otherhand, if an enum is a `Dialect/EnumAttr` then the `EnumAttr` record has the correct `AttrDefName` ("load bearing", i.e., populates `ods.ir.AttributeBuilder('<NAME>')`) but its `enum` field contains the cases, which is an instance of `EnumAttrInfo`. The solution is to generate an one enum class for both `Dialect/EnumAttr` and "independent" `EnumAttrInfo` but to make that class interopable with two builder registrations that both do the right thing (see next sub-bullet). 2. Because we don't have a good connection to cpp `EnumAttr`, i.e., only the `enum class` getters are exposed (like `DimensionAttr::get(Dimension value)`), we have to resort to parsing e.g., `Attribute.parse(f'#gpu<dim {x}>')`. This means that the set of supported `assemblyFormat`s (for the enum) is fixed at compile of MLIR (currently 2, the only 2 I saw). There might be some things that could be done here but they would require quite a bit more C API work to support generically (e.g., casting ints to enum cases and binding all the getters or going generically through the `symbolize*` methods, like `symbolizeDimension(uint32_t)` or `symbolizeDimension(StringRef)`). A few small changes: 1. In addition, since this patch registers default builders for attributes where people might've had their own builders already written, I added a `replace` param to `AttributeBuilder.insert` (`False` by default). 2. `makePythonEnumCaseName` can't handle all the different ways in which people write their enum cases, e.g., `llvm.CConv.Intel_OCL_BI`, which gets turned into `INTEL_O_C_L_B_I` (because `llvm::convertToSnakeFromCamelCase` doesn't look for runs of caps). So I dropped it. On the otherhand regularization does need to done because some enums have `None` as a case (and others might have other python keywords). 3. I turned on `llvm` dialect generation here in order to test `nvvm.WGMMAScaleIn`, which is an enum with [[ https://github.com/llvm/llvm-project/blob/d7e26b56207cbd8995296c5bb7c11ce676b649da/mlir/include/mlir/IR/EnumAttr.td#L22-L25 | no explicit discriminator ]] for the `neg` case. Note, dialects that didn't get a `GEN_ENUM_BINDINGS` don't have any enums to generate. Let me know if I should add more tests (the three trivial ones I added exercise both the supported `assemblyFormat`s and `replace=True`). Reviewed By: stellaraccident Differential Revision: https://reviews.llvm.org/D157934
2023-08-23 10:27:08 -08:00
DIALECT_NAME nvvm
GEN_ENUM_BINDINGS)
[MLIR][python bindings] add vendor gpu dialects Differential Revision: https://reviews.llvm.org/D157820
2023-08-13 10:30:28 -08:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/ROCDLOps.td
SOURCES
dialects/rocdl.py
DIALECT_NAME rocdl)
[mlir] Introduce Python bindings for the quantization dialect So far, only the custom dialect types are exposed. The build and packaging is same as for Linalg and SparseTensor, and in need of refactoring that is beyond the scope of this patch. Reviewed By: stellaraccident Differential Revision: https://reviews.llvm.org/D116605
2022-01-04 07:43:44 -09:00
declare_mlir_python_sources(
MLIRPythonSources.Dialects.quant
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
[mlir][python bindings] generate all the enums This PR implements python enum bindings for *all* the enums - this includes `I*Attrs` (including positional/bit) and `Dialect/EnumAttr`. There are a few parts to this: 1. CMake: a small addition to `declare_mlir_dialect_python_bindings` and `declare_mlir_dialect_extension_python_bindings` to generate the enum, a boolean arg `GEN_ENUM_BINDINGS` to make it opt-in (even though it works for basically all of the dialects), and an optional `GEN_ENUM_BINDINGS_TD_FILE` for handling corner cases. 2. EnumPythonBindingGen.cpp: there are two weedy aspects here that took investigation: 1. If an enum attribute is not a `Dialect/EnumAttr` then the `EnumAttrInfo` record is canonical, as far as both the cases of the enum **and the `AttrDefName`**. On the otherhand, if an enum is a `Dialect/EnumAttr` then the `EnumAttr` record has the correct `AttrDefName` ("load bearing", i.e., populates `ods.ir.AttributeBuilder('<NAME>')`) but its `enum` field contains the cases, which is an instance of `EnumAttrInfo`. The solution is to generate an one enum class for both `Dialect/EnumAttr` and "independent" `EnumAttrInfo` but to make that class interopable with two builder registrations that both do the right thing (see next sub-bullet). 2. Because we don't have a good connection to cpp `EnumAttr`, i.e., only the `enum class` getters are exposed (like `DimensionAttr::get(Dimension value)`), we have to resort to parsing e.g., `Attribute.parse(f'#gpu<dim {x}>')`. This means that the set of supported `assemblyFormat`s (for the enum) is fixed at compile of MLIR (currently 2, the only 2 I saw). There might be some things that could be done here but they would require quite a bit more C API work to support generically (e.g., casting ints to enum cases and binding all the getters or going generically through the `symbolize*` methods, like `symbolizeDimension(uint32_t)` or `symbolizeDimension(StringRef)`). A few small changes: 1. In addition, since this patch registers default builders for attributes where people might've had their own builders already written, I added a `replace` param to `AttributeBuilder.insert` (`False` by default). 2. `makePythonEnumCaseName` can't handle all the different ways in which people write their enum cases, e.g., `llvm.CConv.Intel_OCL_BI`, which gets turned into `INTEL_O_C_L_B_I` (because `llvm::convertToSnakeFromCamelCase` doesn't look for runs of caps). So I dropped it. On the otherhand regularization does need to done because some enums have `None` as a case (and others might have other python keywords). 3. I turned on `llvm` dialect generation here in order to test `nvvm.WGMMAScaleIn`, which is an enum with [[ https://github.com/llvm/llvm-project/blob/d7e26b56207cbd8995296c5bb7c11ce676b649da/mlir/include/mlir/IR/EnumAttr.td#L22-L25 | no explicit discriminator ]] for the `neg` case. Note, dialects that didn't get a `GEN_ENUM_BINDINGS` don't have any enums to generate. Let me know if I should add more tests (the three trivial ones I added exercise both the supported `assemblyFormat`s and `replace=True`). Reviewed By: stellaraccident Differential Revision: https://reviews.llvm.org/D157934
2023-08-23 10:27:08 -08:00
GEN_ENUM_BINDINGS
[mlir] Introduce Python bindings for the quantization dialect So far, only the custom dialect types are exposed. The build and packaging is same as for Linalg and SparseTensor, and in need of refactoring that is beyond the scope of this patch. Reviewed By: stellaraccident Differential Revision: https://reviews.llvm.org/D116605
2022-01-04 07:43:44 -09:00
SOURCES
Revert stubgen-related patches (#157831) Despite several hotfixes, things remain broken, in particular: - installation/distribution (`ninja install / install-distribution`); - downstream projects with bindings exposed. See https://github.com/llvm/llvm-project/pull/157583#issuecomment-3274164357 for more details. Reverts #155741, #157583, #157697. Let's make sure things are fixed and re-land as a unit.
2025-09-10 03:35:38 -08:00
dialects/quant.py
_mlir_libs/_mlir/dialects/quant.pyi)
[mlir] Introduce Python bindings for the quantization dialect So far, only the custom dialect types are exposed. The build and packaging is same as for Linalg and SparseTensor, and in need of refactoring that is beyond the scope of this patch. Reviewed By: stellaraccident Differential Revision: https://reviews.llvm.org/D116605
2022-01-04 07:43:44 -09:00
[mlir][emitc] Add support for C-API/python binding to EmitC dialect (#119476) Added EmitC dialect bindings.
2024-12-11 09:07:21 -09:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/EmitC.td
[mlir][python] add `MLIR_BINDINGS_PYTHON_INSTALL_PREFIX` to make bindings install dir configurable (#124878) This PR parameterizes the install directory of the MLIR Python bindings in the final distribution.
2025-06-23 11:49:01 -08:00
SOURCES
[mlir][emitc] Add support for C-API/python binding to EmitC dialect (#119476) Added EmitC dialect bindings.
2024-12-11 09:07:21 -09:00
dialects/emitc.py
DIALECT_NAME emitc)
[mlir] Fix PDL python bindings build Fixes incorrect build definition for the bindings for the PDL dialect. Reviewed By: ftynse Differential Revision: https://reviews.llvm.org/D117657
2022-01-19 03:42:14 -09:00
declare_mlir_dialect_python_bindings(
[mlir] Introduce Python bindings for the PDL dialect This change adds full python bindings for PDL, including types and operations with additional mixins to make operation construction more similar to the PDL syntax. Reviewed By: ftynse Differential Revision: https://reviews.llvm.org/D117458
2022-01-13 00:53:21 -09:00
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
[mlir] Fix PDL python bindings build Fixes incorrect build definition for the bindings for the PDL dialect. Reviewed By: ftynse Differential Revision: https://reviews.llvm.org/D117657
2022-01-19 03:42:14 -09:00
TD_FILE dialects/PDLOps.td
[mlir] Introduce Python bindings for the PDL dialect This change adds full python bindings for PDL, including types and operations with additional mixins to make operation construction more similar to the PDL syntax. Reviewed By: ftynse Differential Revision: https://reviews.llvm.org/D117458
2022-01-13 00:53:21 -09:00
SOURCES
dialects/pdl.py
Revert stubgen-related patches (#157831) Despite several hotfixes, things remain broken, in particular: - installation/distribution (`ninja install / install-distribution`); - downstream projects with bindings exposed. See https://github.com/llvm/llvm-project/pull/157583#issuecomment-3274164357 for more details. Reverts #155741, #157583, #157697. Let's make sure things are fixed and re-land as a unit.
2025-09-10 03:35:38 -08:00
_mlir_libs/_mlir/dialects/pdl.pyi
[mlir] Fix PDL python bindings build Fixes incorrect build definition for the bindings for the PDL dialect. Reviewed By: ftynse Differential Revision: https://reviews.llvm.org/D117657
2022-01-19 03:42:14 -09:00
DIALECT_NAME pdl)
[mlir] Introduce Python bindings for the PDL dialect This change adds full python bindings for PDL, including types and operations with additional mixins to make operation construction more similar to the PDL syntax. Reviewed By: ftynse Differential Revision: https://reviews.llvm.org/D117458
2022-01-13 00:53:21 -09:00
[mlir][python bindings] turn on openmp Just as in https://reviews.llvm.org/D157820, dialect registration is independent of any vendor specific libs having been linked/built/etc. Reviewed By: rkayaith Differential Revision: https://reviews.llvm.org/D158670
2023-08-23 14:41:04 -08:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/OpenMPOps.td
SOURCES
dialects/openmp.py
[mlir] Fix missing cmake dependency causing non-deterministic build failure (NFC) Fixes #74611
2023-12-07 02:22:45 -09:00
DIALECT_NAME omp
DEPENDS omp_common_td)
[mlir][python bindings] turn on openmp Just as in https://reviews.llvm.org/D157820, dialect registration is independent of any vendor specific libs having been linked/built/etc. Reviewed By: rkayaith Differential Revision: https://reviews.llvm.org/D158670
2023-08-23 14:41:04 -08:00
[mlir][python] provide bindings for the SCF dialect This is an important core dialect that has not been exposed previously. Set up the default bindings generation and provide a nicer wrapper for the `for` loop with access to the loop configuration and body. Depends On D110758 Reviewed By: stellaraccident Differential Revision: https://reviews.llvm.org/D110759
2021-09-29 11:42:53 -08:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/SCFOps.td
SOURCES
dialects/scf.py
DIALECT_NAME scf)
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/ShapeOps.td
SOURCES dialects/shape.py
DIALECT_NAME shape)
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
[mlir][python] provide bindings for ops from the sparse_tensor dialect Previously, the dialect was exposed for linking and pass management purposes, but we did not generate op classes for it. Generate them. Reviewed By: nicolasvasilache Differential Revision: https://reviews.llvm.org/D110819
2021-09-30 05:09:30 -08:00
TD_FILE dialects/SparseTensorOps.td
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
SOURCES dialects/sparse_tensor.py
[mlir][python bindings] generate all the enums This PR implements python enum bindings for *all* the enums - this includes `I*Attrs` (including positional/bit) and `Dialect/EnumAttr`. There are a few parts to this: 1. CMake: a small addition to `declare_mlir_dialect_python_bindings` and `declare_mlir_dialect_extension_python_bindings` to generate the enum, a boolean arg `GEN_ENUM_BINDINGS` to make it opt-in (even though it works for basically all of the dialects), and an optional `GEN_ENUM_BINDINGS_TD_FILE` for handling corner cases. 2. EnumPythonBindingGen.cpp: there are two weedy aspects here that took investigation: 1. If an enum attribute is not a `Dialect/EnumAttr` then the `EnumAttrInfo` record is canonical, as far as both the cases of the enum **and the `AttrDefName`**. On the otherhand, if an enum is a `Dialect/EnumAttr` then the `EnumAttr` record has the correct `AttrDefName` ("load bearing", i.e., populates `ods.ir.AttributeBuilder('<NAME>')`) but its `enum` field contains the cases, which is an instance of `EnumAttrInfo`. The solution is to generate an one enum class for both `Dialect/EnumAttr` and "independent" `EnumAttrInfo` but to make that class interopable with two builder registrations that both do the right thing (see next sub-bullet). 2. Because we don't have a good connection to cpp `EnumAttr`, i.e., only the `enum class` getters are exposed (like `DimensionAttr::get(Dimension value)`), we have to resort to parsing e.g., `Attribute.parse(f'#gpu<dim {x}>')`. This means that the set of supported `assemblyFormat`s (for the enum) is fixed at compile of MLIR (currently 2, the only 2 I saw). There might be some things that could be done here but they would require quite a bit more C API work to support generically (e.g., casting ints to enum cases and binding all the getters or going generically through the `symbolize*` methods, like `symbolizeDimension(uint32_t)` or `symbolizeDimension(StringRef)`). A few small changes: 1. In addition, since this patch registers default builders for attributes where people might've had their own builders already written, I added a `replace` param to `AttributeBuilder.insert` (`False` by default). 2. `makePythonEnumCaseName` can't handle all the different ways in which people write their enum cases, e.g., `llvm.CConv.Intel_OCL_BI`, which gets turned into `INTEL_O_C_L_B_I` (because `llvm::convertToSnakeFromCamelCase` doesn't look for runs of caps). So I dropped it. On the otherhand regularization does need to done because some enums have `None` as a case (and others might have other python keywords). 3. I turned on `llvm` dialect generation here in order to test `nvvm.WGMMAScaleIn`, which is an enum with [[ https://github.com/llvm/llvm-project/blob/d7e26b56207cbd8995296c5bb7c11ce676b649da/mlir/include/mlir/IR/EnumAttr.td#L22-L25 | no explicit discriminator ]] for the `neg` case. Note, dialects that didn't get a `GEN_ENUM_BINDINGS` don't have any enums to generate. Let me know if I should add more tests (the three trivial ones I added exercise both the supported `assemblyFormat`s and `replace=True`). Reviewed By: stellaraccident Differential Revision: https://reviews.llvm.org/D157934
2023-08-23 10:27:08 -08:00
DIALECT_NAME sparse_tensor
GEN_ENUM_BINDINGS_TD_FILE
"../../include/mlir/Dialect/SparseTensor/IR/SparseTensorAttrDefs.td"
)
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
[mlir][SMT] add python bindings (#135674) This PR adds "rich" python bindings to SMT dialect.
2025-04-16 14:17:09 -08:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/SMTOps.td
GEN_ENUM_BINDINGS
SOURCES
dialects/smt.py
DIALECT_NAME smt)
[mlir][spirv] Add support for C-API/python binding to SPIR-V dialect (#76055) Enable bindings. --------- Co-authored-by: jungpark-mlir <jungwook@jungwook-22.04>
2024-01-02 07:11:44 -09:00
declare_mlir_dialect_python_bindings(
[MLIR][Python] remove pybind11 support (#172581) This PR removes pybind which has been deprecated for over a year (https://github.com/llvm/llvm-project/pull/117922).
2025-12-19 08:51:22 -09:00
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/SPIRVOps.td
SOURCES dialects/spirv.py
DIALECT_NAME spirv)
[mlir][spirv] Add support for C-API/python binding to SPIR-V dialect (#76055) Enable bindings. --------- Co-authored-by: jungpark-mlir <jungwook@jungwook-22.04>
2024-01-02 07:11:44 -09:00
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/TensorOps.td
[mlir][tensor][NFC] Rename linalg.init_tensor to tensor.empty tensor.empty/linalg.init_tensor produces an uninititalized tensor that can be used as a destination operand for destination-style ops (ops that implement `DestinationStyleOpInterface`). This change makes it possible to implement `TilingInterface` for non-destination-style ops without depending on the Linalg dialect. RFC: https://discourse.llvm.org/t/rfc-add-tensor-from-shape-operation/65101 Differential Revision: https://reviews.llvm.org/D135129
2022-10-04 00:06:00 -08:00
SOURCES
dialects/tensor.py
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
DIALECT_NAME tensor)
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/TosaOps.td
SOURCES dialects/tosa.py
[mlir][python bindings] generate all the enums This PR implements python enum bindings for *all* the enums - this includes `I*Attrs` (including positional/bit) and `Dialect/EnumAttr`. There are a few parts to this: 1. CMake: a small addition to `declare_mlir_dialect_python_bindings` and `declare_mlir_dialect_extension_python_bindings` to generate the enum, a boolean arg `GEN_ENUM_BINDINGS` to make it opt-in (even though it works for basically all of the dialects), and an optional `GEN_ENUM_BINDINGS_TD_FILE` for handling corner cases. 2. EnumPythonBindingGen.cpp: there are two weedy aspects here that took investigation: 1. If an enum attribute is not a `Dialect/EnumAttr` then the `EnumAttrInfo` record is canonical, as far as both the cases of the enum **and the `AttrDefName`**. On the otherhand, if an enum is a `Dialect/EnumAttr` then the `EnumAttr` record has the correct `AttrDefName` ("load bearing", i.e., populates `ods.ir.AttributeBuilder('<NAME>')`) but its `enum` field contains the cases, which is an instance of `EnumAttrInfo`. The solution is to generate an one enum class for both `Dialect/EnumAttr` and "independent" `EnumAttrInfo` but to make that class interopable with two builder registrations that both do the right thing (see next sub-bullet). 2. Because we don't have a good connection to cpp `EnumAttr`, i.e., only the `enum class` getters are exposed (like `DimensionAttr::get(Dimension value)`), we have to resort to parsing e.g., `Attribute.parse(f'#gpu<dim {x}>')`. This means that the set of supported `assemblyFormat`s (for the enum) is fixed at compile of MLIR (currently 2, the only 2 I saw). There might be some things that could be done here but they would require quite a bit more C API work to support generically (e.g., casting ints to enum cases and binding all the getters or going generically through the `symbolize*` methods, like `symbolizeDimension(uint32_t)` or `symbolizeDimension(StringRef)`). A few small changes: 1. In addition, since this patch registers default builders for attributes where people might've had their own builders already written, I added a `replace` param to `AttributeBuilder.insert` (`False` by default). 2. `makePythonEnumCaseName` can't handle all the different ways in which people write their enum cases, e.g., `llvm.CConv.Intel_OCL_BI`, which gets turned into `INTEL_O_C_L_B_I` (because `llvm::convertToSnakeFromCamelCase` doesn't look for runs of caps). So I dropped it. On the otherhand regularization does need to done because some enums have `None` as a case (and others might have other python keywords). 3. I turned on `llvm` dialect generation here in order to test `nvvm.WGMMAScaleIn`, which is an enum with [[ https://github.com/llvm/llvm-project/blob/d7e26b56207cbd8995296c5bb7c11ce676b649da/mlir/include/mlir/IR/EnumAttr.td#L22-L25 | no explicit discriminator ]] for the `neg` case. Note, dialects that didn't get a `GEN_ENUM_BINDINGS` don't have any enums to generate. Let me know if I should add more tests (the three trivial ones I added exercise both the supported `assemblyFormat`s and `replace=True`). Reviewed By: stellaraccident Differential Revision: https://reviews.llvm.org/D157934
2023-08-23 10:27:08 -08:00
DIALECT_NAME tosa
)
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
[mlir][python] UB dialect python bindings (#157127)
2025-09-05 07:52:01 -08:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/UBOps.td
SOURCES dialects/ub.py
DIALECT_NAME ub
)
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/VectorOps.td
SOURCES dialects/vector.py
[mlir][python bindings] generate all the enums This PR implements python enum bindings for *all* the enums - this includes `I*Attrs` (including positional/bit) and `Dialect/EnumAttr`. There are a few parts to this: 1. CMake: a small addition to `declare_mlir_dialect_python_bindings` and `declare_mlir_dialect_extension_python_bindings` to generate the enum, a boolean arg `GEN_ENUM_BINDINGS` to make it opt-in (even though it works for basically all of the dialects), and an optional `GEN_ENUM_BINDINGS_TD_FILE` for handling corner cases. 2. EnumPythonBindingGen.cpp: there are two weedy aspects here that took investigation: 1. If an enum attribute is not a `Dialect/EnumAttr` then the `EnumAttrInfo` record is canonical, as far as both the cases of the enum **and the `AttrDefName`**. On the otherhand, if an enum is a `Dialect/EnumAttr` then the `EnumAttr` record has the correct `AttrDefName` ("load bearing", i.e., populates `ods.ir.AttributeBuilder('<NAME>')`) but its `enum` field contains the cases, which is an instance of `EnumAttrInfo`. The solution is to generate an one enum class for both `Dialect/EnumAttr` and "independent" `EnumAttrInfo` but to make that class interopable with two builder registrations that both do the right thing (see next sub-bullet). 2. Because we don't have a good connection to cpp `EnumAttr`, i.e., only the `enum class` getters are exposed (like `DimensionAttr::get(Dimension value)`), we have to resort to parsing e.g., `Attribute.parse(f'#gpu<dim {x}>')`. This means that the set of supported `assemblyFormat`s (for the enum) is fixed at compile of MLIR (currently 2, the only 2 I saw). There might be some things that could be done here but they would require quite a bit more C API work to support generically (e.g., casting ints to enum cases and binding all the getters or going generically through the `symbolize*` methods, like `symbolizeDimension(uint32_t)` or `symbolizeDimension(StringRef)`). A few small changes: 1. In addition, since this patch registers default builders for attributes where people might've had their own builders already written, I added a `replace` param to `AttributeBuilder.insert` (`False` by default). 2. `makePythonEnumCaseName` can't handle all the different ways in which people write their enum cases, e.g., `llvm.CConv.Intel_OCL_BI`, which gets turned into `INTEL_O_C_L_B_I` (because `llvm::convertToSnakeFromCamelCase` doesn't look for runs of caps). So I dropped it. On the otherhand regularization does need to done because some enums have `None` as a case (and others might have other python keywords). 3. I turned on `llvm` dialect generation here in order to test `nvvm.WGMMAScaleIn`, which is an enum with [[ https://github.com/llvm/llvm-project/blob/d7e26b56207cbd8995296c5bb7c11ce676b649da/mlir/include/mlir/IR/EnumAttr.td#L22-L25 | no explicit discriminator ]] for the `neg` case. Note, dialects that didn't get a `GEN_ENUM_BINDINGS` don't have any enums to generate. Let me know if I should add more tests (the three trivial ones I added exercise both the supported `assemblyFormat`s and `replace=True`). Reviewed By: stellaraccident Differential Revision: https://reviews.llvm.org/D157934
2023-08-23 10:27:08 -08:00
DIALECT_NAME vector
[mlir][python] smaller scope for vector enumgen (#66992) Don't generate enums from the main VectorOps.td file as that transitively includes enums from Arith. --------- Co-authored-by: Nicolas Vasilache <ntv@google.com>
2023-09-21 02:57:41 -08:00
GEN_ENUM_BINDINGS_TD_FILE
"dialects/VectorAttributes.td")
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
[mlir][x86vector] Python bindings for x86vector dialect (#179958) Registers python bindings for x86vector dialect and transform ops.
2026-02-05 10:05:17 -09:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
[mlir][x86] Rename x86vector to x86 (#183311) Renames 'x86vector' dialect to 'x86'. This is the first PR in series of cleanups around dialects targeting x86 platforms. The new naming scheme is shorter, cleaner, and opens possibility of integrating other x86-specific operations not strictly fitting pure vector representation. For example, the generalization will allow for future merger of AMX dialect into the x86 dialect to create one-stop x86 operations collection and boost discoverability.
2026-02-26 01:21:58 -09:00
TD_FILE dialects/X86.td
SOURCES dialects/x86.py
DIALECT_NAME x86)
[mlir][x86vector] Python bindings for x86vector dialect (#179958) Registers python bindings for x86vector dialect and transform ops.
2026-02-05 10:05:17 -09:00
[MLIR][Python] Add python bindings for IRDL dialect (#158488) In this PR we add basic python bindings for IRDL dialect, so that python users can create and load IRDL dialects in python. This allows users, to some extent, to define dialects in Python without having to modify MLIR’s CMake/TableGen/C++ code and rebuild, making prototyping more convenient. A basic example is shown below (and also in the added test case): ```python # create a module with IRDL dialects module = Module.create() with InsertionPoint(module.body): dialect = irdl.DialectOp("irdl_test") with InsertionPoint(dialect.body): op = irdl.OperationOp("test_op") with InsertionPoint(op.body): f32 = irdl.is_(TypeAttr.get(F32Type.get())) irdl.operands_([f32], ["input"], [irdl.Variadicity.single]) # load the module irdl.load_dialects(module) # use the op defined in IRDL m = Module.parse(""" module { %a = arith.constant 1.0 : f32 "irdl_test.test_op"(%a) : (f32) -> () } """) ```
2025-09-18 18:10:39 -08:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/IRDLOps.td
SOURCES dialects/irdl.py
DIALECT_NAME irdl
GEN_ENUM_BINDINGS
)
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
################################################################################
# Python extensions.
# The sources for these are all in lib/Bindings/Python, but since they have to
# be rebuilt for each package and integrate with the source setup here, we
# just reference them here instead of having ordered, cross package target
# dependencies.
################################################################################
declare_mlir_python_extension(MLIRPythonExtension.Core
MODULE_NAME _mlir
ADD_TO_PARENT MLIRPythonSources.Core
Re-apply "[mlir] Allow out-of-tree python building from installed MLIR." Re-applies D111513: * Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled). * Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing. * Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API. * Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan). * Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time). * Downstreams will need to adapt by: * Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory. * To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension. This reverts commit 1a6c26d1f52999edbfbf6a978ae3f0e6759ea755. Reviewed By: stephenneuendorffer Differential Revision: https://reviews.llvm.org/D113732
2021-11-14 13:44:25 -09:00
ROOT_DIR "${PYTHON_SOURCE_DIR}"
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
SOURCES
Re-apply "[mlir] Allow out-of-tree python building from installed MLIR." Re-applies D111513: * Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled). * Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing. * Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API. * Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan). * Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time). * Downstreams will need to adapt by: * Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory. * To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension. This reverts commit 1a6c26d1f52999edbfbf6a978ae3f0e6759ea755. Reviewed By: stephenneuendorffer Differential Revision: https://reviews.llvm.org/D113732
2021-11-14 13:44:25 -09:00
MainModule.cpp
IRAffine.cpp
IRInterfaces.cpp
Pass.cpp
[mlir] Add PDL C & Python usage (#94714) Following a rather direct approach to expose PDL usage from C and then Python. This doesn't yes plumb through adding support for custom matchers through this interface, so constrained to basics initially. This also exposes greedy rewrite driver. Only way currently to define patterns is via PDL (just to keep small). The creation of the PDL pattern module could be improved to avoid folks potentially accessing the module used to construct it post construction. No ergonomic work done yet. --------- Signed-off-by: Jacques Pienaar <jpienaar@google.com>
2024-06-11 06:45:12 -08:00
Rewrite.cpp
Re-apply "[mlir] Allow out-of-tree python building from installed MLIR." Re-applies D111513: * Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled). * Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing. * Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API. * Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan). * Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time). * Downstreams will need to adapt by: * Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory. * To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension. This reverts commit 1a6c26d1f52999edbfbf6a978ae3f0e6759ea755. Reviewed By: stephenneuendorffer Differential Revision: https://reviews.llvm.org/D113732
2021-11-14 13:44:25 -09:00
# Headers must be included explicitly so they are installed.
Pass.h
[MLIR][python] include Rewrite.h (#95226)
2024-06-12 04:17:13 -08:00
Rewrite.h
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
PRIVATE_LINK_LIBS
LLVMSupport
EMBED_CAPI_LINK_LIBS
MLIRCAPIDebug
MLIRCAPIIR
[mlir] support interfaces in Python bindings Introduce the initial support for operation interfaces in C API and Python bindings. Interfaces are a key component of MLIR's extensibility and should be available in bindings to make use of full potential of MLIR. This initial implementation exposes InferTypeOpInterface all the way to the Python bindings since it can be later used to simplify the operation construction methods by inferring their return types instead of requiring the user to do so. The general infrastructure for binding interfaces is defined and InferTypeOpInterface can be used as an example for binding other interfaces. Reviewed By: gysit Differential Revision: https://reviews.llvm.org/D111656
2021-10-14 07:18:28 -08:00
MLIRCAPIInterfaces
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
# Dialects
[mlir] Rename the Standard dialect to the Func dialect The last remaining operations in the standard dialect all revolve around FuncOp/function related constructs. This patch simply handles the initial renaming (which by itself is already huge), but there are a large number of cleanups unlocked/necessary afterwards: * Removing a bunch of unnecessary dependencies on Func * Cleaning up the From/ToStandard conversion passes * Preparing for the move of FuncOp to the Func dialect See the discussion at https://discourse.llvm.org/t/standard-dialect-the-final-chapter/6061 Differential Revision: https://reviews.llvm.org/D120624
2022-02-26 13:49:54 -09:00
MLIRCAPIFunc
[mlir] Split out Python bindings for dialects into separate libs Historically, the bindings for the Linalg dialect were included into the "core" bindings library because they depended on the C++ implementation of the "core" bindings. The other dialects followed the pattern. Now that this dependency is gone, split out each dialect into a separate Python extension library. Depends On D116649, D116605 Reviewed By: stellaraccident Differential Revision: https://reviews.llvm.org/D116662
2022-01-05 01:21:21 -09:00
)
[mlir] Overhaul C/Python registration APIs to properly scope registration/loading activities. Since the very first commits, the Python and C MLIR APIs have had mis-placed registration/load functionality for dialects, extensions, etc. This was done pragmatically in order to get bootstrapped and then just grew in. Downstreams largely bypass and do their own thing by providing various APIs to register things they need. Meanwhile, the C++ APIs have stabilized around this and it would make sense to follow suit. The thing we have observed in canonical usage by downstreams is that each downstream tends to have native entry points that configure its installation to its preferences with one-stop APIs. This patch leans in to this approach with `RegisterEverything.h` and `mlir._mlir_libs._mlirRegisterEverything` being the one-stop entry points for the "upstream packages". The `_mlir_libs.__init__.py` now allows customization of the environment and Context by adding "initialization modules" to the `_mlir_libs` package. If present, `_mlirRegisterEverything` is treated as such a module. Others can be added by downstreams by adding a `_site_initialize_{i}.py` module, where '{i}' is a number starting with zero. The number will be incremented and corresponding module loaded until one is not found. Initialization modules can: * Perform load time customization to the global environment (i.e. registering passes, hooks, etc). * Define a `register_dialects(registry: DialectRegistry)` function that can extend the `DialectRegistry` that will be used to bootstrap the `Context`. * Define a `context_init_hook(context: Context)` function that will be added to a list of callbacks which will be invoked after dialect registration during `Context` initialization. Note that the `MLIRPythonExtension.RegisterEverything` is not included by default when building a downstream (its corresponding behavior was prior). For downstreams which need the default MLIR initialization to take place, they must add this back in to their Python CMake build just like they add their own components (i.e. to `add_mlir_python_common_capi_library` and `add_mlir_python_modules`). It is perfectly valid to not do this, in which case, only the things explicitly depended on and initialized by downstreams will be built/packaged. If the downstream has not been set up for this, it is recommended to simply add this back for the time being and pay the build time/package size cost. CMake changes: * `MLIRCAPIRegistration` -> `MLIRCAPIRegisterEverything` (renamed to signify what it does and force an evaluation: a number of places were incidentally linking this very expensive target) * `MLIRPythonSoure.Passes` removed (without replacement: just drop) * `MLIRPythonExtension.AllPassesRegistration` removed (without replacement: just drop) * `MLIRPythonExtension.Conversions` removed (without replacement: just drop) * `MLIRPythonExtension.Transforms` removed (without replacement: just drop) Header changes: * `mlir-c/Registration.h` is deleted. Dialect registration functionality is now in `IR.h`. Registration of upstream features are in `mlir-c/RegisterEverything.h`. When updating MLIR and a couple of downstreams, I found that proper usage was commingled so required making a choice vs just blind S&R. Python APIs removed: * mlir.transforms and mlir.conversions (previously only had an __init__.py which indirectly triggered `mlirRegisterTransformsPasses()` and `mlirRegisterConversionPasses()` respectively). Downstream impact: Remove these imports if present (they now happen as part of default initialization). * mlir._mlir_libs._all_passes_registration, mlir._mlir_libs._mlirTransforms, mlir._mlir_libs._mlirConversions. Downstream impact: None expected (these were internally used). C-APIs changed: * mlirRegisterAllDialects(MlirContext) now takes an MlirDialectRegistry instead. It also used to trigger loading of all dialects, which was already marked with a TODO to remove -- it no longer does, and for direct use, dialects must be explicitly loaded. Downstream impact: Direct C-API users must ensure that needed dialects are loaded or call `mlirContextLoadAllAvailableDialects(MlirContext)` to emulate the prior behavior. Also see the `ir.c` test case (e.g. ` mlirContextGetOrLoadDialect(ctx, mlirStringRefCreateFromCString("func"));`). * mlirDialectHandle* APIs were moved from Registration.h (which now is restricted to just global/upstream registration) to IR.h, arguably where it should have been. Downstream impact: include correct header (likely already doing so). C-APIs added: * mlirContextLoadAllAvailableDialects(MlirContext): Corresponds to C++ API with the same purpose. Python APIs added: * mlir.ir.DialectRegistry: Mapping for an MlirDialectRegistry. * mlir.ir.Context.append_dialect_registry(MlirDialectRegistry) * mlir.ir.Context.load_all_available_dialects() * mlir._mlir_libs._mlirAllRegistration: New native extension that exposes a `register_dialects(MlirDialectRegistry)` entry point and performs all upstream pass/conversion/transforms registration on init. In this first step, we eagerly load this as part of the __init__.py and use it to monkey patch the Context to emulate prior behavior. * Type caster and capsule support for MlirDialectRegistry This should make it possible to build downstream Python dialects that only depend on a subset of MLIR. See: https://github.com/llvm/llvm-project/issues/56037 Here is an example PR, minimally adapting IREE to these changes: https://github.com/iree-org/iree/pull/9638/files In this situation, IREE is opting to not link everything, since it is already configuring the Context to its liking. For projects that would just like to not think about it and pull in everything, add `MLIRPythonExtension.RegisterEverything` to the list of Python sources getting built, and the old behavior will continue. Reviewed By: mehdi_amini, ftynse Differential Revision: https://reviews.llvm.org/D128593
2022-07-16 15:09:03 -08:00
# This extension exposes an API to register all dialects, extensions, and passes
# packaged in upstream MLIR and it is used for the upstream "mlir" Python
# package. Downstreams will likely want to provide their own and not depend
# on this one, since it links in the world.
# Note that this is not added to any top-level source target for transitive
# inclusion: It must be included explicitly by downstreams if desired. Note that
# this has a very large impact on what gets built/packaged.
declare_mlir_python_extension(MLIRPythonExtension.RegisterEverything
MODULE_NAME _mlirRegisterEverything
ROOT_DIR "${PYTHON_SOURCE_DIR}"
SOURCES
RegisterEverything.cpp
PRIVATE_LINK_LIBS
LLVMSupport
EMBED_CAPI_LINK_LIBS
MLIRCAPIConversion
MLIRCAPITransforms
MLIRCAPIRegisterEverything
)
[MLIR][Python] remove pybind11 support (#172581) This PR removes pybind which has been deprecated for over a year (https://github.com/llvm/llvm-project/pull/117922).
2025-12-19 08:51:22 -09:00
declare_mlir_python_extension(MLIRPythonExtension.Dialects.Linalg.Nanobind
[mlir] Split out Python bindings for dialects into separate libs Historically, the bindings for the Linalg dialect were included into the "core" bindings library because they depended on the C++ implementation of the "core" bindings. The other dialects followed the pattern. Now that this dependency is gone, split out each dialect into a separate Python extension library. Depends On D116649, D116605 Reviewed By: stellaraccident Differential Revision: https://reviews.llvm.org/D116662
2022-01-05 01:21:21 -09:00
MODULE_NAME _mlirDialectsLinalg
ADD_TO_PARENT MLIRPythonSources.Dialects.linalg
ROOT_DIR "${PYTHON_SOURCE_DIR}"
SOURCES
DialectLinalg.cpp
PRIVATE_LINK_LIBS
LLVMSupport
EMBED_CAPI_LINK_LIBS
MLIRCAPIIR
MLIRCAPILinalg
)
[MLIR][Python] remove pybind11 support (#172581) This PR removes pybind which has been deprecated for over a year (https://github.com/llvm/llvm-project/pull/117922).
2025-12-19 08:51:22 -09:00
declare_mlir_python_extension(MLIRPythonExtension.Dialects.GPU.Nanobind
[mlir][python] add binding to `#gpu.object` (#88992)
2024-04-18 13:31:55 -08:00
MODULE_NAME _mlirDialectsGPU
ADD_TO_PARENT MLIRPythonSources.Dialects.gpu
ROOT_DIR "${PYTHON_SOURCE_DIR}"
SOURCES
DialectGPU.cpp
PRIVATE_LINK_LIBS
LLVMSupport
EMBED_CAPI_LINK_LIBS
MLIRCAPIIR
MLIRCAPIGPU
)
[MLIR][Python] remove pybind11 support (#172581) This PR removes pybind which has been deprecated for over a year (https://github.com/llvm/llvm-project/pull/117922).
2025-12-19 08:51:22 -09:00
declare_mlir_python_extension(MLIRPythonExtension.Dialects.LLVM.Nanobind
[mlir][python] expose LLVMStructType API (#81672) Expose the API for constructing and inspecting StructTypes from the LLVM dialect. Separate constructor methods are used instead of overloads for better readability, similarly to IntegerType.
2024-02-14 05:03:04 -09:00
MODULE_NAME _mlirDialectsLLVM
ADD_TO_PARENT MLIRPythonSources.Dialects.llvm
ROOT_DIR "${PYTHON_SOURCE_DIR}"
SOURCES
DialectLLVM.cpp
PRIVATE_LINK_LIBS
LLVMSupport
EMBED_CAPI_LINK_LIBS
MLIRCAPIIR
MLIRCAPILLVM
[MLIR][Python] expose translate_module_to_llvmir (#163881) This PR exposes `translate_module_to_llvmir` in the Python bindings.
2025-10-20 08:14:52 -08:00
# Misnomer - this is only the LLVMIR translation target.
MLIRCAPITarget
[mlir][python] expose LLVMStructType API (#81672) Expose the API for constructing and inspecting StructTypes from the LLVM dialect. Separate constructor methods are used instead of overloads for better readability, similarly to IntegerType.
2024-02-14 05:03:04 -09:00
)
[MLIR][Python] remove pybind11 support (#172581) This PR removes pybind which has been deprecated for over a year (https://github.com/llvm/llvm-project/pull/117922).
2025-12-19 08:51:22 -09:00
declare_mlir_python_extension(MLIRPythonExtension.Dialects.Quant.Nanobind
[mlir] Split out Python bindings for dialects into separate libs Historically, the bindings for the Linalg dialect were included into the "core" bindings library because they depended on the C++ implementation of the "core" bindings. The other dialects followed the pattern. Now that this dependency is gone, split out each dialect into a separate Python extension library. Depends On D116649, D116605 Reviewed By: stellaraccident Differential Revision: https://reviews.llvm.org/D116662
2022-01-05 01:21:21 -09:00
MODULE_NAME _mlirDialectsQuant
ADD_TO_PARENT MLIRPythonSources.Dialects.quant
ROOT_DIR "${PYTHON_SOURCE_DIR}"
SOURCES
DialectQuant.cpp
PRIVATE_LINK_LIBS
LLVMSupport
EMBED_CAPI_LINK_LIBS
MLIRCAPIIR
MLIRCAPIQuant
)
[MLIR][Python] remove pybind11 support (#172581) This PR removes pybind which has been deprecated for over a year (https://github.com/llvm/llvm-project/pull/117922).
2025-12-19 08:51:22 -09:00
declare_mlir_python_extension(MLIRPythonExtension.Dialects.NVGPU.Nanobind
[NFC] Make NVGPU casing consistent (#91903)
2024-05-12 23:08:04 -08:00
MODULE_NAME _mlirDialectsNVGPU
[mlir][py] Add NVGPU's `TensorMapDescriptorType` in py bindings (#88855) This PR adds NVGPU dialects' TensorMapDescriptorType in the py bindings. This is a follow-up issue from [this PR](https://github.com/llvm/llvm-project/pull/87153#discussion_r1546193095)
2024-04-17 05:59:18 -08:00
ADD_TO_PARENT MLIRPythonSources.Dialects.nvgpu
ROOT_DIR "${PYTHON_SOURCE_DIR}"
SOURCES
DialectNVGPU.cpp
PRIVATE_LINK_LIBS
LLVMSupport
EMBED_CAPI_LINK_LIBS
MLIRCAPIIR
MLIRCAPINVGPU
)
[MLIR][Python] remove pybind11 support (#172581) This PR removes pybind which has been deprecated for over a year (https://github.com/llvm/llvm-project/pull/117922).
2025-12-19 08:51:22 -09:00
declare_mlir_python_extension(MLIRPythonExtension.Dialects.PDL.Nanobind
[mlir] Introduce Python bindings for the PDL dialect This change adds full python bindings for PDL, including types and operations with additional mixins to make operation construction more similar to the PDL syntax. Reviewed By: ftynse Differential Revision: https://reviews.llvm.org/D117458
2022-01-13 00:53:21 -09:00
MODULE_NAME _mlirDialectsPDL
ADD_TO_PARENT MLIRPythonSources.Dialects.pdl
ROOT_DIR "${PYTHON_SOURCE_DIR}"
SOURCES
DialectPDL.cpp
PRIVATE_LINK_LIBS
LLVMSupport
EMBED_CAPI_LINK_LIBS
MLIRCAPIIR
MLIRCAPIPDL
)
[MLIR][Python] remove pybind11 support (#172581) This PR removes pybind which has been deprecated for over a year (https://github.com/llvm/llvm-project/pull/117922).
2025-12-19 08:51:22 -09:00
declare_mlir_python_extension(MLIRPythonExtension.Dialects.SparseTensor.Nanobind
[mlir] Split out Python bindings for dialects into separate libs Historically, the bindings for the Linalg dialect were included into the "core" bindings library because they depended on the C++ implementation of the "core" bindings. The other dialects followed the pattern. Now that this dependency is gone, split out each dialect into a separate Python extension library. Depends On D116649, D116605 Reviewed By: stellaraccident Differential Revision: https://reviews.llvm.org/D116662
2022-01-05 01:21:21 -09:00
MODULE_NAME _mlirDialectsSparseTensor
ADD_TO_PARENT MLIRPythonSources.Dialects.sparse_tensor
ROOT_DIR "${PYTHON_SOURCE_DIR}"
SOURCES
DialectSparseTensor.cpp
PRIVATE_LINK_LIBS
LLVMSupport
EMBED_CAPI_LINK_LIBS
MLIRCAPIIR
MLIRCAPISparseTensor
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
)
[MLIR][Python] remove pybind11 support (#172581) This PR removes pybind which has been deprecated for over a year (https://github.com/llvm/llvm-project/pull/117922).
2025-12-19 08:51:22 -09:00
declare_mlir_python_extension(MLIRPythonExtension.Dialects.Transform.Nanobind
[mlir] add OperationType to the Transform dialect Add a new OperationType handle type to the Transform dialect. This transform type is parameterized by the name of the payload operation it can point to. It is intended as a constraint on transformations that are only applicable to a specific kind of payload operations. If a transformation is applicable to a small set of operation classes, it can be wrapped into a transform op by using a disjunctive constraint, such as `Type<Or<[Transform_ConcreteOperation<"foo">.predicate, Transform_ConcreteOperation<"bar">.predicate]>>` for its operand without modifying this type. Broader sets of accepted operations should be modeled as specific types. Reviewed By: nicolasvasilache Differential Revision: https://reviews.llvm.org/D135586
2022-10-10 06:33:59 -08:00
MODULE_NAME _mlirDialectsTransform
ADD_TO_PARENT MLIRPythonSources.Dialects.transform
ROOT_DIR "${PYTHON_SOURCE_DIR}"
SOURCES
DialectTransform.cpp
[MLIR][Python] Impl XOpInterface(s) from Python, with X=Transform and X=MemoryEffects (#176920) Provides the infrastructure for implementing and late-binding OpInterfaces from Python. * On the mlir-c API declaration side, each `XOpInterface` has a callback struct, with a callback for each method and a userdata member (provided as an arg to each method), and a `mlirXOpInterfaceAttachFallbackModel(ctx, op_name, callbacks)` func. * This CAPI is implemented by defining a subclass of `XOpInterface::FallbackModel` that holds the callback struct and has each method call the corresponding callback (with userdata as an arg). Given a callback struct, a new `FallbackModel` is created and attached, i.e. late bound, to the named op. (MLIR's interface infrastructure is such that the thus registered `FallbackModel` will be returned in case the op gets cast to the `XOpInterface`.) * On the Python side, we expose a stand-in `XOpInterface` base class which has one (class)method: `XOpInterface.attach(cls, op_name, ctx)`. Python users subclass this class (`class MyInterfaceImpl(XOpInterface): ...`) and implement the interface's methods (with the right names and signatures). The user calls `attach` on the subclass (`MyInterfaceImpl.attach("my_dialect.my_op", ctx)`) which prepares the callbacks struct _with userdata set to the subclass_ (as we use it to lookup methods). These callbacks (and userdata) are then registered as an `XOpInterface::FallbackModel` by `mlirXOpInterfaceAttachFallbackModel(...)`. From then on the Python methods will be used to respond to calls to the interface methods (originating in C++). This PR enables implementing the TransformOpInterface and the MemoryEffectsOpInterface, both of which are required for making an op into a transform op. Everything besides the above linked code is there to facilitate exposing the interfaces: the right types for the arguments of the methods are exposed as are functions/methods for manipulating these arguments (e.g. specifying side effects on `OpOperand`s and `OpResult`s and being able to access and set the transform handles associated with args and results).
2026-02-12 05:07:10 -09:00
Rewrite.h
[mlir] add OperationType to the Transform dialect Add a new OperationType handle type to the Transform dialect. This transform type is parameterized by the name of the payload operation it can point to. It is intended as a constraint on transformations that are only applicable to a specific kind of payload operations. If a transformation is applicable to a small set of operation classes, it can be wrapped into a transform op by using a disjunctive constraint, such as `Type<Or<[Transform_ConcreteOperation<"foo">.predicate, Transform_ConcreteOperation<"bar">.predicate]>>` for its operand without modifying this type. Broader sets of accepted operations should be modeled as specific types. Reviewed By: nicolasvasilache Differential Revision: https://reviews.llvm.org/D135586
2022-10-10 06:33:59 -08:00
PRIVATE_LINK_LIBS
LLVMSupport
[MLIR][Python] Impl XOpInterface(s) from Python, with X=Transform and X=MemoryEffects (#176920) Provides the infrastructure for implementing and late-binding OpInterfaces from Python. * On the mlir-c API declaration side, each `XOpInterface` has a callback struct, with a callback for each method and a userdata member (provided as an arg to each method), and a `mlirXOpInterfaceAttachFallbackModel(ctx, op_name, callbacks)` func. * This CAPI is implemented by defining a subclass of `XOpInterface::FallbackModel` that holds the callback struct and has each method call the corresponding callback (with userdata as an arg). Given a callback struct, a new `FallbackModel` is created and attached, i.e. late bound, to the named op. (MLIR's interface infrastructure is such that the thus registered `FallbackModel` will be returned in case the op gets cast to the `XOpInterface`.) * On the Python side, we expose a stand-in `XOpInterface` base class which has one (class)method: `XOpInterface.attach(cls, op_name, ctx)`. Python users subclass this class (`class MyInterfaceImpl(XOpInterface): ...`) and implement the interface's methods (with the right names and signatures). The user calls `attach` on the subclass (`MyInterfaceImpl.attach("my_dialect.my_op", ctx)`) which prepares the callbacks struct _with userdata set to the subclass_ (as we use it to lookup methods). These callbacks (and userdata) are then registered as an `XOpInterface::FallbackModel` by `mlirXOpInterfaceAttachFallbackModel(...)`. From then on the Python methods will be used to respond to calls to the interface methods (originating in C++). This PR enables implementing the TransformOpInterface and the MemoryEffectsOpInterface, both of which are required for making an op into a transform op. Everything besides the above linked code is there to facilitate exposing the interfaces: the right types for the arguments of the methods are exposed as are functions/methods for manipulating these arguments (e.g. specifying side effects on `OpOperand`s and `OpResult`s and being able to access and set the transform handles associated with args and results).
2026-02-12 05:07:10 -09:00
MLIRPythonExtension.Core
[mlir] add OperationType to the Transform dialect Add a new OperationType handle type to the Transform dialect. This transform type is parameterized by the name of the payload operation it can point to. It is intended as a constraint on transformations that are only applicable to a specific kind of payload operations. If a transformation is applicable to a small set of operation classes, it can be wrapped into a transform op by using a disjunctive constraint, such as `Type<Or<[Transform_ConcreteOperation<"foo">.predicate, Transform_ConcreteOperation<"bar">.predicate]>>` for its operand without modifying this type. Broader sets of accepted operations should be modeled as specific types. Reviewed By: nicolasvasilache Differential Revision: https://reviews.llvm.org/D135586
2022-10-10 06:33:59 -08:00
EMBED_CAPI_LINK_LIBS
MLIRCAPIIR
MLIRCAPITransformDialect
)
[MLIR][Python] remove pybind11 support (#172581) This PR removes pybind which has been deprecated for over a year (https://github.com/llvm/llvm-project/pull/117922).
2025-12-19 08:51:22 -09:00
declare_mlir_python_extension(MLIRPythonExtension.Dialects.IRDL.Nanobind
[MLIR][Python] Add python bindings for IRDL dialect (#158488) In this PR we add basic python bindings for IRDL dialect, so that python users can create and load IRDL dialects in python. This allows users, to some extent, to define dialects in Python without having to modify MLIR’s CMake/TableGen/C++ code and rebuild, making prototyping more convenient. A basic example is shown below (and also in the added test case): ```python # create a module with IRDL dialects module = Module.create() with InsertionPoint(module.body): dialect = irdl.DialectOp("irdl_test") with InsertionPoint(dialect.body): op = irdl.OperationOp("test_op") with InsertionPoint(op.body): f32 = irdl.is_(TypeAttr.get(F32Type.get())) irdl.operands_([f32], ["input"], [irdl.Variadicity.single]) # load the module irdl.load_dialects(module) # use the op defined in IRDL m = Module.parse(""" module { %a = arith.constant 1.0 : f32 "irdl_test.test_op"(%a) : (f32) -> () } """) ```
2025-09-18 18:10:39 -08:00
MODULE_NAME _mlirDialectsIRDL
ADD_TO_PARENT MLIRPythonSources.Dialects.irdl
ROOT_DIR "${PYTHON_SOURCE_DIR}"
SOURCES
DialectIRDL.cpp
PRIVATE_LINK_LIBS
LLVMSupport
EMBED_CAPI_LINK_LIBS
MLIRCAPIIR
MLIRCAPIIRDL
)
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
declare_mlir_python_extension(MLIRPythonExtension.AsyncDialectPasses
MODULE_NAME _mlirAsyncPasses
[mlir][python] Fix generation of Python bindings for `async` dialect (#75960) The Python bindings generated for "async" dialect didn't include any of the "async" dialect ops. This PR fixes issues with generation of Python bindings for "async" dialect and adds a test case to use them.
2024-04-20 17:49:39 -08:00
ADD_TO_PARENT MLIRPythonSources.Dialects.async
Re-apply "[mlir] Allow out-of-tree python building from installed MLIR." Re-applies D111513: * Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled). * Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing. * Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API. * Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan). * Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time). * Downstreams will need to adapt by: * Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory. * To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension. This reverts commit 1a6c26d1f52999edbfbf6a978ae3f0e6759ea755. Reviewed By: stephenneuendorffer Differential Revision: https://reviews.llvm.org/D113732
2021-11-14 13:44:25 -09:00
ROOT_DIR "${PYTHON_SOURCE_DIR}"
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
SOURCES
Re-apply "[mlir] Allow out-of-tree python building from installed MLIR." Re-applies D111513: * Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled). * Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing. * Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API. * Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan). * Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time). * Downstreams will need to adapt by: * Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory. * To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension. This reverts commit 1a6c26d1f52999edbfbf6a978ae3f0e6759ea755. Reviewed By: stephenneuendorffer Differential Revision: https://reviews.llvm.org/D113732
2021-11-14 13:44:25 -09:00
AsyncPasses.cpp
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
PRIVATE_LINK_LIBS
LLVMSupport
EMBED_CAPI_LINK_LIBS
MLIRCAPIAsync
)
[MLIR] Fix checks for native arch Using if (TARGET ${LLVM_NATIVE_ARCH}) only works if MLIR is built together with LLVM, but not for standalone builds of MLIR. The correct way to check this is if (${LLVM_NATIVE_ARCH} IN_LIST LLVM_TARGETS_TO_BUILD), as the LLVM build system exports LLVM_TARGETS_TO_BUILD. To avoid repeating the same check many times, add a MLIR_ENABLE_EXECUTION_ENGINE variable. Differential Revision: https://reviews.llvm.org/D131071
2022-08-03 05:28:49 -08:00
if(MLIR_ENABLE_EXECUTION_ENGINE)
Disable the MLIR ExecutionEngine library when the native target is not configured The execution engine would not be functional anyway, we're already disabling the tests, this also disable the rest of the code. Anecdotally this reduces the number of static library built when the builtin target is disabled goes from 236 to 218. Here is the complete list of LLVM targets built when running `ninja check-mlir`: libLLVMAggressiveInstCombine.a libLLVMAnalysis.a libLLVMAsmParser.a libLLVMBinaryFormat.a libLLVMBitReader.a libLLVMBitstreamReader.a libLLVMBitWriter.a libLLVMCore.a libLLVMDebugInfoCodeView.a libLLVMDebugInfoDWARF.a libLLVMDemangle.a libLLVMFileCheck.a libLLVMFrontendOpenMP.a libLLVMInstCombine.a libLLVMIRReader.a libLLVMMC.a libLLVMMCParser.a libLLVMObject.a libLLVMProfileData.a libLLVMRemarks.a libLLVMScalarOpts.a libLLVMSupport.a libLLVMTableGen.a libLLVMTableGenGlobalISel.a libLLVMTextAPI.a libLLVMTransformUtils.a Differential Revision: https://reviews.llvm.org/D117287
2022-01-13 22:47:47 -09:00
declare_mlir_python_extension(MLIRPythonExtension.ExecutionEngine
MODULE_NAME _mlirExecutionEngine
ADD_TO_PARENT MLIRPythonSources.ExecutionEngine
ROOT_DIR "${PYTHON_SOURCE_DIR}"
SOURCES
ExecutionEngineModule.cpp
PRIVATE_LINK_LIBS
LLVMSupport
EMBED_CAPI_LINK_LIBS
MLIRCAPIExecutionEngine
)
endif()
Break apart the MLIR ExecutionEngine from core python module. * For python projects that don't need JIT/ExecutionEngine, cuts the number of files to compile roughly in half (with similar reduction in end binary size). Differential Revision: https://reviews.llvm.org/D106992
2021-07-28 12:00:02 -08:00
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
declare_mlir_python_extension(MLIRPythonExtension.GPUDialectPasses
MODULE_NAME _mlirGPUPasses
ADD_TO_PARENT MLIRPythonSources.Dialects.gpu
Re-apply "[mlir] Allow out-of-tree python building from installed MLIR." Re-applies D111513: * Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled). * Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing. * Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API. * Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan). * Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time). * Downstreams will need to adapt by: * Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory. * To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension. This reverts commit 1a6c26d1f52999edbfbf6a978ae3f0e6759ea755. Reviewed By: stephenneuendorffer Differential Revision: https://reviews.llvm.org/D113732
2021-11-14 13:44:25 -09:00
ROOT_DIR "${PYTHON_SOURCE_DIR}"
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
SOURCES
Re-apply "[mlir] Allow out-of-tree python building from installed MLIR." Re-applies D111513: * Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled). * Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing. * Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API. * Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan). * Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time). * Downstreams will need to adapt by: * Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory. * To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension. This reverts commit 1a6c26d1f52999edbfbf6a978ae3f0e6759ea755. Reviewed By: stephenneuendorffer Differential Revision: https://reviews.llvm.org/D113732
2021-11-14 13:44:25 -09:00
GPUPasses.cpp
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
PRIVATE_LINK_LIBS
LLVMSupport
EMBED_CAPI_LINK_LIBS
MLIRCAPIGPU
)
declare_mlir_python_extension(MLIRPythonExtension.LinalgPasses
MODULE_NAME _mlirLinalgPasses
ADD_TO_PARENT MLIRPythonSources.Dialects.linalg
Re-apply "[mlir] Allow out-of-tree python building from installed MLIR." Re-applies D111513: * Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled). * Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing. * Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API. * Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan). * Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time). * Downstreams will need to adapt by: * Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory. * To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension. This reverts commit 1a6c26d1f52999edbfbf6a978ae3f0e6759ea755. Reviewed By: stephenneuendorffer Differential Revision: https://reviews.llvm.org/D113732
2021-11-14 13:44:25 -09:00
ROOT_DIR "${PYTHON_SOURCE_DIR}"
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
SOURCES
Re-apply "[mlir] Allow out-of-tree python building from installed MLIR." Re-applies D111513: * Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled). * Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing. * Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API. * Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan). * Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time). * Downstreams will need to adapt by: * Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory. * To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension. This reverts commit 1a6c26d1f52999edbfbf6a978ae3f0e6759ea755. Reviewed By: stephenneuendorffer Differential Revision: https://reviews.llvm.org/D113732
2021-11-14 13:44:25 -09:00
LinalgPasses.cpp
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
PRIVATE_LINK_LIBS
LLVMSupport
EMBED_CAPI_LINK_LIBS
MLIRCAPILinalg
)
Move MLIR python sources to mlir/python. * NFC but has some fixes for CMake glitches discovered along the way (things not cleaning properly, co-mingled depends). * Includes previously unsubmitted fix in D98681 and a TODO to fix it more appropriately in a smaller followup. Differential Revision: https://reviews.llvm.org/D101493
2021-04-28 12:04:17 -08:00
[MLIR][Python] remove pybind11 support (#172581) This PR removes pybind which has been deprecated for over a year (https://github.com/llvm/llvm-project/pull/117922).
2025-12-19 08:51:22 -09:00
declare_mlir_python_extension(MLIRPythonExtension.Dialects.SMT.Nanobind
[mlir][SMT] add python bindings (#135674) This PR adds "rich" python bindings to SMT dialect.
2025-04-16 14:17:09 -08:00
MODULE_NAME _mlirDialectsSMT
ADD_TO_PARENT MLIRPythonSources.Dialects.smt
ROOT_DIR "${PYTHON_SOURCE_DIR}"
SOURCES
DialectSMT.cpp
PRIVATE_LINK_LIBS
LLVMSupport
EMBED_CAPI_LINK_LIBS
MLIRCAPIIR
MLIRCAPISMT
MLIRCAPIExportSMTLIB
)
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
declare_mlir_python_extension(MLIRPythonExtension.SparseTensorDialectPasses
MODULE_NAME _mlirSparseTensorPasses
ADD_TO_PARENT MLIRPythonSources.Dialects.sparse_tensor
Re-apply "[mlir] Allow out-of-tree python building from installed MLIR." Re-applies D111513: * Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled). * Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing. * Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API. * Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan). * Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time). * Downstreams will need to adapt by: * Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory. * To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension. This reverts commit 1a6c26d1f52999edbfbf6a978ae3f0e6759ea755. Reviewed By: stephenneuendorffer Differential Revision: https://reviews.llvm.org/D113732
2021-11-14 13:44:25 -09:00
ROOT_DIR "${PYTHON_SOURCE_DIR}"
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
SOURCES
Re-apply "[mlir] Allow out-of-tree python building from installed MLIR." Re-applies D111513: * Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled). * Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing. * Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API. * Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan). * Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time). * Downstreams will need to adapt by: * Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory. * To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension. This reverts commit 1a6c26d1f52999edbfbf6a978ae3f0e6759ea755. Reviewed By: stephenneuendorffer Differential Revision: https://reviews.llvm.org/D113732
2021-11-14 13:44:25 -09:00
SparseTensorPasses.cpp
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
PRIVATE_LINK_LIBS
LLVMSupport
EMBED_CAPI_LINK_LIBS
MLIRCAPISparseTensor
Move MLIR python sources to mlir/python. * NFC but has some fixes for CMake glitches discovered along the way (things not cleaning properly, co-mingled depends). * Includes previously unsubmitted fix in D98681 and a TODO to fix it more appropriately in a smaller followup. Differential Revision: https://reviews.llvm.org/D101493
2021-04-28 12:04:17 -08:00
)
[mlir] expose transform interpreter to Python (#82365) Transform interpreter functionality can be used standalone without going through the interpreter pass, make it available in Python.
2024-02-21 01:01:00 -09:00
declare_mlir_python_extension(MLIRPythonExtension.TransformInterpreter
MODULE_NAME _mlirTransformInterpreter
ADD_TO_PARENT MLIRPythonSources.Dialects.transform
ROOT_DIR "${PYTHON_SOURCE_DIR}"
SOURCES
TransformInterpreter.cpp
PRIVATE_LINK_LIBS
LLVMSupport
EMBED_CAPI_LINK_LIBS
MLIRCAPITransformDialectTransforms
)
[MLIR][Python] remove pybind11 support (#172581) This PR removes pybind which has been deprecated for over a year (https://github.com/llvm/llvm-project/pull/117922).
2025-12-19 08:51:22 -09:00
declare_mlir_python_extension(MLIRPythonExtension.Dialects.AMDGPU.Nanobind
[mlir][amdgpu] Add Python bindings for TDM types (#172309) Add bindings for: - `TDMBaseType` - `TDMDescriptorType` - `TDMGatherBaseType`
2025-12-16 01:43:08 -09:00
MODULE_NAME _mlirDialectsAMDGPU
ADD_TO_PARENT MLIRPythonSources.Dialects.amdgpu
ROOT_DIR "${PYTHON_SOURCE_DIR}"
SOURCES
DialectAMDGPU.cpp
PRIVATE_LINK_LIBS
LLVMSupport
EMBED_CAPI_LINK_LIBS
MLIRCAPIIR
MLIRCAPIAMDGPU
)
[mlir][python] Segment MLIR Python test dialect to avoid testonly dependency. With https://reviews.llvm.org/rG14c9207063bb00823a5126131e50c93f6e288bd3, the build is broken with -DMLIR_INCLUDE_TESTS=OFF. This patch fixes the build and we may want to do a better fix to the layering in a followup. Differential Revision: https://reviews.llvm.org/D112560
2021-10-26 09:14:50 -08:00
# TODO: Figure out how to put this in the test tree.
# This should not be included in the main Python extension. However,
[mlir] support interfaces in Python bindings Introduce the initial support for operation interfaces in C API and Python bindings. Interfaces are a key component of MLIR's extensibility and should be available in bindings to make use of full potential of MLIR. This initial implementation exposes InferTypeOpInterface all the way to the Python bindings since it can be later used to simplify the operation construction methods by inferring their return types instead of requiring the user to do so. The general infrastructure for binding interfaces is defined and InferTypeOpInterface can be used as an example for binding other interfaces. Reviewed By: gysit Differential Revision: https://reviews.llvm.org/D111656
2021-10-14 07:18:28 -08:00
# putting it into MLIRPythonTestSources along with the dialect declaration
# above confuses Python module loader when running under lit.
[mlir][python] Segment MLIR Python test dialect to avoid testonly dependency. With https://reviews.llvm.org/rG14c9207063bb00823a5126131e50c93f6e288bd3, the build is broken with -DMLIR_INCLUDE_TESTS=OFF. This patch fixes the build and we may want to do a better fix to the layering in a followup. Differential Revision: https://reviews.llvm.org/D112560
2021-10-26 09:14:50 -08:00
set(_ADDL_TEST_SOURCES)
if(MLIR_INCLUDE_TESTS)
set(_ADDL_TEST_SOURCES MLIRPythonTestSources)
declare_mlir_python_sources(MLIRPythonTestSources)
declare_mlir_python_sources(MLIRPythonTestSources.Dialects
ADD_TO_PARENT MLIRPythonTestSources)
# TODO: this uses a tablegen file from the test directory and should be
# decoupled from here.
declare_mlir_python_sources(
MLIRPythonTestSources.Dialects.PythonTest
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
ADD_TO_PARENT MLIRPythonTestSources.Dialects
Revert "Revert "[mlir python] Add nanobind support (#119232) Reverts revert #118517 after (hopefully) fixing builders (https://github.com/llvm/llvm-zorg/pull/328, https://github.com/llvm/llvm-zorg/pull/327) This reverts commit 61bf308cf2fc32452f14861c102ace89f5f36fec.
2024-12-09 12:37:43 -09:00
SOURCES
dialects/python_test.py
)
[mlir][python] Segment MLIR Python test dialect to avoid testonly dependency. With https://reviews.llvm.org/rG14c9207063bb00823a5126131e50c93f6e288bd3, the build is broken with -DMLIR_INCLUDE_TESTS=OFF. This patch fixes the build and we may want to do a better fix to the layering in a followup. Differential Revision: https://reviews.llvm.org/D112560
2021-10-26 09:14:50 -08:00
set(LLVM_TARGET_DEFINITIONS
"${MLIR_MAIN_SRC_DIR}/test/python/python_test_ops.td")
mlir_tablegen(
"dialects/_python_test_ops_gen.py"
-gen-python-op-bindings
-bind-dialect=python_test)
add_public_tablegen_target(PythonTestDialectPyIncGen)
declare_mlir_python_sources(
MLIRPythonTestSources.Dialects.PythonTest.ops_gen
ROOT_DIR "${CMAKE_CURRENT_BINARY_DIR}"
ADD_TO_PARENT MLIRPythonTestSources.Dialects.PythonTest
SOURCES "dialects/_python_test_ops_gen.py")
Revert "Revert "[mlir python] Add nanobind support (#119232) Reverts revert #118517 after (hopefully) fixing builders (https://github.com/llvm/llvm-zorg/pull/328, https://github.com/llvm/llvm-zorg/pull/327) This reverts commit 61bf308cf2fc32452f14861c102ace89f5f36fec.
2024-12-09 12:37:43 -09:00
declare_mlir_python_extension(MLIRPythonTestSources.PythonTestExtensionNanobind
MODULE_NAME _mlirPythonTestNanobind
[mlir][python] Segment MLIR Python test dialect to avoid testonly dependency. With https://reviews.llvm.org/rG14c9207063bb00823a5126131e50c93f6e288bd3, the build is broken with -DMLIR_INCLUDE_TESTS=OFF. This patch fixes the build and we may want to do a better fix to the layering in a followup. Differential Revision: https://reviews.llvm.org/D112560
2021-10-26 09:14:50 -08:00
ADD_TO_PARENT MLIRPythonTestSources.Dialects
Re-apply "[mlir] Allow out-of-tree python building from installed MLIR." Re-applies D111513: * Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled). * Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing. * Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API. * Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan). * Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time). * Downstreams will need to adapt by: * Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory. * To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension. This reverts commit 1a6c26d1f52999edbfbf6a978ae3f0e6759ea755. Reviewed By: stephenneuendorffer Differential Revision: https://reviews.llvm.org/D113732
2021-11-14 13:44:25 -09:00
ROOT_DIR "${MLIR_SOURCE_DIR}/test/python/lib"
[mlir][python] Segment MLIR Python test dialect to avoid testonly dependency. With https://reviews.llvm.org/rG14c9207063bb00823a5126131e50c93f6e288bd3, the build is broken with -DMLIR_INCLUDE_TESTS=OFF. This patch fixes the build and we may want to do a better fix to the layering in a followup. Differential Revision: https://reviews.llvm.org/D112560
2021-10-26 09:14:50 -08:00
SOURCES
Revert "Revert "[mlir python] Add nanobind support (#119232) Reverts revert #118517 after (hopefully) fixing builders (https://github.com/llvm/llvm-zorg/pull/328, https://github.com/llvm/llvm-zorg/pull/327) This reverts commit 61bf308cf2fc32452f14861c102ace89f5f36fec.
2024-12-09 12:37:43 -09:00
PythonTestModuleNanobind.cpp
[mlir][python] Segment MLIR Python test dialect to avoid testonly dependency. With https://reviews.llvm.org/rG14c9207063bb00823a5126131e50c93f6e288bd3, the build is broken with -DMLIR_INCLUDE_TESTS=OFF. This patch fixes the build and we may want to do a better fix to the layering in a followup. Differential Revision: https://reviews.llvm.org/D112560
2021-10-26 09:14:50 -08:00
PRIVATE_LINK_LIBS
LLVMSupport
EMBED_CAPI_LINK_LIBS
MLIRCAPIPythonTestDialect
)
endif()
[mlir] support interfaces in Python bindings Introduce the initial support for operation interfaces in C API and Python bindings. Interfaces are a key component of MLIR's extensibility and should be available in bindings to make use of full potential of MLIR. This initial implementation exposes InferTypeOpInterface all the way to the Python bindings since it can be later used to simplify the operation construction methods by inferring their return types instead of requiring the user to do so. The general infrastructure for binding interfaces is defined and InferTypeOpInterface can be used as an example for binding other interfaces. Reviewed By: gysit Differential Revision: https://reviews.llvm.org/D111656
2021-10-14 07:18:28 -08:00
[mlir][Python] create MLIRPythonSupport (#171775) # What This PR adds a shared library `MLIRPythonSupport` which contains all of the CRTP classes ike `PyConcreteValue`, `PyConcreteType`, `PyConcreteAttribute`, as well as other useful code like `Defaulting*` and etc enabling their reuse in downstream projects. Downstream projects can now do ```c++ struct PyTestType : mlir::python::MLIR_BINDINGS_PYTHON_DOMAIN::PyConcreteType<PyTestType> { ... }; class PyTestAttr : public mlir::python::MLIR_BINDINGS_PYTHON_DOMAIN::PyConcreteAttribute<PyTestAttr> { ... } NB_MODULE(_mlirPythonTestNanobind, m) { PyTestType::bind(m); PyTestAttr::bind(m); } ``` instead of using the discordant alternative `mlir_type_subclass`/`mlir_attr_subclass` (same goes for `PyConcreteValue`/`mlir_value_subclass`). # Why This PR is mostly code motion (along with CMake) but before I describe the changes I want to state the goals/benefits: 1. Currently upstream "core" extensions and "dialect" extensions ([all of the `Dialect*` extensions here](https://github.com/llvm/llvm-project/tree/d7c734b5a14bd91e1c76e2ce0014c19f9deef487/mlir/lib/Bindings/Python)) are a two-tier system; **a**. [core extensions](https://github.com/llvm/llvm-project/blob/main/mlir/lib/Bindings/Python/IRTypes.cpp#L361) enjoy first class support as far as type inference[^3], type stub generation, and ease of implementation, while dialect extensions [have poorer support](https://reviews.llvm.org/D150927), incorrect type stub generation much more tedious (boilerplate) implementation; **b**. Crucially, this two-tiered system is reflected in the fact that **the two sets of types/attributes are not in the same Python object hierarchy**. To wit: `isinstance(..., Type)` and `isinstance(..., Attribute)` are not supported for the dialect extensions[^2]; **c**. Since these types are not exposed in public headers, downstream users (dialect extensions or not) cannot write functions that overload on e.g. `PyFloat8*Type` - that's quite a [useful feature](https://github.com/nod-ai/PI/blob/fdbee98df8376f47818e6b47e1cf089528c9d48d/cpp_ext/TorchOps.cpp#L29-L69)! 2. The dialect extensions incur a sizeable performance penalty relative to the core extensions in that every single trip across the wire (either `python->cpp` or `cpp->python`) requires work in addition to nanobind's own casting/construction pipeline; **a**. When going from `python->cpp`, [we extract the capsule object from the Python object](https://github.com/llvm/llvm-project/blob/main/mlir/include/mlir/Bindings/Python/NanobindAdaptors.h#L219C24-L219C46) and then extract from the capsule the `Mlir*` opaque struct/ptr. This side isn't so onerous; **b**. When going from `cpp->python` we call long-hand call Python `import` APIs and construct the Python object using `_CAPICreate`. Note, there at least 2 `attr` calls incurred in addition to `_CAPICreate`; this is already much more [efficiently handled by nanobind itself](https://github.com/wjakob/nanobind/blob/4ba51fcf795971c5d603d875ae4184bc0c9bd8e6/src/nb_internals.h#L381-L382)! 3. This division blocks various features: in some configurations[^1] we trigger a circular import bug because "dialect" types and attributes perform an [import of the root `_mlir` module](https://github.com/llvm/llvm-project/blob/bd9651bf78f2b1713a8203e0bd5b97f7ff199924/mlir/include/mlir/Bindings/Python/NanobindAdaptors.h#L585) when they are created (the types themselves, not even instances of those types). This blocks type stub generation for dialect extensions (i.e., the reason we currently only generate type stubs for `_mlir`). # How Prior this was not done/possible because of "ODR" issues but I have resolved those issues; the basic idea for how we solve this is "move things we want to share into shared libraries": 1. Move IRCore (stuff like `PyConcreteValue`, `PyConcreteType`, `PyConcreteAttribute`) into `MLIRPythonSupport`; - Note, we move the rest of the things in `IRModule.h` (renamed to `IRCore.h`) because `PyConcreteValue`, `PyConcreteType`, `PyConcreteAttribute` depend on them. This makes for a bigger PR than one would hope for but ultimately I think we should give people access to these classes to use as they see fit (specifically inherit from, but also liberally use in bindings signatures instead of the opaque `Mlir*` struct wrappers). 2. Put all of this code into a nested namespace `MLIR_BINDINGS_PYTHON_DOMAIN` which is determined by a compile time define (and tied to `MLIR_BINDINGS_PYTHON_NB_DOMAIN`). This is necessary in order to prevent conflicts on both symbol name **and** typeid (necessary for nanobind to not double register binded types) between multiple bindings libraries (e.g., `torch-mlir`, and `jax`). Note [nanobind doesn't support `module_local` like pybind11](https://nanobind.readthedocs.io/en/latest/porting.html#removed-features). It does support `NB_DOMAIN` but that is not sufficient for disambiguating typeids across projects (to wit: we currently define `NB_DOMAIN` and it was still necessary to move everything to a nested namespace); 3. Build the [nanobind library itself as a shared object](https://github.com/wjakob/nanobind/blob/master/cmake/nanobind-config.cmake#L127) (and link it to both the extensions and `MLIRPythonSupport`). 4. CMake to make this work, in-tree, out-of-tree, downstream, upstream, etc. # Testing Three tests are added here 1. `PythonTestModuleNanobind` is ported to use `PyConcreteType<PyTestType>` instead of `mlir_type_subclass` and `PyConcreteAttribute<PyTestAttr>` instead of `mlir_atrr_subclass`, verifying this works for non-core extensions in-tree; 2. `StandaloneExtensionNanobind` is ported to use `struct PyCustomType : mlir::python::MLIR_BINDINGS_PYTHON_DOMAIN::PyConcreteType<PyCustomType>` instead of `mlir_type_subclass` verifying this works for non-core extensions out-of-tree; 3. `StandaloneExtensionNanobind`'s `smoketest` is extended to also load another bindings package (namely `mlir`) verifying `MLIR_BINDINGS_PYTHON_DOMAIN` successfully disambiguates symbols and typeids. I have also tested this downstream: https://github.com/llvm/eudsl/pull/287 as well run the following builder bots: mlir-nvidia-gcc7: https://lab.llvm.org/buildbot/#/buildrequests/6654424?redirect_to_build=true I have also tested against IREE: https://github.com/iree-org/iree/pull/21916 # Integration It is highly recommended to set the CMake var `MLIR_BINDINGS_PYTHON_NB_DOMAIN` (which will also determine `MLIR_BINDINGS_PYTHON_DOMAIN`) to something unique for each downstream. This can also be passed explicitly to `add_mlir_python_modules` if your project builds multiple bindings packages. I added a `WARNING` to this effect in `AddMLIRPython.cmake`. [^3]: Python values being typed correctly when exiting from cpp; [^1]: Specifically when the modules are imported using `importlib`, which occurs with nanobind's [stubgen](https://github.com/wjakob/nanobind/blob/master/src/stubgen.py#L965); [^2]: The workaround we implemented was a class method for the dialect bindings called `Class.isinstance(...)`;
2026-01-05 08:08:13 -09:00
declare_mlir_python_extension(MLIRPythonExtension.MLIRPythonSupport
_PRIVATE_SUPPORT_LIB
MODULE_NAME MLIRPythonSupport
ADD_TO_PARENT MLIRPythonSources.Core
ROOT_DIR "${PYTHON_SOURCE_DIR}"
SOURCES
Globals.cpp
[mlir][Python] move IRTypes and IRAttributes to MLIRPythonSupport (#174118) This PR continues the work of https://github.com/llvm/llvm-project/pull/171775 by moving more useful types/attributes into MLIRPythonSupport. You can now do ```c++ struct PyTestIntegerRankedTensorType : mlir::python::MLIR_BINDINGS_PYTHON_DOMAIN::PyConcreteType< PyTestIntegerRankedTensorType, mlir::python::MLIR_BINDINGS_PYTHON_DOMAIN::PyRankedTensorType> struct PyTestTensorValue : mlir::python::MLIR_BINDINGS_PYTHON_DOMAIN::PyConcreteValue< PyTestTensorValue> ``` instead of `mlir_type_subclass` and `mlir_value_subclass`; **specifically manual registration of the "value caster" via indirection through the Python interpreter is no longer necessary** . You can also now freely use all such types at the nanobind API level (e.g., overload based on `FP*`): ```c++ using mlir::python::MLIR_BINDINGS_PYTHON_DOMAIN; standaloneM.def("print_fp_type", [](PyF16Type &) { nb::print("this is a fp16 type"); }); standaloneM.def("print_fp_type", [](PyF32Type &) { nb::print("this is a fp32 type"); }); standaloneM.def("print_fp_type", [](PyF64Type &) { nb::print("this is a fp64 type"); }); ``` Note, here we only port `PythonTestModuleNanobind` but there is a follow-up PR that ports **all** in-tree dialect extensions https://github.com/llvm/llvm-project/pull/174156 to use these. After that one we can soft deprecate `mlir_pure_subclass`. Note, depends on https://github.com/llvm/llvm-project/pull/171775
2026-01-05 08:34:58 -09:00
IRAttributes.cpp
IRCore.cpp
IRTypes.cpp
[mlir][Python] create MLIRPythonSupport (#171775) # What This PR adds a shared library `MLIRPythonSupport` which contains all of the CRTP classes ike `PyConcreteValue`, `PyConcreteType`, `PyConcreteAttribute`, as well as other useful code like `Defaulting*` and etc enabling their reuse in downstream projects. Downstream projects can now do ```c++ struct PyTestType : mlir::python::MLIR_BINDINGS_PYTHON_DOMAIN::PyConcreteType<PyTestType> { ... }; class PyTestAttr : public mlir::python::MLIR_BINDINGS_PYTHON_DOMAIN::PyConcreteAttribute<PyTestAttr> { ... } NB_MODULE(_mlirPythonTestNanobind, m) { PyTestType::bind(m); PyTestAttr::bind(m); } ``` instead of using the discordant alternative `mlir_type_subclass`/`mlir_attr_subclass` (same goes for `PyConcreteValue`/`mlir_value_subclass`). # Why This PR is mostly code motion (along with CMake) but before I describe the changes I want to state the goals/benefits: 1. Currently upstream "core" extensions and "dialect" extensions ([all of the `Dialect*` extensions here](https://github.com/llvm/llvm-project/tree/d7c734b5a14bd91e1c76e2ce0014c19f9deef487/mlir/lib/Bindings/Python)) are a two-tier system; **a**. [core extensions](https://github.com/llvm/llvm-project/blob/main/mlir/lib/Bindings/Python/IRTypes.cpp#L361) enjoy first class support as far as type inference[^3], type stub generation, and ease of implementation, while dialect extensions [have poorer support](https://reviews.llvm.org/D150927), incorrect type stub generation much more tedious (boilerplate) implementation; **b**. Crucially, this two-tiered system is reflected in the fact that **the two sets of types/attributes are not in the same Python object hierarchy**. To wit: `isinstance(..., Type)` and `isinstance(..., Attribute)` are not supported for the dialect extensions[^2]; **c**. Since these types are not exposed in public headers, downstream users (dialect extensions or not) cannot write functions that overload on e.g. `PyFloat8*Type` - that's quite a [useful feature](https://github.com/nod-ai/PI/blob/fdbee98df8376f47818e6b47e1cf089528c9d48d/cpp_ext/TorchOps.cpp#L29-L69)! 2. The dialect extensions incur a sizeable performance penalty relative to the core extensions in that every single trip across the wire (either `python->cpp` or `cpp->python`) requires work in addition to nanobind's own casting/construction pipeline; **a**. When going from `python->cpp`, [we extract the capsule object from the Python object](https://github.com/llvm/llvm-project/blob/main/mlir/include/mlir/Bindings/Python/NanobindAdaptors.h#L219C24-L219C46) and then extract from the capsule the `Mlir*` opaque struct/ptr. This side isn't so onerous; **b**. When going from `cpp->python` we call long-hand call Python `import` APIs and construct the Python object using `_CAPICreate`. Note, there at least 2 `attr` calls incurred in addition to `_CAPICreate`; this is already much more [efficiently handled by nanobind itself](https://github.com/wjakob/nanobind/blob/4ba51fcf795971c5d603d875ae4184bc0c9bd8e6/src/nb_internals.h#L381-L382)! 3. This division blocks various features: in some configurations[^1] we trigger a circular import bug because "dialect" types and attributes perform an [import of the root `_mlir` module](https://github.com/llvm/llvm-project/blob/bd9651bf78f2b1713a8203e0bd5b97f7ff199924/mlir/include/mlir/Bindings/Python/NanobindAdaptors.h#L585) when they are created (the types themselves, not even instances of those types). This blocks type stub generation for dialect extensions (i.e., the reason we currently only generate type stubs for `_mlir`). # How Prior this was not done/possible because of "ODR" issues but I have resolved those issues; the basic idea for how we solve this is "move things we want to share into shared libraries": 1. Move IRCore (stuff like `PyConcreteValue`, `PyConcreteType`, `PyConcreteAttribute`) into `MLIRPythonSupport`; - Note, we move the rest of the things in `IRModule.h` (renamed to `IRCore.h`) because `PyConcreteValue`, `PyConcreteType`, `PyConcreteAttribute` depend on them. This makes for a bigger PR than one would hope for but ultimately I think we should give people access to these classes to use as they see fit (specifically inherit from, but also liberally use in bindings signatures instead of the opaque `Mlir*` struct wrappers). 2. Put all of this code into a nested namespace `MLIR_BINDINGS_PYTHON_DOMAIN` which is determined by a compile time define (and tied to `MLIR_BINDINGS_PYTHON_NB_DOMAIN`). This is necessary in order to prevent conflicts on both symbol name **and** typeid (necessary for nanobind to not double register binded types) between multiple bindings libraries (e.g., `torch-mlir`, and `jax`). Note [nanobind doesn't support `module_local` like pybind11](https://nanobind.readthedocs.io/en/latest/porting.html#removed-features). It does support `NB_DOMAIN` but that is not sufficient for disambiguating typeids across projects (to wit: we currently define `NB_DOMAIN` and it was still necessary to move everything to a nested namespace); 3. Build the [nanobind library itself as a shared object](https://github.com/wjakob/nanobind/blob/master/cmake/nanobind-config.cmake#L127) (and link it to both the extensions and `MLIRPythonSupport`). 4. CMake to make this work, in-tree, out-of-tree, downstream, upstream, etc. # Testing Three tests are added here 1. `PythonTestModuleNanobind` is ported to use `PyConcreteType<PyTestType>` instead of `mlir_type_subclass` and `PyConcreteAttribute<PyTestAttr>` instead of `mlir_atrr_subclass`, verifying this works for non-core extensions in-tree; 2. `StandaloneExtensionNanobind` is ported to use `struct PyCustomType : mlir::python::MLIR_BINDINGS_PYTHON_DOMAIN::PyConcreteType<PyCustomType>` instead of `mlir_type_subclass` verifying this works for non-core extensions out-of-tree; 3. `StandaloneExtensionNanobind`'s `smoketest` is extended to also load another bindings package (namely `mlir`) verifying `MLIR_BINDINGS_PYTHON_DOMAIN` successfully disambiguates symbols and typeids. I have also tested this downstream: https://github.com/llvm/eudsl/pull/287 as well run the following builder bots: mlir-nvidia-gcc7: https://lab.llvm.org/buildbot/#/buildrequests/6654424?redirect_to_build=true I have also tested against IREE: https://github.com/iree-org/iree/pull/21916 # Integration It is highly recommended to set the CMake var `MLIR_BINDINGS_PYTHON_NB_DOMAIN` (which will also determine `MLIR_BINDINGS_PYTHON_DOMAIN`) to something unique for each downstream. This can also be passed explicitly to `add_mlir_python_modules` if your project builds multiple bindings packages. I added a `WARNING` to this effect in `AddMLIRPython.cmake`. [^3]: Python values being typed correctly when exiting from cpp; [^1]: Specifically when the modules are imported using `importlib`, which occurs with nanobind's [stubgen](https://github.com/wjakob/nanobind/blob/master/src/stubgen.py#L965); [^2]: The workaround we implemented was a class method for the dialect bindings called `Class.isinstance(...)`;
2026-01-05 08:08:13 -09:00
)
Move MLIR python sources to mlir/python. * NFC but has some fixes for CMake glitches discovered along the way (things not cleaning properly, co-mingled depends). * Includes previously unsubmitted fix in D98681 and a TODO to fix it more appropriately in a smaller followup. Differential Revision: https://reviews.llvm.org/D101493
2021-04-28 12:04:17 -08:00
################################################################################
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
# Common CAPI dependency DSO.
# All python extensions must link through one DSO which exports the CAPI, and
# this must have a globally unique name amongst all embeddors of the python
# library since it will effectively have global scope.
#
# The presence of this aggregate library is part of the long term plan, but its
# use needs to be made more flexible.
#
# TODO: Upgrade to the aggregate utility in https://reviews.llvm.org/D106419
# once ready.
Move MLIR python sources to mlir/python. * NFC but has some fixes for CMake glitches discovered along the way (things not cleaning properly, co-mingled depends). * Includes previously unsubmitted fix in D98681 and a TODO to fix it more appropriately in a smaller followup. Differential Revision: https://reviews.llvm.org/D101493
2021-04-28 12:04:17 -08:00
################################################################################
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
add_mlir_python_common_capi_library(MLIRPythonCAPI
INSTALL_COMPONENT MLIRPythonModules
[mlir][python] add `MLIR_BINDINGS_PYTHON_INSTALL_PREFIX` to make bindings install dir configurable (#124878) This PR parameterizes the install directory of the MLIR Python bindings in the final distribution.
2025-06-23 11:49:01 -08:00
INSTALL_DESTINATION "${MLIR_BINDINGS_PYTHON_INSTALL_PREFIX}/_mlir_libs"
[MLIR][Python] reland (narrower) type stub generation (#157930) This a reland of https://github.com/llvm/llvm-project/pull/155741 which was reverted at https://github.com/llvm/llvm-project/pull/157831. This version is narrower in scope - it only turns on automatic stub generation for `MLIRPythonExtension.Core._mlir` and **does not do anything automatically**. Specifically, the only CMake code added to `AddMLIRPython.cmake` is the `mlir_generate_type_stubs` function which is then used only in a manual way. The API for `mlir_generate_type_stubs` is: ``` Arguments: MODULE_NAME: The fully-qualified name of the extension module (used for importing in python). DEPENDS_TARGETS: List of targets these type stubs depend on being built; usually corresponding to the specific extension module (e.g., something like StandalonePythonModules.extension._standaloneDialectsNanobind.dso) and the core bindings extension module (e.g., something like StandalonePythonModules.extension._mlir.dso). OUTPUT_DIR: The root output directory to emit the type stubs into. OUTPUTS: List of expected outputs. DEPENDS_TARGET_SRC_DEPS: List of cpp sources for extension library (for generating a DEPFILE). IMPORT_PATHS: List of paths to add to PYTHONPATH for stubgen. PATTERN_FILE: (Optional) Pattern file (see https://nanobind.readthedocs.io/en/latest/typing.html#pattern-files). Outputs: NB_STUBGEN_CUSTOM_TARGET: The target corresponding to generation which other targets can depend on. ``` Downstream users should use `mlir_generate_type_stubs` in coordination with `declare_mlir_python_sources` to turn on stub generation for their own downstream dialect extensions and upstream dialect extensions if they so choose. Standalone example shows an example. Note, downstream will also need to set `-DMLIR_PYTHON_PACKAGE_PREFIX=...` correctly for their bindings.
2025-09-20 10:47:32 -08:00
OUTPUT_DIRECTORY "${MLIRPythonModules_ROOT_PREFIX}/_mlir_libs"
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
RELATIVE_INSTALL_ROOT "../../../.."
[mlir] Leverage CMake interface libraries for mlir python This is already partially the case, but we can rely more heavily on interface libraries and how they are imported/exported in other to simplify the implementation of the mlir python functions in Cmake. This change also makes a couple of other changes: 1) Add a new CMake function which handles "pure" sources. This was done inline previously 2) Moves the headers associated with CAPI libraries to the libraries themselves. These were previously managed in a separate source target. They can now be added directly to the CAPI libraries using DECLARED_HEADERS. 3) Cleanup some dependencies that showed up as an issue during the refactor This is a big CMake change that should produce no impact on the build of mlir and on the produced *build tree*. However, this change fixes an issue with the *install tree* of mlir which was previously unusable for projects like torch-mlir because both the "pure" and "extension" targets were pointing to either the build or source trees. Reviewed By: stellaraccident Differential Revision: https://reviews.llvm.org/D128230
2022-06-28 09:39:13 -08:00
DECLARED_HEADERS
MLIRPythonCAPI.HeaderSources
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
DECLARED_SOURCES
MLIRPythonSources
[mlir] Overhaul C/Python registration APIs to properly scope registration/loading activities. Since the very first commits, the Python and C MLIR APIs have had mis-placed registration/load functionality for dialects, extensions, etc. This was done pragmatically in order to get bootstrapped and then just grew in. Downstreams largely bypass and do their own thing by providing various APIs to register things they need. Meanwhile, the C++ APIs have stabilized around this and it would make sense to follow suit. The thing we have observed in canonical usage by downstreams is that each downstream tends to have native entry points that configure its installation to its preferences with one-stop APIs. This patch leans in to this approach with `RegisterEverything.h` and `mlir._mlir_libs._mlirRegisterEverything` being the one-stop entry points for the "upstream packages". The `_mlir_libs.__init__.py` now allows customization of the environment and Context by adding "initialization modules" to the `_mlir_libs` package. If present, `_mlirRegisterEverything` is treated as such a module. Others can be added by downstreams by adding a `_site_initialize_{i}.py` module, where '{i}' is a number starting with zero. The number will be incremented and corresponding module loaded until one is not found. Initialization modules can: * Perform load time customization to the global environment (i.e. registering passes, hooks, etc). * Define a `register_dialects(registry: DialectRegistry)` function that can extend the `DialectRegistry` that will be used to bootstrap the `Context`. * Define a `context_init_hook(context: Context)` function that will be added to a list of callbacks which will be invoked after dialect registration during `Context` initialization. Note that the `MLIRPythonExtension.RegisterEverything` is not included by default when building a downstream (its corresponding behavior was prior). For downstreams which need the default MLIR initialization to take place, they must add this back in to their Python CMake build just like they add their own components (i.e. to `add_mlir_python_common_capi_library` and `add_mlir_python_modules`). It is perfectly valid to not do this, in which case, only the things explicitly depended on and initialized by downstreams will be built/packaged. If the downstream has not been set up for this, it is recommended to simply add this back for the time being and pay the build time/package size cost. CMake changes: * `MLIRCAPIRegistration` -> `MLIRCAPIRegisterEverything` (renamed to signify what it does and force an evaluation: a number of places were incidentally linking this very expensive target) * `MLIRPythonSoure.Passes` removed (without replacement: just drop) * `MLIRPythonExtension.AllPassesRegistration` removed (without replacement: just drop) * `MLIRPythonExtension.Conversions` removed (without replacement: just drop) * `MLIRPythonExtension.Transforms` removed (without replacement: just drop) Header changes: * `mlir-c/Registration.h` is deleted. Dialect registration functionality is now in `IR.h`. Registration of upstream features are in `mlir-c/RegisterEverything.h`. When updating MLIR and a couple of downstreams, I found that proper usage was commingled so required making a choice vs just blind S&R. Python APIs removed: * mlir.transforms and mlir.conversions (previously only had an __init__.py which indirectly triggered `mlirRegisterTransformsPasses()` and `mlirRegisterConversionPasses()` respectively). Downstream impact: Remove these imports if present (they now happen as part of default initialization). * mlir._mlir_libs._all_passes_registration, mlir._mlir_libs._mlirTransforms, mlir._mlir_libs._mlirConversions. Downstream impact: None expected (these were internally used). C-APIs changed: * mlirRegisterAllDialects(MlirContext) now takes an MlirDialectRegistry instead. It also used to trigger loading of all dialects, which was already marked with a TODO to remove -- it no longer does, and for direct use, dialects must be explicitly loaded. Downstream impact: Direct C-API users must ensure that needed dialects are loaded or call `mlirContextLoadAllAvailableDialects(MlirContext)` to emulate the prior behavior. Also see the `ir.c` test case (e.g. ` mlirContextGetOrLoadDialect(ctx, mlirStringRefCreateFromCString("func"));`). * mlirDialectHandle* APIs were moved from Registration.h (which now is restricted to just global/upstream registration) to IR.h, arguably where it should have been. Downstream impact: include correct header (likely already doing so). C-APIs added: * mlirContextLoadAllAvailableDialects(MlirContext): Corresponds to C++ API with the same purpose. Python APIs added: * mlir.ir.DialectRegistry: Mapping for an MlirDialectRegistry. * mlir.ir.Context.append_dialect_registry(MlirDialectRegistry) * mlir.ir.Context.load_all_available_dialects() * mlir._mlir_libs._mlirAllRegistration: New native extension that exposes a `register_dialects(MlirDialectRegistry)` entry point and performs all upstream pass/conversion/transforms registration on init. In this first step, we eagerly load this as part of the __init__.py and use it to monkey patch the Context to emulate prior behavior. * Type caster and capsule support for MlirDialectRegistry This should make it possible to build downstream Python dialects that only depend on a subset of MLIR. See: https://github.com/llvm/llvm-project/issues/56037 Here is an example PR, minimally adapting IREE to these changes: https://github.com/iree-org/iree/pull/9638/files In this situation, IREE is opting to not link everything, since it is already configuring the Context to its liking. For projects that would just like to not think about it and pull in everything, add `MLIRPythonExtension.RegisterEverything` to the list of Python sources getting built, and the old behavior will continue. Reviewed By: mehdi_amini, ftynse Differential Revision: https://reviews.llvm.org/D128593
2022-07-16 15:09:03 -08:00
MLIRPythonExtension.RegisterEverything
[mlir][python] Segment MLIR Python test dialect to avoid testonly dependency. With https://reviews.llvm.org/rG14c9207063bb00823a5126131e50c93f6e288bd3, the build is broken with -DMLIR_INCLUDE_TESTS=OFF. This patch fixes the build and we may want to do a better fix to the layering in a followup. Differential Revision: https://reviews.llvm.org/D112560
2021-10-26 09:14:50 -08:00
${_ADDL_TEST_SOURCES}
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
)
[mlir][python] Add install target for MLIR Python sources. Differential Revision: https://reviews.llvm.org/D155362
2023-07-15 01:44:42 -08:00
################################################################################
# Custom targets.
################################################################################
_flatten_mlir_python_targets(mlir_python_sources_deps MLIRPythonSources)
[MLIR][Python] Skip stubgen while any sanitizer is enabled (#164661) The intention of this PR is described https://github.com/llvm/llvm-project/issues/164197#issuecomment-3432843709 (and https://github.com/llvm/llvm-project/issues/164197#issuecomment-3432935838). When sanitizers are enabled, some additional setup (such as preloading certain libraries) seems to be required for the stubgen step to work properly (https://github.com/llvm/llvm-project/issues/164197#issuecomment-3432924034). In this case, I chose to simply skip the stubgen process, as supporting it would likely require some extra CMake logic, and type stubs don’t appear to be particularly necessary in this configuration.
2025-10-22 21:49:07 -08:00
[MLIR][Python] make sure stubs get installed with LLVM_DISTRIBUTION_COMPONENTS (#168407) Fixes https://github.com/llvm/llvm-project/issues/168393. Also adds top-level `MLIR_PYTHON_STUBGEN_ENABLED` CMake option.
2025-11-19 06:07:28 -09:00
if(MLIR_PYTHON_STUBGEN_ENABLED)
[MLIR][Python] don't generate type stubs when cross-compiling (#160793) Stubgen doesn't work when cross-compiling (stubgen will run in the host interpreter and then fail to find the extension module for the host arch). So disable it when `CMAKE_CROSSCOMPILING=ON`.
2025-09-25 15:59:12 -08:00
# _mlir stubgen
# Note: All this needs to come before add_mlir_python_modules(MLIRPythonModules so that the install targets for the
# generated type stubs get created.
set(_core_type_stub_sources
_mlir/__init__.pyi
_mlir/ir.pyi
_mlir/passmanager.pyi
_mlir/rewrite.pyi
)
[MLIR][Python] reland (narrower) type stub generation (#157930) This a reland of https://github.com/llvm/llvm-project/pull/155741 which was reverted at https://github.com/llvm/llvm-project/pull/157831. This version is narrower in scope - it only turns on automatic stub generation for `MLIRPythonExtension.Core._mlir` and **does not do anything automatically**. Specifically, the only CMake code added to `AddMLIRPython.cmake` is the `mlir_generate_type_stubs` function which is then used only in a manual way. The API for `mlir_generate_type_stubs` is: ``` Arguments: MODULE_NAME: The fully-qualified name of the extension module (used for importing in python). DEPENDS_TARGETS: List of targets these type stubs depend on being built; usually corresponding to the specific extension module (e.g., something like StandalonePythonModules.extension._standaloneDialectsNanobind.dso) and the core bindings extension module (e.g., something like StandalonePythonModules.extension._mlir.dso). OUTPUT_DIR: The root output directory to emit the type stubs into. OUTPUTS: List of expected outputs. DEPENDS_TARGET_SRC_DEPS: List of cpp sources for extension library (for generating a DEPFILE). IMPORT_PATHS: List of paths to add to PYTHONPATH for stubgen. PATTERN_FILE: (Optional) Pattern file (see https://nanobind.readthedocs.io/en/latest/typing.html#pattern-files). Outputs: NB_STUBGEN_CUSTOM_TARGET: The target corresponding to generation which other targets can depend on. ``` Downstream users should use `mlir_generate_type_stubs` in coordination with `declare_mlir_python_sources` to turn on stub generation for their own downstream dialect extensions and upstream dialect extensions if they so choose. Standalone example shows an example. Note, downstream will also need to set `-DMLIR_PYTHON_PACKAGE_PREFIX=...` correctly for their bindings.
2025-09-20 10:47:32 -08:00
[MLIR][Python] don't generate type stubs when cross-compiling (#160793) Stubgen doesn't work when cross-compiling (stubgen will run in the host interpreter and then fail to find the extension module for the host arch). So disable it when `CMAKE_CROSSCOMPILING=ON`.
2025-09-25 15:59:12 -08:00
# Note 1: INTERFACE_SOURCES is a genex ($<BUILD_INTERFACE> $<INSTALL_INTERFACE>)
# which will be evaluated by file(GENERATE ...) inside mlir_generate_type_stubs. This will evaluate to the correct
# thing in the build dir (i.e., actual source dir paths) and in the install dir
# (where it's a conventional path; see install/lib/cmake/mlir/MLIRTargets.cmake).
#
# Note 2: MLIRPythonExtension.Core is the target that is defined using target_sources(INTERFACE)
# **NOT** MLIRPythonModules.extension._mlir.dso. So be sure to use the correct target!
get_target_property(_core_extension_srcs MLIRPythonExtension.Core INTERFACE_SOURCES)
# Why is MODULE_NAME _mlir here but mlir._mlir_libs._mlirPythonTestNanobind below???
# The _mlir extension can be imported independently of any other python code and/or extension modules.
# I.e., you could do `cd $MLIRPythonModules_ROOT_PREFIX/_mlir_libs && python -c "import _mlir"` (try it!).
# _mlir is also (currently) the only extension for which this is possible because dialect extensions modules,
# which generally make use of `mlir_value_subclass/mlir_type_subclass/mlir_attribute_subclass`, perform an
# `import mlir` right when they're loaded (see the mlir_*_subclass ctors in NanobindAdaptors.h).
# Note, this also why IMPORT_PATHS "${MLIRPythonModules_ROOT_PREFIX}/_mlir_libs" here while below
# "${MLIRPythonModules_ROOT_PREFIX}/.." (because MLIR_BINDINGS_PYTHON_INSTALL_PREFIX, by default, ends at mlir).
#
# Further note: this function creates file targets like
# "${CMAKE_CURRENT_BINARY_DIR}/type_stubs/_mlir_libs/_mlir/__init__.pyi". These must match the file targets
# that declare_mlir_python_sources expects, which are like "${ROOT_DIR}/${WHATEVER_SOURCE}".
# This is why _mlir_libs is prepended below.
[MLIR][Python] reland (narrower) type stub generation (#157930) This a reland of https://github.com/llvm/llvm-project/pull/155741 which was reverted at https://github.com/llvm/llvm-project/pull/157831. This version is narrower in scope - it only turns on automatic stub generation for `MLIRPythonExtension.Core._mlir` and **does not do anything automatically**. Specifically, the only CMake code added to `AddMLIRPython.cmake` is the `mlir_generate_type_stubs` function which is then used only in a manual way. The API for `mlir_generate_type_stubs` is: ``` Arguments: MODULE_NAME: The fully-qualified name of the extension module (used for importing in python). DEPENDS_TARGETS: List of targets these type stubs depend on being built; usually corresponding to the specific extension module (e.g., something like StandalonePythonModules.extension._standaloneDialectsNanobind.dso) and the core bindings extension module (e.g., something like StandalonePythonModules.extension._mlir.dso). OUTPUT_DIR: The root output directory to emit the type stubs into. OUTPUTS: List of expected outputs. DEPENDS_TARGET_SRC_DEPS: List of cpp sources for extension library (for generating a DEPFILE). IMPORT_PATHS: List of paths to add to PYTHONPATH for stubgen. PATTERN_FILE: (Optional) Pattern file (see https://nanobind.readthedocs.io/en/latest/typing.html#pattern-files). Outputs: NB_STUBGEN_CUSTOM_TARGET: The target corresponding to generation which other targets can depend on. ``` Downstream users should use `mlir_generate_type_stubs` in coordination with `declare_mlir_python_sources` to turn on stub generation for their own downstream dialect extensions and upstream dialect extensions if they so choose. Standalone example shows an example. Note, downstream will also need to set `-DMLIR_PYTHON_PACKAGE_PREFIX=...` correctly for their bindings.
2025-09-20 10:47:32 -08:00
mlir_generate_type_stubs(
[MLIR][Python] don't generate type stubs when cross-compiling (#160793) Stubgen doesn't work when cross-compiling (stubgen will run in the host interpreter and then fail to find the extension module for the host arch). So disable it when `CMAKE_CROSSCOMPILING=ON`.
2025-09-25 15:59:12 -08:00
MODULE_NAME _mlir
DEPENDS_TARGETS MLIRPythonModules.extension._mlir.dso
[MLIR][Python] reland (narrower) type stub generation (#157930) This a reland of https://github.com/llvm/llvm-project/pull/155741 which was reverted at https://github.com/llvm/llvm-project/pull/157831. This version is narrower in scope - it only turns on automatic stub generation for `MLIRPythonExtension.Core._mlir` and **does not do anything automatically**. Specifically, the only CMake code added to `AddMLIRPython.cmake` is the `mlir_generate_type_stubs` function which is then used only in a manual way. The API for `mlir_generate_type_stubs` is: ``` Arguments: MODULE_NAME: The fully-qualified name of the extension module (used for importing in python). DEPENDS_TARGETS: List of targets these type stubs depend on being built; usually corresponding to the specific extension module (e.g., something like StandalonePythonModules.extension._standaloneDialectsNanobind.dso) and the core bindings extension module (e.g., something like StandalonePythonModules.extension._mlir.dso). OUTPUT_DIR: The root output directory to emit the type stubs into. OUTPUTS: List of expected outputs. DEPENDS_TARGET_SRC_DEPS: List of cpp sources for extension library (for generating a DEPFILE). IMPORT_PATHS: List of paths to add to PYTHONPATH for stubgen. PATTERN_FILE: (Optional) Pattern file (see https://nanobind.readthedocs.io/en/latest/typing.html#pattern-files). Outputs: NB_STUBGEN_CUSTOM_TARGET: The target corresponding to generation which other targets can depend on. ``` Downstream users should use `mlir_generate_type_stubs` in coordination with `declare_mlir_python_sources` to turn on stub generation for their own downstream dialect extensions and upstream dialect extensions if they so choose. Standalone example shows an example. Note, downstream will also need to set `-DMLIR_PYTHON_PACKAGE_PREFIX=...` correctly for their bindings.
2025-09-20 10:47:32 -08:00
OUTPUT_DIR "${CMAKE_CURRENT_BINARY_DIR}/type_stubs/_mlir_libs"
[MLIR][Python] don't generate type stubs when cross-compiling (#160793) Stubgen doesn't work when cross-compiling (stubgen will run in the host interpreter and then fail to find the extension module for the host arch). So disable it when `CMAKE_CROSSCOMPILING=ON`.
2025-09-25 15:59:12 -08:00
OUTPUTS "${_core_type_stub_sources}"
DEPENDS_TARGET_SRC_DEPS "${_core_extension_srcs}"
IMPORT_PATHS "${MLIRPythonModules_ROOT_PREFIX}/_mlir_libs"
[MLIR][Python] reland (narrower) type stub generation (#157930) This a reland of https://github.com/llvm/llvm-project/pull/155741 which was reverted at https://github.com/llvm/llvm-project/pull/157831. This version is narrower in scope - it only turns on automatic stub generation for `MLIRPythonExtension.Core._mlir` and **does not do anything automatically**. Specifically, the only CMake code added to `AddMLIRPython.cmake` is the `mlir_generate_type_stubs` function which is then used only in a manual way. The API for `mlir_generate_type_stubs` is: ``` Arguments: MODULE_NAME: The fully-qualified name of the extension module (used for importing in python). DEPENDS_TARGETS: List of targets these type stubs depend on being built; usually corresponding to the specific extension module (e.g., something like StandalonePythonModules.extension._standaloneDialectsNanobind.dso) and the core bindings extension module (e.g., something like StandalonePythonModules.extension._mlir.dso). OUTPUT_DIR: The root output directory to emit the type stubs into. OUTPUTS: List of expected outputs. DEPENDS_TARGET_SRC_DEPS: List of cpp sources for extension library (for generating a DEPFILE). IMPORT_PATHS: List of paths to add to PYTHONPATH for stubgen. PATTERN_FILE: (Optional) Pattern file (see https://nanobind.readthedocs.io/en/latest/typing.html#pattern-files). Outputs: NB_STUBGEN_CUSTOM_TARGET: The target corresponding to generation which other targets can depend on. ``` Downstream users should use `mlir_generate_type_stubs` in coordination with `declare_mlir_python_sources` to turn on stub generation for their own downstream dialect extensions and upstream dialect extensions if they so choose. Standalone example shows an example. Note, downstream will also need to set `-DMLIR_PYTHON_PACKAGE_PREFIX=...` correctly for their bindings.
2025-09-20 10:47:32 -08:00
)
[MLIR][Python] don't generate type stubs when cross-compiling (#160793) Stubgen doesn't work when cross-compiling (stubgen will run in the host interpreter and then fail to find the extension module for the host arch). So disable it when `CMAKE_CROSSCOMPILING=ON`.
2025-09-25 15:59:12 -08:00
set(_mlir_typestub_gen_target "${NB_STUBGEN_CUSTOM_TARGET}")
list(TRANSFORM _core_type_stub_sources PREPEND "_mlir_libs/")
# Note, we do not do ADD_TO_PARENT here so that the type stubs are not associated (as mlir_DEPENDS) with
# MLIRPythonSources.Core (or something) when a distro is installed/created. Otherwise they would not be regenerated
# by users of the distro (the stubs are still installed in the distro - they are just not added to mlir_DEPENDS).
[MLIR][Python] reland (narrower) type stub generation (#157930) This a reland of https://github.com/llvm/llvm-project/pull/155741 which was reverted at https://github.com/llvm/llvm-project/pull/157831. This version is narrower in scope - it only turns on automatic stub generation for `MLIRPythonExtension.Core._mlir` and **does not do anything automatically**. Specifically, the only CMake code added to `AddMLIRPython.cmake` is the `mlir_generate_type_stubs` function which is then used only in a manual way. The API for `mlir_generate_type_stubs` is: ``` Arguments: MODULE_NAME: The fully-qualified name of the extension module (used for importing in python). DEPENDS_TARGETS: List of targets these type stubs depend on being built; usually corresponding to the specific extension module (e.g., something like StandalonePythonModules.extension._standaloneDialectsNanobind.dso) and the core bindings extension module (e.g., something like StandalonePythonModules.extension._mlir.dso). OUTPUT_DIR: The root output directory to emit the type stubs into. OUTPUTS: List of expected outputs. DEPENDS_TARGET_SRC_DEPS: List of cpp sources for extension library (for generating a DEPFILE). IMPORT_PATHS: List of paths to add to PYTHONPATH for stubgen. PATTERN_FILE: (Optional) Pattern file (see https://nanobind.readthedocs.io/en/latest/typing.html#pattern-files). Outputs: NB_STUBGEN_CUSTOM_TARGET: The target corresponding to generation which other targets can depend on. ``` Downstream users should use `mlir_generate_type_stubs` in coordination with `declare_mlir_python_sources` to turn on stub generation for their own downstream dialect extensions and upstream dialect extensions if they so choose. Standalone example shows an example. Note, downstream will also need to set `-DMLIR_PYTHON_PACKAGE_PREFIX=...` correctly for their bindings.
2025-09-20 10:47:32 -08:00
declare_mlir_python_sources(
[MLIR][Python] don't generate type stubs when cross-compiling (#160793) Stubgen doesn't work when cross-compiling (stubgen will run in the host interpreter and then fail to find the extension module for the host arch). So disable it when `CMAKE_CROSSCOMPILING=ON`.
2025-09-25 15:59:12 -08:00
MLIRPythonExtension.Core.type_stub_gen
[MLIR][Python] reland (narrower) type stub generation (#157930) This a reland of https://github.com/llvm/llvm-project/pull/155741 which was reverted at https://github.com/llvm/llvm-project/pull/157831. This version is narrower in scope - it only turns on automatic stub generation for `MLIRPythonExtension.Core._mlir` and **does not do anything automatically**. Specifically, the only CMake code added to `AddMLIRPython.cmake` is the `mlir_generate_type_stubs` function which is then used only in a manual way. The API for `mlir_generate_type_stubs` is: ``` Arguments: MODULE_NAME: The fully-qualified name of the extension module (used for importing in python). DEPENDS_TARGETS: List of targets these type stubs depend on being built; usually corresponding to the specific extension module (e.g., something like StandalonePythonModules.extension._standaloneDialectsNanobind.dso) and the core bindings extension module (e.g., something like StandalonePythonModules.extension._mlir.dso). OUTPUT_DIR: The root output directory to emit the type stubs into. OUTPUTS: List of expected outputs. DEPENDS_TARGET_SRC_DEPS: List of cpp sources for extension library (for generating a DEPFILE). IMPORT_PATHS: List of paths to add to PYTHONPATH for stubgen. PATTERN_FILE: (Optional) Pattern file (see https://nanobind.readthedocs.io/en/latest/typing.html#pattern-files). Outputs: NB_STUBGEN_CUSTOM_TARGET: The target corresponding to generation which other targets can depend on. ``` Downstream users should use `mlir_generate_type_stubs` in coordination with `declare_mlir_python_sources` to turn on stub generation for their own downstream dialect extensions and upstream dialect extensions if they so choose. Standalone example shows an example. Note, downstream will also need to set `-DMLIR_PYTHON_PACKAGE_PREFIX=...` correctly for their bindings.
2025-09-20 10:47:32 -08:00
ROOT_DIR "${CMAKE_CURRENT_BINARY_DIR}/type_stubs"
[MLIR][Python] don't generate type stubs when cross-compiling (#160793) Stubgen doesn't work when cross-compiling (stubgen will run in the host interpreter and then fail to find the extension module for the host arch). So disable it when `CMAKE_CROSSCOMPILING=ON`.
2025-09-25 15:59:12 -08:00
SOURCES "${_core_type_stub_sources}"
[MLIR][Python] reland (narrower) type stub generation (#157930) This a reland of https://github.com/llvm/llvm-project/pull/155741 which was reverted at https://github.com/llvm/llvm-project/pull/157831. This version is narrower in scope - it only turns on automatic stub generation for `MLIRPythonExtension.Core._mlir` and **does not do anything automatically**. Specifically, the only CMake code added to `AddMLIRPython.cmake` is the `mlir_generate_type_stubs` function which is then used only in a manual way. The API for `mlir_generate_type_stubs` is: ``` Arguments: MODULE_NAME: The fully-qualified name of the extension module (used for importing in python). DEPENDS_TARGETS: List of targets these type stubs depend on being built; usually corresponding to the specific extension module (e.g., something like StandalonePythonModules.extension._standaloneDialectsNanobind.dso) and the core bindings extension module (e.g., something like StandalonePythonModules.extension._mlir.dso). OUTPUT_DIR: The root output directory to emit the type stubs into. OUTPUTS: List of expected outputs. DEPENDS_TARGET_SRC_DEPS: List of cpp sources for extension library (for generating a DEPFILE). IMPORT_PATHS: List of paths to add to PYTHONPATH for stubgen. PATTERN_FILE: (Optional) Pattern file (see https://nanobind.readthedocs.io/en/latest/typing.html#pattern-files). Outputs: NB_STUBGEN_CUSTOM_TARGET: The target corresponding to generation which other targets can depend on. ``` Downstream users should use `mlir_generate_type_stubs` in coordination with `declare_mlir_python_sources` to turn on stub generation for their own downstream dialect extensions and upstream dialect extensions if they so choose. Standalone example shows an example. Note, downstream will also need to set `-DMLIR_PYTHON_PACKAGE_PREFIX=...` correctly for their bindings.
2025-09-20 10:47:32 -08:00
)
[MLIR][Python] make sure stubs get installed with LLVM_DISTRIBUTION_COMPONENTS (#168407) Fixes https://github.com/llvm/llvm-project/issues/168393. Also adds top-level `MLIR_PYTHON_STUBGEN_ENABLED` CMake option.
2025-11-19 06:07:28 -09:00
list(APPEND mlir_python_sources_deps MLIRPythonExtension.Core.type_stub_gen)
[MLIR][Python] don't generate type stubs when cross-compiling (#160793) Stubgen doesn't work when cross-compiling (stubgen will run in the host interpreter and then fail to find the extension module for the host arch). So disable it when `CMAKE_CROSSCOMPILING=ON`.
2025-09-25 15:59:12 -08:00
# _mlirPythonTestNanobind stubgen
if(MLIR_INCLUDE_TESTS)
get_target_property(_test_extension_srcs MLIRPythonTestSources.PythonTestExtensionNanobind INTERFACE_SOURCES)
mlir_generate_type_stubs(
# This is the FQN path because dialect modules import _mlir when loaded. See above.
MODULE_NAME mlir._mlir_libs._mlirPythonTestNanobind
DEPENDS_TARGETS
# You need both _mlir and _mlirPythonTestNanobind because dialect modules import _mlir when loaded
# (so _mlir needs to be built before calling stubgen).
MLIRPythonModules.extension._mlir.dso
MLIRPythonModules.extension._mlirPythonTestNanobind.dso
# You need this one so that ir.py "built" because mlir._mlir_libs.__init__.py import mlir.ir in _site_initialize.
MLIRPythonModules.sources.MLIRPythonSources.Core.Python
OUTPUT_DIR "${CMAKE_CURRENT_BINARY_DIR}/type_stubs/_mlir_libs"
OUTPUTS _mlirPythonTestNanobind.pyi
DEPENDS_TARGET_SRC_DEPS "${_test_extension_srcs}"
IMPORT_PATHS "${MLIRPythonModules_ROOT_PREFIX}/.."
)
set(_mlirPythonTestNanobind_typestub_gen_target "${NB_STUBGEN_CUSTOM_TARGET}")
declare_mlir_python_sources(
MLIRPythonTestSources.PythonTestExtensionNanobind.type_stub_gen
ROOT_DIR "${CMAKE_CURRENT_BINARY_DIR}/type_stubs"
ADD_TO_PARENT MLIRPythonTestSources.Dialects
SOURCES _mlir_libs/_mlirPythonTestNanobind.pyi
)
endif()
[MLIR][Python] reland (narrower) type stub generation (#157930) This a reland of https://github.com/llvm/llvm-project/pull/155741 which was reverted at https://github.com/llvm/llvm-project/pull/157831. This version is narrower in scope - it only turns on automatic stub generation for `MLIRPythonExtension.Core._mlir` and **does not do anything automatically**. Specifically, the only CMake code added to `AddMLIRPython.cmake` is the `mlir_generate_type_stubs` function which is then used only in a manual way. The API for `mlir_generate_type_stubs` is: ``` Arguments: MODULE_NAME: The fully-qualified name of the extension module (used for importing in python). DEPENDS_TARGETS: List of targets these type stubs depend on being built; usually corresponding to the specific extension module (e.g., something like StandalonePythonModules.extension._standaloneDialectsNanobind.dso) and the core bindings extension module (e.g., something like StandalonePythonModules.extension._mlir.dso). OUTPUT_DIR: The root output directory to emit the type stubs into. OUTPUTS: List of expected outputs. DEPENDS_TARGET_SRC_DEPS: List of cpp sources for extension library (for generating a DEPFILE). IMPORT_PATHS: List of paths to add to PYTHONPATH for stubgen. PATTERN_FILE: (Optional) Pattern file (see https://nanobind.readthedocs.io/en/latest/typing.html#pattern-files). Outputs: NB_STUBGEN_CUSTOM_TARGET: The target corresponding to generation which other targets can depend on. ``` Downstream users should use `mlir_generate_type_stubs` in coordination with `declare_mlir_python_sources` to turn on stub generation for their own downstream dialect extensions and upstream dialect extensions if they so choose. Standalone example shows an example. Note, downstream will also need to set `-DMLIR_PYTHON_PACKAGE_PREFIX=...` correctly for their bindings.
2025-09-20 10:47:32 -08:00
endif()
[MLIR][Python] make sure stubs get installed with LLVM_DISTRIBUTION_COMPONENTS (#168407) Fixes https://github.com/llvm/llvm-project/issues/168393. Also adds top-level `MLIR_PYTHON_STUBGEN_ENABLED` CMake option.
2025-11-19 06:07:28 -09:00
add_custom_target("mlir-python-sources" DEPENDS ${mlir_python_sources_deps})
if(NOT LLVM_ENABLE_IDE)
add_llvm_install_targets(install-mlir-python-sources
DEPENDS mlir-python-sources
COMPONENT mlir-python-sources
)
endif()
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
################################################################################
# The fully assembled package of modules.
# This must come last.
################################################################################
[MLIR][Python] don't generate type stubs when cross-compiling (#160793) Stubgen doesn't work when cross-compiling (stubgen will run in the host interpreter and then fail to find the extension module for the host arch). So disable it when `CMAKE_CROSSCOMPILING=ON`.
2025-09-25 15:59:12 -08:00
set(_declared_sources MLIRPythonSources MLIRPythonExtension.RegisterEverything)
[MLIR][Python] make sure stubs get installed with LLVM_DISTRIBUTION_COMPONENTS (#168407) Fixes https://github.com/llvm/llvm-project/issues/168393. Also adds top-level `MLIR_PYTHON_STUBGEN_ENABLED` CMake option.
2025-11-19 06:07:28 -09:00
if(MLIR_PYTHON_STUBGEN_ENABLED)
[MLIR][Python] don't generate type stubs when cross-compiling (#160793) Stubgen doesn't work when cross-compiling (stubgen will run in the host interpreter and then fail to find the extension module for the host arch). So disable it when `CMAKE_CROSSCOMPILING=ON`.
2025-09-25 15:59:12 -08:00
list(APPEND _declared_sources MLIRPythonExtension.Core.type_stub_gen)
endif()
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
add_mlir_python_modules(MLIRPythonModules
[MLIR][Python] reland (narrower) type stub generation (#157930) This a reland of https://github.com/llvm/llvm-project/pull/155741 which was reverted at https://github.com/llvm/llvm-project/pull/157831. This version is narrower in scope - it only turns on automatic stub generation for `MLIRPythonExtension.Core._mlir` and **does not do anything automatically**. Specifically, the only CMake code added to `AddMLIRPython.cmake` is the `mlir_generate_type_stubs` function which is then used only in a manual way. The API for `mlir_generate_type_stubs` is: ``` Arguments: MODULE_NAME: The fully-qualified name of the extension module (used for importing in python). DEPENDS_TARGETS: List of targets these type stubs depend on being built; usually corresponding to the specific extension module (e.g., something like StandalonePythonModules.extension._standaloneDialectsNanobind.dso) and the core bindings extension module (e.g., something like StandalonePythonModules.extension._mlir.dso). OUTPUT_DIR: The root output directory to emit the type stubs into. OUTPUTS: List of expected outputs. DEPENDS_TARGET_SRC_DEPS: List of cpp sources for extension library (for generating a DEPFILE). IMPORT_PATHS: List of paths to add to PYTHONPATH for stubgen. PATTERN_FILE: (Optional) Pattern file (see https://nanobind.readthedocs.io/en/latest/typing.html#pattern-files). Outputs: NB_STUBGEN_CUSTOM_TARGET: The target corresponding to generation which other targets can depend on. ``` Downstream users should use `mlir_generate_type_stubs` in coordination with `declare_mlir_python_sources` to turn on stub generation for their own downstream dialect extensions and upstream dialect extensions if they so choose. Standalone example shows an example. Note, downstream will also need to set `-DMLIR_PYTHON_PACKAGE_PREFIX=...` correctly for their bindings.
2025-09-20 10:47:32 -08:00
ROOT_PREFIX ${MLIRPythonModules_ROOT_PREFIX}
[mlir][python] add `MLIR_BINDINGS_PYTHON_INSTALL_PREFIX` to make bindings install dir configurable (#124878) This PR parameterizes the install directory of the MLIR Python bindings in the final distribution.
2025-06-23 11:49:01 -08:00
INSTALL_PREFIX "${MLIR_BINDINGS_PYTHON_INSTALL_PREFIX}"
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
DECLARED_SOURCES
[MLIR][Python] don't generate type stubs when cross-compiling (#160793) Stubgen doesn't work when cross-compiling (stubgen will run in the host interpreter and then fail to find the extension module for the host arch). So disable it when `CMAKE_CROSSCOMPILING=ON`.
2025-09-25 15:59:12 -08:00
${_declared_sources}
[mlir][python] Segment MLIR Python test dialect to avoid testonly dependency. With https://reviews.llvm.org/rG14c9207063bb00823a5126131e50c93f6e288bd3, the build is broken with -DMLIR_INCLUDE_TESTS=OFF. This patch fixes the build and we may want to do a better fix to the layering in a followup. Differential Revision: https://reviews.llvm.org/D112560
2021-10-26 09:14:50 -08:00
${_ADDL_TEST_SOURCES}
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-22 11:57:41 -08:00
COMMON_CAPI_LINK_LIBS
MLIRPythonCAPI
[mlir] Leverage CMake interface libraries for mlir python This is already partially the case, but we can rely more heavily on interface libraries and how they are imported/exported in other to simplify the implementation of the mlir python functions in Cmake. This change also makes a couple of other changes: 1) Add a new CMake function which handles "pure" sources. This was done inline previously 2) Moves the headers associated with CAPI libraries to the libraries themselves. These were previously managed in a separate source target. They can now be added directly to the CAPI libraries using DECLARED_HEADERS. 3) Cleanup some dependencies that showed up as an issue during the refactor This is a big CMake change that should produce no impact on the build of mlir and on the produced *build tree*. However, this change fixes an issue with the *install tree* of mlir which was previously unusable for projects like torch-mlir because both the "pure" and "extension" targets were pointing to either the build or source trees. Reviewed By: stellaraccident Differential Revision: https://reviews.llvm.org/D128230
2022-06-28 09:39:13 -08:00
)
[MLIR][Python] make sure stubs get installed with LLVM_DISTRIBUTION_COMPONENTS (#168407) Fixes https://github.com/llvm/llvm-project/issues/168393. Also adds top-level `MLIR_PYTHON_STUBGEN_ENABLED` CMake option.
2025-11-19 06:07:28 -09:00
if(MLIR_PYTHON_STUBGEN_ENABLED)
[MLIR][Python] don't generate type stubs when cross-compiling (#160793) Stubgen doesn't work when cross-compiling (stubgen will run in the host interpreter and then fail to find the extension module for the host arch). So disable it when `CMAKE_CROSSCOMPILING=ON`.
2025-09-25 15:59:12 -08:00
add_dependencies(MLIRPythonModules "${_mlir_typestub_gen_target}")
if(MLIR_INCLUDE_TESTS)
add_dependencies(MLIRPythonModules "${_mlirPythonTestNanobind_typestub_gen_target}")
endif()
[MLIR][Python] reland (narrower) type stub generation (#157930) This a reland of https://github.com/llvm/llvm-project/pull/155741 which was reverted at https://github.com/llvm/llvm-project/pull/157831. This version is narrower in scope - it only turns on automatic stub generation for `MLIRPythonExtension.Core._mlir` and **does not do anything automatically**. Specifically, the only CMake code added to `AddMLIRPython.cmake` is the `mlir_generate_type_stubs` function which is then used only in a manual way. The API for `mlir_generate_type_stubs` is: ``` Arguments: MODULE_NAME: The fully-qualified name of the extension module (used for importing in python). DEPENDS_TARGETS: List of targets these type stubs depend on being built; usually corresponding to the specific extension module (e.g., something like StandalonePythonModules.extension._standaloneDialectsNanobind.dso) and the core bindings extension module (e.g., something like StandalonePythonModules.extension._mlir.dso). OUTPUT_DIR: The root output directory to emit the type stubs into. OUTPUTS: List of expected outputs. DEPENDS_TARGET_SRC_DEPS: List of cpp sources for extension library (for generating a DEPFILE). IMPORT_PATHS: List of paths to add to PYTHONPATH for stubgen. PATTERN_FILE: (Optional) Pattern file (see https://nanobind.readthedocs.io/en/latest/typing.html#pattern-files). Outputs: NB_STUBGEN_CUSTOM_TARGET: The target corresponding to generation which other targets can depend on. ``` Downstream users should use `mlir_generate_type_stubs` in coordination with `declare_mlir_python_sources` to turn on stub generation for their own downstream dialect extensions and upstream dialect extensions if they so choose. Standalone example shows an example. Note, downstream will also need to set `-DMLIR_PYTHON_PACKAGE_PREFIX=...` correctly for their bindings.
2025-09-20 10:47:32 -08:00
endif()
Reference in a new issue Copy permalink