Summary:
This provides users with a way to check all symbols present in a given
program. This is useful for many cases where a power user wants to do
something dynamic with the objects present.
PR #205224 changed the path for kernels with no arguments to not use
appendLaunchKernelWithArgs, while at the same time it removed
zeKernelSetGroupSize which resulted in incorrect sizing of the kernel.
This PR removes the alternate path and makes sure all launches go
through appendLaunchKernelWithArgs.
Alternatively we can restore zeKernelSetGroupSize on the alternate path.
This patch does a small cleanup of level zero plugin. Mainly I changed
names of variables to use `CamelCase` whenever they relate to our
internal structures - not objects coming directly from level zero API
etc., so I left some variables and arguments like `zeDevice`, but only
if they are directly related to level zero APIs or structures.
Added a few `auto *` declarations, but again - only when they refer to
our internal objects and not to structures from level zero which might
be pointers underneath, but we use them as transparent objects.
Also did small cleanup of unused headers, making sure that we include
directly what we need rather than use transitive dependencies.
@adurang Please let me know if this patch is okay with you, if you think
this will cause more harm then it's worth I'll close it.
---------
Co-authored-by: Łukasz Plewa <lukasz.plewa@intel.com>
Add the initial Context handle API to liboffload:
- olCreateContext / olDestroyContext
- olGetContextInfo / olGetContextInfoSize
- OL_ERRC_INVALID_CONTEXT error code
- OL_CONTEXT_INFO_{NUM_DEVICES,DEVICES,PLATFORM} queries
This is the first patch in a series. I decided to split the original
change into multiple smaller PRs to make the review process faster and
more manageable.
This patch introduces context management functions, but the context
handle is not used anywhere yet. The following patches will extend the
API to use contexts in:
``` c
ol_result_t olMemAlloc(ol_context_handle_t, ol_device_handle_t, ol_alloc_type_t, size_t, void**);
ol_result_t olMemFree(ol_context_handle_t, void*);
ol_result_t olGetMemInfo(ol_context_handle_t, const void*, ol_mem_info_t, size_t, void*);
ol_result_t olGetMemInfoSize(ol_context_handle_t, const void*, ol_mem_info_t, size_t*);
ol_result_t olCreateQueue(ol_context_handle_t, ol_device_handle_t, ol_queue_handle_t*);
ol_result_t olCreateProgram(ol_context_handle_t, ol_device_handle_t, const void*, size_t, ol_program_handle_t*);
```
You can also take a look at #201398, which contains the first version of
the complete change. I'm cleaning it up as I go, so the patches in this
series are not a direct 1:1 split of that PR.
The message was the opposite of the assert condition.
In addition, when PatternSize == Size the caller wants to fill the
region with exactly one copy of the pattern, which is a valid operation.
The issue was found by AI.
Also check `Size % PatternSize == 0` per L0 spec.
Even though L0, CUDA and HSA do not directly support filling memory
regions with non-power-of-two-sized patterns, plugins explicitly allow
for such fills by falling back to a slow path whenever a direct API call
is not possible.
The Level Zero plugin is the odd one out, forwarding directly to
`zeCommandListAppendMemoryFill`. This fails for non-power-of-two
patterns, causing unit tests for `olMemFill` to fail when run on L0.
This PR adds fallbacks in L0 plugin's `memoryFill` implementation to
support arbitrary pattern sizes:
- if all pattern bytes are the same, substitute `1` for `patternSize`
and proceed with the native fill
- if memory is host-accessible (host or shared alloc), then simply
`std::copy_n` the pattern
- if memory is device-only, upload a seed to the device, and
`memoryCopy` O(log n) times, doubling the pattern each time, until the
entire region is filled. The seed is the input pattern, but extended on
the host to 1-2KiB to potentially eliminate several small `memoryCopy`
calls that would otherwise be almost pure overhead.
Conceptually, this approach has better worst-case overhead than the one
found in the CUDA plugin (decompose the pattern to then run multiple
strided memsets) and the AMDGPU one (call `pushMemoryCopyH2DAsync` with
the pattern, setting `NumTimes`). Aligning the fallback fill approaches
is outside the scope of this PR, however - the main intent is to fix the
unit test failures.
When a Level Zero driver is not present in the environment the current
behavior makes olInit fail if the level_zero plugin is requested. This
changes the behavior to match other plugins where it doesn't fail but
return 0 supported devices.
liboffload plugins would not always exhaustively map the underlying
API's error values to ol_errc_t, resulting in unintended behavior - e.g.
olCreateProgram, given an invalid binary, would return OL_ERRC_UNKNOWN
instead of OL_ERRC_INVALID_BINARY.
This patch largely eliminates the most glaring cases of this by
assigning an ol_errc_t value to every error code of the underlying API
in every plugin. The proper granularity for this mapping is per-API
call, but that is a large rework that belongs in a separate patch.
Assisted-by: Claude
Adds olMemPrefetch and the OL_MEM_MIGRATION_FLAG_* enum, wired through a
new dataPrefetchImpl on the plugin interface with Level Zero and CUDA
implementations. Backends without prefetch support inherit the default
no-op.
Prefetch is a hint that migrates a USM allocation between the host and a
device so subsequent accesses on the target side are faster.
Assisted-by: Claude
The PGO tests expect the produced filename to match the triple.
The tool pulls the triple name from the binary's triple name, and the
RUN lines expect it to match the triple the build was configured with.
Move towards using the triple for representing incompatible
ISA changes. Use the subarch field to represent the various
incompatible cases. Previously we pretended a single triple arch
was universally compatible, and only distinguished by function
level subtargets. Move towards using distinct triples to enable
more sophisticated toolchain handling in the future, like proper
runtime library linking.
Introduce a new subarch per unique ISA, but also introduce
"major subarches" which are compatible by a set of covered
minor ISA versions. These map to the existing generic targets.
There are a few placeholder subarch entries, which currently
have missing backing generic arches for codegen.
This should be the preferred triple arch name going forward,
but is treated as an alias of amdgcn. This does not yet change
clang to emit the new triples.
Part of #154925
This PR allows the user to use JIT with kernel record & replay by
expanding the latter to also work with IR bitcode images.
- Add a new field to `DeviceImageTy` for storing the IR image.
- Expand `RecordReplayTy::FileTy` to also include IR bitcode and
associate it with the file extension `.bc`
- Add the `--load-bitcode` command line option to
`llvm-omp-kernel-replay` tool.
---------
Co-authored-by: Giorgi Gvalia <gvalia1@llnl.gov>
MSVC/clang-cl lack __attribute__((weak)), so GlobalHandler.cpp provided
a strong stub for __llvm_write_custom_profile. When clang_rt.profile is
linked (e.g. instrumented/PGO builds) lld-link rejects the duplicate
strong symbol.
Replace the stub + CMake workaround with the /alternatename linker
pragma, which is the standard Windows weak-symbol equivalent used by
compiler-rt itself. The linker picks compiler-rt's strong definition
when present.
Fix the guard: use _WIN32 && !__MINGW32__ (not _MSC_VER) so the stub and
/alternatename pragma apply to plain clang-on-Windows too, while MinGW
(which supports __attribute__((weak)) natively) continues to use the
real weak declaration.
Also fix the stub signature from 8 to 10 parameters to match the header.
Assisted-by: Claude
Not only is it more declarative than using the `-I` directory, it
additionally adds the given directory to the
`tablegen_compile_commands.yml` which allows the LSP and IDE tooling to
also be aware of the extra include directories used to compile that
TableGen file
On Windows __llvm_write_custom_profile is defined as a strong stub (MSVC
lacks proper weak symbol support) by 09a51b2818, so its address is a
compile-time constant that is never null. The `if
(!__llvm_write_custom_profile)` check therefore triggers
-Wpointer-bool-conversion, which is fatal under -Werror.
Assisted-by: Claude
Add the producer-side format and instrumentation pieces that layer on
top
of the basic HIP offload PGO support.
This extends the raw/indexed profile format with wave size and uniform
counters, teaches llvm-profdata to read and merge them, adds the AMDGPU
sampling runtime, and emits block.uniformity.profile metadata during
PGO instrumentation.
Summary:
In the case of, say, a GPU sanitizer, there could be a pending report
that isn't flushed before the queue dies and the program terminates. Add
an explicit flush to ensure that all work at least posted *before* the
trap fired is cleared in the HSA error callback before actually
quitting.
Summary:
Currently we have two conflicting methods of passing kernel arguments, a
flat pointer + size and an array of pointers. We recently decided to
move the offload API to the latter because it is more generic and lets
you construct the other formats.
This PR primarily just changes the format and the one existing core use.
The uses should be simplier now. Future changes will change the OpenMP
argument parsing.
This is a follow-up to #194333, which changed liboffload's kernel launch
interface to take an array of pointers to arguments instead of a
contiguous argument buffer, but left the old path intact for OpenMP, to
be cleaned up iteratively in later changes.
This patch modifies the offload host kernels, and the OpenMP clang
codegen, to also use argument pointer arrays. The host was the only
plugin that did not support the new submission path. With it converted,
OpenMP can migrate to use the new path in all scenarios, after which
everything related to contiguous argument buffer and its construction
can be removed from plugin interface and the backends.
The change itself is fairly straightforward. The bulk of the diff is
automated test updates. In the runtime, the host kernel signature
becomes an array of pointers (void **), and the codegen is updated to
match. With each argument being a pointer, the kernel function now adds
one load to read the argument's value.
Assisted-by: Claude
This patch is the first step towards introducing alignment support in
memory allocations using liboffload, in order to enable SYCL
implementation of aligned allocations.
At the level of device allocators, it does not modify the Level Zero
code except for forwarding the alignment parameter, since Level Zero
already allows for specifying the alignment in its device allocator
implementation. For AMD and CUDA, it checks whether the alignment passed
by the caller is supported by the given backend; the reasoning behind
this verification is described in the following paragraphs. At the API
level, it adds a new function olMemAllocAligned, which is expected to
work similarly to olMemAlloc, with the difference that the buffers
returned by olMemAllocAligned should be aligned to the alignment passed
by the user. At the level of the plugin interface internal abstractions,
it adds a new argument Alignment to existing functions and delegates
memory allocation between olMemAllocAligned and olMemAlloc
implementations by using a common helper function.
The goal of the anticipated series of patches is to implement handling
of the alignment in the memory manager at the plugin interface level. At
the first stage, presented in this patch, the information about the
passed alignment is used mainly for checking whether the buffer returned
by the device allocators meets the requirements. In the case of the
requested memory size exceeding the thresholds of allocations handled by
the memory manager, the request is forwarded directly to the device.
Otherwise, the memory manager is responsible for allocating memory in
full pages and pooling it according to the requested chunks. In the
first scenario, the requested size, which is greater than the
aforementioned threshold, is usually a multiple of the page size.
Therefore, any alignment smaller than the page size would be correct.
Neither CUDA nor HSA provides users with the ability to specify the
alignment of the allocated memory. Their APIs include the alignment as
one of the possible arguments only in functions that reserve virtual
address space. CUDA enables users only to check the allocation
granularity, which is usually synonymous with the page size. In the case
of HSA, if the memory is allocated using a pool, the user is able to
check not only the granularity, but also the alignment of the buffers
allocated using the given pool. However, these values - granularity and
alignment - are defined only if
HSA_AMD_MEMORY_POOL_INFO_RUNTIME_ALLOC_ALLOWED is set to true, which
should always be set since the current implementation of the memory pool
for the AMD plugin in liboffload uses only hsa_amd_memory_pool_ts for
allocations. Only Level Zero accepts the alignment parameter during the
memory allocation process, but manages returned pointers internally.
Since pooling is also implemented in the memory manager at the plugin
level, such a design in the Level Zero device allocator duplicates
pooling in the whole application.
In the final version, the memory manager at the plugin interface level
will handle memory pooling and the buffer alignment from different
devices instead of delegating memory management to device allocator
implementations, as this would simplify the Level Zero plugin design and
provide AMD and CUDA plugins with support for the alignment in memory
allocations, which is not natively included in their APIs.
Summary:
There's a bit of a chicken and egg problem for the RPC server if we want
to do something creative with the device's image for things like DWARF
dumping. The problem was that destructors can make RPC calls, but the
RPC server also needed the images to be valid. Simple fix is to just
split the destructor calling out and do it first so we can deinitialize
RPC with valid device images.
This is a part of a series of patches that rework OpenMP cross-team
reductions.
This patch changes the cross-team reduction runtime to no longer work
through larger number of teams in chunks. Instead, we allocate a
suitable-sized global buffer for the team values and let all teams run
at once. The last team that finishes uses a strided loop to reduce the
team values from the global buffer.
We also use `mapping::getNumberOfThreadsInBlock()` instead of
`omp_get_num_threads()` because the reduction of the team values runs
outside of the parallel region device code, which would make
`omp_get_num_threads()` always return 1. For Generic-SPMD mode, we also
want to use all available threads, which means that we need to copy the
reduction data from LDS (where it lives in that mode by default) to
scratch in codegen before calling the cross-team reduction.
Performance:
All performance tests can be reproduced with
https://github.com/ro-i/xteam-test @ commit
2f09ae1f566f13189d20b414d8b83bf2986564d2. To reproduce, simply create a
`local.mk` file in the cloned directory with a suitable `OFFLOAD_ARCH`
for your machine and `CXX_trunk` + `CXX_trunk_dev` set to the paths of
the clang++ binaries for llvm/main and this patch. (llvm/main should
best be at the commit that is currently the base for this PR. At the
moment, this is fe87c971bf). Then, run
`make trunk trunk_dev` to build the benchmark binaries for 208 and 10400
teams. Run them with `./run_bench.sh -rq -n10 red_trunk_208
red_trunk_dev_208 red_trunk_10400 red_trunk_dev_10400` to get the avg
performance numbers over 10 rounds.
This tests multiple reduction workloads, including reductions that run
in the Generic-SPMD mode, with 208 teams and with 10400 teams, both à
512 threads, and with a reduction array size of 177,777,777. I tested on
a gfx942 and didn't find any regressions - here are some numbers showing
the performance of this patch relative to the baseline:
```
red_comb_sep_arr_32 double dev_208: +84.37% dev_10400: +54.06%
red_sum_arr_32 double dev_208: +9.72% dev_10400: +51.29%
red_comb double dev_208: +2.73% dev_10400: +8.73%
red_comb_sep double dev_208: +1.89% dev_10400: +87.13%
red_dot double dev_208: +1.46% dev_10400: +25.66%
red_indirect double dev_208: +1.51% dev_10400: +23.54%
red_kernel_part double dev_208: +1.27% dev_10400: +88.28%
red_max double dev_208: +1.29% dev_10400: +22.69%
red_mult double dev_208: +1.96% dev_10400: +22.02%
red_sum double dev_208: +2.03% dev_10400: +22.00%
red_pi double dev_208: +1.08% dev_10400: +1.22%
red_comb_sep_arr_32 uint dev_208: +139.10% dev_10400: +88.02%
red_sum_arr_32 uint dev_208: +49.70% dev_10400: +143.24%
red_dot uint dev_208: +0.77% dev_10400: +24.91%
red_max uint dev_208: +3.43% dev_10400: +36.61%
red_sum uint dev_208: +4.56% dev_10400: +36.95%
red_comb_sep_arr_32 ulong dev_208: +85.22% dev_10400: +58.17%
red_sum_arr_32 ulong dev_208: +9.01% dev_10400: +55.14%
red_dot ulong dev_208: +2.94% dev_10400: +26.89%
red_max ulong dev_208: +2.83% dev_10400: +23.10%
red_sum ulong dev_208: +1.59% dev_10400: +22.82%
red_comb_sep_arr_32 Value dev_208: +29.88% dev_10400: +24.17%
red_sum_arr_32 Value dev_208: +7.26% dev_10400: +38.14%
red_dot Value dev_208: +2.65% dev_10400: +8.60%
red_max Value dev_208: +2.86% dev_10400: +4.77%
red_sum Value dev_208: +0.47% dev_10400: +10.22%
```
This patch uses ideas from Johannes Doerfert, ideas from the AOMP
cross-team reduction implementation, and was assisted by Claude.
The recording phase now sets the teams and threads limits provided by
the user (in the corresponding OpenMP clauses) or zero if not specified.
Additionally, the PR #199483 already enforces that replay's configuration
of threads and teams are respected.
This commit also changes the way we test record and replay when multiple
kernels are recorded in the same test. We use the record report to know how
to associate a json record descriptor file to the target region in the code. We
do not rely anymore on the modification time of the files to know the order,
which was problematic.
Progams that do not use any memory (e.g., no mappings) were failing
because we were trying to execute zero size transfers. This commit adds
handling for this case.
This commit extends the record reporting mechanism:
- `LIBOMPTARGET_RECORD_REPORT_FILENAME` enables the reporting mechanism
and allows specifying the name of the output file.
- The report of recorded kernels are ordered in recording order. This is
really useful for tests that need to record and replay more than one
kernel.
The RPC client can push asynchronous work and then quit. These can be
lost if the kernel returns and is torn down before the RPC server gets
to run to check it. To handle this, we just manually flush the interface
once when we tear the device down. The interface is lock-free so this
should be safe even if the other thread is working through it as well.
Summary:
For whatever reason, HSA copied the questionable choices that OpenCL
made and represents its launch parameters in a threads * blocks grid.
The problem is that you then combine this with an `int32_t` interface,
so you have 31 bits to represent your launch. We were then
pessimistically stating that your launch always had 1024 threads, which
left us with 2^21. This is only about two million which people do all
the time, and this caused us to perform weird clamping in OpenMP. The
effect was that tests like ompx_saxpy_mixed.c was hitting that clamp and
returning wrong results.
Also fix the sanitizer tests failing because of HSA core dumps.
In some cases setting ZE_COMMAND_QUEUE_FLAG_COPY_OFFLOAD_HINT reduces
performance. Here we introduce
LIBOFFLOAD_LEVEL_ZERO_USE_COPY_OFFLOAD_HINT env var to allow users to
control the hint (which continues to be on by default).
Introduce a new L0EventTy type used to implement most of the Event APIs
of PluginInterface (all but getEventElapsedTimeImpl).
---------
Co-authored-by: Kevin Sala Penades <kevinsala.ks@gmail.com>
Return not just the address but also the size of the symbols in getGlobalMetadataFromDevice.
Fixes olGetSymbolInfoSizeGlobalTest.SuccessSize unitt test failure with L0 plugin.
The dataFence device operation logic was still on L0Device. With this all the operations working on command lists have their logic consolidated in L0Queue.
