Summary:
Currently we have two conflicting methods of passing kernel arguments, a
flat pointer + size and an array of pointers. We recently decided to
move the offload API to the latter because it is more generic and lets
you construct the other formats.
This PR primarily just changes the format and the one existing core use.
The uses should be simplier now. Future changes will change the OpenMP
argument parsing.
This is a part of a series of patches that rework OpenMP cross-team
reductions.
This patch changes the cross-team reduction runtime to no longer work
through larger number of teams in chunks. Instead, we allocate a
suitable-sized global buffer for the team values and let all teams run
at once. The last team that finishes uses a strided loop to reduce the
team values from the global buffer.
We also use `mapping::getNumberOfThreadsInBlock()` instead of
`omp_get_num_threads()` because the reduction of the team values runs
outside of the parallel region device code, which would make
`omp_get_num_threads()` always return 1. For Generic-SPMD mode, we also
want to use all available threads, which means that we need to copy the
reduction data from LDS (where it lives in that mode by default) to
scratch in codegen before calling the cross-team reduction.
Performance:
All performance tests can be reproduced with
https://github.com/ro-i/xteam-test @ commit
2f09ae1f566f13189d20b414d8b83bf2986564d2. To reproduce, simply create a
`local.mk` file in the cloned directory with a suitable `OFFLOAD_ARCH`
for your machine and `CXX_trunk` + `CXX_trunk_dev` set to the paths of
the clang++ binaries for llvm/main and this patch. (llvm/main should
best be at the commit that is currently the base for this PR. At the
moment, this is fe87c971bf). Then, run
`make trunk trunk_dev` to build the benchmark binaries for 208 and 10400
teams. Run them with `./run_bench.sh -rq -n10 red_trunk_208
red_trunk_dev_208 red_trunk_10400 red_trunk_dev_10400` to get the avg
performance numbers over 10 rounds.
This tests multiple reduction workloads, including reductions that run
in the Generic-SPMD mode, with 208 teams and with 10400 teams, both à
512 threads, and with a reduction array size of 177,777,777. I tested on
a gfx942 and didn't find any regressions - here are some numbers showing
the performance of this patch relative to the baseline:
```
red_comb_sep_arr_32 double dev_208: +84.37% dev_10400: +54.06%
red_sum_arr_32 double dev_208: +9.72% dev_10400: +51.29%
red_comb double dev_208: +2.73% dev_10400: +8.73%
red_comb_sep double dev_208: +1.89% dev_10400: +87.13%
red_dot double dev_208: +1.46% dev_10400: +25.66%
red_indirect double dev_208: +1.51% dev_10400: +23.54%
red_kernel_part double dev_208: +1.27% dev_10400: +88.28%
red_max double dev_208: +1.29% dev_10400: +22.69%
red_mult double dev_208: +1.96% dev_10400: +22.02%
red_sum double dev_208: +2.03% dev_10400: +22.00%
red_pi double dev_208: +1.08% dev_10400: +1.22%
red_comb_sep_arr_32 uint dev_208: +139.10% dev_10400: +88.02%
red_sum_arr_32 uint dev_208: +49.70% dev_10400: +143.24%
red_dot uint dev_208: +0.77% dev_10400: +24.91%
red_max uint dev_208: +3.43% dev_10400: +36.61%
red_sum uint dev_208: +4.56% dev_10400: +36.95%
red_comb_sep_arr_32 ulong dev_208: +85.22% dev_10400: +58.17%
red_sum_arr_32 ulong dev_208: +9.01% dev_10400: +55.14%
red_dot ulong dev_208: +2.94% dev_10400: +26.89%
red_max ulong dev_208: +2.83% dev_10400: +23.10%
red_sum ulong dev_208: +1.59% dev_10400: +22.82%
red_comb_sep_arr_32 Value dev_208: +29.88% dev_10400: +24.17%
red_sum_arr_32 Value dev_208: +7.26% dev_10400: +38.14%
red_dot Value dev_208: +2.65% dev_10400: +8.60%
red_max Value dev_208: +2.86% dev_10400: +4.77%
red_sum Value dev_208: +0.47% dev_10400: +10.22%
```
This patch uses ideas from Johannes Doerfert, ideas from the AOMP
cross-team reduction implementation, and was assisted by Claude.
Until now, strict behavior in the number of threads and blocks has been
applied only when the kernel is in bare mode. When this mode is enabled,
the values passed in UserNumBlocks and UserThreadLimit are not adjusted
and are the definitive values used to launch the kernel. This commit
detaches the strictness from the kernel mode.
This is going to be used by the kernel replay tool. Additionally, it
starts clearing the path for the upcoming OpenMP dims modifier, used to
configure multidimensional teams and leagues, which will include
strictness choices for teams and threads.
All the bare kernels must indicate strict behavior. Asserts are added to
check this condition.
This PR changes olLaunchKernel to accept an array of pointers to
arguments:
```
void *ArgPtrs[] = {&A, &B, &C};
size_t ArgSizes[] = {sizeof(A), sizeof(B), sizeof(C)};
olLaunchKernel(Queue, Device, Kernel, &LaunchArgs, std::size(ArgPtrs), ArgPtrs, ArgSizes);
```
The newly proposed interface is implementable by existing and
anticipated
backends, is familiar to CUDA programmers, eliminates the extraneous
construction of a contiguous arguments buffer, replacing it with
constructing
an array of pointers, sidesteps the alignment requirements, does not
require reading program image metadata where it's impractical, and
enables
a compliant SYCL implementation to be built on top of it.
The ArgSizes array is required to support OpenCL, which does not have
native support for launching a kernel with an argument pointer array, or
a reliable way of retrieving argument sizes for a kernel.
## Mapping the proposed API to backends
CUDA and Level-Zero both support accepting an array of pointers to
kernel arguments, through `cuLaunchKernel`
(https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__EXEC.html#group__CUDA__EXEC_1gb8f3dc3031b40da29d5f9a7139e52e15)
and `zeCommandListAppendLaunchKernelWithArguments`
(https://oneapi-src.github.io/level-zero-spec/level-zero/latest/core/api.html#zecommandlistappendlaunchkernelwitharguments)
respectively.
For OpenCL, which requires the kernel arguments to be set separately
from the kernel launch, a potential implementation can extract the
number of arguments from the OpenCL API, and then iterate over the
argument pointer and size arrays:
```
cl_uint num_args = 0;
cl_int err = clGetKernelInfo(kernel, CL_KERNEL_NUM_ARGS,
sizeof(num_args), &num_args, NULL);
for (cl_uint i = 0; i < num_args; ++i) {
clSetKernelArg(kernel, i, ArgSizes[i], ArgPtrs[i]);
}
```
The AMDGPU plugin needs to construct a contiguous buffer using the array
of argument pointers. To do this conversion, we need to have offsets
at which to place the arguments. Here, luckily, as mentioned before,
the AMD plugin already reads kernel metadata from the program image.
The implementation simply retrieves size and offset for each argument
from the kernel image, and then uses it to populate a buffer.
OpenMP v6.0 added new enum values to `ompt_callback_target_data_op_t`
for communicating `memset` events. Add a corresponding callback
invokation to `omp_target_memset` calls.
Expand the tests to ensure correct behavior.
Signed-off-by: Jan André Reuter <j.reuter@fz-juelich.de>
Introduce a properties argument to olLaunchKernel to enable future
extensions.
This change adds initial extension for cooperative kernel launch.
The change restores the functionality from:
https://github.com/llvm/llvm-project/pull/184343
which was later reverted in:
https://github.com/llvm/llvm-project/pull/197463#184343 modified the API, and between the last rebase and the final
merge, additional tests were added
upstream(https://github.com/llvm/llvm-project/pull/193927) that still
relied on the previous API version.
Diff between this PR and #184343
```
diff --git a/offload/unittests/OffloadAPI/kernel/olLaunchKernel.cpp b/offload/unittests/OffloadAPI/kernel/olLaunchKernel.cpp
index 0251bffefe60..cd527f18db38 100644
--- a/offload/unittests/OffloadAPI/kernel/olLaunchKernel.cpp
+++ b/offload/unittests/OffloadAPI/kernel/olLaunchKernel.cpp
@@ -241,8 +241,8 @@ TEST_P(olLaunchKernelSingleCounterSyncEventTest, SuccessSyncEvent) {
ArgsSingleCounter Args{LoopRange, NumberToAdd, (uint32_t *)InitValuePassed,
(uint32_t *)ResNum};
- ASSERT_SUCCESS(
- olLaunchKernel(Queue, Device, Kernel, &Args, sizeof(Args), &LaunchArgs));
+ ASSERT_SUCCESS(olLaunchKernel(Queue, Device, Kernel, &Args, sizeof(Args),
+ &LaunchArgs, nullptr));
uint32_t FinalResVal = 0;
ASSERT_SUCCESS(olMemcpy(Queue, &FinalResVal, Host, ResNum, Device, Size));
@@ -294,8 +294,8 @@ TEST_P(olLaunchKernelSingleCounterSyncEventTest, SuccessTwoQueues) {
ArgsSingleCounter Args{LoopRange, NumberToAdd, (uint32_t *)InitValuePassed,
(uint32_t *)ResNum1};
- ASSERT_SUCCESS(
- olLaunchKernel(Queue, Device, Kernel, &Args, sizeof(Args), &LaunchArgs));
+ ASSERT_SUCCESS(olLaunchKernel(Queue, Device, Kernel, &Args, sizeof(Args),
+ &LaunchArgs, nullptr));
ol_event_handle_t Event = nullptr;
ASSERT_SUCCESS(olCreateEvent(Queue, &Event));
@@ -308,7 +308,7 @@ TEST_P(olLaunchKernelSingleCounterSyncEventTest, SuccessTwoQueues) {
// been completed by the time the kernel is executed using the second queue,
// the FinalResVal would be incorrect.
ASSERT_SUCCESS(olLaunchKernel(Queue2, Device, Kernel, &Args2, sizeof(Args2),
- &LaunchArgs));
+ &LaunchArgs, nullptr));
ASSERT_SUCCESS(olSyncQueue(Queue2));
uint32_t FinalResVal = 0;
```
Summary:
This is used in the GPU portion of the build, but really shouldn't be.
Most of these helpers were only used by the GPU portion, which has much
nicer builtins available in new clang. Using these in the main offload
build is broken on Windows and not available on older compilers, so we
split. Also means one less header we share.
The alignment helpers are trivially replaced by an LLVM utility, the
only thing that remains are the pointer arithmetic functions. There's no
conveient place to put these so I just let them stay.
The counterpart fix for clang (as too done here:
[flang-fix](https://github.com/llvm/llvm-project/pull/193669))
The incorrectly interpreted device values in the `target` directive
throws:
```
error: argument to 'device' clause must be a non-negative integer value
#pragma omp target device(-1)
^~
error: argument to 'device' clause must be a non-negative integer value
#pragma omp target device(omp_invalid_device)
^~~~~~~~~~~~~~~~~~
```
This commit extends the kernel replay tool to perform multiple replay
repetitions on the same process. It also prints the execution time of
the kernel replay, which includes the kernel launch and kernel
synchronization (replay I/O time is excluded). Precise kernel timing
should be obtained through the corresponding profiling tools for now.
The output report after recording has been improved as well.
This commit adds a mechanism to return information about a kernel replay
to outer replay tool. This mechanism allows verifying the replay memory
output when using different launch configurations (e.g., different number
of teams or threads) than the one used for recording.
It also adds a new KernelExtraArgsTy structure that is only generated by
the offload/libomptarget runtime components (unlike KernelArgsTy). These
runtime arguments can be used by future extensions, including extensions
not related to kernel record replay.
This commit fixes the kernel record replay on both AMD and CUDA devices. It
also re-organizes the record replay code, moves the whole code to separate
files, and makes it extensible to support other record formats (potentially in
the future). The environment variables for controlling the recording have also
been modified.
This patch adds `olGetEventElapsedTime` to the new LLVM Offload API, as
requested in
[#185728](https://github.com/llvm/llvm-project/issues/185728), and adds
the corresponding support in `plugins-nextgen`.
A main motivation for this change is to make it possible to measure the
elapsed time of work submitted to a queue, especially kernel launches.
This is relevant to the intended use of the new Offload API for
microbenchmarking GPU libc math functions.
### Summary
The new API returns the elapsed time, in milliseconds, between two
events on the same device.
To support the common pattern `create start event → enqueue kernel →
create end event → sync end event → get elapsed time`, `olCreateEvent`
now always creates and records a backend event through the device
interface. For backends that materialize real event state, this gives
the event concrete backend state that can be used for elapsed-time
measurement. For backends that do not materialize backend event state,
`EventInfo` may still remain null and existing event operations continue
to treat such events as trivially complete.
Previously, an event created on an empty queue could be represented only
as a logical event. That representation was sufficient for sync and
completion queries, but it was not suitable for elapsed-time measurement
because there was no backend event state to timestamp. The new behavior
preserves the meaning of completion of prior work while also allowing
backends with timing support to attach real event state.
### Changes in `plugins-nextgen`
#### Common interface
Add elapsed-time support to the common device and plugin interfaces:
* `GenericPluginTy::get_event_elapsed_time`
* `GenericDeviceTy::getEventElapsedTime`
* `GenericDeviceTy::getEventElapsedTimeImpl`
#### AMDGPU
* Add the required ROCr declarations and wrappers.
* Enable queue profiling at queue creation time.
* Record events by enqueuing a real barrier marker packet on the stream.
* Retain the timing signal needed to query the recorded marker later.
* Implement `getEventElapsedTimeImpl` using
`hsa_amd_profiling_get_dispatch_time`, converting the result to
milliseconds with `HSA_SYSTEM_INFO_TIMESTAMP_FREQUENCY`.
This follows the ROCm/HIP approach of enabling queue profiling at HSA
queue creation time, while keeping the AMDGPU queue path simpler than
the lazy-enable alternative discussed during review.
#### CUDA
* Add the required CUDA driver declarations and wrappers.
* Implement `getEventElapsedTimeImpl` with `cuEventElapsedTime`.
#### Host
* Add `getEventElapsedTimeImpl` that stores `0.0f` in the output
pointer, when present, and returns success.
Reason: the host plugin does not materialize backend event state and
already treats event operations as trivially successful. Returning
`0.0f` preserves that model without introducing a new failure mode.
#### Level Zero
* Add `getEventElapsedTimeImpl`, but leave it unimplemented.
Reason: the Level Zero plugin currently does not provide standalone
backend event support for this event model. For example, `waitEventImpl`
/ `syncEventImpl` are still unimplemented there.
---------
Signed-off-by: Leandro Augusto Lacerda Campos <leandrolcampos@yahoo.com.br>
Signed-off-by: Leandro A. Lacerda Campos <leandrolcampos@yahoo.com.br>
Some tests that were checking for prints inside/outside `target` regions
needed to be updated to work on systems where the ordering wasn't
deterministic.
Reverts llvm/llvm-project#184240
Original description from #165494:
-----
OpenMP allows cases like the following:
```c
int *p1, *p2, x;
p1 = p2 = &x;
...
#pragma omp target_exit_data map(delete: p1[:]) from(p2[0])
```
Which means, when the runtime encounters the `from` entry, the ref-count
may
not be zero, but it will go down to zero at the end of the current
construct,
which should cause the "from" transfer to happen.
Similarly, a user may have:
```c
struct S {
int *p;
};
#pragma omp declare_mapper (id1: S s) map(s.p) map(present, alloc: s.p[0:10])
#pragma omp declare_mapper (id2: S s) map(s.p, s.p[0:10])
S s1;
// present-check should fail here
#pragma omp target_enter_data map(alloc: s.p[0:10]) map(mapper(id1), to: s)
// "to" should be honored here
#pragma omp target_enter_data map(alloc: s.p[0:10]) map(mapper(id2), to: s)
```
Where the allocation happens before the "to" entry is encountered by the
runtime. Or, an allocation happens before a "present" entry is
encountered.
To handle cases like this, we need to use the state information of
previously
seen new allocations, deletions, "from" entries, when honoring
`to`/`from`/`present` map entries.
-----
OpenMP allows cases like the following:
```c
int *p1, *p2, x;
p1 = p2 = &x;
...
#pragma omp target_exit_data map(delete: p1[:]) from(p2[0])
```
Which means, when the runtime encounters the `from` entry, the ref-count
may not be zero, but it will go down to zero at the end of the current
construct, which should cause the "from" transfer to happen.
Similarly, a user may have:
```c
struct S {
int *p;
};
#pragma omp declare_mapper (id1: S s) map(s.p) map(present, alloc: s.p[0:10])
#pragma omp declare_mapper (id2: S s) map(s.p, s.p[0:10])
S s1;
// present-check should fail here
#pragma omp target_enter_data map(alloc: s.p[0:10]) map(mapper(id1), to: s)
// "to" should be honored here
#pragma omp target_enter_data map(alloc: s.p[0:10]) map(mapper(id2), to: s)
```
Where the allocation happens before the "to" entry is encountered by the
runtime. Or, an allocation happens before a "present" entry is
encountered.
To handle cases like this, we need to use the state information of
previously seen new allocations, deletions, "from" entries, when
honoring `to`/`from`/`present` map entries.
Allow a to define a set of Types that are not shown by default when
doing default debug loggin (e.g., LIBOMPTARGET_DEBUG=All).
Users can enable output of those types of messages by explicitly adding
them to LIBOMPTARGET_DEBUG.
Used to implement: #180545
---------
Co-authored-by: Michael Klemm <michael.klemm@amd.com>
Summary:
We provide an RPC server to manage calls initiated by the device to run
on the host. This is very useful for the built-in handling we have,
however there are cases where we would want to extend this
functionality.
Cases like Fortran or MPI would be useful, but we cannot put references
to these in the core offloading runtime. This way, we can provide this
as a library interface that registers custom handlers for whatever code
people want.
Eventually we might want to rework the INFO macro to work like the new
ODBG macro but in the meantime at least translate the Info type to the
correct Debug type instead of just using DP directly (which uses the
default type).
To reduce interference between threads, instead of writing the
components of a debug message directly to the underlying stream, write
them to a buffer and flush the buffer to the stream when its completed.
Depends on #174659.
This PR adds a new map-type bit to control the fallback behavior when
when a pointer lookup fails.
For now, this is only meaningful with `RETURN_PARAM`, and can be used
for `need_device_ptr` (for which the default is to use `nullptr` as the
result
when lookup fails), and OpenMP 6.1's `use_device_ptr(fb_nullify)`.
Eventually, this can be extended to work with assumed-size maps on
`target`
constructs, to control what the argument should be set to when lookup
fails (the OpenMP spec does not have a way to control that yet).
Dependent PR: #170578.
These commits fix issues regarding storage of tool data within
libomptarget. Both libomp and libomptarget have been modified to
accommodate this. We differentiate between two cases depending on the
type of the target region:
- merged target regions (default, without `nowait` clause): behavior
remains unchanged, tool data is stored in the thread local
RegionInterface class within libomptarget.
- deferred target regions (using `nowait` clause): tool data is moved to
`ompt_task_info_t` struct within libomp, as `RegionInterface` is thread
local and its data is lost whenever another task is scheduled on the
thread, which happens with deferred target regions.
In the new implementation, `RegionInterface` receives pointers to
`ompt_task_info_t` within libomp which are handled transparently within
libomptarget. Thus, the problem of tool data getting lost when a thread
receives a new task is resolved: `target_data` and `target_task_data`
remain set.
Another issue was the value of `task_data` which is supposed to belong
to the generating task of the region according to the OpenMP standard,
but instead had been set to the `task_data` of the target task itself
until now.
Test cases have been added which check both of these fixes.
---------
Co-authored-by: Joachim <jenke@itc.rwth-aachen.de>
Add a new nextgen plugin that supports GPU devices through the Intel oneAPI Level Zero library. The plugin is not enabled by default and needs to be added to LIBOMPTARGET_PLUGINS_TO_BUILD explicitely.
---------
Co-authored-by: Alexey Sachkov <alexey.sachkov@intel.com>
Co-authored-by: Nick Sarnie <nick.sarnie@intel.com>
Co-authored-by: Joseph Huber <huberjn@outlook.com>
Update debug messages based on the new method from #170425. Added a new
debug type `Tool` and updated the following files.
- include/OffloadPolicy.h
- include/OpenMP/OMPT/Connector.h
- include/Shared/Debug.h
- include/Shared/EnvironmentVar.h
- libomptarget/OpenMP/Mapping.cpp
- libomptarget/OpenMP/OMPT/Callback.cpp
- libomptarget/PluginManager.cpp
This PR adds a new set of debug macros that allow a certain code to be
only executed when certain debug conditions are met. This is useful to
guard things that are not strictly messages but compute and store things
that are related to those messages.
Strictly speaking the existing ODBG_OS could be used as well but that
requires a stream object to be created which is unnecessary in some
cases.
Example of how it works:
```cpp
ODBG_IF("Counters", [&](uint32_t Level) {
someCounter++;
if (Level == 2) moreDetailedCounter += f();
});
ODBG("Counters") << "Counter" = someCounter
<< ODBG_IF(2) << "DetailedCounter" << moreDetailedCounter;
```
This is needed as a way to support older code that was expecting
unconditional attachment to happen for cases like:
```c
int *p;
int x;
#pragma omp targret enter data map(p) // (A)
#pragma omp target enter data map(x) // (B)
p = &x;
// By default, this does NOT attach p and x
#pragma omp target enter data map(p[0:0]) // (C)
```
When the environment variable is set, such maps, where both the pointer
and the pointee already have corresponding copies on the device, but are
not attached to one another, will be attached as-if OpenMP 6.1 TR14's
`attach(always)` map-type-modifier was specified on `(C)`.
This PR adds a new set of debug macros that allow a certain code to be
only executed when certain debug conditions are met. This is useful to
guard things that are not strictly messages but compute and store things
that are related to those messages.
Strictly speaking the existing ODBG_OS could be used as well but that
requires a stream object to be created which is unnecessary in some
cases.
Example of how it works:
```
ODBG_IF("Counters", [&](uint32_t Level) {
someCounter++;
if (Level == 2) moreDetailedCounter += f();
});
ODBG("Counters") << "Counter" = someCounter
<< ODBG_IF(2) << "DetailedCounter" << moreDetailedCounter;
```
* Add compatibility support for DP and REPORT macros
* Define a set of predefined Debug Type for libomptarget
* Start to update libomptarget files (OffloadRTL.cpp, device.cpp)
This is a branch off of
https://github.com/llvm/llvm-project/pull/159856, in which consists of
the runtime portion of the changes required to support indirect function
and virtual function calls on an `omp target device` when the virtual
class / indirect function is mapped to the device from the host.
Key Changes
- Introduced a new flag OMP_DECLARE_TARGET_INDIRECT_VTABLE to mark
VTable registrations
- Modified setupIndirectCallTable to support both VTable entries and
indirect function pointers
Details:
The setupIndirectCallTable implementation was modified to support this
registration type by retrieving the first address of the VTable and
inferring the remaining data needed to build the indirect call table.
Since the Vtables / Classes registered as indirect can be larger than 8
bytes, and the vtables may not be at the first address we either need to
pass the size to __llvm_omp_indirect_call_lookup and have a check at
each step of the binary search, or add multiple entries to the indirect
table for each address registered. The latter was chosen.
Commit: a00def3f20e166d4fb9328e6f0bc0742cd0afa31 is not a part of this
PR and is handled / reviewed in:
https://github.com/llvm/llvm-project/pull/159856,
This is PR (2/3)
Register Vtable PR (1/3):
https://github.com/llvm/llvm-project/pull/159856,
Codegen / _llvm_omp_indirect_call_lookup PR (3/3):
https://github.com/llvm/llvm-project/pull/159857
Split from #158900 it adds a PerThreadContainer that can use STL-like
indexed containers based on a slightly refactored PerThreadTable.
---------
Co-authored-by: Joseph Huber <huberjn@outlook.com>
This PR introduces new debug macros that allow a more fined control of
which debug message to output and introduce C++ stream style for debug
messages.
Changing existing messages (except a few that I changed for testing)
will come in subsequent PRs.
I also think that we should make debug enabling OpenMP agnostic but, for
now, I prioritized maintaing the current libomptarget behavior for now,
and we might need more changes further down the line as we we decouple
libomptarget.
Summary:
This was a lot of code that was only used for upstream LLVM builds of
AMDGPU offloading. We have a generic and fast `malloc` in `libc` now so
just use that. Simplifies code, can be added back if we start providing
alternate forms but I don't think there's a single use-case that would
justify it yet.
