Summary:
This provides users with a way to check all symbols present in a given
program. This is useful for many cases where a power user wants to do
something dynamic with the objects present.
Fails with an error about not being able to find `lld`. AMD team is
investigating, but disable the test in the meantime.
Context: https://github.com/llvm/llvm-project/pull/212860
Signed-off-by: Nick Sarnie <nick.sarnie@intel.com>
At the previous community meeting, we agreed to introduce contexts into
the liboffload API. I also mentioned that, as part of this work, we
could split the allocation function into three separate functions, but
we did not discuss this idea in detail.
This patch only splits the allocation functions, allowing us to discuss
and agree on the proposed interface before introducing contexts into the
`olMemAlloc` API.
Once contexts are added, the device parameter will be removed from host
allocations, and the final interface would look like this:
```c
ol_result_t olMemAllocDevice(ol_context_handle_t, ol_device_handle_t, size_t, void **);
ol_result_t olMemAllocHost(ol_context_handle_t, size_t, void **);
ol_result_t olMemAllocManaged(ol_context_handle_t, ol_device_handle_t, size_t, void **);
```
Currently, for host allocations, the device parameter is required mainly
to determine which plugin should perform the allocation. Once contexts
are introduced, this information will be provided by the context, making
the device parameter redundant.
Requiring both a context and a device for host allocations could also be
misleading, since it is unclear what the device parameter actually
represents in this API.
Assisted by: Claude.
Add the initial Context handle API to liboffload:
- olCreateContext / olDestroyContext
- olGetContextInfo / olGetContextInfoSize
- OL_ERRC_INVALID_CONTEXT error code
- OL_CONTEXT_INFO_{NUM_DEVICES,DEVICES,PLATFORM} queries
This is the first patch in a series. I decided to split the original
change into multiple smaller PRs to make the review process faster and
more manageable.
This patch introduces context management functions, but the context
handle is not used anywhere yet. The following patches will extend the
API to use contexts in:
``` c
ol_result_t olMemAlloc(ol_context_handle_t, ol_device_handle_t, ol_alloc_type_t, size_t, void**);
ol_result_t olMemFree(ol_context_handle_t, void*);
ol_result_t olGetMemInfo(ol_context_handle_t, const void*, ol_mem_info_t, size_t, void*);
ol_result_t olGetMemInfoSize(ol_context_handle_t, const void*, ol_mem_info_t, size_t*);
ol_result_t olCreateQueue(ol_context_handle_t, ol_device_handle_t, ol_queue_handle_t*);
ol_result_t olCreateProgram(ol_context_handle_t, ol_device_handle_t, const void*, size_t, ol_program_handle_t*);
```
You can also take a look at #201398, which contains the first version of
the complete change. I'm cleaning it up as I go, so the patches in this
series are not a direct 1:1 split of that PR.
Even though L0, CUDA and HSA do not directly support filling memory
regions with non-power-of-two-sized patterns, plugins explicitly allow
for such fills by falling back to a slow path whenever a direct API call
is not possible.
The Level Zero plugin is the odd one out, forwarding directly to
`zeCommandListAppendMemoryFill`. This fails for non-power-of-two
patterns, causing unit tests for `olMemFill` to fail when run on L0.
This PR adds fallbacks in L0 plugin's `memoryFill` implementation to
support arbitrary pattern sizes:
- if all pattern bytes are the same, substitute `1` for `patternSize`
and proceed with the native fill
- if memory is host-accessible (host or shared alloc), then simply
`std::copy_n` the pattern
- if memory is device-only, upload a seed to the device, and
`memoryCopy` O(log n) times, doubling the pattern each time, until the
entire region is filled. The seed is the input pattern, but extended on
the host to 1-2KiB to potentially eliminate several small `memoryCopy`
calls that would otherwise be almost pure overhead.
Conceptually, this approach has better worst-case overhead than the one
found in the CUDA plugin (decompose the pattern to then run multiple
strided memsets) and the AMDGPU one (call `pushMemoryCopyH2DAsync` with
the pattern, setting `NumTimes`). Aligning the fallback fill approaches
is outside the scope of this PR, however - the main intent is to fix the
unit test failures.
https://github.com/llvm/llvm-project/pull/207208 caused a CI failure on
openmp-offload-amdgpu-clang-flang in the
olCreateProgramTest.TruncatedBinary test.
This PR removes this test case, as it's unclear whether guarding against
such cases is worth it.
liboffload plugins would not always exhaustively map the underlying
API's error values to ol_errc_t, resulting in unintended behavior - e.g.
olCreateProgram, given an invalid binary, would return OL_ERRC_UNKNOWN
instead of OL_ERRC_INVALID_BINARY.
This patch largely eliminates the most glaring cases of this by
assigning an ol_errc_t value to every error code of the underlying API
in every plugin. The proper granularity for this mapping is per-API
call, but that is a large rework that belongs in a separate patch.
Assisted-by: Claude
Adds olMemPrefetch and the OL_MEM_MIGRATION_FLAG_* enum, wired through a
new dataPrefetchImpl on the plugin interface with Level Zero and CUDA
implementations. Backends without prefetch support inherit the default
no-op.
Prefetch is a hint that migrates a USM allocation between the host and a
device so subsequent accesses on the target side are faster.
Assisted-by: Claude
if olInit fails, it leaves liboffload in an inconsistent state (proper
cleanup will be addressed in a follow-up patch). This can lead to
seemingly unrelated test failures (e.g. the host device is missing).
This patch makes such tests fail immediately with a clear error message.
This patch is the first step towards introducing alignment support in
memory allocations using liboffload, in order to enable SYCL
implementation of aligned allocations.
At the level of device allocators, it does not modify the Level Zero
code except for forwarding the alignment parameter, since Level Zero
already allows for specifying the alignment in its device allocator
implementation. For AMD and CUDA, it checks whether the alignment passed
by the caller is supported by the given backend; the reasoning behind
this verification is described in the following paragraphs. At the API
level, it adds a new function olMemAllocAligned, which is expected to
work similarly to olMemAlloc, with the difference that the buffers
returned by olMemAllocAligned should be aligned to the alignment passed
by the user. At the level of the plugin interface internal abstractions,
it adds a new argument Alignment to existing functions and delegates
memory allocation between olMemAllocAligned and olMemAlloc
implementations by using a common helper function.
The goal of the anticipated series of patches is to implement handling
of the alignment in the memory manager at the plugin interface level. At
the first stage, presented in this patch, the information about the
passed alignment is used mainly for checking whether the buffer returned
by the device allocators meets the requirements. In the case of the
requested memory size exceeding the thresholds of allocations handled by
the memory manager, the request is forwarded directly to the device.
Otherwise, the memory manager is responsible for allocating memory in
full pages and pooling it according to the requested chunks. In the
first scenario, the requested size, which is greater than the
aforementioned threshold, is usually a multiple of the page size.
Therefore, any alignment smaller than the page size would be correct.
Neither CUDA nor HSA provides users with the ability to specify the
alignment of the allocated memory. Their APIs include the alignment as
one of the possible arguments only in functions that reserve virtual
address space. CUDA enables users only to check the allocation
granularity, which is usually synonymous with the page size. In the case
of HSA, if the memory is allocated using a pool, the user is able to
check not only the granularity, but also the alignment of the buffers
allocated using the given pool. However, these values - granularity and
alignment - are defined only if
HSA_AMD_MEMORY_POOL_INFO_RUNTIME_ALLOC_ALLOWED is set to true, which
should always be set since the current implementation of the memory pool
for the AMD plugin in liboffload uses only hsa_amd_memory_pool_ts for
allocations. Only Level Zero accepts the alignment parameter during the
memory allocation process, but manages returned pointers internally.
Since pooling is also implemented in the memory manager at the plugin
level, such a design in the Level Zero device allocator duplicates
pooling in the whole application.
In the final version, the memory manager at the plugin interface level
will handle memory pooling and the buffer alignment from different
devices instead of delegating memory management to device allocator
implementations, as this would simplify the Level Zero plugin design and
provide AMD and CUDA plugins with support for the alignment in memory
allocations, which is not natively included in their APIs.
Depending on system setup, the unit tests could not find the just built
libLLVMOffload.so. Set the BUILD_RPATH to make sure they correctly use
the library from the current build.
Summary:
We need this flag so we can include `stdint.h` without potentially going
through to the system interface. this flag removes all default include
paths excepts the resource directory, which is the intention here.
Return not just the address but also the size of the symbols in getGlobalMetadataFromDevice.
Fixes olGetSymbolInfoSizeGlobalTest.SuccessSize unitt test failure with L0 plugin.
Until now, strict behavior in the number of threads and blocks has been
applied only when the kernel is in bare mode. When this mode is enabled,
the values passed in UserNumBlocks and UserThreadLimit are not adjusted
and are the definitive values used to launch the kernel. This commit
detaches the strictness from the kernel mode.
This is going to be used by the kernel replay tool. Additionally, it
starts clearing the path for the upcoming OpenMP dims modifier, used to
configure multidimensional teams and leagues, which will include
strictness choices for teams and threads.
All the bare kernels must indicate strict behavior. Asserts are added to
check this condition.
This PR changes olLaunchKernel to accept an array of pointers to
arguments:
```
void *ArgPtrs[] = {&A, &B, &C};
size_t ArgSizes[] = {sizeof(A), sizeof(B), sizeof(C)};
olLaunchKernel(Queue, Device, Kernel, &LaunchArgs, std::size(ArgPtrs), ArgPtrs, ArgSizes);
```
The newly proposed interface is implementable by existing and
anticipated
backends, is familiar to CUDA programmers, eliminates the extraneous
construction of a contiguous arguments buffer, replacing it with
constructing
an array of pointers, sidesteps the alignment requirements, does not
require reading program image metadata where it's impractical, and
enables
a compliant SYCL implementation to be built on top of it.
The ArgSizes array is required to support OpenCL, which does not have
native support for launching a kernel with an argument pointer array, or
a reliable way of retrieving argument sizes for a kernel.
## Mapping the proposed API to backends
CUDA and Level-Zero both support accepting an array of pointers to
kernel arguments, through `cuLaunchKernel`
(https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__EXEC.html#group__CUDA__EXEC_1gb8f3dc3031b40da29d5f9a7139e52e15)
and `zeCommandListAppendLaunchKernelWithArguments`
(https://oneapi-src.github.io/level-zero-spec/level-zero/latest/core/api.html#zecommandlistappendlaunchkernelwitharguments)
respectively.
For OpenCL, which requires the kernel arguments to be set separately
from the kernel launch, a potential implementation can extract the
number of arguments from the OpenCL API, and then iterate over the
argument pointer and size arrays:
```
cl_uint num_args = 0;
cl_int err = clGetKernelInfo(kernel, CL_KERNEL_NUM_ARGS,
sizeof(num_args), &num_args, NULL);
for (cl_uint i = 0; i < num_args; ++i) {
clSetKernelArg(kernel, i, ArgSizes[i], ArgPtrs[i]);
}
```
The AMDGPU plugin needs to construct a contiguous buffer using the array
of argument pointers. To do this conversion, we need to have offsets
at which to place the arguments. Here, luckily, as mentioned before,
the AMD plugin already reads kernel metadata from the program image.
The implementation simply retrieves size and offset for each argument
from the kernel image, and then uses it to populate a buffer.
... and disable failing level-zero tests, to be reenabled once the
plugin is fully functional.
Here is the passrate for the level-zero plugin unit tests:
```
Total Discovered Tests: 645
Skipped: 74 (11.47%)
Passed : 571 (88.53%)
```
We are actively working on fixing the issues marked as known failures
here.
This commit modifies the multithreaded kernel launch test to use device
memory instead of managed memory. The test is reported to be failing
intermittently in systems where concurrent managed memory access is
not supported. This is the case for NVIDIA devices that do not support
CU_DEVICE_ATTRIBUTE_CONCURRENT_MANAGED_ACCESS.
The concept of concurrent and coherent managed memory access should
be exposed to liboffload users somehow, e.g., adding it as device property,
so it is clear what execution patterns are allowed with managed memory.
However, this test is just testing concurrent kernel launches. This commit
fixes it until we decide how to proceed with the guarantees on that type of
allocations.
Summary:
Profiling requires getting time stamp information, which required
actually materializing barriers on the event. This caused performance
regressions in some OpenMP applications. Instead, we want to make this
opt in. This just adds a new argument to `olCreateEvent` which takes
some flags. Right now this is just none and profiling. Pretty mechanical
beyond that. Should revert this to the old behavior for OpenMP.
Co-authored-by: Kevin Sala Penades <kevinsala.ks@gmail.com>
Introduce a properties argument to olLaunchKernel to enable future
extensions.
This change adds initial extension for cooperative kernel launch.
The change restores the functionality from:
https://github.com/llvm/llvm-project/pull/184343
which was later reverted in:
https://github.com/llvm/llvm-project/pull/197463#184343 modified the API, and between the last rebase and the final
merge, additional tests were added
upstream(https://github.com/llvm/llvm-project/pull/193927) that still
relied on the previous API version.
Diff between this PR and #184343
```
diff --git a/offload/unittests/OffloadAPI/kernel/olLaunchKernel.cpp b/offload/unittests/OffloadAPI/kernel/olLaunchKernel.cpp
index 0251bffefe60..cd527f18db38 100644
--- a/offload/unittests/OffloadAPI/kernel/olLaunchKernel.cpp
+++ b/offload/unittests/OffloadAPI/kernel/olLaunchKernel.cpp
@@ -241,8 +241,8 @@ TEST_P(olLaunchKernelSingleCounterSyncEventTest, SuccessSyncEvent) {
ArgsSingleCounter Args{LoopRange, NumberToAdd, (uint32_t *)InitValuePassed,
(uint32_t *)ResNum};
- ASSERT_SUCCESS(
- olLaunchKernel(Queue, Device, Kernel, &Args, sizeof(Args), &LaunchArgs));
+ ASSERT_SUCCESS(olLaunchKernel(Queue, Device, Kernel, &Args, sizeof(Args),
+ &LaunchArgs, nullptr));
uint32_t FinalResVal = 0;
ASSERT_SUCCESS(olMemcpy(Queue, &FinalResVal, Host, ResNum, Device, Size));
@@ -294,8 +294,8 @@ TEST_P(olLaunchKernelSingleCounterSyncEventTest, SuccessTwoQueues) {
ArgsSingleCounter Args{LoopRange, NumberToAdd, (uint32_t *)InitValuePassed,
(uint32_t *)ResNum1};
- ASSERT_SUCCESS(
- olLaunchKernel(Queue, Device, Kernel, &Args, sizeof(Args), &LaunchArgs));
+ ASSERT_SUCCESS(olLaunchKernel(Queue, Device, Kernel, &Args, sizeof(Args),
+ &LaunchArgs, nullptr));
ol_event_handle_t Event = nullptr;
ASSERT_SUCCESS(olCreateEvent(Queue, &Event));
@@ -308,7 +308,7 @@ TEST_P(olLaunchKernelSingleCounterSyncEventTest, SuccessTwoQueues) {
// been completed by the time the kernel is executed using the second queue,
// the FinalResVal would be incorrect.
ASSERT_SUCCESS(olLaunchKernel(Queue2, Device, Kernel, &Args2, sizeof(Args2),
- &LaunchArgs));
+ &LaunchArgs, nullptr));
ASSERT_SUCCESS(olSyncQueue(Queue2));
uint32_t FinalResVal = 0;
```
This patch adds:
- a new kernel `void single_counter(int32_t init_loop, int32_t addend,
uint32_t *init_val, uint32_t *out)`
- a new test suite `olLaunchKernelSingleCounterSyncEventTest`, with two
tests
The test `SuccessSyncEvent` verifies whether `olSyncEvent(Event)` awaits
the completion of `Event`.
The test `SuccessTwoQueues` checks the correctness of the
synchronization between queues using events. Enqueueing the kernel on
the queue `Q1` should produce `Result1`, which is used as an initial
value for the queue `Q2`. Therefore, before executing any future work,
`Q2` should wait for the event `Event1`, which is created after
submitting work on `Q1`. The required synchronization is ensured by
`olWaitEvents(Q2, &Event1, 1)`. If `Q2` uses the value passed as
`Result1` before the work on `Q1` has completed, the result of the
kernel enqueued on `Q2` would be incorrect.
Add device info queries to detect support for half-, single-, and
double-precision floating-point formats.
For the AMDGPU, CUDA, and Host plugins, add the new queries alongside
the existing capability reporting without changing current behavior.
For the Level Zero plugin, implement floating-point support detection
and capability querying.
This patch adds `olGetEventElapsedTime` to the new LLVM Offload API, as
requested in
[#185728](https://github.com/llvm/llvm-project/issues/185728), and adds
the corresponding support in `plugins-nextgen`.
A main motivation for this change is to make it possible to measure the
elapsed time of work submitted to a queue, especially kernel launches.
This is relevant to the intended use of the new Offload API for
microbenchmarking GPU libc math functions.
### Summary
The new API returns the elapsed time, in milliseconds, between two
events on the same device.
To support the common pattern `create start event → enqueue kernel →
create end event → sync end event → get elapsed time`, `olCreateEvent`
now always creates and records a backend event through the device
interface. For backends that materialize real event state, this gives
the event concrete backend state that can be used for elapsed-time
measurement. For backends that do not materialize backend event state,
`EventInfo` may still remain null and existing event operations continue
to treat such events as trivially complete.
Previously, an event created on an empty queue could be represented only
as a logical event. That representation was sufficient for sync and
completion queries, but it was not suitable for elapsed-time measurement
because there was no backend event state to timestamp. The new behavior
preserves the meaning of completion of prior work while also allowing
backends with timing support to attach real event state.
### Changes in `plugins-nextgen`
#### Common interface
Add elapsed-time support to the common device and plugin interfaces:
* `GenericPluginTy::get_event_elapsed_time`
* `GenericDeviceTy::getEventElapsedTime`
* `GenericDeviceTy::getEventElapsedTimeImpl`
#### AMDGPU
* Add the required ROCr declarations and wrappers.
* Enable queue profiling at queue creation time.
* Record events by enqueuing a real barrier marker packet on the stream.
* Retain the timing signal needed to query the recorded marker later.
* Implement `getEventElapsedTimeImpl` using
`hsa_amd_profiling_get_dispatch_time`, converting the result to
milliseconds with `HSA_SYSTEM_INFO_TIMESTAMP_FREQUENCY`.
This follows the ROCm/HIP approach of enabling queue profiling at HSA
queue creation time, while keeping the AMDGPU queue path simpler than
the lazy-enable alternative discussed during review.
#### CUDA
* Add the required CUDA driver declarations and wrappers.
* Implement `getEventElapsedTimeImpl` with `cuEventElapsedTime`.
#### Host
* Add `getEventElapsedTimeImpl` that stores `0.0f` in the output
pointer, when present, and returns success.
Reason: the host plugin does not materialize backend event state and
already treats event operations as trivially successful. Returning
`0.0f` preserves that model without introducing a new failure mode.
#### Level Zero
* Add `getEventElapsedTimeImpl`, but leave it unimplemented.
Reason: the Level Zero plugin currently does not provide standalone
backend event support for this event model. For example, `waitEventImpl`
/ `syncEventImpl` are still unimplemented there.
---------
Signed-off-by: Leandro Augusto Lacerda Campos <leandrolcampos@yahoo.com.br>
Signed-off-by: Leandro A. Lacerda Campos <leandrolcampos@yahoo.com.br>
This PR adds extends liboffload olMemRegister API to handle a case when
a memory block may have been mapped before calling olMemRegister to
support some use cases in libomptarget
Summary:
This PR adds a pointer argument to the initialization routine to be used
for global options. Right now this is used to allow the user to
constrain which backends they wish to use.
If a null argument is passed, the same behavior as before is observed.
This is epxected to be extensible by forcing the user to encode the size
of the struct. So, old executables will encode which fields they have
access to.
We use a macro helper to get this struct rather than a runtime call so
that the current state of the size is baked into the executable rather
than something looked up by the runtime. Otherwise it would just return
the size that the (potentially newer) runtime would see
The purpose of this PR is to add support of host as an offloading device
to liboffload. Both OpenMP and sycl support offloading to a host as
their normal workflow and therefore would require such capability from
liboffload library.
Summary:
We provide an RPC server to manage calls initiated by the device to run
on the host. This is very useful for the built-in handling we have,
however there are cases where we would want to extend this
functionality.
Cases like Fortran or MPI would be useful, but we cannot put references
to these in the core offloading runtime. This way, we can provide this
as a library interface that registers custom handlers for whatever code
people want.
Add liboffload asynchronous queue query API for libomptarget migration
This PR adds liboffload asynchronous queue query API that needed to make
libomptarget to use liboffload
Add liboffload memory data locking API for libomptarget migration
This PR adds liboffload memory data locking API that needed to make
libomptarget to use liboffload
LLVM_REQUIRES_* are per-target flags that are never set globally. Yet,
some files used these (undefined) flags for some logic. This patch
emoves these dead checks/unconditionally executes the logic. Note that
the referenced *.exports files are empty, so there is no need to make
related logic conditional on MSVC.
This commit appends a device number after the device name (used as
unittest param name). The number is between 0 and the number of
available non-host devices. In this way, it allows multiple devices of
the same vendor to be tested.
Introduced in OpenMP 6.0, the device UID shall be a unique identifier of
a device on a given system. (Not necessarily a UUID.) Since it is not
guaranteed that the (U)UIDs defined by the device vendor libraries, such
as HSA, do not overlap with those of other vendors, the device UIDs in
offload are always combined with the offload plugin name. In case the
vendor library does not specify any device UID for a given device, we
fall back to the offload-internal device ID.
The device UID can be retrieved using the `llvm-offload-device-info`
tool.
Summary:
This check is unnecessarily restrictive and currently incorrectly fires
for any size less than eight bytes. Just remove it, we do sanity checks
elsewhere and at some point need to trust the ABI.
If olMemAlloc happens to allocate memory that was already allocated
elsewhere (possibly by another device on another platform), it is now
thrown away and a new allocation generated.
A new `AllocBases` vector is now available, which is an ordered list
of allocation start addresses.
Summary:
This exposes the 'isDeviceCompatible' routine for checking if a binary
*can* be loaded. This is useful if people don't want to consume errors
everywhere when figuring out which image to put to what device.
I don't know if this is a good name, I was thining like `olIsCompatible`
or whatever. Let me know what you think.
Long term I'd like to be able to do something similar to what OpenMP
does where we can conditionally only initialize devices if we need them.
That's going to be support needed if we want this to be more
generic.
