`parallelFor` dispatches its work to the thread pool and blocks the
calling thread in the `TaskGroup` destructor until the workers finish.
The caller thus stays idle instead of joining the work as modern parallel
libraries do, and `numactl -C 0-7 ld.lld --threads=8` ends up slower than
`numactl -C 0-8 ld.lld --threads=8`.
Run one worker on the calling thread instead. This requires that no
`parallelFor` body reads `getThreadIndex()`, which is unset on the
caller; the last such user is removed by #209687.
In the jobserver mode: only pool workers acquire job slots, so a
jobserver-limited process may briefly exceed its granted concurrency by
one thread. This is acceptable, better than pessimizing the common
non-jobserver case, and parallelFor is unused by jobserver users.
The DWARF verifier reconstructs simplified template names through
DWARFTypePrinter to compare them against the original DW_AT_name. On
malformed input, a template parameter can have no recoverable name,
which tripped assert(RawName) and aborted instead of letting the
verifier report the problem.
Assisted-by: Claude
rdar://182715403
This patch ports AArch64's negation-aware conjunction algorithm to fix
invalid CCMP emission when OR nested inside an AND.
Example: https://godbolt.org/z/ave7f61hK
Before the change, the above case returns 5 rather the 9 when CCMP
enabled.
Assisted-by: Claude Opus 4.8
Similar to the LegacyPM version, but we need to add a move constructor
to WebAssemblyExceptionInfo and set up some invalidation for the NewPM
given we don't need to invalidate if the CFG doesn't change.
Reviewers: sbc100, aheejin, dschuff
Pull Request: https://github.com/llvm/llvm-project/pull/210246
This is to match the standard style adopted by generic CodeGen analyses
like MachineBlockFrequencyInfo. This makes WebAssemblyExceptionInfo its
own class that we then get through the wrapper pass.
The primary aim of this patch is to add enough modularity such that we
can also provide this analysis through the NewPM in a follow up patch.
Reviewers: dschuff, sbc100, aheejin
Reviewed By: dschuff, aheejin
Pull Request: https://github.com/llvm/llvm-project/pull/210245
The legacy (non-HLFIR) lowering was removed in cc4922b276 (#196205),
but ConvertExpr.cpp and its ComponentPath/IterationSpace support files
were left behind. Their last callers have since been migrated or removed
(#210385: OpenMP iterator subscripts via genExprValue; #210621: initial
values via ConvertConstant; #210639: unreachable FORALL/WHERE machinery
deleted from Bridge.cpp), so nothing outside these files references them
anymore.
- Delete ConvertExpr.{cpp,h}, ComponentPath.{cpp,h},
IterationSpace.{cpp,h}. (createArrayLoads/createArrayMergeStores are
declared in IterationSpace.h but were defined in ConvertExpr.cpp - they
die together; no external callers.)
- CMakeLists.txt: drop the three sources and the ConvertExpr.h /
IterationSpace.h PRECOMPILE_HEADERS entries.
- Remove stale includes (verified symbol-free): IO.cpp,
OpenMP/OpenMP.cpp, DirectivesCommon.h (ConvertExpr.h); Allocatable.cpp,
Support/Utils.cpp (IterationSpace.h).
- Support/Utils.{h,cpp}: remove the ExplicitSpaceArrayBases alias and
its getHashValue/isEqual overloads; the SomeExpr* and Component*
variants stay (live via ExprToValueMap and HLFIR lowering).
This also deletes six small Optimizer/Builder helpers whose last callers
were in ConvertExpr.cpp — fir:🏭:genLenOfCharacter,
getNonDeferredLenParams, getExtentFromTriplet (FIRBuilder),
getExtentAtDimension (BoxValue), fir:🏭:getRealloc
(LowLevelIntrinsics), and the genCharacterCopy template (Factory.h).
Their removal orphans nothing further; Factory.h and LowLevelIntrinsics
retain their live content.
- Docs: drop the ConvertExpr.cpp section from fstack-arrays.md and the
stale ConvertExpr.cpp TODO inventories from ParameterizedDerivedTypes.md
and ProcedurePointer.md.
No functional change: everything removed is unreachable. Downstream
forks using the deleted entry points will break at compile time; this is
intended (precedent: cc4922b276).
Assisted-by: AI
This patch supports writing SecNameTable in Eytzinger layout for
ExtBinary MD5 sample profiles via a hidden command-line option,
-sample-profile-write-eytzinger-name-tables.
Specifically, this patch partitions function GUIDs in SecNameTable into
three mutually exclusive sets:
- CSKeys: Those GUIDs used as keys in context-sensitive SecLBRProfile.
- FlatKeys: Those GUIDs used as keys in flat SecLBRProfile.
- Inlinees: Those GUIDs mentioned in SecLBRProfile but not as keys.
Each set is constructed as an independent Eytzinger array using
llvm::EytzingerTable. The section starts out with the three element
counts followed by the three successive spans.
A subsequent patch will add FuncOffsetTable as arrays parallel to
CSKeys and FlatKeys. It will introduce binary search into the
Eytzinger name table, taking advantage of the cache-friendly layout.
RFC:
https://discourse.llvm.org/t/rfc-faster-sample-profile-loading/90957/8
Assisted-by: Antigravity
`LoopVersioningLICM::instructionSafeForVersioning()` rejects any call
that is
not proven to not access memory. `llvm.pseudoprobe` is declared
`IntrInaccessibleMemOnly` (so the optimizer will not delete or sink it),
so
`AA->doesNotAccessMemory()` returns false and the probe is treated as an
unsafe
call site. This disables loop-versioning LICM for essentially every hot
loop
in a sample-based / CSSPGO profile-guided build (in the presence of
`-fpseudo-probe-for-profiling`), since a pseudo probe is inserted on
every basic block.
Pseudo probes are pure profiling placeholders with no observable effect
on the
loop's memory accesses, so skip them in the call-safety check.
RFC:
https://discourse.llvm.org/t/csspgo-unblocking-pseudo-probe-safe-optimizations/90946
Each Loop stored its blocks twice: the Blocks vector and a DenseBlockSet
used only for O(1) contains(BlockT *). Block lists are inclusive, so building
LoopInfo inserted every block into that set once per enclosing loop --
an O(blocks * depth) cost.
Drop DenseBlockSet and answer contains(BlockT *) from the innermost loop
in LoopInfoBase's block-number-indexed map. This removes the per-build
set maintenance and shrinks sizeof(Loop) from 160 to 72 bytes. Each Loop
keeps a back-pointer to its LoopInfo for the lookup; contains() stays
total for a block from another function (e.g. a global's use).
Note: A pass restructuring the loop nest must not rely on
`contains(BlockT *)` while the block lists and the map are transiently
out of sync, and should scan getBlocks() instead (as FixIrreducible and
LoopSimplifyCFG do). LoopInfoBase::verify() checks the block lists
rather than contains() for the same reason.
Aided by Claude Opus 4.8
`__tgt_register_rpc_callback` unconditionally called
`Plugin::getRPCServer()` which asserts if the RPC server pointer is
null. When a plugin is initialized but its RPC server has not been
created (e.g., `NumDevices == 0` in `GenericPluginTy::init()`), this
triggers a crash.
Rather than introducing a nullable accessor, guard the call by also
checking that the plugin has at least one device. A zero-device plugin
has no RPC server to register callbacks with, so the extra check is both
correct and sufficient.
Suggested by @jhuber6 in the review of the previous attempt (#210215).
PR #210028 added a run of EarlyMachineLICM before StackColoring removed
LIFETIME_START/LIFETIME_END markers, and it ended up hoisting them which
led to miscompiles. This marks them as not safe to move in
`MachineInstr::isSafeToMove`.
---------
Co-authored-by: Matt Arsenault <arsenm2@gmail.com>
For code like
``` cpp
int f(int &a, const int &b) {
a += b + 17;
return a + b;
}
```
the x86 backend emitted three LEAs, rematerializing `a+b` instead of
reusing the value it had just computed and stored:
``` asm
leal (%rdi,%rsi), %eax # a+b (recomputed)
leal 17(%rdi,%rsi), %ecx # a+b+17 (stored)
movl %ecx, (%rdx)
leal 17(%rsi,%rax), %eax # (a+b)+b+17
```
Reason being that when selecting a LEA address for `add2 = add1 + b`
where `add1 = (a+b)+17` is multi-use (it feeds a store), `matchAdd`
looks through `add1` to sink its `+17` into the displacement,
decomposing an already-materialized value and forcing `a+b` to be
recomputed.
The proposed fix: when forming a LEA, if an operand is a value that the
matcher would decompose into several address components **and is already
materialized in a register** (so reusing it is free), use it directly as
a base/index register instead of splitting it.
``` asm
leal 17(%rdi,%rsi), %eax # a+b+17
movl %eax, (%rdx)
addl %esi, %eax # + b — reuses the stored value
```
A value is treated as materialized when it has a use that puts it in a
register **as a value** — a stored value operand, or a `CopyToReg` (a
return value, call argument, or a value live out of the block) — as
opposed to address-only uses (a GEP index, or a load/store pointer)
which fold it for free. This reuses existing idioms: the
`CopyToReg`-means-liveout reasoning mirrors `hasOnlyLiveOutUses`, and
the stored value of a machine store is the operand at
`X86::AddrNumOperands` (as in `X86AvoidStoreForwardingBlocks`).
Identifying whether a value is already materialized is inherently a
heuristic; this PR uses the lightweight use-based check above. The
backend already does this differently for different Ops with varying
levels of sophistication (e.g. the `SUB` case carries a full cost
model). If expert opinion is that this warrants a proper cost model
rather than the boolean check — or a more thorough analysis step — I
will be happy to discuss it and implement if my help is wanted.
No regressions were found on the existing tests.
---
Test changes & MCA
Changed tests: new pr51707.ll; regenerated lea-recursion.ll,
lea-opt-cse1.ll, lea-opt-cse2.ll, lea-opt-cse4.ll. Whole test/CodeGen +
test/DebugInfo sweep: 0 net-new failures (32,757 tests).
`llvm-mca` `Block RThroughput` (cycles/iteration, lower = better),
master → patched, per -mcpu:
```
┌───────────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬───────────┬───────────┐
│ fn \ μarch │ core2 │ nehalem │ sandyb │ haswell │ skylake │ icelake │ alderlake │ sprrapids │
├───────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼───────────┼───────────┤
│ reduced │ 1.5→1.0 │ 1.5→1.0 │ 1.5→1.0 │ 1.5→1.0 │ 1.0→1.0 │ 1.5→0.5 │ 2.0→1.0 │ 3.0→1.0 │
├───────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼───────────┼───────────┤
│ f │ 1.5→1.5 │ 1.5→1.5 │ 2.0→1.5 │ 2.0→1.5 │ 1.3→1.0 │ 1.5→1.0 │ 1.5→1.2 │ 3.0→1.0 │
├───────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼───────────┼───────────┤
│ shl_reuse │ 1.0→1.0 │ 1.0→1.0 │ 1.0→1.0 │ 1.0→1.0 │ 1.0→1.0 │ 1.0→0.7 │ 2.0→1.0 │ 2.0→1.0 │
├───────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼───────────┼───────────┤
│ lea-opt-cse1 │ 2.0→2.0 │ 2.0→2.0 │ 2.3→2.0 │ 2.3→2.0 │ 2.0→2.0 │ 1.5→1.0 │ 2.0→1.5 │ 3.0→1.3 │
├───────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼───────────┼───────────┤
│ lea-recursion │ 7.5→7.5 │ 7.5→7.5 │ 7.5→7.5 │ 7.5→7.0 │ 7.0→7.0 │ 6.5→4.0 │ 12.0→5.2 │ 12.0→7.0 │
└───────────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴───────────┴───────────┘
```
Improvement or neutral on every μarch, growing on newer cores (which
have fewer LEA ports and penalize the 3-source LEAs the patch removes).
Closes#51707.
Add Pre-commit CI to test compiler-rt builtins with libc to check it
works without issues
Split from #197950
Part of #197824
---------
Co-authored-by: Michael Jones <michaelrj@google.com>
Builds on the source/target materialization C bindings (#208934) to
expose the 1:N dialect-conversion functionality through the MLIR C API.
Continues the buildout of the dialect-conversion C bindings (follows
#206146 and #206161).
Assisted by: Claude
Parameters whose type is a standard tag type (e.g. std::in_place_t,
std::nothrow_t,
iterator tags, lock tags) are used purely for overload resolution and
don't need
a name. This patch skips them.
Added an IgnoredTypes option so users can customize the list. Default
covers 24
standard tag types.
Fixes#208448.
AI Usage: This commit is AI-assisted, reviewed and verified by me.
`isLBRSample` in `checkPerfScriptType` currently don't differentiate the
LBR records in hybrid sample (LBR+Stack) or an LBR sample. It checks
"0x" and "/" for the second pair as a unified way to tell whether the
record is an LBR pair. However, this requires a minimal of 2 pairs of
hybrid LBR samples, otherwise, even a valid LBR record will fail the
`isLBRSample` check. This fix adds an initial check for "0x" and "/" for
the first LBR record so we can handle the case where only 1 pair of LBR
record exists in the hybrid sample.
`cir.await` is always eliminated by the FlattenCFG pass before the
DirectToLLVM conversion. Since the operation never reaches LLVM
lowering, remove the unused lowering implementation.
Special case for V_PK_*64 instructions: these do not have OPSEL but SGPR
sources behave like OPSEL is set replicating low 64-bits into high. VGPR
sources in turn read actual 4 registers. To move operand from an SGPR to
a VGPR we need to replicate low half.
Example:
```fortran
integer :: r(:,:)
!$acc parallel loop reduction(+:r)
```
In this code, the reduction accumulator has dynamic dimensions and rank
greater than one.
Fix: use runtime dimension sizes to delinearize flattened
array-reduction indices.
After https://github.com/llvm/llvm-project/pull/211269 a generic
interface name is rejected in acc routine. This PR relaxes this by
allowing it when the interface name and the specific are the same - and
attaches the information to the specific itself.
Example:
```fortran
!$acc parallel loop gang worker reduction(+:sum)
do j = 1, n
!$acc loop vector reduction(+:sum)
do i = 1, n
sum = sum + a(i, j)
end do
end do
```
In this code, each worker has a private partial result, but combine
predication previously allowed only ThreadY row zero to contribute.
Fix: keep ThreadY active only for proven atomic worker combines, while
preserving legacy predication or reporting NYI for unsafe combinations.
Fixes a few places where optimizations were dropping memprof related
metadata when creating new calls. Adds a new facility that can be used
to propagate profile and debug metadata, and employs that in the passes
that were manually specifying just MD_prof and MD_dbg.
The `TypeConverter` materialization tests added in #208934 (`testTypeConverterSourceMaterialization` / `testTypeConverterTargetMaterialization`) freeze their `MlirRewritePatternSet` but never destroy it.
This was caught by LeakSanitizer on the aarch64 HWASan bootstrap bot ([builder 55, build
30763](https://lab.llvm.org/buildbot/#/builders/55/builds/30763)):
Fix: call `mlirRewritePatternSetDestroy(patterns)` after freezing in both tests, matching existing usage.
Assisted by: Claude
When getSimplePointerDependencyFrom scans backward and reaches a
@llvm.lifetime.start, it only treated the marker as a barrier when the
query location MustAliased the lifetime's argument, and otherwise fell
through to `continue`. For a load through a GEP into part of an alloca
the alias result is PartialAlias, so the scan walked past lifetime.start
as if it weren't there, and GVN Load PRE would hoist the load above it.
Inspect the full alias result instead of just isMustAlias: MustAlias
still returns Def and NoAlias still continues, but any partial/may
overlap now returns a Clobber so lifetime.start acts as a barrier and
the access cannot be moved above it.
Fixes#194940
First, replace all the `{eval-rst}` blocks for option documentation with
`:::{option}` fences. This means authors won't have to bounce between
markup languages within one file, and removes more semantic indentation,
which has been a recurring problem.
Second, remove blockquotes that seem unintentional. None of these are
real quotes, and they seem like artifacts of unintended indentation
before the Markdown migration (#208310).
---------
Co-authored-by: Andy Kaylor <akaylor@nvidia.com>
