In #180244 (8e76c3f0d2), handling of Instruction::Sub in
createSCEV declared PtrLHS and PtrRHS as uninitialized local variables.
When only one side matched m_PtrToInt, the other variable remained
uninitialized. Passing it by value to GetOp copied uninitialized shadow
bytes, causing MemorySanitizer to report use-of-uninitialized-value.
Initialize PtrLHS and PtrRHS to nullptr upon declaration.
Assisted-by: Gemini
Use SCEVPtrToAddr instead of SCEVPtrToInt for pointer -> int
conversion in SCEV, like when computing trip count expressions.
getPtrToIntExpr now also creates SCEVPtrToAddrExpr.
Removal of SCEVPtrToIntExpr out of this PR, to limit the scope and
will be done as NFC follow-up.
PR: https://github.com/llvm/llvm-project/pull/180244
Previously, the idiom matcher derived the clamp range only from the raw
`(LHS + C1) u< C2` form and rejected wrapped or full ranges.
Intersect the derived range with the known unsigned range of `LHSUnknown`.
This can reduce wrapped or full ranges to a usable interval and allows the
matcher to extract clamps in more cases.
Related to https://github.com/llvm/llvm-project/issues/208778
ScalarEvolution::getElementSize always derived the element-size SCEV
using a generic address-space-0 pointer index type. On targets whose
data layout uses different pointer index widths per address space (for
example, where some address spaces use 32-bit pointers and others
64-bit), a load/store through a narrow address space produced an access
function SCEV of one width but an element-size SCEV of another.
Delinearization then called SCEVDivision::divide with numerator and
denominator of mismatched types, tripping the assertion added in
a9d295d615 once the implicit sign-extension was removed in
23a32bcedb.
Derive the index type from the memory access's actual pointer operand
address space so the element-size SCEV matches the width of the access
function's SCEV.
Parameterize MatchRangeCheckIdiom over (Predicate, LHS, RHS) so it can
be reused for other predicates discovered during guard collection.
Hoist AddRewrite / GetMaybeRewritten above it and use them in place of
the inline try_emplace, so range-check rewrites chain onto existing ones
the same way other rewrites do.
Teach IsKnownPredicateViaMinOrMax to handle ICMP_ULT/UGT for
SCEVUMinExpr. Since umin(Ops) u<= each Op by definition, proving Op u<
RHS for any single operand is sufficient to establish umin(Ops) u< RHS
by transitivity.
This unblocks loop vectorization for early-exit loops whose trip count
is a umin of a buffer length and an iteration bound.
Fixes https://github.com/llvm/llvm-project/issues/196935
Alive2 proof: https://alive2.llvm.org/ce/z/9DjA2f
Simplify:
```
(K + A) pred (K + B) -> A pred B
(C * A) pred (C * B) -> A pred B
```
when no-wrap flags allow it: NSW for signed predicates, NUW for
unsigned. Also requires C > 0 for signed in the multiplication case.
For equality comparisons, `(K + A) eq/ne (K + B)` simplifies to `A eq/ne
B` without requiring any no-wrap flags.
Alive2 proofs: https://alive2.llvm.org/ce/z/KmyqsD
Add isImpliedCondOperandsViaMatchingDiff to fold equality comparisons
when getMinusSCEV(LHS, RHS) == getMinusSCEV(FoundLHS, FoundRHS).
This handles correlated IV comparisons in loops with multiple pointer
IVs sharing the same stride.
When we're computing the range of the addrec, we already have to reason
about whether it wraps, so we may as well determine the nowrap flags at
the same time.
This is more precise than the previous logic that took the addrec range
and checked whether adding a step to it does not wrap. For example, an
`{0,+,1}` addrec with a full range can still be non-wrapping.
Note that I removed some assertions in the SCEV printed that predicated
exit counts actually have predicates. Due to SCEV's query order
dependence, this can happen, also prior to this change, see for example
https://llvm.godbolt.org/z/cWK1MMEqv. While this indicates suboptimal
results, it's not a bug, and we should not assert.
Fixes https://github.com/llvm/llvm-project/issues/200788.
Add an overload of getPtrStride and update isNoWrap to collect no-wrap
SCEV predicates in a vector, and have the caller add them to PSE if
needed.
This also requires to explicitly build the wrap predicate via
ScalarEvolution::getWrapPredicate (clearing statically implied flags)
rather than going through PredicatedScalarEvolution::setNoOverflow,
which can now be removed together with the flags map.
This also requires generalizing SCEVUnionPredicate::implies so a wrap
predicate can be recognized as implied by an existing wrap predicate
after applying equal predicates, avoiding redundant predicates when the
collected predicates are committed.
The only functional changes is re-ordering of runtime check groups in a
single test case; no functional IR changes across a large IR corpus (32k
modules)
To be used to avoid adding unnecessary wrap predicates during interleave
group analysis: https://github.com/llvm/llvm-project/pull/200807
PR: https://github.com/llvm/llvm-project/pull/203787
Make the CanBeFreed argument of getPointerDereferenceableBytes() a
pointer, so that nullptr can be passed if we're not interested in
whether frees are possible or not.
Nearly all places don't actually care about frees, including BasicAA,
which is the hottest caller of this API. This improves compile-time when
deref-at-point semantics are enabled.
I've kept the argument required so that callers still have to make an
explicit choice to ignore frees. (I'd be open to making it optional
though, given that only a single caller actually cares...)
Application code on AMDGPUs often uses the following pattern:
for (int i = blockDim.x / 2; i >= 1; i >>= 1)
if (threadIdx.x < i) {
<STMT based on threadIdx.x>
}
__syncthreads();
}
but this is currently not unrolled by LLVM for two reasons:
1. We overestimate the loop count with the number of bits in the
iteration variable (32).
2. Runtime unrolling is not applicable because of convergent
instructions.
For the specific pattern above, we can compute a tighter bound by using
range information, which for blockDim.x is (0,1025) and already emitted
by clang, resulting in 10 as upper bound as opposed to 32.
The next step in unrolling the loop is to adjust the threshold used to
control full loop unrolling with convergent instructions.
Assisted-by: Cursor
This patch simplifies the iterative worklist handling in createSCEVIter.
In the previous implementation, a `Value` that could not be materialized
immediately would typically be pushed onto the stack twice and popped
twice: first in the initial unvisited state, and then again after being
re-queued in the visited state once its operands had to be processed
first.
This change avoids that redundant re-queue step by updating the existing
stack entry in place before pushing unresolved operands. Use
`setInt(true)` to update the existing `Stack` entry in place instead of
pushing a separate `(V, true)` entry. This saves one
`emplace_back`/`pop_back` pair for each `Value`, and reduces memory
consumption of the `Stack` worklist.
Every cached SCEV varies with Loop `L` transitively contains an
`AddRec`, and every `AddRec` for the loop is recorded in `LoopUsers[L]`.
`forgetMemoizedResults` already closes this set transitively through
`SCEVUsers` and `ExprValueMap`.
Therefore `forgetLoop` does not need to walk the def-use chain starting
from header PHIs, it only needs to initialize `ToForget` with
`LoopUsers[L]` and explicitly remove each header PHI's entries from
`ValueExprMap` and `ConstantEvolutionLoopExitValue`, push its cached
SCEVs into `ToForget`.
As a side effect, cached SCEVs that reside in the loop body but do not
depend on any `AddRec` for the loop (e.g. a `SCEVUnknown` for an icmp)
are no longer invalidated.
The existing pairwise common-factor fold in getAddExpr handles two
patterns:
`W + X + (X * Y * Z) --> W + (X * ((Y*Z)+1))`
`X + (A*B*C) + (A*D*E) --> X + (A*(B*C+D*E))`
Both fold exactly two terms sharing a common factor, then re-enter
getAddExpr() with the partially-simplified Ops. When n terms share a
common factor X, this requires n-1 re-entries through the full
getAddExpr normalization pipeline.
Replace this with a single-pass scan that collects all terms sharing the
common factor and folds them in one shot:
`A1*X + A2*X + ... + An*X --> X * (A1 + A2 + ... + An)`
This reduces the number of top-level re-entries into getAddExpr() for
this fold from n-1 to 1, improving compile time for expressions with
many terms sharing a common factor.
Note that in some cases this change causes benign operand order
changes.
By the end of ScalarEvolution::getBackEdgeTakenInfo():
```
...
BackedgeTakenInfo Result = computeBackedgeTakenCount(L);
if (Result.hasAnyInfo()) {
// Invalidate any expression using an addrec in this loop.
...
}
return BackedgeTakenCounts.find(L)->second = std::move(Result);
```
BackedgeTakenCounts may no longer contain an entry for L, resulting in a
crash (https://github.com/llvm/llvm-project/issues/195176). This patch
fixes the issue by inserting the entry if it does not exist (and
otherwise keeping the same behavior).
This adds a regression test (crash on opt 19.1.0 onwards e.g.,
https://godbolt.org/z/Wq3n3rEqT), metarenamed from
https://github.com/dtcxzyw/llvm-autoreduce/issues/106, which in turn was
derived from https://github.com/llvm/llvm-project/issues/195176.
Since #174380, instcombine can clear some set bits in the added constant
in expressions like this, when A has some known-clear low order bits.
(A + 15) & ~15
This transformation is valid, but can make it harder for later passes to
recognise this idiom for rounding up to a power of 2. This is causing
the ARM MVE tail predication pass to fail on loops with a trip count
which is a multiple of a small power of 2.
The fix is to reverse the transformation when building SCEV expressions,
canonicalising to always use the largest valid value for the added
constant.
Alive proofs:
https://alive2.llvm.org/ce/z/hhndoWhttps://alive2.llvm.org/ce/z/_JYVat
In preparation for switching DenseMap from tombstone deletion to
backward-shift deletion, update call sites that reuse an iterator or a
bucket reference after erasing another entry from the same map.
These work under tombstone deletion because unrelated buckets stay put,
but backward-shift deletion relocates entries to close the gap.
Add DenseMap::remove_if, similar to SmallPtrSet::remove_if, as
replacement for erase-while-iterating, and use it where applicable.
Aided by Claude Opus 4.7
Simplify zext(C+A)<nsw> -> (sext(C) + zext(A))<nsw> if
* zext (C + A)<nsw> >=s 0 and
* A >=s V.
For now this is limited to cases where the first operand is a constant,
so the SExt can be folded to a new constant. This can be relaxed in the
future.
The initial version checks for non-negative manually to limit compile-time,
supporting only A = smax(C2, ..) where C2 >= abs(C)
Alive2 proof of the general pattern and the test changes in zext-nuw.ll
(times out in the online instance but verifies locally)
https://alive2.llvm.org/ce/z/_BtyGy
PR: github.com/llvm/llvm-project/pull/142599
This reverts commit 45d94f67b8.
This recommit contains a small fix: when `loop-computable` are mixed
with `loop-uniform`, it returns `loop-variant (unknown)`.
Original message:
This patch extends `ScalarEvolution::LoopDisposition` with a new
`LoopUniform` state to describe SCEVs that are invariant across all
iterations of a given loop, but may still depend on inner-loop induction
variables.
Unlike `LoopInvariant`, which requires the value to be fully invariant
with respect to the loop, LoopUniform captures expressions that do not
depend on the loop’s own induction variables, yet may vary in nested
loops. This distinction is useful for analyses and optimizations that
reason about per-iteration stability at a specific loop level.
Example:
```
for (i)
for (j)
dep(j); // uniform w.r.t. i
dep(i, j); // not uniform w.r.t. i
```
This enables us to more precisely reason about the behavior of SCEV in
the context of nested loops, which is very useful for optimizations such
as `LoopUnrollAndJam` and `OuterLoopVectorization`.
PR: https://github.com/llvm/llvm-project/pull/194304
The Scalar Evolution analysis converts the `or disjoint` instruction
into a synthetic add with the `IsNSW` and `IsNUW` flags set, but the
real instruction reference set to `nullptr`. The flags propagation logic
only reads the real instruction flags, when the instruction reference is
present. This behavior silently ignores the flags for the synthetic add,
which can lead to missed optimization opportunities.
This patch inserts the pointer to the actual instruction into the
`BinaryOp` object created for the synthetic add. It also adjust the
flags handling logic to threat the `disjoint` flag as both `nuw` and
`nsw` as if they were set on the `add` instruction.
Assisted-by: Claude Opus 4.6 <noreply@anthropic.com>
The udiv exact handling for constant RHS is already partially handling
in the generic udiv code:
8a0c5d3f43/llvm/lib/Analysis/ScalarEvolution.cpp (L3641-L3657)
Move two remaining folds there as well:
* `(%a * %b)<nuw> / %b` to `%a`
* `(A*GCD * %b)<nuw> / (C*GCD)` to `(A * %b)<nuw> / C`
Removing common factors from a multiply does not require exact if the
multiply is nuw (and conversely, being exact is not sufficient without
nuw).
This makes getUDivExactExpr() equivalent to getUDivExpr(). I've retained
the method for now in case we want to add exact specific logic in the
future.
Proof: https://alive2.llvm.org/ce/z/iQeb6v
This patch extends `ScalarEvolution::LoopDisposition` with a new
`LoopUniform` state to describe SCEVs that are invariant across all
iterations of a given loop, but may still depend on inner-loop induction
variables.
Unlike `LoopInvariant`, which requires the value to be fully invariant
with respect to the loop, LoopUniform captures expressions that do not
depend on the loop’s own induction variables, yet may vary in nested
loops. This distinction is useful for analyses and optimizations that
reason about per-iteration stability at a specific loop level.
Example:
```
for (i)
for (j)
dep(j); // uniform w.r.t. i
dep(i, j); // not uniform w.r.t. i
```
This enables us to more precisely reason about the behavior of SCEV in
the context of nested loops, which is very useful for optimizations such
as `LoopUnrollAndJam` and `OuterLoopVectorization`.
As far as I can tell, ControlsOnlyExit is used during reasoning to
assume
that we stay in the loop as long as the condition is true/false, and are
guaranteed to exit otherwise.
But unless I am missing something, a sub-condition of an AND/OR never
solely controls the exit, whether we exit always depends on both
conditions.
Pass false to ControlsOnlyExit, as otherwise we would incorrectly assume
that we must exit if either conditions is true, when we would only exit
if both are true.
For now, ControlsOnlyExit is only used for the neutral element case, it
may be worth removing this.
Alive2 proof of IndVars mis-compile: https://alive2.llvm.org/ce/z/kWs4hE
PR: https://github.com/llvm/llvm-project/pull/194831
The patch moves out of SCEV's scope so they can be re-used for SCEVUse.
SCEVUse gets an additional getNoWrapFlags helper that returns the union
of the expressions SCEV flags and the use-specific flags.
SCEVExpander has been updated to use this new helper.
In order to avoid other changes, the original names are exposed via
constexpr in SCEV. Not sure if there's a nicer way. One alternative
would be to define the enum in struct, and have SCEV inherit from it.
The patch also clarifies that the SCEVUse flags encode NUW/NSW, and
hides getInt, setInt, etc to avoid potential mis-use.
PR: https://github.com/llvm/llvm-project/pull/190199
Add SCEVUseVisitor, a new visitor class where all visit methods receive
a SCEVUse instead of a const SCEV*. Use it for SCEVExpander, so it can
use use-specific flags in the future.
PR: https://github.com/llvm/llvm-project/pull/188863
This was only called on CondBr instructions, where it is always faster
to access the successors directly than to use successors().
Multi-edges don't dominate anything, so this rare case is often already
handled by dominates().
There is also a very small (hardly measurable) performance
improvement here (it did show up in profiles at 0.03% or so).
Restrict PHI nodes that getRangeRef is allowed to recursively examine so
we don't need a "visited" set. And fix createSCEVIter so it creates all
the relevant SCEV nodes before getRangeRef tries to examine them.
The tests that are affected have induction variables that aren't
AddRecs. (Other cases are theoretically affected, but don't seem to show
up in our tests.)
Add a ConstantExpr::getPtrAdd() API that creates a getelementptr i8
constant expression, similar to IRBuilder::CreatePtrAdd(). In the future
this will create a ptradd expression.
