Commit graph

2985 commits

Author SHA1 Message Date
Adam Bzowski
30611cae03
[BOLT] Fix RELR addend updates for non-zero image bases (#213000)
This patch fixes an AArch64 BOLT failure exposed by LLVM commit
`fbba327208d7f6994f9a28a51dca3e913e3b444a`, which enabled packed
relative relocations (RELR) globally. BOLT incorrectly used a virtual
relocation address as a file offset when updating RELR addends,
corrupting instrumented executables when their image base was non-zero.
The observed CMake compiler-check failure was only a consequence of the
corrupted BOLT-instrumented Clang executable crashing; the defect itself
is in BOLT, not CMake.

The regression test links an AArch64 PIE with a non-zero image base and
verifies that BOLT updates the RELR addend at the correct file offset.

Assisted-by: Codex
2026-07-30 23:03:38 +01:00
Rafael Auler
5c57064076
[BOLT] Stop materializing .dwo DIE vectors early in the pipeline (#207292)
Summary: preprocessDWODebugInfo() eagerly force-extracted every .dwo
compile unit's DIE tree (getNonSkeletonUnitDIE(false)) very early in
BOLT pipeline, way before DWARFRewriter kicked in. Those vectors then
sit in memory throughout the entire rewrite pipeline, directly
contributing to BOLT's RSS peak. I did a fair amount of digging and
didn't find any reason as to why we need to keep all DIEs of DWO CU
materialized at all, since DWARFRewriter won't even read this vector
(the #197359 concurrency fix did use that, but that is unnecessary). The
problem is that these DIE trees are a massive contribution to RSS when
processing large binaries where we have 10s of K of dwos, storing
complete trees for each processed dwo.

This diff changes the #197359 concurrency fix to not rely on the DIE
sibling/children structure. It parses DWP type units selectively per
compile unit (DIEBuilder::buildDWPTypeUnitsForUnit ->
collectReferencedTypeSignatures) by finding the DW_FORM_ref_sig8
references in a unit's DIEs to decide which type units belong in that
unit's output .dwo. That walk previously used DWARFDie::children(),
which requires the unit's full DIE vector. Here we rewrite the walk to
stream the unit's DIEs one at a time with
DWARFDebugInfoEntry::extractFast (the same technique already used by
DIEBuilder::constructFromUnit and DWARFRewriter::partitionCUs), reading
DW_FORM_ref_sig8 attributes off a single reusable transient entry. The
tree structure is irrelevant -- every DIE in the unit is visited
regardless -- so no DIE vector is
built. collectReferencedTypeSignatures now takes a DWARFUnit& instead of
a DWARFDie.

With the walk self-sufficient:
- preprocessDWODebugInfo() now extracts only the .dwo CU DIE
(getNonSkeletonUnitDIE(true)); nothing reads the full array off it
anymore (constructFromUnit and the signature walk both stream).
- BinaryContext::collectDebugScopeBoundaries() drops its split-DWARF
fast-path, which called DWARFUnit::dies() (= full extraction); DWO units
are now streamed like monolithic ones.

The result is that .dwo DIE vectors are never materialized during BOLT
processing.

Expected to be a ~10% RSS win on large split-dwarf binaries.

Depends on #207291
2026-07-29 11:50:02 -07:00
Paschalis Mpeis
b6b9603c76
[RFC][BOLT] Enable lite mode by default on AArch64 (#205545) 2026-07-28 15:52:44 +01:00
Rafael Auler
63e60a2853
[BOLT] Fix strict mode bug in removal of block referenced by JT (#211940)
Summary:
BOLT can create a jump table object from a PC-relative operand reference
(e.g. leaq JT(%rip)) whenever the referenced memory looks like a PIC
jump table -- see
BinaryContext::handleAddressRef. This is independent of whether BOLT
recognizes the indirect jump dispatch itself. The jump table annotation
is only attached to the jmp instruction later, in
BinaryFunction::analyzeIndirectBranch, and only when the target-specific
analyzeIndirectBranch matcher recognizes the dispatch pattern.

If the matcher does not recognize the pattern, the annotation is never
attached to the jmp, even though the jump table object and its entries
exist and reference basic blocks in the function.

In strict mode BOLT still fully processes such a function: it stays
simple, the block is marked as having unknown control flow, and the jump
table object is kept intact. Because the terminator carries no jump
table annotation, BinaryBasicBlock::hasJumpTable() returns false for it.
Later passes then transform the function normally: remove-nops can empty
a jump table target block that consists only of nops, and NormalizeCFG
then redirects the block's predecessor and deletes the empty block - it
does not recognize the block as a jump table target, since the
predecessor's hasJumpTable() is false. The jump table object still
references the deleted block by label, which produces an "Undefined
temporary symbol" error and fails emission.

Fix NormalizeCFG so it does not redirect/remove a block whose
predecessor ends in an indirect branch, not only one flagged by
hasJumpTable().
2026-07-27 12:35:54 -07:00
Rafael Auler
c881d81f6b
[BOLT] Release pseudo-probe decoder after emit (#211935)
PseudoProbeRewriter builds sizable data structures and then hold onto
them after they are needed.
PseudoProbeRewriter::postEmitFinalizer() parses the input .pseudo_probe
sections into an MCPseudoProbeDecoder whose address-to-probe and
GUID-to-function-desc maps can be very large (tens of GiB on big
binaries with many probes). This is not used again once probes have been
updated, yet they survive into the memory-heavy DWARF rewrite
(updateDebugInfo), directly inflating BOLT's peak RSS at the worst
possible time.

This frees them at the end of the postEmitFinalizer() calls, before
updateDebugInfo runs.

Stacked on top of the .dwo DIE diff, in large binaries you should
observe ~17% peak RSS wins if your
build uses pseudo probe maps.
2026-07-27 12:34:36 -07:00
Elvina Yakubova
f901499b11
[BOLT] Split unmarked tail code after CFI-bounded functions into separate functions (#176173)
Detect trailing unmarked code after CFI-bounded predecessors
(__BOLT_FDE_FUNC* or symbols whose FDE address range equals their size)
and promote it into separate BinaryFunction objects named
__BOLT_UNMARKED_TAILat<addr>.

A predecessor is only considered when:
  - the target is AArch64
  - it is not a pseudo or fragment function
  - it has slack beyond its recorded size (MaxSize > Size)
  - it is CFI-bounded
  - no function is already registered at the tail start address
  
The trailing region is measured conservatively and we bail out (leaving
the slack attached to the predecessor) when:
- the tail range is not fully contained in the predecessor's origin
section
- a data marker ($d) or a constant island overlaps the region
(data-in-code)
  - disassembly fails or yields a zero-length instruction
- after trimming trailing padding, the tail is empty (i.e. it contained
only padding) or does not end in a return
- any bytes past the decoded code are non-zero, i.e. not pure zero
padding.

The bug was introduced in commit
[cd27741](cd27741c11)
2026-07-27 10:17:02 +01:00
Fangrui Song
571a1deb31
[LoopInfo] Store blocks using Euler tour representation (#211485)
Block list construction appends each block to all enclosing loops
(O(depth) per block), and each loop owns a separately allocated vector.

Switch to an Euler tour representation: one BlockLayout array per
LoopInfo holding the in-loop blocks in a loop-contiguous reverse
postorder, each loop's list a [begin, end) slice of it, subloop slices
nested inside their parent's. Headers remain first;
`SubLoops`/`TopLevelLoops` orders are unchanged. A few tests observe the
order and are updated. (Previously a loop's list was the function's
reverse postorder restricted to its members, so a subloop's blocks were
interleaved with the parent's own blocks at their RPO positions.)

Mutations first copy a borrowed slice into private storage from
LoopInfo's allocator. `contains(BlockT *)` remains map-based: a
materialized loop's slice is not a membership test. This copy is rare:
across default<O2> over dagcombiner and sqlite3, only 12--14% of loops
built by analyze() are ever mutated; the rest keep the borrowed slice.

AllocateLoop() no longer forwards constructor arguments; analyze() uses
allocateLoop(Header), removing the header-taking loop constructors.

Aided by Claude Fable 5
2026-07-25 01:57:01 +00:00
Jinjie Huang
82488f0e71
[BOLT] Fix pwrite assertion failure via a new safePWrite wrapper (#198569)
Background: Currently, BOLT seems to implicitly assume that the .dynsym
section is located at a low offset within the binary, calling pwrite()
directly to update it.

Issue: In scenarios where the binary has been modified by tools like
patchelf, sections like .dynsym may be moved to a high offset area. This
can lead to a violation of the Offset + Size <= Pos assertion in
pwrite(). A typical scenario is when the previous eh_frame_header update
moves the stream cursor (pos) back to a low
offset([code](https://github.com/llvm/llvm-project/blob/llvmorg-23-init/bolt/lib/Rewrite/RewriteInstance.cpp#L6387)).

Fix: This patch resolves the pwrite assertion failure via a new
safePWrite wrapper, which introduces a defensive check that verifies and
conditionally adjusts the stream position. A corresponding test case has
also been added.
2026-07-24 19:55:26 +08:00
Thrrreeee
9e1be60f66
[BOLT][DebugInfo] Add DWARF64 support for BOLT (#206437)
BOLT currently assumes the DWARF32 format when it rewrites debug
sections. As a result, rewriting a binary that was compiled with
`-gdwarf64` either produces a corrupted output or fails `llvm-dwarfdump
--verify`. This patch adds DWARF64 support to the BOLT DWARF rewriter,
honoring each unit's DwarfFormat and emitting correct encodings DWARF64
for binaries, including mix DWARF32 and DWARF64.
2026-07-24 15:07:52 +08:00
Amir Ayupov
a8228369ca
[BOLT][docs] New backend guide (#211382)
Add a brief doc with BOLT-specific guidance for adding a new backend.
2026-07-22 14:44:36 -07:00
aokblast
46858d55e8
[BOLT] Implement platform independent distribution and shuffle algorithm (#206554)
std::uniform_int_distribution and std::shuffle are
implementation-defined, so binary built with libc++ and libstdc++ can
produce different fragment layouts when using SplitStrategy. In this
case, the FreeBSD build emits only one fragment instead of three.

The output geenrate by --bolt-seed should be deterministic for the same
binary regardless of the standard library implementation. Implement a
portable shuffle and uniform distribution to guarantee identical results
for a given seed.

After this change, we generate 1 fragment, which decrease the strength
of the testcase. As a result, we change the seed to different value to
allow it generate different number of fragments.

Also, replace in-place file modifcation with a write-and-replace
approach for better portability in sed.
2026-07-22 11:25:26 +08:00
Thrrreeee
3f0919e8d9
[BOLT][RISCV] Support the TLS global-dynamic relocation (#209995)
This patch adds BOLT support for the RISC-V TLS global-dynamic
relocation `R_RISCV_TLS_GD_HI20`. Classifying it as both a **TLS
relocation** and a **GOT-style relocation** allows BOLT to preserve the
relocation, recover the actual GOT entry address from the linked
instructions, and symbolize the instruction pair.

The RISC-V TLS relocation test is extended to cover a global-dynamic
sequence and verify that BOLT reconstructs the `AUIPC`/`PCREL_LO12` pair
correctly.
2026-07-21 17:25:01 -07:00
Jon Roelofs
3421dc247e
[MC][AArch64] Add .cfi_set_ra_state for PAuth_LR (#209949)
The new CFI directive directly assigns an RA signing state to the RA_SIGN_STATE DWARF pseudo-register for use when unwinding, to indicate whether the value of PC has been used as a diversifier for return address signing. The new directive subsumes and replaces .cfi_negate_ra_state_with_pc, which was found to be unsuitable for descibing some block layouts [1], particularly in hot-cold-split functions.

1: https://github.com/ARM-software/abi-aa/pull/346
2026-07-21 14:22:36 -07:00
Fangrui Song
9a49fc244b
[LoopInfo] Replace raw loop-container accessors with semantic operations (#210653)
getBlocksVector, mutable getBlocksSet, getSubLoopsVector, and
getTopLevelLoopsVector expose LoopInfo's internal containers, so every
caller open-codes loop-nest surgery, keeps the block vector and set in
sync by hand, and freezes the representation. Replace them with two
LoopInfoBase operations:

* removeBlocksIf(LoopT &, Pred): drop matching blocks from a loop.
* takeChildrenIf(LoopT *Parent, Pred): detach and return matching
  children, clearing their parent; a null Parent means the top level.

Both preserve the relative order of what remains. Mutation inside
LoopInfoBase also lets the analysis repair derived state once per edit
and finish a multi-step edit before any query observes it, which an
accessor returning a container cannot.

For the same reason, move hasNoExitBlocks, getExitEdges, and
getUniqueLatchExitBlock onto LoopInfoBase.

Two behavior changes: SimpleLoopUnswitch's deleteDeadBlocksFromLoop runs
its per-child deletion callbacks while the forest is still consistent,
before detaching and destroying dead children; FixIrreducible's
reconnectChildLoops keeps children in their original order rather than
std::partition's.

Aided by Claude Fable 5
2026-07-21 07:01:19 +00:00
YongKang Zhu
9c024f3a83
[BOLT] Drop option --instrument-funcs-file (#210217) 2026-07-20 08:35:14 -07:00
Amir Ayupov
6b9929001d
[BOLT] Add fdata symbols mode (#209932)
Add a new fdata profile mode accepting symbol names as-is.

It addresses the UX gap surfaced by #208313 that all methods of selecting
functions to process (symbolized profile types, `-funcs*` options) expect
disambiguated local symbols. 

Introduce fdata mode activated by a `symbols` header line, where each subsequent
line is a (mangled) symbol name. The names are matched against restored function
names (with disambiguation suffix stripped), and every matching function gets an
execution count of 1. 

This can be used to drive selective instrumentation (`--instrument-hot-only`) or
basic linker-script style optimizations.

Symbols are stored as branch entry data so matching reuses the existing default
branch profile path.

Test Plan:
Update docs/profiles.md and add profile-symbols-mode.s
2026-07-16 17:31:30 -07:00
Reid Kleckner
03ab16cb41
[docs] Finish MyST migration for BOLT docs (#210199)
Tracking issue: #201242
See the [migration guide] for more information. 

[migration guide]:
https://llvm.org/docs/SphinxQuickstartTemplate.html#markdown-migration-guidelines
This is a stacked PR based on #210198, which will be a standalone commit
that
renames *.rst -> *.md before this PR lands for history preservation
purposes.

This was prepared with rst2myst plus LLM-assisted cleanup. I paged
through all the generated HTML looking for migration artifacts, and all
of the differences I could find appear to be formatting error
corrections. Please spot check my work and approve if it looks good. You
can use the HTML links below to confirm it renders properly.

-----

Before/after validation links:
| Source file | Before HTML | After HTML |
| --- | --- | --- |
| `bolt/docs/BOLTAArch64OptimizationStatus.md` |
[before](https://llvmdocs.staging.reidkleckner.dev/bolt/docs-old/BOLTAArch64OptimizationStatus.html)
|
[after](https://llvmdocs.staging.reidkleckner.dev/bolt/docs/BOLTAArch64OptimizationStatus.html)
|
| `bolt/docs/GettingStarted.md` |
[before](https://llvmdocs.staging.reidkleckner.dev/bolt/docs-old/GettingStarted.html)
|
[after](https://llvmdocs.staging.reidkleckner.dev/bolt/docs/GettingStarted.html)
|
| `bolt/docs/index.md` |
[before](https://llvmdocs.staging.reidkleckner.dev/bolt/docs-old/index.html)
|
[after](https://llvmdocs.staging.reidkleckner.dev/bolt/docs/index.html)
|
2026-07-16 23:57:41 +00:00
Reid Kleckner
0fc03f0454
[bolt][docs] Rename BOLT docs to Markdown (#210198)
Tracking issue: #201242
See the [migration guide] for more information. 

[migration guide]:
https://llvm.org/docs/SphinxQuickstartTemplate.html#markdown-migration-guidelines

This is the initial straight rename commit. It will probably break the
docs build, but it has to be a separate PR for blame preservation
purposes.
2026-07-16 16:46:21 -07:00
nitro
e5c19b0ebc
[BOLT] Skip function instead of aborting on jump table analysis failure (#206742)
`analyzeJumpTable()` runs twice on a candidate jump table: once from
`analyzeMemoryAt` while the referencing function is being disassembled,
and
again from `populateJumpTables()` after disassembly. The second run is
stricter,
because the check that an entry points at an instruction
(`getInstructionAtOffset`) only fires once the target function reaches
the
`Disassembled` state. A table can therefore be accepted the first time
and
rejected the second -- for instance when an entry lands at an address
with no
instruction (a symbol whose declared size is larger than its
disassembled
extent). `populateJumpTables()` reached `llvm_unreachable("jump table
heuristic
failure")` and aborted in that case.

Handle it the way BOLT already handles indirect branches it can't
analyze: drop
the table and skip the owning function(s) via `addFragmentsToSkip()`.
The
`JumpTable` is deliberately left in the maps so it is still freed by
`~BinaryContext` (an earlier attempt at this, 52cd00ca, was reverted in
468d4f6d
because it erased the table without deallocating it).

New test `jump-table-failed-reanalysis.s` builds such a table; before
this change
BOLT aborts, after it the function is skipped and the rest of the binary
is
processed normally.
2026-07-15 14:50:26 -07:00
Aiden Grossman
b1a35022e0
[BOLT] Remove ability to use external shell
The external shell is getting removed soon, so remove the ability to
force the use of it within BOLT.

Reviewers:
aaupov, petrhosek, maksfb, yozhu, paschalis-mpeis, yavtuk, ilovepi, rafaelauler, ayermolo

Pull Request: https://github.com/llvm/llvm-project/pull/209567
2026-07-14 18:12:11 -07:00
Rafael Auler
6dc5473fa6
[BOLT] Fix shifted DWARF inline-scope ranges; track scope boundaries (#207291)
Summary:
BOLT updated DWARF lexical-scope ranges (DW_TAG_inlined_subroutine /
lexical_block low_pc/high_pc and DW_AT_ranges) via
translateInputToOutputRange(), which mapped a boundary using its input
offset relative to the start of the containing basic block:

OutAddr = BB.getOutputAddressRange().first + (InputOffset -
BB.getOffset())

This assumes intra-block byte offsets are preserved input->output. Any
pass that changes instruction sizes within a block ahead of a scope
boundary breaks that assumption. With --plt=all, each `call foo@PLT` (5
bytes, e8+rel32) is rewritten to `call *foo@GOT(%rip)` (6 bytes, ff
15+rel32); N such calls before a boundary shift its emitted low_pc/
high_pc N bytes too early, onto the preceding instruction. The range
stays within the parent so `llvm-dwarfdump --verify` does not catch it;
symbolizers then attribute samples on those instructions to the wrong
inlined frames.

Fix is to resolve scope-range boundaries through the precise
per-instruction input-to-output AddressMap (the same map BAT already
uses) instead of input-relative block offsets. Boundary instructions are
arbitrary (not just calls/branches), so they are normally absent from
that map; to make them resolvable, disassembly now keeps an offset for
the boundary instructions (only the boundaries, to keep the AddressMap
small) and those offsets are emitted as AddressMap entries whenever an
address map is required (requiresAddressMap(), which includes
--update-debug-sections via requiresPreciseAddressMap()), not only under
BAT.

Boundary collection (BinaryContext::collectDebugScopeBoundaries) runs in
readDebugInfo() after preprocessDebugInfo(). It streams the DIEs of the
CUs that will be updated directly from the already-loaded DWARFContext
with DWARFDebugInfoEntry::extractFast, decoding one DIE at a time into a
single reusable entry, so no DIE forest is materialized and no second
DWARF context is created. For split DWARF the .dwo DIEs are already
extracted by preprocessDWODebugInfo(), so the cached array is reused
instead. Boundaries are stored per function in a sorted, deduplicated
vector (BinaryFunction::DebugScopeBoundaryOffsets), queried during
disassembly with a monotonic cursor, and freed once the function is
disassembled, so the feature adds no global state.

The behavior is on by default and can be disabled with
--accurate-debug-ranges=0, which falls back to the old block-relative
mapping.

Test on big binaries with split dwarf, -lite=0:
                                wall                Max RSS
      -accurate-debug-ranges=0  1070.1s             94.91
      -accurate-debug-ranges=1  1173.0s             100.07
      delta                     +102.9s             +5.16   (+4.6% CPU,
                                                             +5.4% RSS)

Test on big binaries with split dwarf, lite=1:
      -accurate-debug-ranges=0  433.9s              83.23
      -accurate-debug-ranges=1  434.7s              83.79   (+0.7% RSS)

    --time-rewrite shows the boundary-collection phase itself is cheap:
    "read debug ranges" is 5.8s of 4960s total user time on one binary
    (0.1%) and 18.8s of 6307s on another binary; the feature's
cost is dominated by emitting/translating the extra AddressMap entries,
    not by collection.
2026-07-14 16:40:05 -07:00
Sergey Shcherbinin
3f80df8698
[llvm-profgen][test] Split tests by target architecture (#207724)
The top-level llvm-profgen lit configuration previously disabled the
entire test directory unless the X86 target was registered. As a result,
target-independent tests and tests for ARM and AArch64 also implicitly
depended on X86.

Split target-specific tests and inputs into X86, ARM, and AArch64
directories with per-directory lit.local.cfg files. Leave help.test at
the top level because it is target-independent. Existing X86 tests
remain guarded by X86 and run as before, while ARM and AArch64 tests no
longer require the X86 target. The X86 non-ARM ETM rejection case also
no longer requires the ARM target; it only requires X86 and OpenCSD.

Split the former etm-arch.test into ARM/etm-arch.test, containing the
two ARM-specific checks, and X86/etm-non-arm.test, containing the
existing X86 rejection check. The tested inputs and expected diagnostics
are preserved.

Update match-blocks-with-pseudo-probes-inline.test,
pseudoprobe-decoding-inline.test, and pseudoprobe-decoding-noinline.test
in BOLT to refer to the relocated X86 inputs. All other llvm-profgen
tests and inputs are pure renames.

This does not change llvm-profgen runtime behavior; it only removes
unnecessary cross-target dependencies from its test suite.

This is a preparatory change for extending llvm-profgen profiling
support to AArch64 and for introducing the universal profile-guided
automatic software loop data prefetcher described in the RFC:


https://discourse.llvm.org/t/rfc-universal-profile-guided-automatic-software-loop-data-prefetcher-for-llvm/90916
2026-07-13 11:17:29 +05:30
Thrrreeee
171ba71128
[BOLT] fix DIE traversal incorrect loop termination condition. (#208450)
The DIE traversal loop in partitionCUs() used an incorrect termination
condition, causing it to read past the end of the CU. Fix the loop to
stop at NextCUOffset so traversal no longer runs beyond the unit's
boundary.

Fixed [#208440](https://github.com/llvm/llvm-project/issues/208440).
2026-07-13 10:59:06 +08:00
Fabian Parzefall
fe80624dc6
[BOLT] Treat relr relocations like rela (#206830)
The linker can emit certain relative .dyn.rela relocations in .dyn.relr.
This includes relocations for jumptables emitted for indirect goto. The
current processing recognizes these relocations as jump targets only for
relative relocations from .dyn.rela, but not from .dyn.relr. Give
.dyn.relr the same treatment, so CFG construction does not dismiss
blocks that are only reachable indirectly.
2026-07-10 09:26:48 -07:00
Nikita Popov
8d590969ba
[LLD][AArch64] Make adrp+ldr relaxation per-symbol all-or-nothing (#208396)
We can't relax only some adrp+ldr pairs for a symbol, because there may
be a branch target between the adrp and ldr of this form:

        adrp    x1, :got:sym
    .Lfoo:
        ldr     x1, [x1, :got_lo12:sym]
        # ...

        adrp    x1, :got:sym
        ldr     x2, [x1, :got_lo12:sym]
        b       .Lfoo

Relaxing the first adrp+ldr here would be invalid. This was clarified in
the ARM ABI in:
11fb4ef428.

The implementation already performed a pre-scan to check that the
relevant relocations occur in pairs. Change this scan to a) check all
the preconditions for the relaxation and b) make the decision per
symbol.

Fixes https://github.com/llvm/llvm-project/issues/138254.
2026-07-10 07:09:08 +00:00
aokblast
dac462d85c
[BOLT] Fix use-old-text-zero-padding on FreeBSD (#206581)
BSD od supports only decimal value to -N parameter. To fix the test
failure, we use decimal value instead of hex value in this test case.
2026-07-10 15:06:22 +08:00
Fangrui Song
be40b8ec4e
[BOLT] Give BinaryBasicBlock a GraphTraits block number (#207899)
LoopInfoBase (and DominatorTreeBase) keep a DenseMap fallback for block
types whose GraphTraits lacks getNumber() (#103400); BOLT's
BinaryLoopInfo is one of the last two users.

BinaryBasicBlock already carries a dense [0, size) index (getIndex(),
assigned by updateBBIndices()). Expose it as the GraphTraits node
number, mirroring llvm::BasicBlock/Function.

This makes `GraphHasNodeNumbers<BinaryBasicBlock *>` true, moving BOLT's
LoopInfo and DominatorTree onto the number-indexed (SmallVector) path.
Prerequisite for requiring GraphHasNodeNumbers in LoopInfoBase.

Aided by Claude Opus 4.8
2026-07-08 23:46:17 -07:00
Fabian Parzefall
0bd936eee2
[BOLT] Register PIE indirect goto relocations (#206819)
PIE binaries use R_*_RELATIVE dynamic relocations for indirect goto jump
tables. BOLT creates entry points for their targets but does not mark
these addresses as targets from data relocations, so indirect jumps with
unknown control flow (which originates from indirect goto) have no CFG
successors and are interpreted as potential tail calls, while the jump
targets are incorrectly identified as additional entry points.

Add these offsets to the function's list of non-entry relocation
references instead of marking them as entry points. Extend relocation
rewriting to also check whether a block is externally referenced. This
mirrors behavior of data-to-code relocations in non-pie binaries.
2026-07-08 14:21:12 -07:00
Rafael Auler
c7c7cab93e
[BOLT] Fix DW_FORM_implicit_const values lost during DWARF5 rewriting (#192166)
Summary:
Fix two bugs in DIEBuilder that caused DW_FORM_implicit_const values to
be zeroed out when rewriting DWARF5 debug sections
(--update-debug-sections).

1. In constructDIEFast(), DWARFFormValue was constructed with just the
form code, leaving the value at 0. For DW_FORM_implicit_const,
extractValue() is a no-op since the value is expected to be pre-set.
Fix: use AttrSpec.getFormValue() which initializes the value from the
abbreviation table.

2. In assignAbbrev(), AddAttribute(Attr.getAttribute(), Attr.getForm())
used the two-argument overload which discards the implicit_const value.
Fix: use AddAttribute(Attr) to copy the full DIEAbbrevData.

Fixes https://github.com/llvm/llvm-project/issues/192084
2026-07-08 12:27:23 -07:00
Haibo Jiang
c279890c85
[BOLT][AArch64] Support call ICP (#208158)
* extends MCPlusBuilder with comparision between registers.
* updates the ICP pass to request a scavenged register for AArch64
callsites.
* wires ICP code generation on AArch64.
2026-07-08 15:58:30 +08:00
Ádám Kallai
74eebbe02b
[BOLT] Fixed PerfScript unittest for X86 (#207408)
The test was not prepared to run on X86 target.
This change parameterized the test function to ensure proper
initialization for different targets.
2026-07-06 10:44:29 +02:00
David CARLIER
2c4816cc7e
[BOLT][Passes] reserve layout vectors in ExtTSP block reordering. (#206402) 2026-07-02 18:12:18 +01:00
Kito Cheng
09d944e919
[BOLT] Make ICF bucket iteration order deterministic for single-threaded mode (#200706)
`CongruentBuckets` is an unordered_map, so iterating it directly
produces non-deterministic folding order across different build
environments. Sort buckets by the binary address of the representative
function before iterating to guarantee a stable debug output order.

That order isn't matter for the output binary, but it cause randomly
different debug output across different build environments, which makes
it hard to write stable tests for ICF.

Also the debug output of multi-threaded ICF is already non-deterministic
even we sort that by address, so we only try to stabilize the
single-threaded ICF output.

Update the icf-safe tests to reflect the new deterministic ordering.

Assisted-by: Sonnet 4.6
2026-07-01 16:04:17 +08:00
Alexis Engelke
0b413b7d0f
[MC][TableGen] Make MCRegisterClasses relocation-free (#206753)
MCRegisterClasses currently store pointers to the register list and the
bit set. Store these three types together in one data structure and use
relative offsets to avoid these relocations and move the large
MCRegisterClasses array from .data.rel.ro into .data. This reduces the
amount of data that needs to be relocated by 86 KB.

This has two side effects: first, MCRegisterClass is not copyable and
the few uses that did copy were changed. Second, the MCRegisterClasses
array is no longer easily accessible as a global (well, it *technically*
is, but that requires the type of the entire storage struct, which I
don't want to expose). Therefore, these accesses need to go through a
function; which shouldn't be too costly and be inlined in an LTO build.
2026-07-01 07:40:06 +02:00
Fabian Parzefall
9cdea3b9d5
[BOLT] Preserve relocation to rela/relr association (#206612)
BOLT emits all dynamic relative relocations in .relr.dyn if it exists in
the input, including relative relocations from .rela.dyn. This can fail
for two reasons: first, relative relocations can target odd addresses.
These can never be emitted as relr relocations. Second, BOLT reuses the
existing .relr.dyn section. Even if a relative relocation from .rela.dyn
can be emitted as relr, there is simply no space in the existing
.relr.dyn section. This causes an overflow. Prevent this by emitting
relative relocations in the same section they originate from.
2026-06-30 12:49:40 -07:00
Petr Hosek
e192ec42b4
[CMake] Warn when the version of CMake is older than 3.31.0 (#204203)
This is a preparation to raise the minimum required CMake version to
3.31.0 in LLVM 24.

See
https://discourse.llvm.org/t/rfc-raising-minimum-required-cmake-version-to-3-31/91086.
2026-06-29 11:16:27 -07:00
Amina Chabane
a9a58990ed
[BOLT][Documentation] Prepare documentation for hosting (#201378)
This PR aims to organise BOLT's docs in preparation for hosting.

- Added a landing page into `index.rst`, much like
[clang.llvm.org](https://clang.llvm.org/docs/index.html).
- Moved Getting Started content from `index.rst` into its own
`GettingStarted.rst`.
- Enables Markdown content to be rendered. A current
[discussion](https://discourse.llvm.org/t/rfc-make-myst-markdown-the-llvm-docs-format-rip-rest/90840)
is being had about MD/RST on discourse, but for now I have decided to
support the mix of formats currently in BOLT.
- Included AArch64 optimization status page.
- Fixes some links/anchors.

**Landing Page**
<img width="531" height="311" alt="image"
src="https://github.com/user-attachments/assets/4fa376bf-cdf1-4ad1-8be5-1de57597ac10"
/>


**Build with Sphinx**
`sphinx-build -b html bolt/docs bolt-docs-html`
Requires Sphinx and myst_parser.
2026-06-28 21:20:57 +08:00
Alexis Engelke
3655c98da1
[Support][Object/ELF][NFC] Use new enum table (#206068)
Straight forward replacement.
2026-06-28 07:13:53 +02:00
aokblast
f99a584a7c
[BOLT] Work around BSD sed's lack of in-place editing support (#206183)
BSD sed does not implement `-i` the same way as GNU sed. Use a
copy-and-replace approach instead of in-place editing to ensure
compatibility.
2026-06-28 00:29:54 +08:00
Scott Linder
42ff45b865
[docs] Create utils/docs (#203962)
llvm-project is home to many sphinx documentation sites, each with
configuration quirks and bespoke extensions.

The sphinx config model makes sharing code somewhat difficult. There
are options like sphinx-multiproject, but some of our docs builds are
out of the source tree while some are done out of the binary tree, so
the multiproject configuration itself would need to be generated. It
also would impose more uniformity around extensions than required.

This change instead creates a python package at utils/docs/llvm_sphinx
and makes it available to all sphinx-build processes via PYTHONPATH.
Each conf.py does not modify its own sys.path because not all builds are
out of the source tree, so there isn't a stable relative path to use to
refer to the utils/docs/ directory.

Type checking via pyright in new package is pinned to being python 3.8
compatible.

The myst slug script from llvm/docs is promoted to be the default,
with a slight edit to targets needed in FlangDriver.md to adapt to the
change.

The ghlinks extension from clang gets lifted to be the first generic
extension, although it is still only used in clang currently. I used it
as an example of a general extension that might be useful for other
projects, but I also intend to add a new generic extension in a future
patch.

As part of the move ghlinks is refactored a bit:

* Rather than have a main function directly to run its tests, those
  tests are now run via the command-line entrypoint to the new package,
  e.g. `python utils/docs --test` will now replace
  `python clang/docs/ghlinks.py --test`. This makes the command-line
  more pithy (`python utils/docs/llvm_sphinx/ext/ghlinks --test` is a
  bit long), and makes the default behavior when running be to
  include utils/docs in the sys.path, so imports can be absolute and
  the behavior is as close to the CMake targets as possible.
* The static files that were previously written to a tempdir during
  testing are just committed directly in git. The assumption here is
  that the original version was taking pains to avoid polluting the
  clang/docs directory with test files that would confuse sphinx-build,
  and without that limitation the workaround can be removed.
* Some typing-related changes to make it strictly python 3.8 compatible.

Change-Id: Ie16de273f74973f163dbf7fe9d22d80a466d051f
2026-06-24 15:12:48 -04:00
Fabian Parzefall
89ff26442d
[BOLT] Replace partial instructions with traps in patched entries (#205211)
Overwriting a function entry with a jump is likely to not perfectly
align with the instruction stream. If the end of the patch does not
fall onto an instruction boundary, the bytes following the jump are
orphaned and will have nonsensical interpretations. This can leave
other tools confused, especially since these orphaned bytes can
decode to instructions that do not nicely rejoin the still intact
part of the instructions stream. Overwrite these bytes with traps
in the PatchEntry pass.

Fixes #198455.
2026-06-24 11:48:20 -07:00
Paul Osmialowski
55b45709a8
[BOLT] Increase BufSize in runtime/common.h (#204607)
During my work towards bolring the flang binary, I've encountered a
frequently occuring problem with running out of the buffer space.

The problem affects C++ programs with a decent number of very long
symbol names, which is inevitable when using template metaprogramming.
As one can clearly see, flang is one of such programs.

The proposed BufSize value is an effect of the trial-and-error process
aiming at finiding the smallest reasonable increase.
2026-06-24 18:00:27 +01:00
David CARLIER
4de5455b77
[NFC] use DenseMap/SmallPtrSet in CacheMetrics and TailDupli… (#205480)
…cation

Swap pointer-keyed std::unordered_map/std::set for their ADT equivalents
on hot paths.
2026-06-24 13:25:09 +01:00
DmitriiMartynov
18ce6f449e
[BOLT][AArch64] reproducible output with constant islands (#204546)
Optimized binaries from subsequent llvm-bolt runs may sometimes differ
due to the unordered set (SmallPtrSet), even if the input binary and
parameters are the same. Usage of SetVector guarantees a deterministic
sequence of binary functions while keeping each function as a single
instance.

Below you can see two different main functions before the fix after two
llvm-bolt runs (same input binaries, same arguments).

```
0000000000210400 <main>:
  210400: 10000140      adr     x0, 0x210428 <main+0x28>
  210404: f9400000      ldr     x0, [x0]
  210408: 10000140      adr     x0, 0x210430 <main+0x30>
  21040c: f9400000      ldr     x0, [x0]
  210410: 10000180      adr     x0, 0x210440 <main+0x40>
  210414: f9400000      ldr     x0, [x0]
  210418: 10000180      adr     x0, 0x210448 <main+0x48>
  21041c: f9400000      ldr     x0, [x0]
  210420: d65f03c0      ret
  210424: d503201f      nop
  210428: 00000010      udf     #0x10
  21042c: 00000000      udf     #0x0
  210430: 00000020      udf     #0x20
  210434: 00000000      udf     #0x0
  210438: d503201f      nop
  21043c: d503201f      nop
  210440: 00000030      udf     #0x30
  210444: 00000000      udf     #0x0
  210448: 00000040      udf     #0x40
  21044c: 00000000      udf     #0x0
  210450: d503201f      nop
  210454: d503201f      nop

0000000000210400 <main>:
  210400: 100001c0      adr     x0, 0x210438 <main+0x38>
  210404: f9400000      ldr     x0, [x0]
  210408: 100001c0      adr     x0, 0x210440 <main+0x40>
  21040c: f9400000      ldr     x0, [x0]
  210410: 100000c0      adr     x0, 0x210428 <main+0x28>
  210414: f9400000      ldr     x0, [x0]
  210418: 100000c0      adr     x0, 0x210430 <main+0x30>
  21041c: f9400000      ldr     x0, [x0]
  210420: d65f03c0      ret
  210424: d503201f      nop
  210428: 00000030      udf     #0x30
  21042c: 00000000      udf     #0x0
  210430: 00000040      udf     #0x40
  210434: 00000000      udf     #0x0
  210438: 00000010      udf     #0x10
  21043c: 00000000      udf     #0x0
  210440: 00000020      udf     #0x20
  210444: 00000000      udf     #0x0
  210448: d503201f      nop
  21044c: d503201f      nop
```
2026-06-23 13:54:45 +03:00
Daniel Hill
ef752a0780
[BOLT] Emit .eh_frame_hdr before .eh_frame to prevent table scans from libdw (#201917)
resolves https://github.com/llvm/llvm-project/issues/201701 by moving
the `.eh_frame_hdr` section-header entry before `.eh_frame` before
assigning final section indices.

This lets BOLT binaries work with existing [libdw library logic
expecting the index before the
table](67199e1c97/libdw/dwarf_getcfi_elf.c (L301)),
preventing linear `.eh_frame` scanning.
2026-06-22 14:15:51 -07:00
YongKang Zhu
1f3d8cba70
[BOLT] Route alignment options through BinaryContext (NFC) (#204902)
Consolidate all alignment-related `cl::opt` definitions into
`CommandLineOpts.cpp`, expose matching public members on `BinaryContext`,
and populate them from `RewriteInstance::adjustCommandLineOptions` (and
mirrored in `MachORewriteInstance`). Switch all readers in Aligner,
BinaryEmitter, LongJmp, BinaryFunction and the use-old-text logs to
`BC.*` instead of `opts::*`.
2026-06-22 13:47:07 -07:00
Fangrui Song
2bb8a56e7d
[MC] emitCodeAlignment: take MCSubtargetInfo by reference. NFC (#205140)
The fragment member cannot be null, and the sibling streamer hooks
(emitInstruction, initSections, emitPrefAlign) already take it by
reference.
2026-06-22 17:48:02 +00:00
YongKang Zhu
e47530b9f4
[BOLT][AArch64] Align tentative layout bases using per-section alignment (#204262)
Move `AssignSections` pass before `AlignerPass` so it can record the max
code alignment per output section, then align the tentative hot/cold
section bases using the recorded alignment, which makes tentative layout
better match actually emitted.
2026-06-19 19:35:43 -07:00
David CARLIER
abbb031d38
[BOLT][rewrite] warn about functions without CFG before binary analyses. (#197294) 2026-06-19 17:02:11 +01:00
Petar Avramovic
bf18f6f430
[CommandLine] Make cl::boolOrDefault a scoped enum (#204553)
Prevents implicit conversion to bool/int, where BOU_FALSE wrongly
evaluated as true. All uses qualified as cl::boolOrDefault::BOU_*.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-06-19 16:25:25 +02:00