This patch fixes an AArch64 BOLT failure exposed by LLVM commit
`fbba327208d7f6994f9a28a51dca3e913e3b444a`, which enabled packed
relative relocations (RELR) globally. BOLT incorrectly used a virtual
relocation address as a file offset when updating RELR addends,
corrupting instrumented executables when their image base was non-zero.
The observed CMake compiler-check failure was only a consequence of the
corrupted BOLT-instrumented Clang executable crashing; the defect itself
is in BOLT, not CMake.
The regression test links an AArch64 PIE with a non-zero image base and
verifies that BOLT updates the RELR addend at the correct file offset.
Assisted-by: Codex
Summary:
BOLT can create a jump table object from a PC-relative operand reference
(e.g. leaq JT(%rip)) whenever the referenced memory looks like a PIC
jump table -- see
BinaryContext::handleAddressRef. This is independent of whether BOLT
recognizes the indirect jump dispatch itself. The jump table annotation
is only attached to the jmp instruction later, in
BinaryFunction::analyzeIndirectBranch, and only when the target-specific
analyzeIndirectBranch matcher recognizes the dispatch pattern.
If the matcher does not recognize the pattern, the annotation is never
attached to the jmp, even though the jump table object and its entries
exist and reference basic blocks in the function.
In strict mode BOLT still fully processes such a function: it stays
simple, the block is marked as having unknown control flow, and the jump
table object is kept intact. Because the terminator carries no jump
table annotation, BinaryBasicBlock::hasJumpTable() returns false for it.
Later passes then transform the function normally: remove-nops can empty
a jump table target block that consists only of nops, and NormalizeCFG
then redirects the block's predecessor and deletes the empty block - it
does not recognize the block as a jump table target, since the
predecessor's hasJumpTable() is false. The jump table object still
references the deleted block by label, which produces an "Undefined
temporary symbol" error and fails emission.
Fix NormalizeCFG so it does not redirect/remove a block whose
predecessor ends in an indirect branch, not only one flagged by
hasJumpTable().
Detect trailing unmarked code after CFI-bounded predecessors
(__BOLT_FDE_FUNC* or symbols whose FDE address range equals their size)
and promote it into separate BinaryFunction objects named
__BOLT_UNMARKED_TAILat<addr>.
A predecessor is only considered when:
- the target is AArch64
- it is not a pseudo or fragment function
- it has slack beyond its recorded size (MaxSize > Size)
- it is CFI-bounded
- no function is already registered at the tail start address
The trailing region is measured conservatively and we bail out (leaving
the slack attached to the predecessor) when:
- the tail range is not fully contained in the predecessor's origin
section
- a data marker ($d) or a constant island overlaps the region
(data-in-code)
- disassembly fails or yields a zero-length instruction
- after trimming trailing padding, the tail is empty (i.e. it contained
only padding) or does not end in a return
- any bytes past the decoded code are non-zero, i.e. not pure zero
padding.
The bug was introduced in commit
[cd27741](cd27741c11)
Block list construction appends each block to all enclosing loops
(O(depth) per block), and each loop owns a separately allocated vector.
Switch to an Euler tour representation: one BlockLayout array per
LoopInfo holding the in-loop blocks in a loop-contiguous reverse
postorder, each loop's list a [begin, end) slice of it, subloop slices
nested inside their parent's. Headers remain first;
`SubLoops`/`TopLevelLoops` orders are unchanged. A few tests observe the
order and are updated. (Previously a loop's list was the function's
reverse postorder restricted to its members, so a subloop's blocks were
interleaved with the parent's own blocks at their RPO positions.)
Mutations first copy a borrowed slice into private storage from
LoopInfo's allocator. `contains(BlockT *)` remains map-based: a
materialized loop's slice is not a membership test. This copy is rare:
across default<O2> over dagcombiner and sqlite3, only 12--14% of loops
built by analyze() are ever mutated; the rest keep the borrowed slice.
AllocateLoop() no longer forwards constructor arguments; analyze() uses
allocateLoop(Header), removing the header-taking loop constructors.
Aided by Claude Fable 5
Background: Currently, BOLT seems to implicitly assume that the .dynsym
section is located at a low offset within the binary, calling pwrite()
directly to update it.
Issue: In scenarios where the binary has been modified by tools like
patchelf, sections like .dynsym may be moved to a high offset area. This
can lead to a violation of the Offset + Size <= Pos assertion in
pwrite(). A typical scenario is when the previous eh_frame_header update
moves the stream cursor (pos) back to a low
offset([code](https://github.com/llvm/llvm-project/blob/llvmorg-23-init/bolt/lib/Rewrite/RewriteInstance.cpp#L6387)).
Fix: This patch resolves the pwrite assertion failure via a new
safePWrite wrapper, which introduces a defensive check that verifies and
conditionally adjusts the stream position. A corresponding test case has
also been added.
BOLT currently assumes the DWARF32 format when it rewrites debug
sections. As a result, rewriting a binary that was compiled with
`-gdwarf64` either produces a corrupted output or fails `llvm-dwarfdump
--verify`. This patch adds DWARF64 support to the BOLT DWARF rewriter,
honoring each unit's DwarfFormat and emitting correct encodings DWARF64
for binaries, including mix DWARF32 and DWARF64.
std::uniform_int_distribution and std::shuffle are
implementation-defined, so binary built with libc++ and libstdc++ can
produce different fragment layouts when using SplitStrategy. In this
case, the FreeBSD build emits only one fragment instead of three.
The output geenrate by --bolt-seed should be deterministic for the same
binary regardless of the standard library implementation. Implement a
portable shuffle and uniform distribution to guarantee identical results
for a given seed.
After this change, we generate 1 fragment, which decrease the strength
of the testcase. As a result, we change the seed to different value to
allow it generate different number of fragments.
Also, replace in-place file modifcation with a write-and-replace
approach for better portability in sed.
This patch adds BOLT support for the RISC-V TLS global-dynamic
relocation `R_RISCV_TLS_GD_HI20`. Classifying it as both a **TLS
relocation** and a **GOT-style relocation** allows BOLT to preserve the
relocation, recover the actual GOT entry address from the linked
instructions, and symbolize the instruction pair.
The RISC-V TLS relocation test is extended to cover a global-dynamic
sequence and verify that BOLT reconstructs the `AUIPC`/`PCREL_LO12` pair
correctly.
Add a new fdata profile mode accepting symbol names as-is.
It addresses the UX gap surfaced by #208313 that all methods of selecting
functions to process (symbolized profile types, `-funcs*` options) expect
disambiguated local symbols.
Introduce fdata mode activated by a `symbols` header line, where each subsequent
line is a (mangled) symbol name. The names are matched against restored function
names (with disambiguation suffix stripped), and every matching function gets an
execution count of 1.
This can be used to drive selective instrumentation (`--instrument-hot-only`) or
basic linker-script style optimizations.
Symbols are stored as branch entry data so matching reuses the existing default
branch profile path.
Test Plan:
Update docs/profiles.md and add profile-symbols-mode.s
`analyzeJumpTable()` runs twice on a candidate jump table: once from
`analyzeMemoryAt` while the referencing function is being disassembled,
and
again from `populateJumpTables()` after disassembly. The second run is
stricter,
because the check that an entry points at an instruction
(`getInstructionAtOffset`) only fires once the target function reaches
the
`Disassembled` state. A table can therefore be accepted the first time
and
rejected the second -- for instance when an entry lands at an address
with no
instruction (a symbol whose declared size is larger than its
disassembled
extent). `populateJumpTables()` reached `llvm_unreachable("jump table
heuristic
failure")` and aborted in that case.
Handle it the way BOLT already handles indirect branches it can't
analyze: drop
the table and skip the owning function(s) via `addFragmentsToSkip()`.
The
`JumpTable` is deliberately left in the maps so it is still freed by
`~BinaryContext` (an earlier attempt at this, 52cd00ca, was reverted in
468d4f6d
because it erased the table without deallocating it).
New test `jump-table-failed-reanalysis.s` builds such a table; before
this change
BOLT aborts, after it the function is skipped and the rest of the binary
is
processed normally.
The external shell is getting removed soon, so remove the ability to
force the use of it within BOLT.
Reviewers:
aaupov, petrhosek, maksfb, yozhu, paschalis-mpeis, yavtuk, ilovepi, rafaelauler, ayermolo
Pull Request: https://github.com/llvm/llvm-project/pull/209567
Summary:
BOLT updated DWARF lexical-scope ranges (DW_TAG_inlined_subroutine /
lexical_block low_pc/high_pc and DW_AT_ranges) via
translateInputToOutputRange(), which mapped a boundary using its input
offset relative to the start of the containing basic block:
OutAddr = BB.getOutputAddressRange().first + (InputOffset -
BB.getOffset())
This assumes intra-block byte offsets are preserved input->output. Any
pass that changes instruction sizes within a block ahead of a scope
boundary breaks that assumption. With --plt=all, each `call foo@PLT` (5
bytes, e8+rel32) is rewritten to `call *foo@GOT(%rip)` (6 bytes, ff
15+rel32); N such calls before a boundary shift its emitted low_pc/
high_pc N bytes too early, onto the preceding instruction. The range
stays within the parent so `llvm-dwarfdump --verify` does not catch it;
symbolizers then attribute samples on those instructions to the wrong
inlined frames.
Fix is to resolve scope-range boundaries through the precise
per-instruction input-to-output AddressMap (the same map BAT already
uses) instead of input-relative block offsets. Boundary instructions are
arbitrary (not just calls/branches), so they are normally absent from
that map; to make them resolvable, disassembly now keeps an offset for
the boundary instructions (only the boundaries, to keep the AddressMap
small) and those offsets are emitted as AddressMap entries whenever an
address map is required (requiresAddressMap(), which includes
--update-debug-sections via requiresPreciseAddressMap()), not only under
BAT.
Boundary collection (BinaryContext::collectDebugScopeBoundaries) runs in
readDebugInfo() after preprocessDebugInfo(). It streams the DIEs of the
CUs that will be updated directly from the already-loaded DWARFContext
with DWARFDebugInfoEntry::extractFast, decoding one DIE at a time into a
single reusable entry, so no DIE forest is materialized and no second
DWARF context is created. For split DWARF the .dwo DIEs are already
extracted by preprocessDWODebugInfo(), so the cached array is reused
instead. Boundaries are stored per function in a sorted, deduplicated
vector (BinaryFunction::DebugScopeBoundaryOffsets), queried during
disassembly with a monotonic cursor, and freed once the function is
disassembled, so the feature adds no global state.
The behavior is on by default and can be disabled with
--accurate-debug-ranges=0, which falls back to the old block-relative
mapping.
Test on big binaries with split dwarf, -lite=0:
wall Max RSS
-accurate-debug-ranges=0 1070.1s 94.91
-accurate-debug-ranges=1 1173.0s 100.07
delta +102.9s +5.16 (+4.6% CPU,
+5.4% RSS)
Test on big binaries with split dwarf, lite=1:
-accurate-debug-ranges=0 433.9s 83.23
-accurate-debug-ranges=1 434.7s 83.79 (+0.7% RSS)
--time-rewrite shows the boundary-collection phase itself is cheap:
"read debug ranges" is 5.8s of 4960s total user time on one binary
(0.1%) and 18.8s of 6307s on another binary; the feature's
cost is dominated by emitting/translating the extra AddressMap entries,
not by collection.
The top-level llvm-profgen lit configuration previously disabled the
entire test directory unless the X86 target was registered. As a result,
target-independent tests and tests for ARM and AArch64 also implicitly
depended on X86.
Split target-specific tests and inputs into X86, ARM, and AArch64
directories with per-directory lit.local.cfg files. Leave help.test at
the top level because it is target-independent. Existing X86 tests
remain guarded by X86 and run as before, while ARM and AArch64 tests no
longer require the X86 target. The X86 non-ARM ETM rejection case also
no longer requires the ARM target; it only requires X86 and OpenCSD.
Split the former etm-arch.test into ARM/etm-arch.test, containing the
two ARM-specific checks, and X86/etm-non-arm.test, containing the
existing X86 rejection check. The tested inputs and expected diagnostics
are preserved.
Update match-blocks-with-pseudo-probes-inline.test,
pseudoprobe-decoding-inline.test, and pseudoprobe-decoding-noinline.test
in BOLT to refer to the relocated X86 inputs. All other llvm-profgen
tests and inputs are pure renames.
This does not change llvm-profgen runtime behavior; it only removes
unnecessary cross-target dependencies from its test suite.
This is a preparatory change for extending llvm-profgen profiling
support to AArch64 and for introducing the universal profile-guided
automatic software loop data prefetcher described in the RFC:
https://discourse.llvm.org/t/rfc-universal-profile-guided-automatic-software-loop-data-prefetcher-for-llvm/90916
The DIE traversal loop in partitionCUs() used an incorrect termination
condition, causing it to read past the end of the CU. Fix the loop to
stop at NextCUOffset so traversal no longer runs beyond the unit's
boundary.
Fixed [#208440](https://github.com/llvm/llvm-project/issues/208440).
The linker can emit certain relative .dyn.rela relocations in .dyn.relr.
This includes relocations for jumptables emitted for indirect goto. The
current processing recognizes these relocations as jump targets only for
relative relocations from .dyn.rela, but not from .dyn.relr. Give
.dyn.relr the same treatment, so CFG construction does not dismiss
blocks that are only reachable indirectly.
We can't relax only some adrp+ldr pairs for a symbol, because there may
be a branch target between the adrp and ldr of this form:
adrp x1, :got:sym
.Lfoo:
ldr x1, [x1, :got_lo12:sym]
# ...
adrp x1, :got:sym
ldr x2, [x1, :got_lo12:sym]
b .Lfoo
Relaxing the first adrp+ldr here would be invalid. This was clarified in
the ARM ABI in:
11fb4ef428.
The implementation already performed a pre-scan to check that the
relevant relocations occur in pairs. Change this scan to a) check all
the preconditions for the relaxation and b) make the decision per
symbol.
Fixes https://github.com/llvm/llvm-project/issues/138254.
PIE binaries use R_*_RELATIVE dynamic relocations for indirect goto jump
tables. BOLT creates entry points for their targets but does not mark
these addresses as targets from data relocations, so indirect jumps with
unknown control flow (which originates from indirect goto) have no CFG
successors and are interpreted as potential tail calls, while the jump
targets are incorrectly identified as additional entry points.
Add these offsets to the function's list of non-entry relocation
references instead of marking them as entry points. Extend relocation
rewriting to also check whether a block is externally referenced. This
mirrors behavior of data-to-code relocations in non-pie binaries.
Summary:
Fix two bugs in DIEBuilder that caused DW_FORM_implicit_const values to
be zeroed out when rewriting DWARF5 debug sections
(--update-debug-sections).
1. In constructDIEFast(), DWARFFormValue was constructed with just the
form code, leaving the value at 0. For DW_FORM_implicit_const,
extractValue() is a no-op since the value is expected to be pre-set.
Fix: use AttrSpec.getFormValue() which initializes the value from the
abbreviation table.
2. In assignAbbrev(), AddAttribute(Attr.getAttribute(), Attr.getForm())
used the two-argument overload which discards the implicit_const value.
Fix: use AddAttribute(Attr) to copy the full DIEAbbrevData.
Fixes https://github.com/llvm/llvm-project/issues/192084
* extends MCPlusBuilder with comparision between registers.
* updates the ICP pass to request a scavenged register for AArch64
callsites.
* wires ICP code generation on AArch64.
`CongruentBuckets` is an unordered_map, so iterating it directly
produces non-deterministic folding order across different build
environments. Sort buckets by the binary address of the representative
function before iterating to guarantee a stable debug output order.
That order isn't matter for the output binary, but it cause randomly
different debug output across different build environments, which makes
it hard to write stable tests for ICF.
Also the debug output of multi-threaded ICF is already non-deterministic
even we sort that by address, so we only try to stabilize the
single-threaded ICF output.
Update the icf-safe tests to reflect the new deterministic ordering.
Assisted-by: Sonnet 4.6
BOLT emits all dynamic relative relocations in .relr.dyn if it exists in
the input, including relative relocations from .rela.dyn. This can fail
for two reasons: first, relative relocations can target odd addresses.
These can never be emitted as relr relocations. Second, BOLT reuses the
existing .relr.dyn section. Even if a relative relocation from .rela.dyn
can be emitted as relr, there is simply no space in the existing
.relr.dyn section. This causes an overflow. Prevent this by emitting
relative relocations in the same section they originate from.
Overwriting a function entry with a jump is likely to not perfectly
align with the instruction stream. If the end of the patch does not
fall onto an instruction boundary, the bytes following the jump are
orphaned and will have nonsensical interpretations. This can leave
other tools confused, especially since these orphaned bytes can
decode to instructions that do not nicely rejoin the still intact
part of the instructions stream. Overwrite these bytes with traps
in the PatchEntry pass.
Fixes#198455.
This PR implements a new parallel DWARF debug info processing pipeline
for BOLT that significantly speeds up `--update-debug-sections` for
large binaries. It is the second part of the split from the overall RFC
changes
RFC - [[RFC][BOLT] A New Parallel DWARF Processing Approach in
BOLT](https://discourse.llvm.org/t/rfc-bolt-a-new-parallel-dwarf-processing-approach-in-bolt/90736)
(The overall changes.)
This PR does the following:
1. **Equivalence-class CU partitioning:** Replaces batchsize grouping
with union-find over DW_FORM_ref_addr references. Connected CUs share a
bucket; isolated CUs become singletons.
> For the non-LTO case, CUs have no cross-CU dependencies, so each CU is
placed into its own singleton bucket and processed fully in parallel.
> For the LTO case, CUs with cross-CU dependencies are grouped into the
same bucket and processed sequentially within that bucket, while
different buckets are still processed in parallel.
2. **Per-bucket parallel processing with in-order merge:** Each bucket
gets its own DIEBuilder and local range/loc writers; completed buckets
are then merged sequentially in partition order, applying offset fixups
and emitting CUs to ensure deterministic output.
3. **Add main-binary parallel process** : Main-binary CU update and
split-DWO processing both happen within the same per-bucket task.
Static PIEs contain DT_INIT and DT_FINI. However, statically linked libc
does not read the dynamic entries, so patching those entries has no
effect. Check DT_FLAGS_1 to detect input static PIEs and rely on entry
point patching accordingly.
Fixes#201371.
There is a race in the instrumentation runtime during setup. The setup
initializes the function pointers for indirect call instrumentation
before the indirect call counters array. If the application spawns a
background thread through a constructor (as does jemalloc), the
background thread has a chance to derefence that uninitialized array
pointer. Defer initialization of these function pointers to prevent this
race.
Fixes#198181.
Co-authored-by: Fabian Parzefall <parzefall@meta.com>
With --use-old-text, the output starts as a byte-for-byte copy of the
input. Alignment padding between sections could retain stale data from
the original binary. Zero the padding so the result matches writing
sections to new file offsets.
1. In `filterBinaryMMapInfo`, replace `incovertibleErrorCode` with errc
code as `parseMainEvents` converts returned Error to std::error_code.
2. In `parsePerfData`, pass through Error returned by `prepareToParse`
for memory events.
Test Plan: updated perf_test.test
External samples (X:0) were breaking heatmap printing, e.g.
```
0x00000000: O0x00000000: ........
```
Explicitly track `IsFirst` instead of relying on zero.
Test Plan:
updated heatmap-preagg.test
To avoid mixed up error messages in multi-perf case, provide diagnostics
buffer and stream for each aggregator job.
Test Plan: updated pre-aggregated-perf.test
Reviewers: yavtuk, maksfb, rafaelauler, ayermolo, paschalis-mpeis, yozhu
Reviewed By: yavtuk
Pull Request: https://github.com/llvm/llvm-project/pull/203464
Propagate perf/preaggregated input parsing errors through DataAggregator
instead of terminating from per-input aggregation jobs.
This lets multi-input aggregation report failed inputs as warnings when
at least one input succeeds while returning errors when all inputs fail.
It also converts pre-aggregated parsing diagnostics to returned Error
values and removes worker-path exits from perf setup and parsing.
Test Plan: updated pre-aggregated-perf.test
Reviewers: maksfb, rafaelauler, ayermolo, yozhu, yavtuk
Reviewed By: rafaelauler
Pull Request: https://github.com/llvm/llvm-project/pull/200476
Parse branch type and set `IsReturn` when using perf data input.
This makes profile attaching more accurate for external returns.
Test Plan: updated perf_brstack.test
Reviewers: maksfb, rafaelauler, yozhu, ayermolo, yavtuk, paschalis-mpeis
Reviewed By: yozhu
Pull Request: https://github.com/llvm/llvm-project/pull/202813
Extend pre-aggregated format with an optional mispredictions field.
Update parsing and pre-aggregated profile writer.
Test Plan:
added PreAggregatedX86TestHelper.OptionalMispredField
updated pre-aggregated-records.s
This reverts commit 6b13656fd8.
We have identified various bugs hence reverting:
* relaxLocalBranches() should account for BB growth and adjust subsequent BB offsets in fragment
* multiple parallel workers are sharing the same allocator in DataflowInfoManager
* liveness is run lazily, potentially after the CFG has been modified
When reordering blocks we may have to invert branches. Sometimes this
isn't possible for compare-and-branch instructions because the immediate
value would overflow/underflow after the adjustment. In such cases I am
splitting the instruction into a compare followed by a branch. For this
to be legal we should be sure that the condition flags are not being
clobbered. Liveness analysis may help here.
## Summary
This PR fixes a race condition in LLVM BOLT's
DIEBuilder::buildTypeUnits() that is triggered when DWARF5 split-DWARF
(.dwo/.dwp) inputs are processed with multi-threaded CU processing.
Concurrent invocations from different worker threads share the same DWP
type-unit state, which results in duplicated DIE extraction, assertion
failures, and intermittent crashes. The fix serializes buildTypeUnits()
for DWP inputs via a function-local static std::mutex, leaving the
non-DWO fast path unchanged.
## Problem Description
When BOLT processes DWARF debug info with --debug-thread-count=4
--cu-processing-batch-size=4 on testcase
dwarf5-df-types-dup-dwp-input.test, multiple threads concurrently call
DIEBuilder::buildTypeUnits() on shared DWP type units. Since type units
within a DWP file are shared across compilation units, multiple threads
may attempt to extract DIEs from the same type unit simultaneously,
violating the assertion.
**Note:** This is an intermittent failure — due to its dependence on
thread scheduling and interleaving, it does not reproduce on every run,
and may require multiple executions under the multi-threaded
configuration to surface, which is why it escaped earlier
single-threaded testing.
Add a pre-aggregated profile output format (`--profile-format=preagg`)
so perf.data can be pre-parsed/aggregated and used as input with -pa.
Supports branch (T traces) and basic samples (S records).
Currently only covers main binary, can be extended to cover multi-DSO.
Test Plan: Updated perf_test.test, added perf_brstack.test
Reviewers:
yota9, ayermolo, yozhu, maksfb, yavtuk, paschalis-mpeis, rafaelauler
Reviewed By: paschalis-mpeis
Pull Request: https://github.com/llvm/llvm-project/pull/199465
Allow multiple -p/-perfdata args (comma-separated and repeated).
Process them in parallel with `--perfdata-jobs/-pj` (default 4,
0 for all available hardware threads). Keep YAML/DataReader
inputs single-profile only.
This also enables density computation with multiple perf files.
Prior to that, it was impossible to compute total density as
it wasn't computed from symbolized profiles.
Depends on
#199323#199322#199321#199320
Test Plan:
Updated pre-aggregated-perf.test and perf_test
For a large binary (300MB text) and 7 ETM perf data files
(6GB each), compared to parallel perf2bolt + merge-fdata:
* processing time is equal: 670+3s -> 669s (within noise),
* output is equivalent: sorted fdata is equal.
Reviewers:
paschalis-mpeis, rafaelauler, yozhu, ayermolo, yavtuk, maksfb, yota9
Reviewed By: paschalis-mpeis, rafaelauler
Pull Request: https://github.com/llvm/llvm-project/pull/199324
Simplify LitConfig initialization and setter to allow None values.
TestingConfig.maxIndividualTestTime is initialized to 0 (or resolved to
0 if None) strictly during initialization.
This fixes an issue where the aggressive BOLT timeout of 60s (previously
set globally on lit_config) was leaking and affecting libc++ tests. By
moving the timeout configuration from the global lit_config to the
individual test suite config, we ensure that timeouts are isolated and
respect suite-local settings without leaking.
PR Stack:
* https://github.com/llvm/llvm-project/pull/198192
* https://github.com/llvm/llvm-project/pull/199886
* ➤ https://github.com/llvm/llvm-project/pull/199996
* https://github.com/llvm/llvm-project/pull/198193
The diff from #198192 is 39b632f741012bfbff9858407765e45435ad95ff
Assisted-by: Gemini
