`__grow_by` is an internal function, and we deprecated it to avoid
accidental use of it inside the library. `_LIBCPP_DEPRECATED_` is meant
for user-configuration, which doesn't apply here.
Fixes#124940
This allows users to influence how much we overalign `string`s
allocations and tune it to the new/delete implementation via
`-fnew-alignment`. If we don't have `__STDCPP_DEFAULT_NEW_ALGINMENT__`
or we're not using `std::allocator`, we default to an alignment of
`sizeof(void*)`.
Allocators are generally very cheap to copy, so avoiding copies by
having separate overloads is not that useful. Defaulting them
significanlty reduces the overload set the compiler has to consider and
simplifies the code, since we can remove some functions in the future.
Closes#87368
This patch introduces `_LIBCPP_{BEGIN/END}_EXPLICIT_ABI_ANNOTATIONS` and
marks everything within an
`_LIBCPP_{BEGIN,END}_UNVERSIONED_NAMESPACE_STD` (and any derivatives
like `_LIBCPP_{BEGIN,END}_NAMESPACE_STD`) implicitly by default. This
allows us to drop `_LIBCPP_HIDE_FROM_ABI` in most of the code base,
except for functions which shouldn't be `_LIBCPP_HIDE_FROM_ABI`.
This patch doesn't remove any `_LIBCPP_HIDE_FROM_ABI`s, since we have
over 13k of them in the code base. Actually dropping them will happen
over some time to avoid too many merge conflicts.
This was only necessary for ASan container annotations for short
strings. Remove it now that we do not support ASan container annotations
on short strings to:
1. Not suppress Asan on other violations, e.g. std::string::size() on
dead object.
2. Make the code a bit more clean.
Short string annotations were removed in
https://github.com/llvm/llvm-project/pull/194208.
Essentially a revert of #79536, although manual given how much the code
has changed.
Doing container overflow checks on SSO strings is fundamentally unsound
due to how they interact with compiler optimizations within LLVM, which
assume that the entire object is dereferencable when it is live. LLVM
can sometimes insert speculative loads that will create false positives
when they try and load a partially filled SSO buffer. This was
previously prevented using inline asm barriers within std::string
itself. However, this can also manifest in other ways, with the
motivating example for the revert being GlobalOpt inserting a
speculative load of a std::variant containing a SSO std::string, causing
an ASan failure.
`std::exchange` is helpful for simplifying move operations because we
can couple the resetting component with the move operation. This both
halves the number of necessary lines for each move, and also ensures we
haven't forgotten to reset the moved-from object. Since `std::exchange`
is a C++14 feature, we'll need to use `__exchange` for anything that's
from C++11.
This commit also applies `std::__exchange` to a some trivial use-cases
to show its utility. Applying this project-wide requires a census over
all of our types. It's hoped that this will start a community-wide
clean-up that happens over time.
Since befaa35212, the CI stably failed for
the generic-no-wide-characters build, because in no-`wchar_t` modes, the
header for `__remove_cv_t` wasn't properly included.
This PR adds the missing include of `<__type_traits/remove_cv.h>`.
As drive-by, `<__cstddef/size_t.h>` and
`<__type_traits/is_constant_evaluated.h>`, which are included by
`<cwchar>`, are also made included by `<string>` to avoid potential
regression as we're using `size_t` and
`__libcpp_is_constant_evaluated()` in `<string>`.
There is no reason to double-wrap iterators, since we can already
achieve anything we want within `__bounded_iter`itself.
This is technically ABI breaking, but people using bounded iterators
shouldn't require ABI stability currently.
Fixes#178521
The previous `_LIBCPP_ASAN_VOLATILE_WRAPPER` approach was used to
prevent
speculative loads of string data before the short/long state was
determined. This patch replaces that mechanism with a more explicit
`__annotate_memory_barrier()` using an empty volatile assembly block.
This PR is inspired by #183457 and by downstream false positive on
`__get_long_size`. It fails same way as `__get_long_pointer` before we
have
`_LIBCPP_ASAN_VOLATILE_WRAPPER`. Barrier approach avoids
expanding `_LIBCPP_ASAN_VOLATILE_WRAPPER` for size_t, and to
in general looks more readable.
I failed to create reasonable reproducer for test, I suspect it requires
precise set of compiler flags, and libc++ site_config which will be hard
to maintain in test.
`std::__to_address(__get_pointer())` is an extremely common pattern
inside `string` and is basically equivalent to `data()`, except that
`data()` only returns a non-const pointer since C++17. This patch
introduces `__data()` to back-port returning a non-const pointer.
Address sanitizer recently got a new macro __SANITIZER_DISABLE_CONTAINER_OVERFLOW__
which is intended to disable container overflow checks in libraries (either the
standard library or user libraries that might provide such checks). This patch makes
libc++ honor that macro and, in addition, cleans up how these checks are enabled for
string (which is special) by introducing a macro just for string.
rdar://166234942
When doing amortized growth we currently have separate functions for
calculating the new capacity and allocating. However, we set the size to
the calculated capacity that way instead of the actually required size.
This makes the interface quite confusing, since the size has to be set
manually. Instead, this patch refactors the function to get the
amortized growth capacity to instead allocate and set the size
correctly.
Instead of unwrapping the arguments to string's observer functions
everywhere, simply unwrap the specific argument and then forward all of
them to a single function which does the final unwrapping of `*this`.
This simplifies the code a bit.
`__erase_external_with_move` is in the dylib, so the compiler doesn't
see the definition. Marking it `noexcept` sometimes allows clang to
remove exceptions related code, improving code size slightly.
Flags that should be used both for compiling and for linking are
provided through the %{flags} substitution. Our clang-tidy tests should
be using them, not only %{compile_flags}.
Introducing this utility makes the `__grow_by{,_and_replace}`
significantly easier to understand and allows us to migrate away from
these functions in the future.
This does a couple of things:
- code that is only useful for `shrink_to_fit` is moved into that
function
- `shrink_to_fit` is simplified a bit
- `__recommend` is renamed to better reflect what the function actually
does
- `__allocate_long_buffer` asserts that the passed capacity doesn't fit
into the SSO
The Trivial Relocation feature has been removed from the C++26 working
draft. Based on discussions in Kona, it is unlikely that the
"replaceable" type concept will come back in the C++29 time frame.
Since we don't have a use for the type trait in the library at the
moment, remove the code associated to it. If we end up needing something
like it in the future, we can always add it back.
8dae17be29 refactors basic_string for more
code reuse. This makes sense in most cases, but has performance overhead
in the case of ~basic_string. The refactoring of ~basic_string to call
__reset_internal_buffer() added a redundant (inside the destructor)
reset of the object, which the optimizer is unable to optimize away in
many cases. This patch prevents a ~1% regression we observed on an
internal workload when applying the original refactoring. This does
slightly pessimize the code readability, but I think this change is
worth it given the performance impact.
I'm hoping to add a benchmark(s) to the upstream libc++ benchmark suite
around string construction/destruction to ensure that this case does not
regress as it seems common in real world applications. I will put up a
separate PR for that when I figure out a reasonable way to write it.
Verify that the operation passed to resize_and_overwrite returns an
integer-like type, matching the behavior of other standard library
implementations like GCC's libstdc++
Fixes#160577
There is a tradition to use U.S. English spellings for APIs. For
example, it's uninitialized_fill and not uninitialised_fill,
specialization not specialisation, etcetera.
This patch adds additional overloads to `map::at` in case its known that
the argument is transparently comparable to the key type. This avoids
actually constructing the key type in some cases, potentially removing
allocations.
```
--------------------------------------------------------
Benchmark old new
--------------------------------------------------------
BM_map_find_string_literal 12.8 ns 2.68 ns
```
This patch introduces a string-internal API to make the allocation and
deallocation the long string simpler. Before this we had a lot of code
duplication, so ensuring that things were actually correct was
non-trivial.
Since `__resize_default_init` is only ever used inside the dylib we can
remove the libc++-internal API and switch to the public one. This patch
inlines a bunch of functions that aren't required anymore and simplifies
the code that way.
This uses the new `__builtin_lt_synthesises_from_spaceship` builtin from
clang to use three way comparison for arbitrary user-defined types that
only provide a spaceship operator.
We've built up quite a few links directly to github within the code
base. We should instead use `llvm.org/PR<issue-number>` to link to bugs,
since that is resilient to the bug tracker changing in the future. This
is especially relevant for tests linking to bugs, since they will
probably be there for decades to come. A nice side effect is that these
links are significantly shorter than the GH links, making them much less
of an eyesore.
This patch also replaces a few links that linked to the old bugzilla
instance on llvm.org.
LLVM 20 contained an ABI break that can result in the size of
`std::unordered_{map,set,multimap,multiset}` and `std::deque` changing
when used with an allocator type that is empty and contains a base class
that is the same across rebound allocator instantiations (e.g.
``Allocator<int>`` and ``Allocator<char>`` are both empty and contain
the same base class).
In addition, the layout of a user-defined type that:
- contains one of the following containers:
`std::unordered_{map,set,multimap,multiset}`, `std::deque`, `std::map`,
`std::set`, `std::multimap`, `std::multiset`, `std::list` or
`std::vector`, and
- passes an empty allocator, comparator or hasher type to that
container, and
- has a member of that same empty allocator, comparator or hasher type
inside the enclosing struct, and
- that member is either marked with `[[no_unique_address]]` or optimized
out via the EBO (empty base optimization) technique
saw its size increase from LLVM 19 to LLVM 20. This was caused by the
usage of `[[no_unique_address]]` within some of libc++'s containers in a
way that allowed subtle interactions with enclosing objects. This is
fixed in LLVM 21 on Clang (returning to the LLVM 19 ABI), however that
implies an ABI break from LLVM 20 to LLVM 21.
Furthermore, fixing this causes a slight regression to constant
evaluation support in `std::unique_ptr`. Specifically, constant
evaluation will now fail when the deleter relies on being
value-initialized for constant-evaluation admissibility. If a
default-initialized deleter can be used during constant evaluation, or
if the default constructor is non-trivial, the `unique_ptr` is not
affected by this regression. In particular, this regression does not
impact any `unique_ptr` using the default deleter.
Note that there is currently no way to realistically fix this ABI break
on GCC, therefore GCC will remain on the ABI introduced in LLVM 19. That
also means that Clang and GCC will have a slightly different ABI for the
small subset of types listed above until we are able to apply the same
fix we did with Clang on GCC.
We fix this regression by surrounding the members of the
`_LIBCPP_COMPRESSED_PAIR` with an anonymous struct. This restricts the
shifting of empty types to the front of the `_LIBCPP_COMPRESSED_PAIR`
instead of throughout the surrounding object. This "frees up" the zero
offset to contain another object of the same type, restoring the ability
to perform EBO or to elide the storage for a type with
`[[no_unique_address]]` in the enclosing (user-defined) struct.
Fixes#154146
Co-authored-by: Louis Dionne <ldionne.2@gmail.com>
The use-case for `__is_same_uncvref` seems rather dubious, since not a
single use-cases needed the `remove_cvref_t` to be applied to both of
the arguments. Removing the alias makes it clearer what actually
happens, since we're not using an internal name anymore and it's clear
what the `remove_cvref_t` should apply to.
Currently, when the string shrink into the SSO buffer, the `__rep_.__s`
member isn't activated before the `traits_type::copy` call
yet, so internal `__builtin_memmove` call writing to the buffer causes
constant evaluation failure. The existing test coverage seems a bit
defective and doesn't cover this case - `shrink_to_fit` is called on the
copy of string after erasure, not the original string object.
This PR reorders the `__set_short_size` call, which starts the lifetime
of the SSO buffer, before the copy operation. Test coverage is achieved
by calling `shrink_to_fit` on the original erased string.
This reverts commit c861fe8a71.
Unfortunately, this use of hidden visibility attributes causes
user-defined specializations of standard-library types to also be marked
hidden by default, which is incorrect. See discussion thread on #131156.
...and also reverts the follow-up commits:
Revert "[libc++] Add explicit ABI annotations to functions from the block runtime declared in <__functional/function.h> (#140592)"
This reverts commit 3e4c9dc299.
Revert "[libc++] Make ABI annotations explicit for windows-specific code (#140507)"
This reverts commit f73287e623.
Revert "[libc++][NFC] Replace a few "namespace std" with the correct macro (#140510)"
This reverts commit 1d411f27c7.
