`b05a5d0a` added an arity-trimming step to the shared
`ScriptedPythonInterface::Dispatch`: extensions are now allowed to
define methods with trailing parameters as optional
(`num_children(self)` vs. `num_children(self, max_count)`), so before
calling into a method, `Dispatch` needs to know how many positional
arguments it actually accepts and drop any trailing ones we'd otherwise
pass.
That check calls `PythonCallable::GetArgInfo`, which ran a whole
embedded Python script through `inspect.signature` on every call, since
every scripted-extension dispatch goes through it.
For the common case `GetArgInfo()` actually needs to handle fast (plain
Python functions/methods, classes used as constructors, and callable
instances defining `__call__`, i.e. everything `Dispatch<T>()` and
`CreatePluginObject()` ever pass it), the answer is available as plain
data attributes, with no Python bytecode execution required:
`__func__`/`__self__` to unwrap bound methods, and `__code__`'s
`co_argcount`/`co_flags` for the actual positional-argument count and
varargs bit. For a class, that means `__init__` -- except a class may
customize `__new__` instead and leave `__init__` untouched, in which
case `object.__init__` becomes lenient about extra arguments.
Anything else still lacking `__code__` can fall back to the original
`inspect.signature()`-based implementation, now exposed as
`PythonCallable::GetArgInfoFromInspectSignature()` so it can be used
separately.
rdar://183776556
---------
Signed-off-by: Med Ismail Bennani <ismail@bennani.ma>
Give `type synthetic add -l` a formal
`ScriptedSyntheticChildrenInterface`, matching the architecture used
elsewhere in this series: a C++ interface header, a Python-backed
implementation, `PluginManager` registration with CLI/API usages, and a
generatable ABC template (`scripted_synthetic_children.py`) wired into
`scripting extension generate`.
Every method goes through the shared `Dispatch<T>()` machinery instead
of hand-rolling its own Locker/raw-SWIG calls. `Dispatch<T>()` is taught
to introspect the target method's arity via
`PythonCallable::GetArgInfo()` and drop trailing args before calling, so
providers that legitimately define an argument as optional
(`num_children(self)` vs. `num_children(self, max_count)`) still work
through the generic dispatch path.
This retires the ad-hoc `LLDBSwigPython_*` synthetic-children bridge
functions entirely.
Signed-off-by: Med Ismail Bennani <ismail@bennani.ma>
Give both `command script add -c` (raw class) and `-p` (ParsedCommand) a
single, shared
`ScriptedCommandInterface`/`ScriptedCommandPythonInterface`, built on
the existing `ScriptedPythonInterface` machinery: `CreatePluginObject`
delegates to the base template, and every other method goes through
`Dispatch`.
Add a lighter `ScriptedCommand` ABC template (`scripted_command.py`) for
raw mode; the existing `ParsedCommand`/`LLDBOptionValueParser` classes
are kept as-is.
Extend `ScriptedPythonInterface` with a few `Transform` overloads
(`DebuggerSP`, `std::vector<std::string>`, `CommandReturnObject &`) so
those argument types route through `Dispatch` without extra plumbing.
Delete the standalone SWIG bridge functions this plugin was the sole
caller of. `CommandObjectScriptingObjectRaw` and
`CommandObjectScriptingObjectParsed` now hold a
`ScriptedCommandInterfaceSP` instead of a raw
`StructuredData::GenericSP`; `command script add`'s `DoExecute`
constructs the interface once via
`CreateScriptedCommandInterface()->CreatePluginObject` for either flag.
Signed-off-by: Med Ismail Bennani <ismail@bennani.ma>
Give `frame recognizer add -l` a formal
ScriptedStackFrameRecognizerInterface, matching the architecture already
used by ScriptedProcess, ScriptedBreakpoint, etc.: a C++ interface
header, a Python-backed implementation built on
ScriptedPythonInterface's CreatePluginObject/Dispatch machinery,
PluginManager registration, and a generatable Python ABC template
(scripted_stackframe_recognizer.py).
get_recognized_arguments' documented contract returns a plain list of
lldb.SBValue (not an lldb.SBValueList), so it bypasses the generic
Dispatch<ValueObjectListSP>() extractor and calls the SWIG bridge
directly, matching the legacy behavior exactly.
ScriptedStackFrameRecognizer now holds a single interface object created
once in its constructor and reused across every RecognizeFrame() call,
same lifecycle as before.
Signed-off-by: Med Ismail Bennani <ismail@bennani.ma>
A `breakpoint set -P <class>` with an invalid-UTF-8 class name crashes
lldb with `EXC_BAD_ACCESS`. The class name is passed through
unmodified, so the byte sequence `\xd0p` (an incomplete two-byte UTF-8
sequence) reaches the Python layer:
```
./bin/lldb -b -o $'breakpoint set -P \xd0p -f main.cpp' ./bin/lldb
...
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/
Stack dump:
#4 PythonDictionary::GetItem(PythonObject const&) const
#6 PythonDictionary::GetItemForKey(PythonObject const&) const
#7 PythonObject::ResolveNameWithDictionary(StringRef, PythonDictionary const&)
#9 ScriptedPythonInterface::CreatePluginObject<...>(...)
```
`ResolveNameWithDictionary` builds a `PythonString` from the class name
and looks it up in the module dictionary. When the name is not valid
UTF-8, `PyUnicode_FromStringAndSize` returns `NULL`, so the
`PythonString` is left with `m_py_obj == NULL`. `GetItemForKey` forwards
that object to `GetItem`, which passed the `NULL` `PyObject *` straight
to `PyDict_GetItemWithError`.
Debugging the crashing lldb under lldb shows the `NULL` key arriving in
`GetItem`:
```
(lldb) break set -n lldb_private::python::PythonDictionary::GetItem
(lldb) run --no-use-colors -b -s /tmp/bp_cmds.txt
* stop reason = breakpoint 1.1
(lldb) up
(lldb) frame variable key.m_py_obj
(PyObject *) key.m_py_obj = nullptr
```
Continuing lands on the dereference of that `NULL` inside CPython:
```
(lldb) continue
* stop reason = EXC_BAD_ACCESS (code=1, address=0x8)
frame #0: PyDict_GetItemWithError + 40
-> ldr x8, [x1, #0x8]
```
`x1` holds the key argument, which is `NULL`, so reading the type field
at offset `0x8` faults at address `0x8`.
`GetItem` already guards against an invalid dictionary (`!IsValid()`);
extend that guard to reject an invalid key as well and return
`nullDeref()`. `GetItemForKey` consumes the error and returns an empty
`PythonObject`, so existing callers such as `ResolveNameWithDictionary`
keep working and the breakpoint command now fails cleanly instead of
crashing.
This was found by `lldb-commandinterpreter-fuzzer`.
Adds `PythonDataObjectsTest.TestPythonDictionaryGetItemWithInvalidKey`,
which feeds an invalid-UTF-8 key to `GetItem`. Without the fix the test
dereferences the `NULL` key and crashes (`SIGSEGV`).
This is a general mechanism that allows a custom breakpoint resolver to
"override" the resolver lldb would have used. It is implemented by
making a scripted breakpoint resolver class, implementing an extra
"overrides_resolver" method. When lldb has decided what resolver it is
going to use for the breakpoint, if there are any override resolvers
registered, it will pass the serialized form of the resolver to the
`overrides_resolver` method of the registered resolvers. If any of the
override resolvers return true from the overrides_resolver method, that
resolver will be used in place of the one lldb would have used.
The original motivation for this feature is to make the support of
"debugging a scripting language from the implementation side". We added
the notion of "facade breakpoints" so that a location given in the
context of the scripting language can be handled by the emulation code.
But when you are using lldb in some IDE, when "stopped" in the scripting
language, the IDE will show the scripting language source. It is natural
in that case for the user to expect the UI gesture to set a file and
line breakpoint should work the same way.
To implement that, the implementor would make their facade breakpoint
resolver also an override resolver that checks to see if the incoming
resolver was a file and line breakpoint for a source file of the
scripting language type. If it is, it returns true, and lldb will use
that resolver instead. Note, this will also work when breakpoints are
set on the command line, so you can transparently support "scripting
language" breakpoints from the command line as well.
Add support for building the script interpreter plugins as shared
libraries, controlled by the `LLDB_ENABLE_DYNAMIC_SCRIPTINTERPRETERS`
CMake variable. This allows the plugins, rather than `liblldb`, to link
against their respective dependencies such as Python and Lua, which has
several benefits:
1. It makes it possible to gracefully handle the Python or Lua
dependencies not being present, instead of crashing.
2. It enables programmatically deciding which Python or Lua library to
load.
3. Combined with (2), this opens up a path towards not statically
linking with Python at all, which is required in order to load the LLDB
module in a different Python interpreter (at least this is a hard
requirement on Darwin due to two-level namespaces and install names).
This PR completes #183791
This revert #181334 and its follow-up PRs (including #181488, #181492,
#181493, #181494 and #181498) as well as Ismail's documentation changes
(#181594, #181717). The original commit causes a test failure in CI
(https://github.com/llvm/llvm-project/issues/181938) but the more I look
at the patch, the more I'm convinced it was not ready to land. It will
be easier to iterate on the feedback by re-landing this than by using
post-commit review.
## Summary
Based on discussion from
[RFC](https://discourse.llvm.org/t/rfc-python-callback-for-source-file-resolution/83545),
this PR adds a new `SymbolLocatorScripted` plugin that allows Python
scripts to implement custom symbol and source file resolution logic.
This enables downstream users to build custom symbol servers, source
file remapping, and build artifact resolution entirely in Python.
### Changes
- Adds `LocateSourceFile()` to the SymbolLocator plugin interface,
called during source path resolution with a fully loaded `ModuleSP`, so
the plugin has access to the module's UUID, file paths, and symbols.
- Adds `SymbolLocatorScripted` plugin that delegates all four
SymbolLocator methods (`LocateExecutableObjectFile`,
`LocateExecutableSymbolFile`, `DownloadObjectAndSymbolFile`,
`LocateSourceFile`) to a user-provided Python class.
- Adds `ScriptedSymbolLocatorPythonInterface` to bridge C++ calls to
Python, with proper GIL management and error handling.
- Results for `LocateSourceFile` are cached per (module UUID, source
file) pair.
- The Python class is configured via: `settings set
plugin.symbol-locator.scripted.script-class module.ClassName`
### Python class interface
```python
class MyLocator:
def __init__(self, exe_ctx, args): ...
def locate_source_file(self, module, original_source_file):
...
def locate_executable_object_file(self, module_spec): ...
def locate_executable_symbol_file(self, module_spec,
default_search_paths): ...
def download_object_and_symbol_file(self, module_spec,
force_lookup, copy_executable): ...
```
### Test plan
```
Added TestScriptedSymbolLocator.py with 3 test cases:
- test_locate_source_file — verifies the locator resolves source
files, receives a valid SBModule with UUID, and remaps paths correctly
- test_locate_source_file_none_fallthrough — verifies returning
None falls through to default LLDB resolution, and that having no script
class set works normally
- test_invalid_script_class — verifies graceful handling of
invalid class names without crashing
```
Co-authored-by: Rahul Reddy Chamala <rachamal@fb.com>
This patch extends ScriptedFrame to work with real (non-scripted)
threads,
enabling frame providers to synthesize frames for native processes.
Previously, ScriptedFrame only worked within
ScriptedProcess/ScriptedThread
contexts. This patch decouples ScriptedFrame from ScriptedThread,
allowing
users to augment or replace stack frames in real debugging sessions for
use
cases like custom calling conventions, reconstructing corrupted frames
from
core files, or adding diagnostic frames.
Key changes:
- ScriptedFrame::Create() now accepts ThreadSP instead of requiring
ScriptedThread, extracting architecture from the target triple rather
than ScriptedProcess.arch
- Added SBTarget::RegisterScriptedFrameProvider() and
ClearScriptedFrameProvider() APIs, with Target storing a
SyntheticFrameProviderDescriptor template for new threads
- Added "target frame-provider register/clear" commands for CLI access
- Thread class gains LoadScriptedFrameProvider(),
ClearScriptedFrameProvider(),
and GetFrameProvider() methods for per-thread frame provider management
- New SyntheticStackFrameList overrides FetchFramesUpTo() to lazily
provide
frames from either the frame provider or the real stack
This enables practical use of the SyntheticFrameProvider infrastructure
in
real debugging workflows.
rdar://161834688
Signed-off-by: Med Ismail Bennani <ismail@bennani.ma>
Signed-off-by: Med Ismail Bennani <ismail@bennani.ma>
This patch implements the base and python interface for the
ScriptedFrameProvider class.
This is necessary to call python APIs from the ScriptedFrameProvider
that will come in a follow-up.
Signed-off-by: Med Ismail Bennani <ismail@bennani.ma>
Signed-off-by: Med Ismail Bennani <ismail@bennani.ma>
Modify the python wrapper to return uint32_t,
which prevents incorrect child name-to-index mapping and avoids
performing redundant operations on non-existent SBValues.
This patch adds the notion of "Facade" locations which can be reported
from a ScriptedResolver instead of the actual underlying breakpoint
location for the breakpoint. Also add a "was_hit" method to the scripted
resolver that allows the breakpoint to say which of these "Facade"
locations was hit, and "get_location_description" to provide a
description for the facade locations.
I apologize in advance for the size of the patch. Almost all of what's
here was necessary to (a) make the feature testable and (b) not break
any of the current behavior.
The motivation for this feature is given in the "Providing Facade
Locations" section that I added to the python-reference.rst so I won't
repeat it here.
rdar://152112327
If we're not touching them, we don't need to do anything special to pass
them along -- with one important caveat: due to how cmake arguments
work, the implicitly passed arguments need to be specified before
arguments that we handle.
This isn't particularly nice, but the alternative is enumerating all
arguments that can be used by llvm_add_library and the macros it calls
(it also relies on implicit passing of some arguments to
llvm_process_sources).
So the dSYM can be told what target it has been loaded into.
When lldb is loading modules, while creating a target, it will run
"command script import" on any Python modules in Resources/Python in the
dSYM. However, this happens WHILE the target is being created, so it is
not yet in the target list. That means that these scripts can't act on
the target that they a part of when they get loaded.
This patch adds a new python API that lldb will call:
__lldb_module_added_to_target
if it is defined in the module, passing in the Target the module was
being added to, so that code in these dSYM's don't have to guess.
If your arguments or option values are of a type that naturally uses one
of our common completion mechanisms, you will get completion for free.
But if you have your own custom values or if you want to do fancy things
like have `break set -s foo.dylib -n ba<TAB>` only complete on symbols
in foo.dylib, you can use this new mechanism to achieve that.
...and "[lldb/Interpreter] Introduce `ScriptedStopHook{,Python}Interface` & make use of it (#105449)"
This reverts commit 76b827bb4d, and commit 1e131ddfa8
because the first commit caused the test command-stop-hook-output.test to fail.
This patch introduces new `ScriptedStopHook{,Python}Interface` classes
that make use of the Scripted Interface infrastructure and makes use of
it in `StopHookScripted`.
It also relax the requirement on the number of argument for initializing
scripting extension if the size of the interface parameter pack contains
1 less element than the extension maximum number of positional arguments
for this initializer.
This addresses the cases where the embedded interpreter session
dictionary is passed to the extension initializer which is not used most
of the time.
---------
Signed-off-by: Med Ismail Bennani <ismail@bennani.ma>
Among other things, returning an empty string as the repeat command
disables auto-repeat, which can be useful for state-changing commands.
There's one remaining refinement to this setup, which is that for parsed
script commands, it should be possible to change an option value, or add
a new option value that wasn't originally specified, then ask lldb "make
this back into a command string". That would make doing fancy things
with repeat commands easier.
That capability isn't present in the lldb_private side either, however.
So that's for a next iteration.
I haven't added this to the docs on adding commands yet. I wanted to
make sure this was an acceptable approach before I spend the time to do
that.
This patch makes ScriptedThreadPlan conforming to the ScriptedInterface
& ScriptedPythonInterface facilities by introducing 2
ScriptedThreadPlanInterface & ScriptedThreadPlanPythonInterface classes.
This allows us to get rid of every ScriptedThreadPlan-specific SWIG
method and re-use the same affordances as other scripting offordances,
like Scripted{Process,Thread,Platform} & OperatingSystem.
To do so, this adds new transformer methods for `ThreadPlan`, `Stream` &
`Event`, to allow the bijection between C++ objects and their python
counterparts.
This just re-lands #70392 after fixing test failures.
Signed-off-by: Med Ismail Bennani <ismail@bennani.ma>
This patch makes ScriptedThreadPlan conforming to the ScriptedInterface
& ScriptedPythonInterface facilities by introducing 2
ScriptedThreadPlanInterface & ScriptedThreadPlanPythonInterface classes.
This allows us to get rid of every ScriptedThreadPlan-specific SWIG
method and re-use the same affordances as other scripting offordances,
like Scripted{Process,Thread,Platform} & OperatingSystem.
To do so, this adds new transformer methods for `ThreadPlan`, `Stream` &
`Event`, to allow the bijection between C++ objects and their python
counterparts.
This just re-lands #70392 after fixing test failures.
Signed-off-by: Med Ismail Bennani <ismail@bennani.ma>
The unit tests only test the Python objects and don't actually use
anything from the LLDB module. That means that all the additional
complexity in ScriptInterpreterPythonImpl::Initialize is overkill.
By doing the initialization by hand, we avoid the annoying
ModuleNotFoundError.
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'lldb'
The error is the result of us stubbing out the SWIG (specifically
`PyInit__lldb`) because we cannot link against libLLDB from the unit
tests.
The downside of doing the initialization manually is that we do lose a
bit of test coverage. For example, issue #70453 also manifested itself
in the unit tests.
Use a Python Bytes and ByteArray object instead of Integers for
TestOwnedReferences and TestBorrowedReferences. These two tests were
failing when building against Python 3.12 because these Integer objects
had a refcount of 4294967296 (-1). I didn't dig into it, but I suspect
the Python runtime has adopted an optimization to decrease refcounting
traffic for these simple objects.
This allows you to specify options and arguments and their definitions
and then have lldb handle the completions, help, etc. in the same way
that lldb does for its parsed commands internally.
This feature has some design considerations as well as the code, so I've
also set up an RFC, but I did this one first and will put the RFC
address in here once I've pushed it...
Note, the lldb "ParsedCommand interface" doesn't actually do all the
work that it should. For instance, saying the type of an option that has
a completer doesn't automatically hook up the completer, and ditto for
argument values. We also do almost no work to verify that the arguments
match their definition, or do auto-completion for them. This patch
allows you to make a command that's bug-for-bug compatible with built-in
ones, but I didn't want to stall it on getting the auto-command checking
to work all the way correctly.
As an overall design note, my primary goal here was to make an interface
that worked well in the script language. For that I needed, for
instance, to have a property-based way to get all the option values that
were specified. It was much more convenient to do that by making a
fairly bare-bones C interface to define the options and arguments of a
command, and set their values, and then wrap that in a Python class
(installed along with the other bits of the lldb python module) which
you can then derive from to make your new command. This approach will
also make it easier to experiment.
See the file test_commands.py in the test case for examples of how this
works.
Temporarily revert to unblock the CI bots, this is breaking the -DLLVM_ENABLE_MODULES=On
modules style build. I've notified Ismail.
This reverts commit 888501bc63.
This patch makes ScriptedThreadPlan conforming to the ScriptedInterface
& ScriptedPythonInterface facilities by introducing 2
ScriptedThreadPlanInterface & ScriptedThreadPlanPythonInterface classes.
This allows us to get rid of every ScriptedThreadPlan-specific SWIG
method and re-use the same affordances as other scripting offordances,
like Scripted{Process,Thread,Platform} & OperatingSystem.
To do so, this adds new transformer methods for `ThreadPlan`, `Stream` &
`Event`, to allow the bijection between C++ objects and their python
counterparts.
Signed-off-by: Med Ismail Bennani <ismail@bennani.ma>
This patch enforces that every scripted object implements all the
necessary abstract methods.
Every scripted affordance language interface can implement a list of
abstract methods name that checked when the object is instanciated.
Since some scripting affordances implementations can be derived from
template base classes, we can't check the object dictionary since it
will contain the definition of the base class, so instead, this checks
the scripting class dictionary.
Previously, for the various python interfaces, we used
`ABC.abstractmethod` decorators but this is too language specific and
doesn't work for scripting affordances that are not derived from
template base classes (i.e OperatingSystem, ScriptedThreadPlan, ...), so
this patch provides generic/language-agnostic checks for every scripted
affordance.
Signed-off-by: Med Ismail Bennani <ismail@bennani.ma>
This patch enforces that every scripted object implements all the
necessary abstract methods.
Every scripted affordance language interface can implement a list of
abstract methods name that checked when the object is instanciated.
Since some scripting affordances implementations can be derived from
template base classes, we can't check the object dictionary since it
will contain the definition of the base class, so instead, this checks
the scripting class dictionary.
Previously, for the various python interfaces, we used
`ABC.abstractmethod` decorators but this is too language specific and
doesn't work for scripting affordances that are not derived from
template base classes (i.e OperatingSystem, ScriptedThreadPlan, ...), so
this patch provides generic/language-agnostic checks for every scripted
affordance.
Signed-off-by: Med Ismail Bennani <ismail@bennani.ma>
This patch makes ScriptedThreadPlan conforming to the ScriptedInterface
& ScriptedPythonInterface facilities by introducing 2
ScriptedThreadPlanInterface & ScriptedThreadPlanPythonInterface classes.
This allows us to get rid of every ScriptedThreadPlan-specific SWIG
method and re-use the same affordances as other scripting offordances,
like Scripted{Process,Thread,Platform} & OperatingSystem.
To do so, this adds new transformer methods for `ThreadPlan`, `Stream` &
`Event`, to allow the bijection between C++ objects and their python
counterparts.
Signed-off-by: Med Ismail Bennani <ismail@bennani.ma>
This patch aims to consolidate the OperatingSystem scripting affordance
by introducing a stable interface that conforms to the
Scripted{,Python}Interface.
This unify the way we call into python methods from lldb while
also improving its capabilities by allowing us to pass lldb_private
objects are arguments.
Differential Revision: https://reviews.llvm.org/D159314
Signed-off-by: Med Ismail Bennani <ismail@bennani.ma>
This patch changes the way plugin objects used with Scripted Interfaces
are created.
Instead of implementing a different SWIG method to create the object for
every scripted interface, this patch makes the creation more generic by
re-using some of the ScriptedPythonInterface templated Dispatch code.
This patch also improves error handling of the object creation by
returning an `llvm::Expected`.
Signed-off-by: Med Ismail Bennani <ismail@bennani.ma>
Signed-off-by: Med Ismail Bennani <ismail@bennani.ma>
This patch refactors the `StructuredData::Integer` class to make it
templated, makes it private and adds 2 public specialization for both
`int64_t` & `uint64_t` with a public type aliases, respectively
`SignedInteger` & `UnsignedInteger`.
It adds new getter for signed and unsigned interger values to the
`StructuredData::Object` base class and changes the implementation of
`StructuredData::Array::GetItemAtIndexAsInteger` and
`StructuredData::Dictionary::GetValueForKeyAsInteger` to support signed
and unsigned integers.
This patch also adds 2 new `Get{Signed,Unsigned}IntegerValue` to the
`SBStructuredData` class and marks `GetIntegerValue` as deprecated.
Finally, this patch audits all the caller of `StructuredData::Integer`
or `StructuredData::GetIntegerValue` to use the proper type as well the
various tests that uses `SBStructuredData.GetIntegerValue`.
rdar://105575764
Differential Revision: https://reviews.llvm.org/D150485
Signed-off-by: Med Ismail Bennani <ismail@bennani.ma>
Many SB classes have public constructors or methods involving types that
are private. Some are more obvious (e.g. containing lldb_private in the
name) than others (lldb::FooSP is usually std::shared_pointer<lldb_private::Foo>).
This commit explicitly does not address FileSP, so I'm leaving that one
alone for now.
Some of these were for other SB classes to use and should have been made
protected/private with a friend class entry added. Some of these were
public for some of the swig python helpers to use. I put all of those
functions into a class and made them static methods. The relevant SB
classes mark that class as a friend so they can access those
private/protected members.
I've also removed an outdated SBStructuredData test (can you guess which
constructor it was using?) and updated the other relevant tests.
Differential Revision: https://reviews.llvm.org/D150157
