`b05a5d0a` added an arity-trimming step to the shared
`ScriptedPythonInterface::Dispatch`: extensions are now allowed to
define methods with trailing parameters as optional
(`num_children(self)` vs. `num_children(self, max_count)`), so before
calling into a method, `Dispatch` needs to know how many positional
arguments it actually accepts and drop any trailing ones we'd otherwise
pass.
That check calls `PythonCallable::GetArgInfo`, which ran a whole
embedded Python script through `inspect.signature` on every call, since
every scripted-extension dispatch goes through it.
For the common case `GetArgInfo()` actually needs to handle fast (plain
Python functions/methods, classes used as constructors, and callable
instances defining `__call__`, i.e. everything `Dispatch<T>()` and
`CreatePluginObject()` ever pass it), the answer is available as plain
data attributes, with no Python bytecode execution required:
`__func__`/`__self__` to unwrap bound methods, and `__code__`'s
`co_argcount`/`co_flags` for the actual positional-argument count and
varargs bit. For a class, that means `__init__` -- except a class may
customize `__new__` instead and leave `__init__` untouched, in which
case `object.__init__` becomes lenient about extra arguments.
Anything else still lacking `__code__` can fall back to the original
`inspect.signature()`-based implementation, now exposed as
`PythonCallable::GetArgInfoFromInspectSignature()` so it can be used
separately.
rdar://183776556
---------
Signed-off-by: Med Ismail Bennani <ismail@bennani.ma>
Give `type synthetic add -l` a formal
`ScriptedSyntheticChildrenInterface`, matching the architecture used
elsewhere in this series: a C++ interface header, a Python-backed
implementation, `PluginManager` registration with CLI/API usages, and a
generatable ABC template (`scripted_synthetic_children.py`) wired into
`scripting extension generate`.
Every method goes through the shared `Dispatch<T>()` machinery instead
of hand-rolling its own Locker/raw-SWIG calls. `Dispatch<T>()` is taught
to introspect the target method's arity via
`PythonCallable::GetArgInfo()` and drop trailing args before calling, so
providers that legitimately define an argument as optional
(`num_children(self)` vs. `num_children(self, max_count)`) still work
through the generic dispatch path.
This retires the ad-hoc `LLDBSwigPython_*` synthetic-children bridge
functions entirely.
Signed-off-by: Med Ismail Bennani <ismail@bennani.ma>
Give both `command script add -c` (raw class) and `-p` (ParsedCommand) a
single, shared
`ScriptedCommandInterface`/`ScriptedCommandPythonInterface`, built on
the existing `ScriptedPythonInterface` machinery: `CreatePluginObject`
delegates to the base template, and every other method goes through
`Dispatch`.
Add a lighter `ScriptedCommand` ABC template (`scripted_command.py`) for
raw mode; the existing `ParsedCommand`/`LLDBOptionValueParser` classes
are kept as-is.
Extend `ScriptedPythonInterface` with a few `Transform` overloads
(`DebuggerSP`, `std::vector<std::string>`, `CommandReturnObject &`) so
those argument types route through `Dispatch` without extra plumbing.
Delete the standalone SWIG bridge functions this plugin was the sole
caller of. `CommandObjectScriptingObjectRaw` and
`CommandObjectScriptingObjectParsed` now hold a
`ScriptedCommandInterfaceSP` instead of a raw
`StructuredData::GenericSP`; `command script add`'s `DoExecute`
constructs the interface once via
`CreateScriptedCommandInterface()->CreatePluginObject` for either flag.
Signed-off-by: Med Ismail Bennani <ismail@bennani.ma>
This patch adds a `scripting extension generate <ExtensionType>...`
command that introspects a Python extension base class and emits a
skeleton subclass with `# TODO: Implement` stubs for its abstract
methods (or, with `-a`, every method), then opens the result in an
editor.
Generated imports use `from <module> import <class>`, and the generated
`__init__` forwards its arguments to `super().__init__(...)` since every
base class relies on its constructor to set up attributes
(`self.target`, `self.process`, ...) that inherited, non-overridden
methods depend on. When the host can't open an external editor (e.g.
non-macOS), the command reports that as a message rather than an error,
since the file is already written.
To catch regressions in the generator itself instead of just checking
that a file was produced, this patch adds
`TestScriptingExtensionGenerate.py`, which generates a template for
every extension kind on the fly and drives it through a live lldb
session (`process launch -C`, `breakpoint set -P`, `target stop-hook add
-P`, etc.).
Signed-off-by: Med Ismail Bennani <ismail@bennani.ma>
Give `frame recognizer add -l` a formal
ScriptedStackFrameRecognizerInterface, matching the architecture already
used by ScriptedProcess, ScriptedBreakpoint, etc.: a C++ interface
header, a Python-backed implementation built on
ScriptedPythonInterface's CreatePluginObject/Dispatch machinery,
PluginManager registration, and a generatable Python ABC template
(scripted_stackframe_recognizer.py).
get_recognized_arguments' documented contract returns a plain list of
lldb.SBValue (not an lldb.SBValueList), so it bypasses the generic
Dispatch<ValueObjectListSP>() extractor and calls the SWIG bridge
directly, matching the legacy behavior exactly.
ScriptedStackFrameRecognizer now holds a single interface object created
once in its constructor and reused across every RecognizeFrame() call,
same lifecycle as before.
Signed-off-by: Med Ismail Bennani <ismail@bennani.ma>
`ScriptedHookInterface` already carries a `HandleStop` entry point, so
`ScriptedStopHookInterface` is a proper subset. Delete
`ScriptedStopHookInterface`, `ScriptedStopHookPythonInterface`, and
`ScriptInterpreter::CreateScriptedStopHookInterface`, and route
`Target::StopHookScripted` through `ScriptedHookInterface` /
`CreateScriptedHookInterface` instead.
The `target stop-hook add -P <classname>` command is unchanged; it now
constructs a `ScriptedHookInterface` internally, so any existing
`ScriptedStopHook` subclass keeps working as long as it exposes a
`handle_stop` method. New scripts should subclass `ScriptedHook`
directly (which is a superset).
Signed-off-by: Med Ismail Bennani <ismail@bennani.ma>
A Python command's print() writes to sys.stdout, which the interpreter
session pointed at the debugger's raw terminal descriptor via
PyFile_FromFd. Those writes bypass the debugger's output lock, so they
can race with the statusline, resulting in truncated output. Back the
session's terminal stdout and stderr with a pipe instead. A reader
thread drains the pipe and writes to the terminal through
Debugger::PrintAsync, which takes the same output lock as the
statusline.
Only redirect when a statusline could actually be drawing: the pipe and
reader thread are set up only when StatuslineSupported() holds
(show-statusline is enabled and the output is an escape-code-capable
terminal) and the target is the debugger's own terminal. The interactive
interpreter opts out entirely, since input()'s readline line editing and
echo need both sys.stdin and sys.stdout to be the real terminal.
Python still gets a real line-buffered text file over the pipe, so
output stays live and file objects passed to SB APIs
(SetImmediateOutputFile, subprocess, fileno) keep working. On session
teardown the wrappers are flushed before the pipe is closed so a
retained sys.stdout does not strand a trailing line or write into a
closed descriptor.
rdar://181841574
This finishes the removal of ConstString from FileSpec's interface. Note
that FileSpec is still _backed_ by ConstStrings, that will be changed in
a follow-up.
A non-goal for this change is to rewrite business logic, so I kept the
type of some variables as ConstString where a refactor would necessary
to change that. I converted printf-style formatting to formatv-style
formatting where required, and otherwise tried to leave callsites alone
to the extent that was possible.
`PosixApi.h` typedef'd `pid_t` as `uint32_t`, while Python's
`pyconfig.h` on Windows typedef's it as `int`. C++ forbids redeclaring a
typedef with a different type, so the two headers cannot coexist. The
`NO_PID_T` macro in `lldb-python.h` suppressed LLDB's typedef, but only
if `lldb-python.h` got included before `PosixApi.h`.
`pid_t` on Windows was originally defined in d87fc157d2. At this time,
there was no Python support for LLDB on Windows and `uint32_t` matches
the `DWORD` type used by the Win32 API for process IDs.
This patch matches the Python type in `PosixApi.h`, removing the need
for the include ordering.
This is a follow up to https://github.com/llvm/llvm-project/pull/197048.
The comment was preserved even though the header was almost never
actually placed first. For the most part that didn't matter, but you can
through bad luck uncover the conflict between Python typedefs (in this
case of pid_t) and ones from include/lldb/Host/windows/PosixAPI.h.
So this patch just moves this header to the front in all its
appearances.
Add support for building the script interpreter plugins as shared
libraries, controlled by the `LLDB_ENABLE_DYNAMIC_SCRIPTINTERPRETERS`
CMake variable. This allows the plugins, rather than `liblldb`, to link
against their respective dependencies such as Python and Lua, which has
several benefits:
1. It makes it possible to gracefully handle the Python or Lua
dependencies not being present, instead of crashing.
2. It enables programmatically deciding which Python or Lua library to
load.
3. Combined with (2), this opens up a path towards not statically
linking with Python at all, which is required in order to load the LLDB
module in a different Python interpreter (at least this is a hard
requirement on Darwin due to two-level namespaces and install names).
This PR completes #183791
Using Python's built-in help shows an empty screen when the statusline
is enabled. The issue is the pydoc pager (e.g. less) which doesn't play
nice with the statusline. Use the "plain" pager instead.
I considered making this conditional on the statusline, but to do that
right you would need to register a callback that toggles it every time
the setting changes and that doesn't seem worth the complexity.
Fixes#166610
Introduce a unified target hook command system for LLDB that handles
multiple target lifecycle events (module load, module unload, process
stop) through a single Hook class.
The existing target stop-hook commands remain unchanged for now, meaning
stop-hooks can now be created through either target stop-hook add or
target hook add -S.
A future follow-up can alias target stop-hook to the unified system to
consolidate the two paths.
New commands:
- target hook add [-o cmd] [-P class] [-u] [-S] - create a hook (fires
on module load by default, optionally on unload/stop)
- target hook add-filter [filter-opts] <hook-id> - attach stop-event
filters (shlib, function, file/line, thread) to an existing hook
- target hook list / delete / enable / disable - manage hooks
- target hook {enable,disable} {module-loaded|module-unloaded|stop-hook}
<id> - per-event toggling
top-event filter options live on a separate add-filter command rather
than on target hook add, keeping the add command clean and making the
system safe for filters on other event types.
Ran the tests + Made sure stop tests are passing as well.
---------
Co-authored-by: Bar Soloveychik <barsolo@fb.com>
Follow up to #193168.
This call adds a newline if there isn't one. Changing these will
eventually let us always add a newline, which is in line with
the other methods on CommandReturnObject.
This is a small part of calls found with:
* VSCode search for
`(\.AppendErrorWithFormat\(([\s\r\n]+)?"(?:(?:\\.|[^"\\])*))\\n"` and
replace with `$1"`.
* Asserting that the last character of the format string is not a
newline.
* Manual inspection.
The LLVM Coding Standards [1] specify that:
> [T]o match error message styles commonly produced by other tools,
> start the first sentence with a lowercase letter, and finish the last
> sentence without a period, if it would end in one otherwise.
Historically, that hasn't been something we've enforced in LLDB, but in
the past year or so I've started to pay more attention to this in code
reviews. This PR brings more error messages in compliance, further
increasing consistency.
I also adopted `createStringErrorV` where it improved the code as a
drive-by for lines I was already touching.
[1] https://llvm.org/docs/CodingStandards.html#error-and-warning-messages
Assisted-by: Claude Code
lldb had three preprocessor defines for logging,
LLDB_LOG - formatv style argument
LLDB_LOGF - printf style argument
LLDB_LOGV - formatv style argument, only when verbose enabled
If you weren't looking at Log.h and the definition of these three, and
wanted to log something with formatv, it was easy to use LLDB_LOGV by
accident. We just had a situation where an important log statement
wasn't logging and it turned out to be this. This is fragile if you
aren't looking at the header directly, so I'd like to make this more
explicit. My proposal:
LLDB_LOG - formatv style argument
LLDB_LOG_VERBOSE - formatv style argument, only when verbose enabled
LLDB_LOGF - printf style argument
LLDB_LOGF_VERBOSE - printf style argument, only when verbose enabled
The new fouth one is to remove several places where we do `if (log &&
log->GetVerbose()) LLDB_LOGF (...)` in the sources today, and make both
styles consistent.
This PR implements that change, mechanically changing all LLDB_LOGV's to
LLDB_LOG_VERBOSE.
It also updates many of the `if (log && log->GetVerbose()) LLDB_LOGF`'s.
Some uses of this conditional expression do extra calculations in
addition to logging, and so those were left as-is so we're not doing
throwaway work when running without verbose logging.
There were many instances throughout lldb where callers are still doing
`if (log) LLDB_LOG*(...)`, a remnant of when all calls were to the `Log`
object's `Printf()` method, and you had to check if your local Log*
pointer was non-nullptr before calling the method. I removed those,
again keeping ones where work for logging is done in the block of code.
The code changes are all mechanical and uninteresting, but the question
of whether this naming change is widely agreed on is maybe worth
discussing.
Fix missing calls to UnregisterPlugin and add an assert in the
PluginManager that ensures all plugins have been unregistered by the
time the plugin manager is destroyed.
Remove call_once wrappers around PluginManager::RegisterPlugin. Plugins
can be registered and unregistered in Initialize and Terminate
respectively. In its current state, after having called Terminate, a
plugin can never be re-initialized.
This PR changes the way we set the shlib directory helper. Instead of
setting it while initializing the Host plugin, we register it when
initializing the Python plugin. The motivation is that the current
approach is incompatible with the dynamically linked script
interpreters, as they will not have been loaded at the time the Host
plugin is initialized.
The downside of the new approach is that we set the helper after having
initialized the Host plugin, which theoretically introduces a small
window where someone could query the helper before it has been set.
Fortunately the window is pretty small and limited to when we're
initializing plugins, but it's less "pure" than what we had previously.
That said, I think it balances out with removing the plugin include.
Make the interfaces part of lldbPluginScriptInterpreterPython instead of
putting them into their own static library. This avoids the need for an
extra static archive and more importantly a bunch of code duplication
between the two CMakeLists.txt.
Avoid directly including `ScriptInterpreterPython.h` in `SBHostOS`, and
instead go through the plugin interface to obtain the scripting path.
This also deprecates the ``SBHostOS::GetLLDBPythonPath`` method in favor
of the more generic GetScriptPath variant.
The header guard is redundant because
`source/Plugins/ScriptInterpreter/CMakeLists.txt` already gates the
entire `Python/` subdirectory behind `LLDB_ENABLE_PYTHON`.
This revert #181334 and its follow-up PRs (including #181488, #181492,
#181493, #181494 and #181498) as well as Ismail's documentation changes
(#181594, #181717). The original commit causes a test failure in CI
(https://github.com/llvm/llvm-project/issues/181938) but the more I look
at the patch, the more I'm convinced it was not ready to land. It will
be easier to iterate on the feedback by re-landing this than by using
post-commit review.
## Summary
Based on discussion from
[RFC](https://discourse.llvm.org/t/rfc-python-callback-for-source-file-resolution/83545),
this PR adds a new `SymbolLocatorScripted` plugin that allows Python
scripts to implement custom symbol and source file resolution logic.
This enables downstream users to build custom symbol servers, source
file remapping, and build artifact resolution entirely in Python.
### Changes
- Adds `LocateSourceFile()` to the SymbolLocator plugin interface,
called during source path resolution with a fully loaded `ModuleSP`, so
the plugin has access to the module's UUID, file paths, and symbols.
- Adds `SymbolLocatorScripted` plugin that delegates all four
SymbolLocator methods (`LocateExecutableObjectFile`,
`LocateExecutableSymbolFile`, `DownloadObjectAndSymbolFile`,
`LocateSourceFile`) to a user-provided Python class.
- Adds `ScriptedSymbolLocatorPythonInterface` to bridge C++ calls to
Python, with proper GIL management and error handling.
- Results for `LocateSourceFile` are cached per (module UUID, source
file) pair.
- The Python class is configured via: `settings set
plugin.symbol-locator.scripted.script-class module.ClassName`
### Python class interface
```python
class MyLocator:
def __init__(self, exe_ctx, args): ...
def locate_source_file(self, module, original_source_file):
...
def locate_executable_object_file(self, module_spec): ...
def locate_executable_symbol_file(self, module_spec,
default_search_paths): ...
def download_object_and_symbol_file(self, module_spec,
force_lookup, copy_executable): ...
```
### Test plan
```
Added TestScriptedSymbolLocator.py with 3 test cases:
- test_locate_source_file — verifies the locator resolves source
files, receives a valid SBModule with UUID, and remaps paths correctly
- test_locate_source_file_none_fallthrough — verifies returning
None falls through to default LLDB resolution, and that having no script
class set works normally
- test_invalid_script_class — verifies graceful handling of
invalid class names without crashing
```
Co-authored-by: Rahul Reddy Chamala <rachamal@fb.com>
This patch implements the base and python interface for the
ScriptedFrameProvider class.
This is necessary to call python APIs from the ScriptedFrameProvider
that will come in a follow-up.
Signed-off-by: Med Ismail Bennani <ismail@bennani.ma>
Signed-off-by: Med Ismail Bennani <ismail@bennani.ma>
Modify the python wrapper to return uint32_t,
which prevents incorrect child name-to-index mapping and avoids
performing redundant operations on non-existent SBValues.
This patch introduces a new scripting affordance in lldb:
`ScriptedFrame`.
This allows user to produce mock stackframes in scripted threads and
scripted processes from a python script.
With this change, StackFrame can be synthetized from different sources:
- Either from a dictionary containing a load address, and a frame index,
which is the legacy way.
- Or by creating a ScriptedFrame python object.
One particularity of synthezising stackframes from the ScriptedFrame
python object, is that these frame have an optional PC, meaning that
they don't have a report a valid PC and they can act as shells that just
contain static information, like the frame function name, the list of
variables or registers, etc. It can also provide a symbol context.
rdar://157260006
Signed-off-by: Med Ismail Bennani <ismail@bennani.ma>
Signed-off-by: Med Ismail Bennani <ismail@bennani.ma>
The current implementation tries to (1) patch the existing readline
module definition if it's already present in the inittab and (2) append
our patched readline module to the inittab. The former (1) uses the
non-stable Python API and I can't find a situation where this is
necessary.
We do this work before initialization, so for the readline
module to exist, it either needs to be added by Python itself (which
doesn't seem to be the case), or someone would have had to have added it
without initializing.
Use `PyThread_get_thread_ident`, which is part of the Stable API,
instead of accessing a member of the PyThreadState, which is opaque when
using the Stable API.
PyConfig and friends are not part of the stable API. We could switch
back to Py_SetPythonHome, which has been deprecated, but still part of
the stable API. For now, limit the use of PyConfig to when
LLDB_EMBED_PYTHON_HOME is enabled, which essentially means Windows.
Changing the order doesn't seem to matter.
Eliminate calls to PyGILState_Check, which is not part of the Python
Limited C API. In the Locker, we can use PyGILState_Ensure directly. We
could do something similar to replace the assert, but I don't think it's
worth it. We don't assert that we hold the GIL anywhere else.
The behavior of thread initialization changed in Python 3.7. The minimum
supported Python version is now 3.8. That means that
`PyEval_ThreadsInitialized` always returns true and `PyEval_InitThreads`
is never called.
The helper function existed to coordinate initializing the threads and
acquiring the GIL, which is no longer necessary. With that, there's no
point in having the helper at all. This PR eliminates the function and
inlines to GIL acquisition into the caller.
This is a continuation of 68fd102, which did the same thing but only for
StopInfo. Using make_shared is both safer and more efficient:
- With make_shared, the object and the control block are allocated
together, which is more efficient.
- With make_shared, the enable_shared_from_this base class is properly
linked to the control block before the constructor finishes, so
shared_from_this() will be safe to use (though still not recommended
during construction).
So the dSYM can be told what target it has been loaded into.
When lldb is loading modules, while creating a target, it will run
"command script import" on any Python modules in Resources/Python in the
dSYM. However, this happens WHILE the target is being created, so it is
not yet in the target list. That means that these scripts can't act on
the target that they a part of when they get loaded.
This patch adds a new python API that lldb will call:
__lldb_module_added_to_target
if it is defined in the module, passing in the Target the module was
being added to, so that code in these dSYM's don't have to guess.
Since the inner wrapper call might have removed one of the entries from
the global dict that the outer wrapper ALSO was going to delete, make
sure that we check that the key is still in the global dict before
trying to act on it.
Should fix the following compile error on Windows:
C:\Python312\include\pyconfig.h(225): error C2371: 'pid_t': redefinition; different basic types
C:\buildbot\as-builder-10\lldb-x-aarch64\llvm-project\lldb\include\lldb/Host/windows/PosixApi.h(80): note: see declaration of 'pid_t'
This patch improves the synchronization of the debugger's output and error
streams using two new abstractions: `LockableStreamFile` and
`LockedStreamFile`.
- `LockableStreamFile` is a wrapper around a `StreamFile` and a mutex. Client
cannot use the `StreamFile` without calling `Lock`, which returns a
`LockedStreamFile`.
- `LockedStreamFile` is an RAII object that locks the stream for the duration
of its existence. As long as you hold on to the returned object you are
permitted to write to the stream. The destruction of the object
automatically flush the output stream.
Remove Debugger::GetOutputStream and Debugger::GetErrorStream in
preparation for replacing both with a new variant that needs to be
locked and hence can't be handed out like we do right now.
The patch replaces most uses with GetAsyncOutputStream and
GetAsyncErrorStream respectively. There methods return new StreamSP
objects that automatically get flushed on destruction.
See #126630 for more details.
