Rust spoils me, I rarely need to reach for a debugger thanks to structured logging via
tracing and errors with spantraces thanks to
eyre. When I do find myself wanting to really dig my grubby little paws into some code, configuring the debugger targets can feel intimidating.
Experience has taught me that I can conquer intimidating things though, and sometimes that is best done with a little help. While hacking on a
pgx bug, I realized I was using a newly installed system, and needed to reconfigure my debugger settings.
This made a good opportunity for me to share with you! This short article will cover how to configure Visual Studio Code with CodeLLDB so you can visually debug your
pgx extensions. This includes being able to step into the
postgres sources. While the instructions may be Arch Linux specific, they should be able to be adapted to a different Linux.
Before we start, set the following
sysctl to ensure you can actually do a debug as a non-root user:
Now, the base packages we need:
If you haven't already, setup
yay or another AUR helper:
Next, install the Microsoft branded Visual Studio Code (
visual-studio-code-bin) from the AUR. (The Open Source build,
code doesn't have the extensions we need.)
Don't be brash, read the
yayshows you, ensure you are comfortable with them. Do this every time, no matter what.
Let's install some extensions,
rust-analyzer, CodeLLDB, C/C++, and Hex Editor:
We'll then pull the PostgreSQL sources locally and build them into a
Now let's clone the
pgx sources locally and install from there:
# Checkout a specific revision if desired.
Finally, we'll create an extension to explore with and open up
Update the default PostgreSQL version feature. Edit the
[features] block to have:
# debuggable/Cargo.toml # ...  = ["pg14"] # ...
code window has the
debuggable extension open, we're going to make a 'Workspace' with
pgx in the same window.
Hit Ctrl+Shift+P (That is -- hold down the Control key while then holding the Shift key, then also the P key, releasing them afterward) then enter into the command pallette
add folder to workspace and ensure the picked selection is indeed the option:
Workspaces: Add folder to workspace....
In this file chooser, select the
~/git/tcdi/pgx folder we cloned earlier.
Do the same with
At this point, consider saving the workspace via the "File" menu.
Debugging the SQL generation process
While it's (hopefully) unlikely you'll encounter bugs with the SQL generation process of
pgx, let's cover that first.
pgx's SQL generation occurs via
schema subcommand, usually invoked via the
cargo pgx schema call. The
cargo-pgx binary will call
cargo to build the library,
postmaster 'mock', inspect the symbols which the various
pgx macros generator, then invoke the symbols to gather metadata. Finally, it feeds that metadata into the
As far as I am aware, there isn't a good way to hook up
lldb to a running proc macro.
The best info I could find was in 'Testing Proc Macros' and "Debugging tips".
If you happen to know of a way, please email me! I'll give you credit in this section.
Instead, you will likely rely on
Now you can use
cargo expand on the entire extension (with no args), or select specific symbols to expand:
$ cargo expand hello_debuggable Checking debuggable v0.0.0 Finished dev target in 0.22s
pgx macros will generate other symbols to expand, too. Typically
#[pg_extern] macros produce a
$FUNCNAME_wrapper function which is what PostgreSQL calls:
$ cargo expand hello_debuggable_wrapper Checking debuggable v0.0.0 Finished dev target in 0.22s unsafe extern "C"
For SQL generation, most macros will also produce like a
$ cargo expand __pgx_internals_fn_hello_debuggable Checking debuggable v0.0.0 Finished dev target in 0.22s pub extern "C"
cargo-pgx's SQL generation
To debug the
cargo-pgx side of SQL generation, you can create a debug configuration in
debuggable/.vscode/launch.json with this:
Navigate over to
tracing::trace!(sql = %ext_sql);,, it should be in the
to_sql function, click the red dot that appears when you hover to the left of the line number there, and set a breakpoint.
In the "Run & Debug" pane (selected in screenshot above on the left), hit the "Play" button beside "SQL generation" at the top. In a terminal you should see
cargo pgx schema get run and begin to process the SQL, it will hit the breakpoint during this process and allow you to inspect and work with it:
At the top middle of the screenshot above, you can see the 'Continue', 'Step over', 'Step into' and related buttons. To the left side we can see 'Variables', the 'Watch' pane, as well as the call stack which caused us to arrive at a particular position.
Debugging running extension code is a little bit different. Let's recall something about how PostgreSQL works:
The PostgreSQL server can handle multiple concurrent connections from clients. To achieve this it starts (“forks”) a new process for each connection. From that point on, the client and the new server process communicate without intervention by the original postgres process. Thus, the supervisor server process is always running, waiting for client connections, whereas client and associated server processes come and go. (All of this is of course invisible to the user. We only mention it here for completeness.)
-- PostgreSQL Architectural Fundamentals
So PostgreSQL starts a supervisor which then forks a new process each connection. This means, if we're debugging extension, we want to attach to that fork.
CodeLLDB will let us attach to a running process this way, so we'll add a new job to the
Then we'll change up the
#[pg_extern] function in the code to make it more interesting:
Set a breakpoint on the highlighted line, then in a terminal run
cargo pgx run.
Once you reach a
Select the 'Attach' option from the debugger dropdown. (Where we hit 'SQL Generation' before in the 'Run & Debug' pane.) You should see a
postgres process with your extensio name.
Attaching won't immediately break on anything, but when you start running commands from
psql you can trigger breakpoints within the extension:
CREATE EXTENSION debuggable; SELECT hello_debuggable ; -- Breakpoint hit!!!
On the breakpoint, you'll see that familiar screen again:
pgx and PostgreSQL code
Having now learnt to debug
cargo-pgx and our own extension, how do we debug if code slips inside
pgx, or even PostgreSQL?
Let's modify our
hello_debuggable to call something over the Server Programming Interface (SPI):
cargo pgx run again, and attach to the process like before. Then in
DROP EXTENSION IF EXISTS debuggable CASCADE; CREATE EXTENSION debuggable; SELECT hello_debuggable ;
Break! Now we're going to 'Step into' this
Now, navigate over to
postgres/src/backend/executor/spi.c and set a breakpoint on the
int SPI_connect(void) function:
Now hit the 'Continue' on the debugger and we should break inside PostgreSQL:
Fantastic, now we can debug almost every part of our
pgx extension. That's all for this article today!
I hope you manage to find your bugs! If not, let us know in Discord