Debugging under python 3.11 will skip the breakpoint

[xxc-zsz]

Sample code below:

#Bt_Connection
#python -m pip install brainflow
#pip install pybluez
#python.exe -m pip install --upgrade pip

import argparse
import time
import socket
import os

from brainflow.board_shim import BoardShim, BrainFlowInputParams, BoardIds
from brainflow.data_filter import DataFilter, FilterTypes, DetrendOperations, AggOperations

msg="X"
print(msg)

test= "test"
print(test)

launch.json

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Debug Current File",
            "type": "python",
            "request": "launch",
            "program": "${file}",
            "console": "integratedTerminal",
            "stopOnEntry": true,
            "justMyCode": false
        }
    ]
}

When I choose to use python 3.11 as the interpreter, there is an error in debugging. After the first breakpoint stops, I click Continue, skip the next two breakpoints and output the result in the terminal.

Debugging with the python 3.10 version then everything works fine.

You can see the python version in the lower right corner.

Related StackOverflow questions.

[ahmdgawad]

I have the same issue, btw what's the font you're using?

[xxc-zsz]

btw what's the font you're using?

@ahmdgawad SimSun. this is my settings.json:

    "editor.fontFamily": "SimSun, Consolas, 'Courier New', monospace",

[davetapley]

See also ⬇️ where @AdamYoblick has a failing test which might be related.

• Add Python 3.11 to the ci #1107

[AdamYoblick]

Hi all, I'm planning in looking into this, but I have some other tasks with deadlines that I have to get to first. This won't get missed, I promise 😄

[airportyh]

I can repro this bug, and I also see this warning:

Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off

[airportyh]

I determined that setting -Xfrozen_modules=off did not fix this bug.

[int19h]

The frozen modules warning is due to #1251 and should be unrelated.

Could you clarify which version of debugpy this is?

[airportyh]

Version 1.8.0

[airportyh]

More info for you: if single step through op's code example, it goes until the line:
from brainflow.board_shim import BoardShim, BrainFlowInputParams, BoardIds and then stepping over that line exits.

If I remove references to brainflow and instead replace them with import flask or import requests, everything works as expected. There must be something within brainflow that is triggering this bug.

[airportyh]

I've isolated it to pkg_resources. You can repro the bug with:

import pkg_resources
print('hello world')

Set a breakpoint on line 2, start the debugger, and it fails to pause on line 2.

• edited/fixed code example *

[xu20160924]

I have the same issue under Python 3.11, but it's worse in my situation it won't even pause at the first breakpoint.

[tboddyspargo]

Hi, folks! I'm running into the same thing, but I haven't pinpointed the contributing factors or catalysts in my environment.

Do we have a sense if the fix will be within debugpy itself (and which area), or if other packages need to make adjustments to comply with debugpy + 3.11 expectations?

[int19h]

Could you share the logs ("logToFile":true in launch.json for launch, set DEBUGPY_LOG_DIR=... for target process for attach)? I would be especially interested in debugpy.pydevd*.log.

[tboddyspargo]

@int19h - This time I put a breakpoint on a function name that I knew was as early in my third party entrypoint code (huey_consumer.py) as possible and added the --wait-for-client option. I DID hit that breakpoint, but after continuing the execution, I hit none of the other breakpoints in my own code that I expected to hit.

Here are the log files - let me know if I missed something:
debugpy.server-1.log
debugpy.pydevd.1.log
debugpy.adapter-10.log

[int19h]

Thank you! There's no "smoking gun" in the log, but one thing that I've noticed is that a bunch of threads are getting spawned. Are those breakpoints expected to be hit on other threads? If so, are the threads started using stdlib threading, or through the OS APIs or some other native library? In the latter case, you may need to use debugpy.trace_this_thread() to ensure that breakpoints apply there.

[tboddyspargo]

Are those breakpoints expected to be hit on other threads? If so, are the threads started using stdlib threading, or through the OS APIs or some other native library?

I'm not 100% sure, but it does look like huey's default behavior is to execute tasks in a thread (most of my breakpoints were inside of a Thread). I can confirm that that is a stdlib threading.Thread. There are also third-party libraries spawning processes and threads (e.g. fsspec) where I don't have any breakpoints.

However, I'll also mention that I did have one breakpoint inside of an __init__.py of a package that's one of the first things to be imported by the entrypoint script and I didn't hit that one while I did hit breakpoints in the entrypoint script itself (huey_consumer.py).

UPDATE: After successfully hitting a breakpoint in the huey_consumer.py, I started stepping through the code to see when I would "lose" control and if I could get into my own code or not.

When I got to the place where debugpy tried to step into my code, I saw this:

So, for some additional context for this situation: This debugger is running in a docker container with the source code bind-mounted to the container and I've specified a pathMapping in VSCode's launch "attach" configuration (and I've confirmed that the paths are correct in the logs I attached). VSCode is using a debgupy client to "attach" to the process running in the container. Could debugpy be failing to hit these breakpoints due to changes in how python 3.11 and debugpy look for and find the source code files where it's supposed to hit breakpoints?

FWIW, so far, I haven't (I don't think) had any issues hitting breakpoints during local test execution of the same code. I originally chalked that up to other differences in the test environment (no threading in huey, etc.).

[int19h]

What does it show on the call stack when this message appears? You should be able to see the full path by hovering over the corresponding stack frame or the tab with the error message in the editor.

The error does imply that path mappings are off, but yes, the logs do indeed seem to show correct mapping for all breakpoints. Just about the only thing I could think of that could still throw it off potentially is if the remote paths have a casing difference somewhere...

Another possibility would be to use the Python breakpoint() builtin to induce a breakpoint where you expect one to be hit when set normally - since that mechanism doesn't rely on paths, it should work even if path mapping is off, and the paths that you'll then see on the call stack (and in the corresponding logs) once it breaks should reflect the remote filesystem as seen by the debugger - any mismatches would also be visible there.

[tboddyspargo]

Thanks so much for working with me on this, @int19h!

What does it show on the call stack when this message appears? You should be able to see the full path by hovering over the corresponding stack frame or the tab with the error message in the editor.

... Just about the only thing I could think of that could still throw it off potentially is if the remote paths have a casing difference somewhere...

Indeed, the full path that debugpy/VSCode was trying to display when it hit the breakpoint was incorrect. In my case, it was ~/dev/workspace/pipeliner/core/utils/timer.py, while the correct path should have been ~/dev/workspace/pipeliner/pipeliner/core/utils/timer.py (note the additional pipeliner folder in the path).

That puts me hot on the trail of a mapping issue, because the pathMapping that I'm using takes advantage of ${workspaceFolder} variable from VSCode. The config lives in ~/dev/workspace/.vscode/launch.json, but the area of code I'm debugging (and which is mapped to the container/debugger) lives in a separate folder of my multi-root workspace (~/dev/workspace/pipeliner). I'm guessing that VSCode is expanding ${workspaceFolder} in the context of the pipeliner subfolder of the multi-root workspace instead of the . folder, where the launch configuration actually resides.

I even tried to work around the issue by using ${workspaceFolder:workspace} to force a specific folder of the multi-root workspace to be interpolated there, but i still seems to result in the same incorrect file path being used...

Given that the path mappings logged in the debugpy.pydevd*.log seem correct and my ${workspaceFolder:workspace} experiment didn't change the outcome, my best guess is that VSCode is performing some EXTRA (and incorrect) path mapping in the multi-root workspace case when it tries to pass control to the IDE debugger and display the file with the breakpoint. The failure to find the corresponding file causes a continue request to be sent back to the debug server, making it appear as though the breakpoints aren't getting hit (speculating a lot more here...).

@int19h - Is this still the right project and issue to be using for my situation? I don't want to muddy the waters too much if this is getting into different territory from the focus of this particular issue page.

Another possibility would be to use the Python breakpoint() builtin to induce a breakpoint where you expect one to be hit when set normally - since that mechanism doesn't rely on paths, it should work even if path mapping is off, and the paths that you'll then see on the call stack (and in the corresponding logs) once it breaks should reflect the remote filesystem as seen by the debugger - any mismatches would also be visible there.

I gave this a shot, but it didn't seem to work. I just added a breakpoint() builtin function call to a spot in my mapped code paths that I know gets executed and my attached debugger never stopped there. I'm not sure if I missed another step for that work-around.

[tboddyspargo]

@int19h - I'm very sorry to have taken your time with what was ultimately "user error." It turns out we accidentally had TWO COPIES of the source code in the container, the one that was properly mapped to the debugger was not the one that was actually being executed. After fixing that, I'm able to use the debugger without issue.

Please feel free to hide my comments as "off topic" since they are probably just distracting for whatever might actually be going on for other folks.

Thanks again!

[xu20160924]

I don't know why, but after selecting interpreter to xxx/venv/bin/python manually, it worked. It's so strange and I am beginning to doubt my problem maybe is not the same as yours? But hope it is useful for you.