No 'initialised' event when using 'startDebugging'

[puremourning]

Environment data

• debugpy version: 1.6.7
• OS and version: macOS 13.2.1
• Python version (& distribution if applicable, e.g. Anaconda): CPython 3.11
• Using VS Code or Visual Studio: no

Actual behavior

I'm trying to migrate from the custom 'debugpyAttach' event to the new 'startDebugging' request in DAP. Despite all the discussions on the DAP repo, we have ended up with a DAP requirement says:

A client implementation of startDebugging should start a new debug session (of the same type as the caller) in the same way that the caller’s session was started. If the client supports hierarchical debug sessions, the newly created session can be treated as a child of the caller session.

(emphasis mine).

Previously, with the custom message, there was a 'connect' block that you had to pull out and use as the host/port to connect the 'attach' request to. Now that's forbidden. The above statement is that the client must start another instance of debugpy and just send it the 'configuration' supplied on the 'startDebugging' request. That's what I've implemented.

i.e., given the following simple program:

import time
import multiprocessing as mp


def First():
  for i in range( 10 ):
    print( f"in first x {i}" )
    time.sleep( 0.1 )


if __name__ == '__main__':
  print( "main" )
  p1 = mp.Process( target=First )

  p1.start()
  p1.join()

  print( "Done" )

On execution of p1.start(), I receive 2 (for some reason) startDebugging requests. They have a request of attach and a configuration which itself contains the subProcessId and connect blocks. Vimspector then does per the specification and:

1. starts a new instance of debugpy
2. Sends initialise request
3. On receipt of initialise reply, sends 'attach' request with the above 'configuration' block as-is
4. awaits the 'initialized' event and 'attach' reply

The actual behaviour is that I receive a 'debugpyWaitingForServer' event (whatever that is), then the 'attach' reply, followed by a 'terminated' event. But no 'initialized' event. My assumption therefore is that it's not actually attaching to the subprocess.

Here's a very brief summary of the actual messages:

Parent session:

• start debugpy
• ->initialize / <-reply
• ->launch
• <-initialized event / ->breakpoints / ->configuration done
• <-launch reply
• ...
• <-startDebugging [config 1] / reply [child 1]
• <-startDebugging [config 2] / reply [child 2]

Child 1:

• start debugpy
• ->initialize/<-reply
• ->attach [config 1]
• <-debugpywaitingforserver
• <-attach reply (?!)
• <-terminated

Child 2:

• start debugpy
• ->initialize/<-reply
• ->attach [config 2]
• <-debugpywaitingforserver
• <-attach reply (?!)
• <-terminated

Full Message traces of the parent (top level) session and the 2 "child" sessions are here:
https://gist.github.com/puremourning/24160d0683e2c65dc81d36e8c37db1d9

Expected behavior

Clients should be able to start a new debugpy and send the request per the specification. Then a full initialisation sequence should occur.

It's entirely possible that I have a bug here, but I'd like to at least confirm that what I'm doing/saying is inline with your expectations.

Steps to reproduce:

1. Using the above script, initialise and step through he multi-process creation
2. on receipt of the 'startdebugging' request, copy the 'configuration' to the 'attach' request's 'arguments'

[int19h]

I don't think that's necessarily the only interpretation of the spec - it's certainly not how I took it. My understanding is that "in the same way that the caller’s session was started" just means that the new debug configuration has to be interpreted in the same manner it would have been interpreted if it were used for the original session. Thus, the expected behavior for "attach" would be to interpret the host/port and connect directly, not to spawn a new debugpy instance (since for a top-level session you would only do that for "launch" and not for "attach"). This is also how VSCode interprets it based on my testing, so it seems to have been the intent.

"debugpyWaitingForServer" is a custom event that is used to indicate that the adapter reached the state where it's waiting for a debug server hosted in the debuggee process to connect to it on the specified host/port. So far as I know, the only current use is in tests, so that the test runner can spawn the debuggee and tell it to connect to the adapter running in "listen" mode at the right time. Since it's only used internally, it's not really documented or considered stable, but if these semantics are useful, we could change that.

[puremourning]

Thus, the expected behavior for "attach" would be to interpret the host/port and connect directly, not to spawn a new debugpy instance (since for a top-level session you would only do that for "launch" and not for "attach"). This is also how VSCode interprets it based on my testing, so it seems to have been the intent.

Maybe that's what' VSCode does, but the contents of the 'configuration' block may not contain client-specific data. From the spec:

• Arguments passed to the new debug session. The arguments must only contain
• properties understood by the launch or attach requests of the debug
• adapter and they must not contain any client-specific properties (e.g.
• type) or client-specific features (e.g. substitutable 'variables').

This is also how VSCode interprets it

This is the point; The 'connect' block is a vscode-specific implementation detail and is not specified in the protocol. VScode != the protocol and neither is the protocol defined as "de facto" what vscode does or doesn't do.

I'm almost certain it was not the intent, because I recall the discussion on the DAP repo (at length) about this. And the DAP guys were adamant not to 'standardise' any concept of 'transport' and thus the only implementable protocol assumed nothing about transport and required that clients be dumb about it. Indeed, @connor4312 agreed that's a correct interpretation.

To put it another way, with reference only to what is specified in DAP, exactly what should a client do when receiving a startDebugging request from debugpy?

[puremourning]

To explain why this is difficult, it means that I go from having custom code just-for-debugpy "debugpyAttach" (to pull out the host/port and construct some 'multi-session' configuration) to having custom code just-for-debugpy 'startDebugging' request (to pull out the host/port and construct some 'multi-session' configuration), which seems to defeat the purpose of having a standardised protocol

(And FWIW I actually argued in favour of your original-debugpyAttach style request where some details of the transport could be specified, and standardised, because I completely see the value of that, but the DAP guys weren't having it)

[int19h]

It looks like I missed much of that conversation, unfortunately. Reading through it, it still feels vague to me as to what client and adapter responsibilities actually are wrt each other. And, given the way it is implemented in VSCode by the same people who designed the spec, it seems that it's not just me. :)

Anyway, it seems that the immediate issue is that debugpy doesn't even attempt to handle "connect" in the adapter and expects the client to handle it instead (so you can't spin it up in stdio and then do "attach" / "connect" and expect it to work) - which is different from all other configurations including "attach"/"listen". It could be fixed by treating such an "attach" session as pass-through to the specified address; that would be rather inefficient, but I don't see the alternative given the spec as designed.

OTOH the other possibility is to follow the "same as initial session" rule but connect over TCP for that initial session as well - the client host/port for the adapter remains the same, so parsing "connect" is redundant if you already know them from the initial session. And debugpy itself doesn't care if you connect over TCP and then do "launch". So I guess debugpy is technically compliant, but only in this mode, not if you're using it over stdio.

[puremourning]

but connect over TCP for that initial session as well

you mean as in start the adapter with python3 debugpy/build/lib/debugpy/adapter --port ${some_port}, then connect to that self-same port for each startDebugging request ? If that would work, I can do that; that's actually what vscode-js-debug requires too.

[int19h]

Yep, exactly. Although I should note that none of the debugpy tests actually test this particular combination of TCP + "launch", so there may be some surprises. But the code itself is all transport-agnostic, and subprocess debugging already works TCP-only in attach scenarios, so launch shouldn't be any different.