Breakpoints in Dart SDK code are not hit

[bleroux]

I tried to put breakpoints in Dart SDK code but those breakpoints are ignored.

To reproduce this I setup an environment with :

• A fresh SDK installation : Dart SDK version: 2.15.0-68.0.dev (dev) (Wed Sep 1 05:51:32 2021 -0700) on "linux_x64"
• A fresh Dart project created from DartCode command Dart: New Project and selecting simple-console template
• Try to put a breakpoint in print source code

I used two ways to put this breakpoint :
1 - Using 'Go to definition' and add the breakpoint
2 - Launching a debug session and 'Step into' to reach Dart SDK source code and add the breakpoint

Two breakpoints were created with different paths, neither works :

DartCode debugging logs :
Dart-Code-Log-2021-08-05 13-41-24.txt

Corresponding session in video :
https://user-images.githubusercontent.com/840911/132003058-ce293e04-b9d7-4642-bc9e-2889701ea086.mp4

[DanTup]

@bkonyi do you know if this is expected to work? I was certain it did (except for Flutter), but I can repro this and it doesn't work. When you Go-to-Definition you end up in a file like inside the SDK folder which results in us sending a breakpoint like this:

{
	"id": "10",
	"jsonrpc": "2.0",
	"method": "addBreakpointWithScriptUri",
	"params": {
		"isolateId": "isolates/996847484333475",
		"scriptUri": "file:///Users/danny/Dev/Dart%20SDKs/v2.13.4/lib/core/print.dart",
		"line": 9
	}
}

The breakpoint is never resolved/never hit when printing though.

[bleroux]

Comments in flutter/flutter#27189 are very informative and still relevant.

I tried the following dirty hack in Dart-Code/src/debug/dart_debug_impl.ts -> setBreakPointsRequest :

		if (uri === "dart:core/print.dart") {
		  uri = "org-dartlang-sdk:///sdk/lib/core/print.dart";
		}

With this, I'm able to add a breakpoint in print method for a Dart project. But it doesn't work for a Flutter project.

It seems there are two distinct issues :

• dart: URIs should be converted before calling VM Service addBreakpointWithScriptUri RPC
• org-dartlang-sdk:///sdk/lib/core/print.dart is not the right URI when running on Flutter embedded SDK

[DanTup]

dart: URIs should be converted before calling VM Service addBreakpointWithScriptUri RPC

I'm not certain this is the correct place to fix this versus the VM. When we step into dart:core/print.dart, the VM uses dart:core/print.dart as the URI of the script:

{
	"jsonrpc": "2.0",
	"result": {
		"type": "Stack",
		"frames": [
			{
				"type": "Frame",
				"kind": "Regular",
				"location": {
					"type": "SourceLocation",
					"script": {
						"type": "@Script",
						"fixedId": true,
						"id": "libraries/@0150898/scripts/dart%3Acore%2Fprint.dart/0",
						"uri": "dart:core/print.dart", // <------------------------------ see here
						"_kind": "kernel"
					},
					"tokenPos": 321
				},

It seems sensible that addBreakpointWithScriptUri should accept this as a script URI, which is what we're currently sending:

{
	"id": "49",
	"jsonrpc": "2.0",
	"method": "addBreakpointWithScriptUri",
	"params": {
		"isolateId": "isolates/3306004115768255",
		"scriptUri": "dart:core/print.dart",
		"line": 10
	}
}

Handling this in the VM would fix it for all editors rather than just VS Code (I also don't think VS Code should need to understand the internals of what org-dartlang-sdk is - it feels like an implementation detail we shouldn't rely on). I'd like @bkonyi's opinion about this before adding workarounds here (not least because the DAP is moving to the SDK so any workaround would end up being replicated there too).

[bleroux]

I fully agree with you @DanTup : when I wrote "URIs should be converted before calling VM Service addBreakpointWithScriptUri RPC", I described the issue not what should be done to fix it 😄

I too think that it would be better to fix this issue at the VM level, otherwhise all debugging clients (DartCode, IntelliJ Plugin, DevTools, ...) will have to provide their own workaround.

I will try to dig deeper to see what can be done with other VM Service RPC (getScripts, getSourceReport, ...)

[bkonyi]

Yeah, this should almost certainly be fixed in the VM.

Dealing with breakpoint resolution within SDK code is a pain since the VM doesn't always reference the code on disk. At the very least, for the standalone VM, I'm 99% sure we always load SDK code from a pre-built snapshot so breakpoints need to be set using the org-dartlang-sdk:// URIs, as the VM doesn't know (or care) where the actual source exists on disk. As for Flutter, I'm not sure exactly how those are referenced as things are packaged differently. I'll have to take a closer look.

[bleroux]

@bkonyi Yes, breakpoint resolution is a pain 😅
I'm digging in the VM code and it seems there are some inconsistencies in Script URLs resolution. When done with my investigation I will create an issue on Dart SDK repo to have your opinion.

@DanTup Found some code in flutter/intellij dealing with BreakpointsUri : https://github.com/flutter/flutter-intellij/blob/4bd532c5dd021e88a6ad664e8ff8f929475a7796/src/io/flutter/run/FlutterPositionMapper.java#L183
I don't think it's a fully working workaround for this issue, but it's funny they choosed to add several breakpoints at once.

[DanTup]

@bkonyi

Yeah, this should almost certainly be fixed in the VM.

Ok, I will file some bugs in the SDK later with clear examples of the issue and how to repro.

Dealing with breakpoint resolution within SDK code is a pain since the VM doesn't always reference the code on disk.

Yeah, though it's also a pain in the IDE where (while not debugging) we only have file paths from the editor and no good way to map them (at least not without edge cases :-)).

For one of the cases above though, there's no file path involved - the VM gives us dart:core/print.dart and we download the source from the VM to show the user. Then they add a breakpoint and we send it back as dart:core/print.dart and it still fails. This one seems like it shouldn't really need any mapping anywhere.

@bleroux

I don't think it's a fully working workaround for this issue, but it's funny they choosed to add several breakpoints at once.

We used to do the same in VS Code, but it brings some complications of mapping between a single breakpoint in VS Code and multiple in the VM. Most of the bugs that required doing this (which were breakpoints only working if you sent either file:/// or package: depending on how the file was loaded in the VM) have been fixed by the VM so my preference is to try to continue down that path. Any logic that goes in editors needs duplicating in multiple places, which raises the barrier to a new editor adding debugging support (although, hopefully an SDK-provided DAP will improve that a little).

[bleroux]

After several days of digging (well worth it, as I learned many things: building a custom Dart SDK, building a custom Dart Code, playing with Dart VM C++ code, and more), I found a very simple hack 🥁 :

When calling addBreakpointWithScriptUri just replace dart: prefix with / 🤣

Found this from this VM Code : https://github.com/dart-lang/sdk/blob/main/runtime/vm/object.cc#L12842

That works nicely for Flutter app. But for Dart apps, it works partially (works only when the breakpoint is added after 'stepping in' and it creates a bug : can't find source in editor when selecting the breakpoint).

It's just hack in DartCode, but maybe it could be included, considering that :

• it is a small code change
• it is a step forward compared to now (breakpoints in SDK don't work at all)
• fixing this at the VM level is far more complicated (possible regressions, difficulty to test Flutter apps with a custom SDK, etc)

@DanTup, I will create a PR so you can test this easily, but I don't have a strong opinion on releasing such a hack (maybe behind a flag ?).

I will also pursue investigation on trying to fix the VMService code.

[DanTup]

Thanks for the digging! My only concern with shipping this workaround here, is that if it's fixed in the VM in a way that stops /dart:core from working, things wouldn't work right in the version with the SDK fix.

There's work in progress to move the debug adapters to the SDK which would actually avoid that issue, since the debug adapter code is coupled to the SDK/VM version so it might be less weird there.

I am curious why it works this way though. If it's just an accidental bug and nobody else is doing this, then perhaps fixing it isn't going to regress anything. I'll file an SDK bug about this shortly though with some details, and wait to hear @bkonyi's opinion before deciding what to do (although if we do ship a workaround, I think it might be better to do just in the SDK DAP, which I hope we're not too far off completing and migrating Dart-Code too).

[bleroux]

Hi @DanTup , I agree with your concern with shipping this. I think that people really interested by this will be able to use my PR to build a custom DartCode.
Concerning fixing this at VM level, I will comment on the issue you created to give some details about my digging.
And great to hear about DAP, I might use it to test the VM Service, as its integration tests look very nice 👍 .

[DanTup]

@bleroux is this now resolved by dart-lang/sdk#47204 or are there still issues?

[bleroux]

Hi @DanTup, with this PR, adding a Breakpoint with a 'dart:' URI is fixed.

In Dart Code context, it means that when the VM Service is live, breakpoints could be added after stepping into SDK Code (breakpoints added with the 'dart:' prefix). For flutter app, the breakpoint will be correctly persisted and it will be hit in current and future debug session.

Issues remaining (visible with Dart Code, maybe related to VM or Dart Code) :

1. Dart apps : the breakpoint is persisted but not hit in future debug session (maybe Dart Code related, i will have a look).
2. Flutter app : before launching the app, breakpoints added in SDK Code will have a file Uri file:///home/danny/flutter/bin/cache/pkg/sky_engine/lib/core/print.dart and those breakpoints are not hitted (have to investigate if it can be fixed in the VM Code)
3. Dart and Flutter apps : when the VM is not started, clicking on a 'dart:' breakpoint have no effect
4. Dart and Flutter apps : when the VM is started, clicking on a 'dart:' breakpoint open an editor with the following exception
   Could not load source 'dart:core/print.dart': TypeError: Cannot read property 'data' of undefined at DartDebugSession.sourceRequest (/home/bruno/.vscode/extensions/dart-code.dart-code-3.26.0/out/dist/debug.js:4164:32)
   I had a look in DartDebugSession.sourceRequest, problem here is that it uses cached data(this.threadManager.getStoredData(sourceReference);) and the data id is obsolete.

I will investigate 1 & 2, maybe you can have a look to 3 & 4.

[DanTup]

I think 3&4 are are slightly related to microsoft/vscode#131502. However fixing that probably means those breakpoints will disappear, which also feels akward. I'll note that on the issue and see whether maybe there's a better way to fix this.

[DanTup]

Actually, I think I might be able to fix 4 by returning new sourceReferences when the breakpoints are re-sent.

I don't think 3 is generally possible to fix (any source code that was fetched from the VM in a debug session we have no way fo fetching when the session is gone). However, if we could map the dart: URI back to a location on disk for the majority of of cases (in the SDK lib folder), then that would allow us to do away with most of the dart: URIs during debugging - although we'd need the VM to accept those paths for setting breakpoints (although this also fixes the other issue, of Go-to-Definition into the SDK and set breakpoints there). I think that would be a great change, but I've no idea how complicated/feasible it is (since it's in the VM side).

[bleroux]

Hi @DanTup, after looking at the VM Code, I think that 2. can't be fixed at the VM level.

My understandings are that, for Flutter apps, snapshots files are created ('.dill' & 'snapshot' files, more details here https://mrale.ph/dartvm/). Those files don't store orginal file paths for dart:.
For flutter app, sky_engine package describes in sky_engine/lib/_embedder.yaml how it replaces some dart: lib :

  "dart:async": "async/async.dart"
  "dart:collection": "collection/collection.dart"
  "dart:convert": "convert/convert.dart"
  "dart:core": "core/core.dart"

If it can't be fixed at the VM Level, I see some options :

• avoid setting a breakpoint with a file Uri to sky_engine. Is it the analysis server that returns this path to Dart Code ?
• convert the Uri when calling setBreakpoints (would probably be hacky)

[DanTup]

avoid setting a breakpoint with a file Uri to sky_engine. Is it the analysis server that returns this path to Dart Code ?

Yep. Although I don't entirely understand the mechanics of this (why sky_engine exists, or why the server goes to its source and not the SDK sources inside Flutter's Dart SDK for things like print). It might be correct that it goes into sky_engine, but if so, it feels like a breakpoint set in sky_engine should be hit at runtime.

It would be best if any fixes didn't involve workarounds inside VS Code, because they'd need to be reproduced in each editor (or at least DAP), and it feels like these are implementation details that could change over time, and things may break.

It feels like between the analysis server and the VM, there should be some way for us to get a path from Go-to-Definition that just works at runtime (and if we don't have the correct source code at development time that will run at runtime, then perhaps Go-to-Definition shouldn't be going anywhere). I don't know enough about how the VM works to understand how complex this is though.

[bleroux]

My understandings for your questions :

• "why sky_engine exists" : sky_engine is used to override or extend the Dart SDK.
• "why the server goes to its source not the SDK sources inside Flutter's Dart SDK" : to improve startup performances binary snapshots are used. In those snapshots, Dart SDK is already loaded (with cached source code I think). And snapshot files are generally downloaded from Flutter infrastructure (it could be overrided locally for development, but it seems that for common use cases they are downloaded).

I could have a look to the analysis server. It would be helpful, if you can help me locate where Dart Code calls the analysis server when 'Go-to-Definition' is selected.

[DanTup]

Something I'm not clear on from the above - is which version of the code is the one that will actually run at runtime on the device? What's in the sky_engine package, or what's in the Dart SDK's lib folder? If it's the former, then shouldn't the debugger accept package:sky_engine for breakpoints? If the latter, then why is there sky_engine source code at all? (or, are they always the same?)

It would be helpful, if you can help me locate where Dart Code calls the analysis server when 'Go-to-Definition' is selected.

Go-to-Definition is powered by the analysis servers navigation classes (https://github.com/dart-lang/sdk/blob/master/pkg/analyzer_plugin/lib/utilities/navigation/navigation_dart.dart), although I think it's just using resolution info from general analysis, so the reason it's getting sky_engine is likely through the analysis of the files rather than specifically the navigation implementation.

That said, even if the analysis server could easily be changed to not use sky_engine, it's not going to be able to return dart: URIs, so the best it could do would be map it to the SDK's lib folder - and that means the running VM would still need to be able to handle breakpoints for those paths? (although I think it should do that anyway, because this issue still exists for non-Flutter where there's no sky_engine).

[bleroux]

Hi @DanTup , I have yet to understand Dart and Flutter internals, but here are my understandings for your questions : "which version of the code is the one that will actually run at runtime on the device?" . I think a comparison to Java or Typescript could be made :

• Java : at runtime .class files are loaded by the JVM. For debugger or other tooling, .java files have to be reachable.
• TypeScript : at runtime .js files are loaded by the VM (V8 or other). To debug TS code, sourcemap files are required.
• Dart : at runtime (JIT), core libraies (and more) are loaded from binary snapshots files. The application code is loaded from source files (not sure). Source files for SDK and others are there for build convenience and for tooling
  (https://mrale.ph/dartvm/ , is very interesting, I have to read this again to fully understand)

For instance, in Dart Code we can select the 'print' function and 'go-to-definition'. If we edit its source code and launch the app, the changes are not loaded. We have to rebuild snapshots to see the changes in action.

Second question "What's in the sky_engine package, or what's in the Dart SDK's lib folder?" :

• Dart SDK lib is a full Dart SDK. I think its main use is to run Flutter tools.
• sky_engine is a partial Dart SDK, extended with dart:ui lib and it contains some SDK overrides (such as its own Object.toString function for debugging purpose). In summary, it is a custom Dart SDK tailored for Flutter apps needs.

I had a first look at analysis server code, it contains code to load SDK libs and find 'dart:' mapping in embedder.yaml files. I will try to find if there is a way to get the mapping (between files and 'dart:' libs) that analysis server build internally.
My hope is that, if analysis server exposes this mapping, tools might have a way to find if a given file uri match a 'dart:' lib.

[bleroux]

@DanTup
Analysis server exposes an API to resolve mapping ('execution.mapUri', see closed issue above).
I have some clues on how it could be use in Dart Code, but I don't know if it's worth trying as it means significant work compared to the benefit (breakpoints in SDK for a Flutter app are hit since my commit on dart-sdk).

[DanTup]

Interesting! I tried calling that API with dart:core/print.dart and got back /Users/danny/Dev/Dart SDKs/nightly/lib/core/print.dart which is accurate. I'm not sure there are cases that wouldn't work, but it does seem like it does the mapping I'd like. The only issue is it's in the wrong place for the debugger to use 😞

I don't know if it's worth trying as it means significant work compared to the benefit (breakpoints in SDK for a Flutter app are hit since my commit on dart-sdk).

Although that's a nice improvement, I don't think it's the ideal solution (even ignoring that it's Flutter-only) because it leaves some problems:

1. If you Go-to-Definition into the SDK source, the breakpoint will still be recorded against the file path (not dart: URI) and I believe that still doesn't work
2. If you Go-to-Definition into SDK source, and then while debugging step into the same file, you end up with two files in your editor that seem like they should be the same (one is the local file path, one is the VM-downloaded dart:xxx source)
3. If you set a breakpoint in a dart: URI during debugging, that URI is only valid for that session. For example if you stop debugging, and then click on the breakpoints in VS Code, you'll get an error - because we have no way of getting dart: source code when you're not in a debugging session.

I still think the ideal solution would be that when debugging and you break inside dart:core/foo.dart, the debug adapter reports the actual file location on disk to the editor, rather than "you can ask me (the debug adapter) for this source code". If the debug adapter had access to this same API the analysis server has, that might go a long way towards that - however I'm not sure if:

a) that's feasible (does the VM have enough info? if not, is it realistic to give it?)
b) whether this works with Flutter (eg. does that analyzer API return the correct thing? is it guaranteed there is source on disk that matches what will run?)

I'll see if I can get some time with people that know this better than me for a discussion and see if we can come up with something. This issue has bugged me forever and it'd be good to have a plan (or a concrete reason why it can't work).

[bleroux]

@DanTup

"a) does the VM have enough info?"

• No the VM doesn't have enough info (it loads sources from .dill files that could be created from any computer). But it knows the relative path : org-dartlang-sdk:///sdk/lib/core/print.dart. So replacing org-dartlang-sdk:///sdk by the current SDK folder is the way to get the absolute path.

"a) if not, is it realistic to give it? "

• That's one solution I considered : get the SDK lib folder, call a VM Service Api that combined this path with the relative path known by the VM (or expose this internal relative path on an existing API). It would mean updating VM Service code, calling the analysis Api on the client side and using the absolute path when adding a breakpoint (or when opening a source file). That what I meant by "significant work" 😅

"b)` whether this works with Flutter (eg. does that analyzer API return the correct thing?"

• Yes, see this comment, I tested execution.mapUri on a Flutter app.

"b) is it guaranteed there is source on disk that matches what will run?)"

• No guarantee : sources on disk could be edited and the VM won't use them. But I don't think it's a big deal compared to current state.

If we want to go further, we can try to split this on several step. First one would be to fully understand how and where client code would be impacted.
Considering we would be able to compute the absolute path, do you think that one step would be to update the following code ?

Dart-Code/src/debug/dart_debug_impl.ts

Lines 1085 to 1091 in e037571

	// Download the source if from a "dart:" uri.
	let sourceReference: number | undefined;
	if (uri.startsWith("dart:") || uri.startsWith("org-dartlang-app:")) {
	sourcePath = undefined;
	sourceReference = thread.storeData(location.script);
	canShowSource = true;
	}

So that the editor will open source file on disk ? And breakpoint would be registered with a source path instead of a source reference ?

If yes, another step would be that setBreakPointsRequest call the VM Service after converting the breakpoint file path to a dart: URI.

[DanTup]

Sorry for the delay!

Considering we would be able to compute the absolute path, do you think that one step would be to update the following code ?

FWIW - all of the code in dart_debug_impl is going to go away soon. This code is the "debug adapter" which is a layer over the VM Service that implements the Debug Adapter Protocol. I've been working to upstream this into the Dart SDK here:

https://github.com/dart-lang/sdk/tree/main/pkg/dds/tool/dap#debug-adapter-protocol

That means any solution should be implementable there instead, although its implementation is fairly similar, and the equiv of the code above is here:

https://github.com/dart-lang/sdk/blob/6535fe9fd099c5b2752942fe7ee7df8382a42ab1/pkg/dds/lib/src/dap/protocol_converter.dart#L395

If we can reliably convert the non-file URIs into file URIs here (and as you say, back in setBreakPointsRequest) I think that would be a good step - although I'm not sure if we can reliably do that? (how do we know if an incoming file path should be mapped to dart:?).

That said, I know @bkonyi was looking at adding VM Service to map URIs (intended for package-to-file/file-to-package). If that could be extended to handle dart: URIs (I suspect it's much more complicated, but it feels like the right place to have it) it would be usable by all debuggers and not only those going through DAP. I think it may be worth seeing if that's a possibility before doing too much here (unless it's fairly trivial to do and later revert if no longer required).

[DanTup]

I'm expecting dart-lang/sdk@08db718 to handle this, but it'll only work when we've moved over to the SDK DAPs so I'll keep this issue open to verify once that happens.

I hope to add an experimental flag to opt in to this after the next SDK release, and then start moving over by default based on feedback from those using it.

[DanTup]

This issue has been fixed for some upcoming releases. I tested it today using the current beta version of the Dart SDK (2.18.0), the pre-release v3.47.20220808 version of the Dart extension, and with the dart.previewSdkDaps: true setting.

When using all of these, the Dart SDK libraryes (such as dart:core) are not mapped back to your local SDK sources (the same ones you get to using Go-to-Definition in the editor), and breakpoints set in those files will be mapped over to dart:core by the debug adapter.

Some of these pieces are not released in stable releases yet, and the SDK DAPs will be a gradual roll out over the coming releases (although you can opt-in explicitly with the dart.previewSdkDaps: true setting).

Please file issues if you still have issues with this once you have confirmed you're on:

• = v3.47.20220808 of the Dart extension

• = v2.18.0-271.8.beta of the Dart SDK (or a Flutter SDK that includes this version)

• The dart.previewSdkDaps setting set to true, or you have confirmed that you've been enabled automatically as part of the gradual rollout (this will include the text "Running SDK DAP" in the log opened by the Dart: Open Extension Log command).

[DanTup]

I noticed today that this fix does not entirely work when running Flutter apps - they return a different org-darlang-app:/// URI to the Dart SDK.

I've filed #4128 to track this, and dart-lang/sdk#49863 to find out if it's a bug, and if not exactly what the mapping rules are.