Can not go to definition of non local libraries (i.e. dependencies)

[i-am-the-slime]

Is this because it is downloaded in some binary form?

[ndmitchell]

Currently goto definition only works for the local package. Where should it take you for the base library? Typically people don't even have a copy of the source installed. The Haddock documentation is one option, but it's no longer really goto definition...

[cocreature]

fwiw we do have some logic for cross-package goto definition based on HIE files that works for DAML. It allows you to configure the location of the sources so it would be possible to point it to something like ~/.cabal/packages/hackage.haskell.org. But it definitely requires more work and the location of the sources should probably come from hie-bios as there is no way we can figure out where they are located (if they are present at all).

[jacg]

The Haddock documentation is one option, but it's no longer really goto definition...

Could we add support for goto documentation?

[seagreen]

Currently goto definition only works for the local package. Where should it take you for the base library? Typically people don't even have a copy of the source installed. The Haddock documentation is one option, but it's no longer really goto definition...

This doesn't help in the short term, but it's good to remember these aren't immutable restrictions.

haskell-code-explorer allows cross-package jumping (so long as you've indexed each dependency) and it is insanely nice. Seriously, go to a random function and try it out. I really, really want this in my editor as well.

That said, I know that ghcide is new, and at the moment I'm just delighted that it works so well. But I'd like to keep the sights for Haskell editor tooling set high in the long run. I'd be 100% OK with downloading the source for each of my dependencies (and waiting X hours for them all to be indexed) if it meant jump-to-definition would always work in my editor.

PS In the meantime jumping to the Haddocks would be totally fine with me.

[hasrthur]

I recommend using codex meanwhile. It can generate ctags for all the dependencies

[seagreen]

@arthurborisow: Great point as a short term solution.

In the long run I'm really hoping for something better than codex though. Lots of terms (especially ones with common names like Success) appear in many different packages, and it's jarring to be taken to the wrong one so often.

[ndmitchell]

Note that we have the fully qualified name, as base-version.Prelude.Just or whatever, it's just where to send you that we need to figure out.

[jneira]

This feature is included in @wz1000 hiedb branch of ghcide and merge it would bring it to master

[jneira]

@wz1000 commented in a duplicate issue in how to use its hiedb branch, that includes this feature: https://github.com/haskell/ghcide/issues/745#issuecomment-683334743

[jneira]

The new pull request that will add hiedb support in hls is #704

[wz1000]

This will not be addressed in #704 but in an upcoming PR.

[jneira]

@wz1000 hi! had you the chance to prepare that pr finally?

[jneira]

There is a closely related ghc proposal: https://discourse.haskell.org/t/extended-dependency-generation-support/2811
Not sure if it is necessary to implement the feature (as there is a wip since quite time ago)

[jneira]

@pepeiborra @wz1000 it would be great to have some idea about the plans for this, as there are lot of people eager to see this land. Otoh i am not sure if there is a wip about or not.
thanks!

[wz1000]

I had a somewhat working implementation of this, but there were some issues. In principle, most of what needs to be done is provide an implementation for lookupMod here - possibly taking inspiration from this commit which reverted support for this feature from my WIP branch (there were some unresolved problems with the feature and I decided to focus on getting the rest of the branch merged first).

The feature as before it was reverted by that commit worked in the following fashion

• The user to build their dependencies with -fwrite-ide-info and then index the resulting .hie files with ghcide/HLS. This can be done by cding to the project root directory and calling ghcide hiedb index dir/with/hie/files.

• Once the dependencies are indexed into the database, when we get a go to definition request for something in an indexed module, we check if we have a source file registered for the module in the database. If so, we can just return that location. (this is still the code path as implemented today)

  Otherwise lookupMod is called. Currently lookupMod always fails. To implement the non-local definition feature, lookupMod needs to get the contents of the module from the indexed .hie files and then write out a file somewhere onto the filesystem with those contents. It then needs to register this file in the database as the source file for this module (so that future lookups would directly jump to the file we wrote). Finally, it returns the location of this file so the client can jump to it.

  The file has to be written somewhere in the users project directory or we won't be able to handle LSP requests for it from the same server.

The major problems which need to be solved are:

• A nicer workflow for allowing the user to access this feature. Currently they need to add something like the following to their cabal.project:

  package *
    ghc-options: -fwrite-ide-info -hiedir /some/absolute/directory
  

  And then manually index /some/absolute/directory/ whenever it changes.

• We have a new class of read-only haskell source files which we can't compile. We have to somehow ensure that most rules
  and plugins cannot try to run on these files (Add formatting plugin for cabal files which uses cabal-fmt #2047 seems to be facing similar issues, a solution could be shared between these). However, rules that only depend on GetHieAST and direct derivatives can probably run just fine. For example, hover, references, document symbols and go to definition should work on these files.

  Also, we must ignore all edits to these files. This ties into the previous point about not trying to compile them with GHC, which we must not do. Unfortunately LSP doesn't support read only files (Specify read-only locations microsoft/language-server-protocol#1150) or virtual documents (Proposal: Allow a language server to provide a reference content for a specific uri scheme microsoft/language-server-protocol#336) or our life would be much easier. We can mark the source files for dependencies as write-protected on the file system, but this doesn't prevent most editors from editing them in memory without saving them on disk (which we will get change notifications for).

• There are also inconsistencies with how cabal handles .hie files, the assumption is the same for stack as well, but has not been explored. For example, on first compilation using cabal, it is possible to generate the exhaustive .hie files for all external dependencies. However cabal will not track the state of .hie files and will not recompile an external dependency if the .hie is removed.

  - This is most likely due to error, but @drsooch encountered strange results attempting to capture external dependencies with -fwrite-ide-info. In most scenarios, it was almost impossible to regenerate external library .hie files, other than the first compilation. User Error :) -- use absolute paths.

• The main problem faced was the time it took to index a large project. For example HLS generated ~2500 .hie files, and it took ~20 minutes to index all of them. There is no great solution to abstract this away from the user. If it is possible to do it internally, is it possible to spin off a long-running process for that long?

[jneira]

many thanks for the detailed description of the state of the festure, I hope it will help to make progress on this

[jneira]

The comment about the plan to get the feature has been updated by @drsooch after work in its analysis, thanks!

I wonder how other langs has resolved the issue (for example ruts), maybe it would worth to have a look.

[michael-swan]

@wz1000 Several thoughts come to mind. You've listed several blockers:

1. No way to advertise file as read-only with LSP (or virtual document)
2. It will take 20+ minutes to index large projects.
3. cabal and stack do not respect .hie files as build obligations like it does .hi and .o binaries.
4. Users shouldn't have to include specific ghc-options because that's confusing.

These may all be facile points coming from someone who doesn't maintain any of this software, but here are the thoughts that jump out at me:

1. Why not chmod a=r files extracted from HIE's (or equivalent on Windows resp.) since we are already the sole user of these extracted files? It doesn't look like a read-only notify interface is coming from the looks of Specify read-only locations microsoft/language-server-protocol#1150. Most editors respect read-only settings of files anyways though there is always the option to override. This in-memory read-only override behaviour should be considered out of scope as far as HLS is concerned because users must explicitly do this and more than likely realize that their tooling isn't obligated to update its internal model according to those changes. With respect to the virtual document proposal, there is no timeline for that being merged and shouldn't be anticipated.
2. This is then a question of bottlenecks in indexing. Why does it take half a second on average to index a single HIE in a project like HLS? Does the serialization of writes to SQLite in hiedb also inhibit parallel indexing? To resolve some of these bottlenecks, we may want to take a page from rust-analyzer which I believe doesn't even have the concept of "ahead-of-time indexing" and begs the question: what end is indexing achieving that couldn't be achieve by lazily evaluating queries through information already provided to hiedb? Beyond performance bottlenecks though, there are some behaviours we should expect from indexing in HLS: backgrounding this indexing work and prioritizing indexing of pertinent files. These two suggestions would make waiting for such indexing seem more reasonable, though this only pertain to HLS if ahead-of-time indexing continues to be necessary.
3. This is clearly a blocker, but it seems like it is out of scope for HLS. I think the best course of action is to bug folks working on stack and cabal to get them to add HIE's to their build obligations if -fwrite-ide-info is known to be a GHC build option.
4. Can't HLS just add custom GHC command-line options through hie-bios? What is stopping us from amending GHC options this way? Is it that hie-bios options only enter the picture after a build has complete?

[eyeinsky]

With regarding to this issue, do any of the circumstances change to make go-to-definition work for packages added via source-repository-package in cabal.project?

[kr3v]

I've developed this - https://github.com/kr3v/haskell-gtd-nl - as an attempt to implement the requested feature ('go to definition for non-local libraries'). It is independent from HLS / vscode-haskell (separate VS Code extension, separate server (low memory usage mostly, can be used with HLS on low-memory machines)). I hope it will fill the gap until the feature is added to HLS.

[xave]

haskell-docs-cli allows a variant of this. It puts the non-local files in a temporary cache directory. It seems to me that the ultimate solution is along those lines. You'd need an ability to clear and manage the cache as well. You'd want a way to handle different versions of packages likely by referencig the cabal file (at least), though there should be some ghc compile time info to say exactly which versions of which functions are used in the project. That could be captured and referenced when doing the jump.

Until then, haskell-docs-cli is pretty nice for jumping to source and searching hoogle/haddocks.

[Geomathic]

So we still can't jump to definitions on base package functions?
I did cabal unpack base in my project any way to make goto definition jump to source code in ./base-4.20.0.1?

[xave]

I just tried that and it works: cabal unpack base-4.20.0.1

Any nice way to get it to pull in all build-depends packages in a cabal file?

Alternatively, to have cabal output those as a list to stdout.

[Geomathic]

I just tried that and it works: cabal unpack base-4.20.0.1

Any nice way to get it to pull in all build-depends packages in a cabal file?

Alternatively, to have cabal output those as a list to stdout.

What works? You pulled it and then you were able to jump to to definition on say standard functions? For me it just says "No definition found.".
Could you provide cabal file and steps to reproduce?

[xave]

My apologies for the confusion. I do not use hls because it does not yet suit my needs (performance on large repos, features not complete). I use vim, ctags, and grep.

I had posted that comment about haskell-docs-cli to show that it is possible to get this information programatically. Your comment about cabal unpack make that even more apparent. Thus, the steps to close this issue should be straightforward and it is an important base feature for an LSP/IDE. This issue should be solved up to actually doing the implementation itself.

The "works for me" part is appreciation for the cabal unpack, which I had not known about.