Meta discussion about unclear error messages

[giordano]

I feel like Pkg has a general problem with unclear error messages. I figured I'd open a meta issue to discuss this problem to address it at a more general level, rather than trying to solve individual bugs.

By going through the list of issues I found (before getting tired):

• Artifact system swallows too many errors into a "failed to install error" #1659
• New error: can not merge projects #1714
• More helpful error messages if TOML files are corrupted #1818
• Unhelpful error message: test dependency is not present in extras #1821
• Unhelpful error message about failure to parse compatibility version #1853
• Strange error message from Pkg.add #1935
• Better error message for already dev'ed package that doesn't have a Project.toml #1956
• AssertionError: sourcepath !== nothing #2097
• give better diagnostic for missing package #2191
• instantiate aborts with assertion error #2214
• More helpful error messages #2308
• Unclear error message when adding/updating dependencies on broken module #2381
• Misleading error message on dev with wrong directory #2406
• Bad error message "already exists as a direct dependency" #2476
• lazy artifacts error reporting is hard to read and hard to use #2481
• Running tests fail with “sourcepath !== nothing” #2513
• Error message for missing project file could be improved #2528
• installing packages runs into a keyerror: key UUID("...") not found #2554
• A project without module gives a very misleading error  #2560
• Helpful message for Pkg.add(url=...#forked_branch) #2588
• KeyError: key UUID("...") not found in CI on nightly (1.7) #2602
• Error message for dev when the path exist could be better #2672
• Vague warnings about dependency graph not being a DAG #2697
• Cloning repo using Pkg.add(url) results in a non-descriptive error #2701
• Improve the "empty intersection" error message #2738
• Bad error message for incorrect compat format in dev'ed package #2774
• Circular dependencies cause error key UUID("...") not found #2776

Some issues are already closed, that's not the point, I just wanted to show that very often users, including experienced ones, face error messages that are unactionable, confusing or also completely misleading.

I totally understand that these things can go wrong in uncountable many ways and it's impossible to cater for all possible situations, in fact many times the problems are on the user's side, rather than Pkg's, but users have no idea of what they did wrong or how to fix their issue. They are only left with unclear messages and no clues of what to do.

For example, when there is something wrong in an environment, it'd be useful to explicitly point out which is the faulty environment.

[IanButterworth]

I agree. It'd be good to do a push to fix the open issues, and I wonder if the error handling tooling could be bolstered to make it easier. For instance make pkgerror accept some standard kwargs like a project/manifest filepath, a sourcefile, a cachefile, etc. that print in a fixed way, to make errors more consistent and easier for Pkg devs to make richer errors?

[KristofferC]

#2776 this is not an error message problem, it is a bug. Same with #2214.

What's the point of a meta issue? When do we close this, when all the linked issues are closed? I don't see how this helps get any work done. It comes off as mostly a gripe.

[IanButterworth]

Perhaps a GitHub project instead?

[giordano]

#2776 this is not an error message problem, it is a bug.

The user created by mistake a circular dependency that shouldn't have been there in the first place. Pkg knows that there is a circular dependency but it doesn't tell you (or not in all cases).

Same with #2214.

I'm pretty sure #2214 is the same as #2097, so that one can be closed as well. The solution was to rework the manifest to provide...more informative error messages and let the users know that they're using incompatible manifests.

What's the point of a meta issue? When do we close this, when all the linked issues are closed? I don't see how this helps get any work done. It comes off as mostly a gripe.

It is a gripe if you want. I had solved myself issues like #1853 and #2776 before opening the tickets, I could have just moved on, but try to think about how frustrated can be users who are not willing to spend time debugging these issues.

I wanted to show this is a general problem, it isn't specific to a single failure. The list of issues isn't the list of issues that have to be closed, but it's there to show that users just keep running into different errors and don't have a clue of what to do. Maybe general error handling/reporting should be reconsidered, instead of (well, in addition to) looking to individual bugs.

[KristofferC]

The user created by mistake a circular dependency that shouldn't have been there in the first place

I don't see why Pkg should not handle it? I've fixed other problems in Pkg where it has bugged on circular dependencies.

I wanted to show this is a general problem, it isn't specific to a single failure

I mean, it's like opening an issue on Base julia and saying that compilation is slow. OK, now what?

Maybe general error handling/reporting should be reconsidered,

PRs implementing some more structured error handling in Julia (or just Pkg) that improves the experience for users are welcome as well as targeted issues/PRs that aim to improve individual error messages.

But again, I don't see how this issue does anything constructively unless you think that the people that regularly work on Pkg has some notion that the error messages are perfect and they just need to be informed of their badness to realize they should prioritise differently.

[DilumAluthge]

How about we convert this issue to a discussion? Then we don't need to determine the criteria for closing.

[IanButterworth]

Converting this to a discussion seems like a good idea. I'll do that.

I should add that before this issue, @giordano and I were discussing some of these pkg error issues/gripes and I made a suggestion to make this meta issue to prompt some ideas on how to handle things in general.

On a related note, I recently took a pass through all of the open Pkg issues and was struck by how many diverse ideas and open bugs there are (I closed anything that I considered resolved). I'm wondering if there would be merit to organizing them into some targeted projects, perhaps including one for improving error messaging? - Projects seem to be disabled on this repo, someone with admin would need to enable them

[DilumAluthge]

Projects seem to be disabled on this repo, someone with admin would need to enable them

I enabled GitHub Projects on this repository.