Poor performance with semantic highlighting. #2881
Labels
fixed in next version (main)
A fix has been implemented and will appear in an upcoming version
Comments
|
Here are a couple of ways we might be able to approach this problem. First, if you'd be willing to give me temporary access to a private github repo which contains the file(s) that exhibit the problem, I could look into the perf issue. If that's not feasible, I'll need to ask you to do some additional work to narrow down the problem. Based on the logging, the perf issue appears to be related to the file |
|
@erictraut, Thank you for the tips and the offer for help. For some reason i was stuck with the paradigm that the code would need to be able to run to test subsets of it, so i was daunted at the prospect of testing that way. Clearly that's not the case, and it is easy to test subsets of it by commenting sections. Anyway, commenting blocks of the data.py seemed to help the checking and analyzing time roughly(really roughly) linearly, but commenting the This makes me suspect that the slowdown is related to an update somewhere since i've been using numpy extensively in this package for a while, but the slowdown happened sometime last week. |
|
Make sure you're using recent versions of numpy. Earlier versions had no type information, so all types needed to be inferred from the source, which was expensive. The numpy maintainers have started to add type annotations. The latest published version of numpy (1.22.4) has a "type completeness score" of 27.2%. That means about a quarter of the symbols that comprise its public interface are fully annotated. That may sound low, but they've started with the most commonly-used functionality. It's possible that you're using some aspect of numpy that is not yet annotated, and that might contribute to performance issues because the types need to be inferred if they are not annotated. Let me know if you narrow it down any further. |
|
Copy that, i tried updating everything to the latest, and spent a bunch of time cleaning up the type hints related to numpy but that only made marginal if any gains in the performance. I'll keep an eye out for issues, but im running short on ideas. Thanks, |
|
I got a cleaner log output this morning where i only opened the simpler of the 2 problem files. I'm not sure what to be looking for in this trace, when i looked before the only long times i saw reported were on the Checking and Analyzing of my problem files. Its not clear how these times stack up to get the total time. |
|
Ok it seems that the Anyway, i would appreciate any further tips on what to check for. |
|
Thanks for posting the logs. Unfortunately, I don't see any clues that help get us closer to understanding the problem. Here's the next thing you can try. Add the following to the Then open the file When you're done with the experiment, make sure to remove the above two lines from your settings. |
|
Thanks for the tip. It seems like the worst offending line is the following which is making use of scipy sparse arrays which are new in scipy 1.8.0. model_matrix = scipy.sparse.csr_array(
(intf_array, (rows, ind_array)),
shape=(n_samples, n_grid),
)Another interesting one is np.newaxis, which i would have thought would be pretty commonly used. points = points[:, np.newaxis]This is curious too, I use list.append() a lot so its hard to tell which instance this refers to. |
|
I don't see anything obviously wrong here. I'm happy to dig into it deeper, but I'll need a self-contained sample that I can use to repro locally. If you're willing to give me temporary private access to your repo, I could diagnose directly. Or if you want to set up a temporary repo that contains simplified or redacted versions, that would work as well. |
|
@erictraut I appreciate the help. I'll make up a simplified repo and give you access. |
|
We've had issues with scipy in the pasts. previously |
bschnurr
added
needs stub
and removed
needs investigation
|
@erictraut Just checking in if you've had a chance to look at this, i've tried to share a simplified version of the repo with you. Again, thanks for the help. |
|
@CmpCtrl, I didn't receive an invitation from github, so I wasn't aware that you had created a simplified repro. Could you confirm that I have access to the repo. If so, perhaps you could try sending the invitation again? |
|
@CmpCtrl, I received your github invite. Thanks for the repro. I'll look at it when I have time, probably later this week. |
|
Copy that, thanks for the help. |
|
you can try disabling semantic highlighting.
|
|
I realize that i've not been making a clear distinction (either in my head or in these comments/issues) between syntax highlighting and semantic highlighting and what I'm concerned with is the performance of the semantic highlighting. I apologize for the confusion. I've posted a gif in the microsoft/pyright#3582 of the slow performance that I'm hoping to improve. |
CmpCtrl
changed the title
erictraut
added
fixed in next version (main)
|
I've introduced several more rounds of optimizations to pyright that incrementally improve performance for this code. Overall, I cut the time in roughly half. For more details, refer to the tracking bug in pyright. |
|
This issue has been fixed in version 2022.7.41, which we've just released. You can find the changelog here: CHANGELOG.md |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Environment data
Expected behaviour
I expect semantic highlighting to be sub second performance.
** Edited to correct my mistake of referring to syntax highlighting when i really mean semantic highlighting **
Actual behaviour
Im working on a package with several modules in it. Two of the modules seem to have very poor performance in Pylance, one taking around 5s and another taking around 2-3s. So the total time for semantic highlighting ends up near 10s. Clearly i've got something wrong in these 2 files. The first is around 3600 lines of code, the second around 1000. I noticed the slowdown last week, but haven't chased it much until now. I've been working on the files, so i cant rule out a change i made, nor can i recall if it started with an update to something.
Unfortunately i cant share this code, so i am looking for some more tips on how to diagnose where the slowdown is.
I looked thru the log trace and didnt see anything that stuck out except for the lines below. It seems like the majority of the lines in the log happen fast except for these.
Thanks for the help.
Logs
Python Language Server Log
Code Snippet / Additional information
The text was updated successfully, but these errors were encountered: