Reset cursor at dedent after whitespace #11630
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
CodSpeed Performance ReportMerging #11630 will create unknown performance changesComparing Summary
Benchmarks breakdown
|
|
| code | total | + violation | - violation | + fix | - fix |
|---|---|---|---|---|---|
| E117 | 152372 | 0 | 152372 | 0 | 0 |
| E113 | 73488 | 0 | 73488 | 0 | 0 |
| E116 | 6080 | 0 | 6080 | 0 | 0 |
| RUF100 | 1 | 1 | 0 | 0 | 0 |
Formatter (stable)
✅ ecosystem check detected no format changes.
Formatter (preview)
✅ ecosystem check detected no format changes.
dhruvmanila
force-pushed
the
dhruv/parser-phase-2
branch
from
2d34e13
to
e8ab801
Compare
dhruvmanila
force-pushed
the
dhruv/dedent-after-whitespace
branch
from
4faed6a
to
b600b42
Compare
MichaReiser
reviewed
May 31, 2024
| // | ||
| // Here, the cursor is at `^` and the `indentation` contains the whitespaces before | ||
| // the `pass` token. | ||
| self.cursor.start_token(); |
There was a problem hiding this comment.
I must admit that reading self.cursor.start_token() is hard to understand. Why are we starting a token when we're just about to return it. Your comment certainly helps. That makes me wonder if we can instead add a method on Lexer with a more meaningful name.
self.update_token_start_to_current_offset() // A bi tlong
self.reset_token_start()but I must admit, I don't find them much better than what we have. But maybe you have an idea for a better name?
There was a problem hiding this comment.
Yeah, I agree. Let me think about this.
I did review them, sorry for not mentioning in the PR description. This isn't needed in all places, it's only require if the lexer skips whitespaces. For posterity, the usages are: ruff/crates/ruff_python_parser/src/lexer.rs Lines 853 to 863 in 685d11a ⬆️ This emits any pending dedent tokens. There are no whitespaces consumed by the lexer before this, so no need to reset the cursor. ruff/crates/ruff_python_parser/src/lexer.rs Lines 1031 to 1041 in 685d11a ⬆️ We already reset the cursor before this function is called ⬇️ ruff/crates/ruff_python_parser/src/lexer.rs Lines 877 to 898 in b600b42 |
dhruvmanila
added a commit
that referenced
this pull request
Jun 3, 2024
## Summary This PR fixes a bug to reset the cursor before emitting the `Dedent` token which had preceding whitespaces. Previously, we would just use `TextRange::empty(self.offset())` because the range for each token would be computed at the place where it was emitted. Now, the lexer handles this centrally in `next_token` which means we require special handling at certain places. This is one such case. Computing the range centrally in `next_token` has a benefit that the return type of lexer methods is a simple `TokenKind` but it does add a small complexity to handle such cases. I think it's worth doing this but if anyone thinks otherwise, feel free to comment. ## Test Plan Add a test case and update the snapshot.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes a bug to reset the cursor before emitting the
Dedenttoken which had preceding whitespaces.Previously, we would just use
TextRange::empty(self.offset())because the range for each token would be computed at the place where it was emitted. Now, the lexer handles this centrally innext_tokenwhich means we require special handling at certain places. This is one such case.Computing the range centrally in
next_tokenhas a benefit that the return type of lexer methods is a simpleTokenKindbut it does add a small complexity to handle such cases. I think it's worth doing this but if anyone thinks otherwise, feel free to comment.Test Plan
Add a test case and update the snapshot.