-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: Join adjacent inlineText tokens #1926
Conversation
This pull request is being automatically deployed with Vercel (learn more). 🔍 Inspect: https://vercel.com/markedjs/markedjs/fk98enxmj |
CI is not passing but all the spec tests and benchmarks pass just fine on my machine. Not sure what's going on. Any help? |
Ah I see there's a separate Lexer unit test suite. It expects adjacent but separate text tokens in a couple of cases. I assume that isn't a requirement anymore.... |
I think I fixed the tests in calculuschild#2. I also changed the block text token to do the merge in the lexer instead of the tokenizer and merge text tokens returned by other tokenizers. |
Join adjacent innerText tokens
Thanks for looking into it!
@UziTech I was going to ask about this as well so I'm glad you had the same idea. Seemed to make more sense in the Lexer. Do we also want to do the same for the block |
Ya it is probably a good time to change the tokenizer signatures if we need to since this and #1864 should be released as v2 soon. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These changes are from my code
Co-authored-by: Tony Brix <[email protected]>
Co-authored-by: Tony Brix <[email protected]>
Can you update the tokenizer signatures for |
Yay! All the Block Tokenizers have the same signature now! 🎉 |
lastToken = tokens[tokens.length - 1]; | ||
lastToken = tokens[tokens.length - 1]; | ||
// An indented code block cannot interrupt a paragraph. | ||
if (lastToken && lastToken.type === 'paragraph') { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder if we should check for a paragraph before we call the code
tokenizer? That might save some work that doesn't need to be done.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Probably worth trying. It's a tradeoff of always checking LastToken
and sometimes calling codeTokenizer
vs always calling codeTokenizer
and sometimes checking LastToken
. I'm not sure how often this situation comes up that one would be better than the other.
Edit: If LastToken
is a paragraph though what do we do? Just continue? or call the "text" tokenizer out of sequence?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If LastToken is a paragraph though what do we do? Just continue? or call the "text" tokenizer out of sequence?
Good point. I suppose we would still need to call code tokenizer to see if we should skip the other tokens. Maybe it is still better to check code tokenizer first.
🎉 This PR is included in version 2.0.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
Marked version: 1.2.9
Description
text
tokens similar to how block text tokens are merged. HTML output doesn't change, but this makes it easier to make a custom renderer since text tokens aren't broken up.Contributor
Committer
In most cases, this should be a different person than the contributor.