fix: Join adjacent inlineText tokens #1926

calculuschild · 2021-02-04T23:14:03Z

Marked version: 1.2.9

Description

Fixes Paragraph text is split into separate tokens when using backslash #1906. Joins together adjacent inline text tokens similar to how block text tokens are merged. HTML output doesn't change, but this makes it easier to make a custom renderer since text tokens aren't broken up.

Contributor

Test(s) exist to ensure functionality and minimize regression (if no tests added, list tests covering this PR); or,
no tests required for this PR.
If submitting new feature, it has been documented in the appropriate places.

Committer

In most cases, this should be a different person than the contributor.

CI is green (no forced merge required).
Squash and Merge PR following conventional commit guidelines.

vercel · 2021-02-04T23:14:08Z

This pull request is being automatically deployed with Vercel (learn more).
To see the status of your deployment, click below or on the icon next to each commit.

🔍 Inspect: https://vercel.com/markedjs/markedjs/fk98enxmj
✅ Preview: https://markedjs-git-fork-calculuschild-joininnertexttokens.markedjs.vercel.app

calculuschild · 2021-02-04T23:27:31Z

CI is not passing but all the spec tests and benchmarks pass just fine on my machine. Not sure what's going on. Any help?

calculuschild · 2021-02-05T02:43:54Z

Ah I see there's a separate Lexer unit test suite. It expects adjacent but separate text tokens in a couple of cases. I assume that isn't a requirement anymore....

UziTech · 2021-02-05T02:59:04Z

I think I fixed the tests in calculuschild#2. I also changed the block text token to do the merge in the lexer instead of the tokenizer and merge text tokens returned by other tokenizers.

Join adjacent innerText tokens

calculuschild · 2021-02-05T03:12:10Z

Thanks for looking into it!

I also changed the block text token to do the merge in the lexer instead of the tokenizer and merge text tokens returned by other tokenizers.

@UziTech I was going to ask about this as well so I'm glad you had the same idea. Seemed to make more sense in the Lexer. Do we also want to do the same for the block code tokenizer? It also does this same merge thing.

UziTech · 2021-02-05T03:14:57Z

Ya it is probably a good time to change the tokenizer signatures if we need to since this and #1864 should be released as v2 soon.

UziTech

These changes are from my code

lib/marked.esm.js

Co-authored-by: Tony Brix <[email protected]>

UziTech · 2021-02-05T04:05:52Z

Can you update the tokenizer signatures for text and code in the docs

…d/marked into joinInnerTextTokens

Made requested changes.

calculuschild · 2021-02-05T04:11:09Z

Yay! All the Block Tokenizers have the same signature now! 🎉

UziTech · 2021-02-06T07:28:05Z

src/Lexer.js

-          lastToken = tokens[tokens.length - 1];
+        lastToken = tokens[tokens.length - 1];
+        // An indented code block cannot interrupt a paragraph.
+        if (lastToken && lastToken.type === 'paragraph') {


I wonder if we should check for a paragraph before we call the code tokenizer? That might save some work that doesn't need to be done.

Probably worth trying. It's a tradeoff of always checking LastToken and sometimes calling codeTokenizer vs always calling codeTokenizer and sometimes checking LastToken. I'm not sure how often this situation comes up that one would be better than the other.

Edit: If LastToken is a paragraph though what do we do? Just continue? or call the "text" tokenizer out of sequence?

If LastToken is a paragraph though what do we do? Just continue? or call the "text" tokenizer out of sequence?

Good point. I suppose we would still need to call code tokenizer to see if we should skip the other tokens. Maybe it is still better to check code tokenizer first.

# [2.0.0](v1.2.9...v2.0.0) (2021-02-07) ### Bug Fixes * Join adjacent inlineText tokens ([#1926](#1926)) ([f848e77](f848e77)) * Total rework of Emphasis/Strong ([#1864](#1864)) ([7293251](7293251)) ### BREAKING CHANGES * `em` and `strong` tokenizers have been merged into one `emStrong` tokenizer

github-actions · 2021-02-07T22:26:45Z

🎉 This PR is included in version 2.0.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

Join adjacent innerText tokens

65b7a35

vercel bot deployed to Preview February 4, 2021 23:14 View deployment

Update marked.esm.js

0ade5ac

vercel bot deployed to Preview February 5, 2021 02:27 View deployment

fix lexer unit tests

72e7cc8

Merge pull request #2 from UziTech/pr/1926

87f4704

Join adjacent innerText tokens

vercel bot deployed to Preview February 5, 2021 03:06 View deployment

calculuschild changed the title ~~Join adjacent innerText tokens~~ Join adjacent inlineText tokens Feb 5, 2021

Move merge logic for block code to Lexer

d7bb213

vercel bot deployed to Preview February 5, 2021 03:50 View deployment

UziTech previously requested changes Feb 5, 2021

View reviewed changes

lib/marked.esm.js Outdated Show resolved Hide resolved

lib/marked.esm.js Outdated Show resolved Hide resolved

Update lib/marked.esm.js

1c5cf59

Co-authored-by: Tony Brix <[email protected]>

vercel bot deployed to Preview February 5, 2021 04:00 View deployment

Update lib/marked.esm.js

ead6e66

Co-authored-by: Tony Brix <[email protected]>

vercel bot deployed to Preview February 5, 2021 04:00 View deployment

calculuschild requested a review from UziTech February 5, 2021 04:04

calculuschild added 2 commits February 4, 2021 23:09

Update code & text signatures in Docs

3b1130e

Merge branch 'joinInnerTextTokens' of https://github.com/calculuschil…

296e327

…d/marked into joinInnerTextTokens

vercel bot deployed to Preview February 5, 2021 04:09 View deployment

UziTech approved these changes Feb 5, 2021

View reviewed changes

UziTech requested a review from davisjam February 5, 2021 04:14

UziTech requested review from joshbruce and styfle February 5, 2021 04:14

UziTech mentioned this pull request Feb 5, 2021

fix: Total rework of Emphasis/Strong #1864

Merged

5 tasks

UziTech reviewed Feb 6, 2021

View reviewed changes

styfle approved these changes Feb 7, 2021

View reviewed changes

UziTech changed the title ~~Join adjacent inlineText tokens~~ fix: Join adjacent inlineText tokens Feb 7, 2021

UziTech merged commit f848e77 into markedjs:master Feb 7, 2021

github-actions bot added the released label Feb 7, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: Join adjacent inlineText tokens #1926

fix: Join adjacent inlineText tokens #1926

calculuschild commented Feb 4, 2021

vercel bot commented Feb 4, 2021 •

edited

Loading

calculuschild commented Feb 4, 2021

calculuschild commented Feb 5, 2021

UziTech commented Feb 5, 2021

calculuschild commented Feb 5, 2021 •

edited

Loading

UziTech commented Feb 5, 2021

UziTech left a comment

UziTech commented Feb 5, 2021

calculuschild commented Feb 5, 2021

UziTech Feb 6, 2021

calculuschild Feb 6, 2021 •

edited

Loading

UziTech Feb 6, 2021

github-actions bot commented Feb 7, 2021

fix: Join adjacent inlineText tokens #1926

fix: Join adjacent inlineText tokens #1926

Conversation

calculuschild commented Feb 4, 2021

Description

Contributor

Committer

vercel bot commented Feb 4, 2021 • edited Loading

calculuschild commented Feb 4, 2021

calculuschild commented Feb 5, 2021

UziTech commented Feb 5, 2021

calculuschild commented Feb 5, 2021 • edited Loading

UziTech commented Feb 5, 2021

UziTech left a comment

Choose a reason for hiding this comment

UziTech commented Feb 5, 2021

calculuschild commented Feb 5, 2021

UziTech Feb 6, 2021

Choose a reason for hiding this comment

calculuschild Feb 6, 2021 • edited Loading

Choose a reason for hiding this comment

UziTech Feb 6, 2021

Choose a reason for hiding this comment

github-actions bot commented Feb 7, 2021

vercel bot commented Feb 4, 2021 •

edited

Loading

calculuschild commented Feb 5, 2021 •

edited

Loading

calculuschild Feb 6, 2021 •

edited

Loading