feat: add processAllTokens hook

[UziTech]

Marked version: 11.0.0

Description

Add tokens hook that receives all tokens and can change the tokens.

This hook can allow extensions that change all tokens before walkTokens functions are called. For example an extension that adds the parent and sibling properties to tokens.

Contributor

• Test(s) exist to ensure functionality and minimize regression (if no tests added, list tests covering this PR); or,
• no tests required for this PR.
• If submitting new feature, it has been documented in the appropriate places.

Committer

In most cases, this should be a different person than the contributor.

• CI is green (no forced merge required).
• Squash and Merge PR following conventional commit guidelines.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
marked-website	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Dec 12, 2023 6:01am

[calculuschild]

Ok, so this is a "middle Process" hook along the lines of Preprocess and Postprocess? I like that. Does this also give access to the tokens.links property so we can manipulate the reflink definitions? For example to inject a common link that can be referenced globally without being defined within the markdown input?

[UziTech]

Yes it would. It is passed the tokens object that is returned by the lexer after all tokens have been created and before walkTokens functions are called.

[calculuschild]

Adding to this then: is there a benefit to splitting into two hooks? one immediately after block tokens, and another immediately after inline tokens? I'm thinking of a case where you might want to alter the block-level tokens before their internal tokens get lexed.

[UziTech]

A hook that captures block tokens before creating the inline tokens will need to take a bit more thought because of the way we do async marked.

The hook will have to be in the Lexer.lex method and have to return a promise if options.async is true but it currently doesn't return a promise. That might have to be a breaking change.

I think it could be done in another PR.

[UziTech]

@calculuschild should I rename this to allTokens so we can add blockTokens later?

[calculuschild]

Yeah, probably a good idea. I might suggest we use a verb like we have with the other steps, i.e. "preprocess", "walktokens", "parse", etc.

• Preprocess
• Lex
• BlockTokens????
• AllTokens????
• WalkTokens
• Parse
• Postprocess

Maybe "ProcessTokens" or something?

Or like, ProcessBlockTokens, ProcessAllTokens.

Something like that?

[UziTech]

Ya I think I like processAllTokens then we can do processBlockTokens later

[kapouer]

For reference, this also solves #2639 and proves mind communication.