Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add processAllTokens hook #3114

Merged
merged 4 commits into from
Dec 12, 2023
Merged

Conversation

UziTech
Copy link
Member

@UziTech UziTech commented Nov 30, 2023

Marked version: 11.0.0

Description

Add tokens hook that receives all tokens and can change the tokens.

This hook can allow extensions that change all tokens before walkTokens functions are called. For example an extension that adds the parent and sibling properties to tokens.

Contributor

  • Test(s) exist to ensure functionality and minimize regression (if no tests added, list tests covering this PR); or,
  • no tests required for this PR.
  • If submitting new feature, it has been documented in the appropriate places.

Committer

In most cases, this should be a different person than the contributor.

Copy link

vercel bot commented Nov 30, 2023

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
marked-website ✅ Ready (Inspect) Visit Preview 💬 Add feedback Dec 12, 2023 6:01am

@calculuschild
Copy link
Contributor

Ok, so this is a "middle Process" hook along the lines of Preprocess and Postprocess? I like that. Does this also give access to the tokens.links property so we can manipulate the reflink definitions? For example to inject a common link that can be referenced globally without being defined within the markdown input?

@UziTech
Copy link
Member Author

UziTech commented Nov 30, 2023

Yes it would. It is passed the tokens object that is returned by the lexer after all tokens have been created and before walkTokens functions are called.

@calculuschild
Copy link
Contributor

calculuschild commented Nov 30, 2023

Adding to this then: is there a benefit to splitting into two hooks? one immediately after block tokens, and another immediately after inline tokens? I'm thinking of a case where you might want to alter the block-level tokens before their internal tokens get lexed.

@UziTech
Copy link
Member Author

UziTech commented Dec 1, 2023

A hook that captures block tokens before creating the inline tokens will need to take a bit more thought because of the way we do async marked.

The hook will have to be in the Lexer.lex method and have to return a promise if options.async is true but it currently doesn't return a promise. That might have to be a breaking change.

I think it could be done in another PR.

@UziTech
Copy link
Member Author

UziTech commented Dec 10, 2023

@calculuschild should I rename this to allTokens so we can add blockTokens later?

@calculuschild
Copy link
Contributor

Yeah, probably a good idea. I might suggest we use a verb like we have with the other steps, i.e. "preprocess", "walktokens", "parse", etc.

  • Preprocess
  • Lex
  • BlockTokens????
  • AllTokens????
  • WalkTokens
  • Parse
  • Postprocess

Maybe "ProcessTokens" or something?

Or like, ProcessBlockTokens, ProcessAllTokens.

Something like that?

@UziTech
Copy link
Member Author

UziTech commented Dec 10, 2023

Ya I think I like processAllTokens then we can do processBlockTokens later

@UziTech UziTech merged commit faae243 into markedjs:master Dec 12, 2023
8 checks passed
@UziTech UziTech changed the title feat: add tokens hook feat: add processAllTokens hook Dec 12, 2023
@UziTech UziTech deleted the postlexer-preparser branch December 12, 2023 06:02
github-actions bot pushed a commit that referenced this pull request Dec 12, 2023
# [11.1.0](v11.0.1...v11.1.0) (2023-12-12)

### Features

* add processAllTokens hook ([#3114](#3114)) ([faae243](faae243))
@kapouer
Copy link
Contributor

kapouer commented Dec 12, 2023

For reference, this also solves #2639 and proves mind communication.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants