feat: allow disabling tokenizers

[kellyjosephprice]

	RM-215

🧰 Changes

Adds the option of disabling tokenizers. This is in service of the editor's markdown decorators. During deserialization, we can leave the inlines as plain text. Then, during the decoration step, we can parse the markdown with just the inline tokenizers.

Usage

  const opts = { tokenizerSet: 'blocks' };
  markdown.mdast(md, opts); // won't parse inline code, emphasis, etc.

[rafegoldberg]

Can you add a brief usage example to the PR description?

[rafegoldberg]

👍🏼 I had some suggestions, but nothing I'd hold merging this over. Also added this option to the docs in #129, so feel free to merge that whenever!

[rafegoldberg]

Can we rename the public-facing option to disableTokenizers? It feels a bit redundant here, but I think it'd be a bit more descriptive of what it actually does for the end user.

Suggested change

	if (opts.tokenizerSet in disableTokenizers) {
	return { ...disableTokenizers[opts.tokenizerSet], ...opts };
	} else if (opts.tokenizerSet) {
	throw new Error(`opts.tokenizerSet "${opts.tokenizerSet}" not one of "${Object.keys(disableTokenizers)}"`);
	}
	if (opts.disableTokenizers in disableTokenizers) {
	return { ...disableTokenizers[opts.disableTokenizers], ...opts };
	} else if (opts.disableTokenizers) {
	throw new Error(`opts.disableTokenizers "${opts.disableTokenizers}" not one of "${Object.keys(disableTokenizers)}"`);
	}

[rafegoldberg]

This PR was released!

🚀 Changes included in v6.26.0-next.1

[rafegoldberg]

This PR was released!

🚀 Changes included in v6.26.0