Add Tokens newtype wrapper, TokenKind iterator

[dhruvmanila]

Summary

Alternative to #11237

This PR adds a new Tokens struct which is a newtype wrapper around a vector of lexer output. This allows us to add a kinds method which returns an iterator over the corresponding TokenKind. This iterator is implemented as a separate TokenKindIter struct to allow using the type and provide additional methods like peek directly on the iterator.

This exposes the linter to access the stream of TokenKind instead of Tok.

Edit: I've made the necessary downstream changes and plan to merge the entire stack at once.

[github-actions]

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

ℹ️ ecosystem check encountered linter errors. (no lint changes; 1 project error)

lnbits/lnbits (error)

ruff check --no-cache --exit-zero --ignore RUF9 --output-format concise --preview

Failed to pull: From https://github.com/lnbits/lnbits
 + 365f9a39...e9e69d9d main       -> origin/main  (forced update)
hint: You have divergent branches and need to specify how to reconcile them.
hint: You can do so by running one of the following commands sometime before
hint: your next pull:
hint: 
hint:   git config pull.rebase false  # merge
hint:   git config pull.rebase true   # rebase
hint:   git config pull.ff only       # fast-forward only
hint: 
hint: You can replace "git config" with "git config --global" to set a default
hint: preference for all repositories. You can also pass --rebase, --no-rebase,
hint: or --ff-only on the command line to override the configured default per
hint: invocation.
fatal: Need to specify how to reconcile divergent branches.

Formatter (stable)

✅ ecosystem check detected no format changes.

Formatter (preview)

✅ ecosystem check detected no format changes.

[dhruvmanila]

I might possibly need to maintain a Vec<TokenKind> because certain references utilizes lex_starts_at which returns the tokens within a certain range. An iterator would make it a O(n) while a vector would allow us to do a binary search. I could possibly use a BTreeMap of token start to the (TokenKind, TextRange) to directly use the range method.

Edit: I'm using binary search to get the start and end index.

[dhruvmanila]

So, this isn't currently being used anywhere. This would basically replace the usages of lex_starts_at but it turns out all of the usages of the function is being done in the AST checker where we don't have access to the token stream.

One solution here would be to store a TokenKinds struct which contains Vec<(TokenKind, TextRange)> on the Checker. This way the rules which utilizes the lex_starts_at or lex function can still get the tokens.

Another would be to club this change with the parser. I'm leaning more towards this.