Improve handling of raw fstrings (#462)

[thatch]

Summary

Fixes the reported issue by reverting to the regex used before 02fc440 when raw.

I am not 100% sure checking just top of stack is enough, however. I wish there was a conformance suite for f-strings.

Test Plan

Includes new test case

[codecov-io]

Codecov Report

Merging #466 (9fc8a77) into master (507b453) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master     #466   +/-   ##
=======================================
  Coverage   94.64%   94.64%           
=======================================
  Files         234      234           
  Lines       22731    22735    +4     
=======================================
+ Hits        21514    21518    +4     
  Misses       1217     1217           

Impacted Files	Coverage Δ
libcst/_nodes/tests/test_atom.py	100.00% <ø> (ø)
libcst/_parser/parso/python/tokenize.py	83.28% <100.00%> (+0.09%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 507b453...9fc8a77. Read the comment docs.

[bgw]

It doesn't seem like this problem is unique to raw f-strings? It seems that { and } characters are processed before escape characters are:

>>> a = "test"
>>> f"\{a}"
'\\test'

I wish there was a conformance suite for f-strings.

[rant]

CPython's algorithm for this is much simpler than ours (and therefore, much less bug-prone). The decision to split f-strings in the tokenizer is one of my regrets in LibCST's design.

CPython doesn't try to split f-strings in the tokenizer. Instead, it detects f-strings when building the AST and then recursively calls the tokenizer and parser. This moves complexity out of the (already way too complex) tokenizer. We could do the same, but we didn't, and at this point, fixing it is harder than just patching these bugs when they come up.

It might make sense to change approaches when we inevitably have to rewrite much of the parser to support 3.10's soft-keywords (e.g. using a PEG grammar).

I might try to tackle this as part of my Rust porting attempts (#452). My new Rust-based tokenizer implementation supports enabling/disabling the f-string splitting behavior with a feature flag.

[/rant]

[zsol]

I'm not super familiar with this part but if there's no other reviews by Monday I'll accept and merge