Allow trailing whitespace without newline at EOF

[zsol]

Summary

The native tokenizer was a bit too strict about indentation checking at EOF. This is valid Python: if 1:\n pass\n (note the extra space at the end without a newline), but the tokenizer was trying to match the single space at the end as if it was a dedent from level 4 -> 1, which failed (there was never a level 1 indentation).

This PR relaxes the tokenizer a bit to allow for such cases. Indentation checking is skipped if after consuming all whitespace at the beginning of the line, we encounter EOF.

Fixes #608

Test Plan

Added unit test.

[zsol]

cc @bgw AFAICT this was a bug in your original implementation as well

[zsol]

CPython pure python tokenizer does the same: https://github.com/python/cpython/blob/main/Lib/tokenize.py#L495-L496

[codecov-commenter]

Codecov Report

Merging #611 (3bb68a8) into main (332710d) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #611   +/-   ##
=======================================
  Coverage   94.77%   94.77%           
=======================================
  Files         245      245           
  Lines       25140    25140           
=======================================
  Hits        23826    23826           
  Misses       1314     1314           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 332710d...3bb68a8. Read the comment docs.

[stroxler]

Looks good to me