CSTValidationError Must have at least one space before 'for' keyword.

[tushar-deepsource]

Code

[i if True else ""for i in "abc"]

Parser output

>>> import libcst
>>> libcst.parse_expression('[i if True else ""for i in "abc"]')
Traceback (most recent call last):
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/libcst/_parser/base_parser.py", line 151, in _add_token
    plan = stack[-1].dfa.transitions[transition]
KeyError: ReservedString(])

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/libcst/_parser/base_parser.py", line 156, in _add_token
    self._pop()
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/libcst/_parser/base_parser.py", line 220, in _pop
    new_node = self.convert_nonterminal(tos.dfa.from_rule, tos.nodes)
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/libcst/_parser/python_parser.py", line 44, in convert_nonterminal
    return self.nonterminal_conversions[nonterminal](self.config, children)
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/libcst/_parser/conversions/expression.py", line 1152, in convert_testlist_comp_list
    return _convert_testlist_comp(
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/libcst/_parser/conversions/expression.py", line 1183, in _convert_testlist_comp
    comprehension_type(elt=elt.value, for_in=for_in, lpar=(), rpar=()),
  File "<string>", line 9, in __init__
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/libcst/_nodes/base.py", line 118, in __post_init__
    self._validate()
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/libcst/_nodes/expression.py", line 3511, in _validate
    raise CSTValidationError(
libcst._nodes.base.CSTValidationError: Must have at least one space before 'for' keyword.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/libcst/_parser/entrypoints.py", line 160, in parse_expression
    result = _parse(
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/libcst/_parser/entrypoints.py", line 56, in _parse
    return _pure_python_parse(
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/libcst/_parser/entrypoints.py", line 89, in _pure_python_parse
    result = parser.parse()
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/libcst/_parser/base_parser.py", line 110, in parse
    self._add_token(token)
  File "/Users/tusharsadhwani/code/marvin-python/venv3/lib/python3.10/site-packages/libcst/_parser/base_parser.py", line 170, in _add_token
    raise ParserSyntaxError(
libcst._exceptions.ParserSyntaxError: Syntax Error @ 1:33.
Internal error: Must have at least one space before 'for' keyword.

[i if True else ""for i in "abc"]
                                ^

[zsol]

Note that the parse error doesn't happen anymore with the native parser, but the validation error is still there. That should be reasonably easy to fix

[mx781]

@zsol Looks like there might still be some uncovered cases - the following fails on the latest PyPI version:

cst.parse_module("[(b if 1 else a)for x in []]")

[mx781]

@zsol seems like it's something specific to ternary statements that's not getting parsed correctly as the following also fail with a similar error:

cst.parse_module('assert(1 if True else 0)')
cst.parse_module('1 if True else(1 if True else 0)')

[zsol]

We'll probably need to relax the validation rules even further than #935, if it's still broken in recent releases