Avoid tokenizer exceptions for qualified expressions

[skaupper]

Currently qualified expressions trigger a tokenizer exception.

This PR should take care of these, by recognizing that the character pattern '( may not be followed by another single quote (which would produce the character literal '('). In those cases both characters are output as instances of CharacterToken.

An additional test case ensure a correct parsing of qualified expressions (e.g. UNSIGNED'(x"0")) and the relevant character literal (.

[Paebbels]

Should we have a test for these sequences:

foo'('0')
foo'('(')

[Paebbels]

Suggested change

	if buffer[0] == "(": # Relevant for qualified expressions
	if buffer[0] == "(": # Relevant for qualified expressions

Python (styling) requires at least 2 spaces before #. I'm following Python coding styles loosely, but this is a good rule :).

[Paebbels]

@skaupper are you planning more fixes like this or should I prepare for a release of that code to main?

[skaupper]

Should we have a test for these sequences:

foo'('0')
foo'('(')

Well, without trying it I am pretty sure these cases will fail with the current solution.
It will probably be tricky to unambiguously support all variations of character literals, qualified expressions and attributes. Maybe with a deeper lookahead?

Can you think of an example of the form foo'(',')' being valid VHDL?

I will have another look at it!

[skaupper]

@skaupper are you planning more fixes like this or should I prepare for a release of that code to main?

Currently I focus on fixing things which occur while parsing some generated VHDL sources. As of now it seems qualified expressions are the last issue with the tokenizer for my use cases.

I may have a look into some issues with the block parser afterwards (e.g. #9 and #16). Personally, I do not need a release though.

[Paebbels]

Can you think of an example of the form foo'(',')' being valid VHDL?

I don't think so.

F****

Type, qualified expression on character , followed by an attribute ...

character'(',')'length

[skaupper]

Type, qualified expression on character , followed by an attribute ...

character'(',')'length

After digging through the LRM and using ModelSim to confirm some ideas, it seems like character'(',')'length is not valid VHDL, since a qualified expression is neither a name nor a function_call according to the syntax.

On the other hand these snippets ARE valid VHDL: character'(''') and ''', both representing a value of '.

A similiar issue arises with the string literal "a""b" representing the string a"b (both a and b can be omitted for a string literal of """") but that should be part of another issue/PR, i guess.

[skaupper]

For this PR, I will focus on solving the foo'('(') problem. Using a look-behind tokens seems promising, since '( may only be a qualified expression if a WordToken preceeds it (any counter examples are welcome).

Any shenanigans involving ''' and """" should probably be handled in a separate PR.

[Paebbels]

Please remove the draft state when ready for review / merge.