Support indented here-docs (fix incorrect string highlighting)

[aeschli]

From @skaurus on October 20, 2017 14:34

I'm talking about this: http://perldoc.perl.org/perlop.html#Indented-Here-docs

Steps to Reproduce:

1. write perl code like that:

if ($cond) {
    my $string = <<~EOF;
        indented
        text
        will be
        indented
        just by 4 spaces
        and totally not 8
    EOF
    my $res = $self->method;
}
...

2. code after closing EOF will be colored as if it's still inside the string.

Currently it works only if opening here-doc marker does not have a tilde; AND closing marker is not indented (at all).

Reproduces without extensions: Yes

Copied from original issue: microsoft/vscode#36640

[aeschli]

From @roblourens on October 20, 2017 17:31

@mjbvz This is perl, not PHP :D

[skaurus]

Maybe I can help?
Issues like this will pop up from time to time (Perl is not getting much love these days...) and I am motivated to fix them.

[skaurus]

Really, could you please point me to the relevant source file, or docs, or anything and I'll try squash that motherfbug. I'm a bit pissed by it at the moment. I have to create a multiline comment and write inside specific incantation to make syntax highlighting happy, and that looks ugly.

[rdwampler]

@skaurus:
I believe this the relevant section in the grammar.

perl.tmbundle/Syntaxes/Perl.plist

Lines 2042 to 2113 in 7390bf7

	<dict>
	<key>begin</key>
	<string>(((&lt;&lt;) *'([^']*)'))(.*)\n?</string>
	<key>captures</key>
	<dict>
	<key>1</key>
	<dict>
	<key>name</key>
	<string>punctuation.definition.string.perl</string>
	</dict>
	<key>2</key>
	<dict>
	<key>name</key>
	<string>string.unquoted.heredoc.quote.perl</string>
	</dict>
	<key>3</key>
	<dict>
	<key>name</key>
	<string>punctuation.definition.heredoc.perl</string>
	</dict>
	<key>4</key>
	<dict>
	<key>patterns</key>
	<array>
	<dict>
	<key>include</key>
	<string>$self</string>
	</dict>
	</array>
	</dict>
	</dict>
	<key>contentName</key>
	<string>string.unquoted.heredoc.quote.perl</string>
	<key>end</key>
	<string>(^\4$)</string>
	</dict>
	<dict>
	<key>begin</key>
	<string>(((&lt;&lt;) *\\((?![=\d\$\( ])[^;,'"`\s\)]*)))(.*)\n?</string>
	<key>captures</key>
	<dict>
	<key>1</key>
	<dict>
	<key>name</key>
	<string>punctuation.definition.string.perl</string>
	</dict>
	<key>2</key>
	<dict>
	<key>name</key>
	<string>string.unquoted.heredoc.quote.perl</string>
	</dict>
	<key>3</key>
	<dict>
	<key>name</key>
	<string>punctuation.definition.heredoc.perl</string>
	</dict>
	<key>4</key>
	<dict>
	<key>patterns</key>
	<array>
	<dict>
	<key>include</key>
	<string>$self</string>
	</dict>
	</array>
	</dict>
	</dict>
	<key>contentName</key>
	<string>string.unquoted.heredoc.quote.perl</string>
	<key>end</key>
	<string>(^\4$)</string>
	</dict>

I believe you'll need to add a regex to match the indented terminator. An example can be found in the ruby bundle:
https://github.com/textmate/ruby.tmbundle/blob/300b9961332c6dc8e282bd8c3192c2ea528f95a6/Syntaxes/Ruby.plist#L1765-L1805

[skaurus]

Thanks!

[skaurus]

So I finally got to it.

It seems that regexes that match the start of the here-docs can have one capture group less, since groups 1 and 2 are identical. So line 2044 from the code fragment above would be:
<string>((&lt;&lt;) *'([^']*)')(.*)\n?</string>

Now, to actually support indented here-docs it would be nice to use regex conditionals: if opening here-doc identifier have ~, then and only then closing identifier can have \s* before it. Wikipedia says that TextMate uses oniguruma regex library, and it have support for regex conditionals since version 6.5.0. But it seems that current release of TextMate uses some older version because conditional regexes do not work.

Right now we can allow spaces regardless of ~ presence, but that can broke highlighting in some cases.
Or, I can create copies of all existing here-doc definitions, with ~ and spaces allowed.

Which way is better do you think?

[skaurus]

Commit with an extra capture groups removed (only in here-docs definitions) and spaces allowed regardless of tilda presence would look like this: skaurus@bba43a9

With regex conditionals, ~? in start regexes would be replaced with (~)?, and (^\3$) in end regexes with (^(?(3)\s*|)\4$).

[skaurus]

Hey?

[nezirin]

m(._.)m

[camtauxe]

Looks like there's another pull request for the same thing #41, (though it's just as neglected)

[PatrickCronin]

There's also #42, which is a bit more thorough than 41.

[namotco]

Any movement on this PR?

[PatrickCronin]

I believe #42 (merged) has addressed this issue.

[namotco]

I see thanks!

…

On Fri, Jul 26, 2024 at 7:11 AM Patrick Cronin ***@***.***> wrote: I believe #42 <#42> (merged) has addressed this issue. — Reply to this email directly, view it on GitHub <#26 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADJNXAHSI3WLVQQE3JPTUTZOI4FJAVCNFSM6AAAAABLPYIOVWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJSGYZTAMJUGM> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- Ryan Perry Technology Director www.pharmetika.com 800-222-4767 x700 Middleton, WI 53562