Adopt a new Perl6/Raku grammar

[bscan]

Remove Perl6 from the set of built-in languages. Perl6 was renamed to "Raku" in 2019, so the current display name is wrong. Further, the syntax highlighting is outdated relative to the latest text-mate grammars. Removing Perl6 entirely would reduce this maintenance burden, and allow extension developers to iterate independently. There are many languages far more popular than Raku that require extensions to define the language configuration and syntax highlighting (e.g. Scala, Kotlin, COBOL, Fortran, Haskell, Erlang, Lisp, Scheme).

If Perl6 is removed, then vscode would automatically prompt users to download the extension for Raku, which provides updated syntax highlighting and a language server. https://github.com/bscan/RakuNavigator

Example of incorrect syntax highlighting can be seen here where a user was using the old built-in "Perl6" highlighting instead of the updated Raku highlighting. bscan/RakuNavigator#1

GitHub's highlighting has also already shifted from using the old Perl6 grammars to using the newer Raku grammars as can be seen here: https://github.com/github/linguist/blob/master/vendor/README.md

Thanks!

[alexr00]

We don't often get complaints about the current Perl6 grammar, so I don't want to completely remove it as it is a better experience to have syntax highlighting built in.

Instead of removing the existing Perl6 grammar, we could adopt a new, better, grammar. It looks like you have one: https://github.com/bscan/RakuNavigator/blob/master/syntaxes/raku.tmLanguage.json

If it's ok with you @bscan, once https://marketplace.visualstudio.com/items?itemName=bscan.raku-navigator gets a few more installs I'd like to switch VS Code over to using https://github.com/bscan/RakuNavigator/blob/master/syntaxes/raku.tmLanguage.json as the built in Perl6 grammar. I'll also change the display name to Raku at that time.

[alexr00]

Assigning to a later milestone to check back in then.

[bscan]

Sure, keeping Raku syntax highlighting sounds good too. However, I'd recommend tracking the upstream repo here https://github.com/Raku/atom-language-perl6 , which is part of the official Raku organization . This is also the repo that Github/Linguist are using for Raku syntax highlighting. The raku.tmLanguage.json file in the RakuNavigator repo is essentially https://github.com/Raku/atom-language-perl6/blob/master/grammars/raku.cson , except converted to json.

Once vscode has the updated grammar, raku name, and raku file associations, I will remove them from the Raku extension, and the extension will then have the language server as its primary focus. Thanks!

[alexr00]

https://github.com/Raku/atom-language-perl6 makes sense to me. Unfortunately that repo hasn't had a commit in over 2 years :(.

[alexr00]

The current perl grammar at https://github.com/textmate/perl.tmbundle has been more recently maintained then the grammar at https://github.com/Raku/atom-language-perl6. I don't want to update to a less well maintained grammar. Transferring to the backlog to check in again later.

[bscan]

Hi @alexr00 , the Perl (Perl5) grammar is actively maintained in the textmate repo, but the Raku (formerly known as Perl6) grammar in that repo is unmaintained with a last commit of more than 6 years ago. The grammar in the Raku/atom repo is 3.5 years newer, and actively utilized by GitHub (via Linguist) as well.

say 'This is highlighted with the "new" 2020 Raku grammar' ;

Although I agree with your concerns of the Raku repo not being updated since 2020. I can work with the Raku team to update that repo, and ideally make it more vscode friendly (e.g. convert .cson files to .tmLanguage.json and be more generic than an atom extension). Thanks!

[2colours]

@alexr00 hello, after the recent development of things, https://github.com/Raku/atom-language grammar is not only more up-to-date content-wise (which it had been previously as well, mind you) but it's more maintainable as well, after some renovation.

Keeping it in mind that the renaming of the language has happened some 4 years ago, please consider switching to that grammar so that the Raku community at least gets a chance to take charge of presenting a consistent, more appealing vision.

[alexr00]

@2colours thanks for the ping! I will try out the new grammar. One thing that we will need to consider is that the name of the new grammar is "Raku" and the old grammar is "Perl 6". The scopeName is also different: source.raku vs. source.perl.6. Changing this scope name might break extensions that rely on it.

[alexr00]

@2colours I've tried out the Raku grammar. I'm seeing a lot of surprising coloring:

The old Perl6 grammar is on the left, the new Raku grammar is on the right:

• The $!x and $!y on line 7 are not colored properly
• The to on line 10 is unexpectedly colored
• the first letter of the $variable point is always miscolored

For now, we will not adopt the new grammar. I will just rename the current Perl 6 grammar to Raku.

[2colours]

class Point is rw {
    has $.x;
    has $.y;
    
    method distance( Point $p ) {
        sqrt(($!x - $p.x) ** 2 + ($!y - $p.y) ** 2)
    }
    
    method distance-to-center {
        self.distance: Point.new(x => 0, y => 0)
    }
}

my $point = Point.new( x => 1.2, y => -3.7 );
say "Point's location: (", $point.x, ', ', $point.y, ')';
# OUTPUT: Point's location: (1.2, -3.7)

# Changing x and y (note methods "x" and "y" used as lvalues):
$point.x = 3;
$point.y = 4;
say "Point's location: (", $point.x, ', ', $point.y, ')';
# OUTPUT: Point's location: (3, 4)

my $other-point = Point.new(x => -5, y => 10);
$point.distance($other-point); #=> 10
$point.distance-to-center;     #=> 5

Github Linguist uses the same highlighter data and does a third thing. Not sure what you are using or how they are handling the textmate json files. All in all, I would be suspicious of the provided screenshots and how to get them.

@alexr00 anyway, it's disheartening more than by a bit because that package is likely to never get updated, and as somebody completely unfamiliar with Textmate highlighters, I specifically worked towards making this package maintainable. Not "better" than an abandoned package that most people wouldn't want to read, based on some ad-hoc, hardly representative testing.

[alexr00]

@2colours, you can see how I tried out the grammar in this PR, which I created for transparency and to save the changes I made so that the Raku grammar can easily be adopted later if it's fixed: #192877.

Having a sample file that we try the grammar out on is how we test all the grammars we depend on. We have a test that uses these sample files to easily tell what changed when we pull in updates to grammars. You can also see this in #192877. I suspect there is some difference in how GitHub Linguist's TextMate interpreter works (VS Code's is here: https://github.com/microsoft/vscode-textmate).

I am in favor of more maintainable, but we also can't give users a worse experience. If you want to try out the grammar in VS Code, you can easily create an extension and copy over everything that I put in the Raku extension in my PR.

[alexr00]

@2colours I made a repo for testing the grammar with VS Code so you can very easily try it: https://github.com/alexr00/testraku

[alexr00]

To verify:

1. Create a new file and save it with the .raku file extension.
2. Verify that the language mode of the editor is Raku