... (yada-yada) parsing is inconsistent #16165

p5pRT · 2017-09-23T11:39:49Z

Migrated from rt.perl.org#132150 (status was 'resolved')

Searchable as RT132150$

p5pRT · 2017-09-23T11:39:49Z

From @mauke

Created by @mauke

Perl parses '...' as 'die("Unimplemented")':

$ perl -MO=Deparse -e '...'
die 'Unimplemented';
-e syntax OK

But it only does this at the beginning of a statement:

$ perl -MO=Deparse -e '+ ...'
syntax error at -e line 1, near "+ ..."
-e had compilation errors.

However, '...' is not by itself a statement:

$ perl -MO=Deparse -e '... sub foo {}'
syntax error at -e line 1, near "... sub foo "
-e had compilation errors.

In fact, it's an expression:

$ perl -MO=Deparse -e '... . "hello"'
die('Unimplemented') . 'hello';
-e syntax OK

$ perl -MO=Deparse -e '... lt "hello"'
die('Unimplemented') lt 'hello';
-e syntax OK

$ perl -MO=Deparse -e '..., foo()'
die('Unimplemented'), foo();
-e syntax OK

However, it's a naughty expression that doesn't tell the parser that the next
token should be an operator:

$ perl -MO=Deparse -e '... + 0'
syntax error at -e line 1, near "... +"
-e had compilation errors.

Here '+ 0' is parsed as a unary + applied to 0. The syntax error occurs because
you can't have two terms in a row ('...', '+0') without an operator in between.

This is especially noticeable with '/':

$ perl -MO=Deparse -e '... / 1'
Search pattern not terminated at -e line 1.

$ perl -MO=Deparse -e '... / 1 /'
syntax error at -e line 1, near "... / 1 /"
-e had compilation errors.

For consistency, either '...' should be made a statement of its own, or it
should give operator context to the following token.

Perl Info


Flags:
    category=core
    severity=low

Site configuration information for perl 5.26.0:

Configured by mauke at Fri Sep 22 13:28:36 CEST 2017.

Summary of my perl5 (revision 5 version 26 subversion 0) configuration:
   
  Platform:
    osname=linux
    osvers=4.9.41-1-lts
    archname=i686-linux
    uname='linux simplicio 4.9.41-1-lts #1 smp mon aug 7 17:57:02 cest 2017 i686 gnulinux '
    config_args=''
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=undef
    usemultiplicity=undef
    use64bitint=undef
    use64bitall=undef
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
    bincompat5005=undef
  Compiler:
    cc='cc'
    ccflags ='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'
    optimize='-O2 -march=native'
    cppflags='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
    ccversion=''
    gccversion='7.2.0'
    gccosandvers=''
    intsize=4
    longsize=4
    ptrsize=4
    doublesize=8
    byteorder=1234
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=12
    longdblkind=3
    ivtype='long'
    ivsize=4
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=4
    prototype=define
  Linker and Libraries:
    ld='cc'
    ldflags ='-fstack-protector-strong -L/usr/local/lib'
    libpth=/usr/local/lib /usr/lib/gcc/i686-pc-linux-gnu/7.2.0/include-fixed /usr/lib /lib
    libs=-lpthread -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat
    perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
    libc=libc-2.26.so
    so=so
    useshrplib=false
    libperl=libperl.a
    gnulibc_version='2.26'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=so
    d_dlsymun=undef
    ccdlflags='-Wl,-E'
    cccdlflags='-fPIC'
    lddlflags='-shared -O2 -march=native -L/usr/local/lib -fstack-protector-strong'



@INC for perl 5.26.0:
    /home/mauke/usr/lib/perl5/site_perl/5.26.0/i686-linux
    /home/mauke/usr/lib/perl5/site_perl/5.26.0
    /home/mauke/usr/lib/perl5/5.26.0/i686-linux
    /home/mauke/usr/lib/perl5/5.26.0


Environment for perl 5.26.0:
    HOME=/home/mauke
    LANG=en_US.UTF-8
    LANGUAGE=en_US
    LC_COLLATE=C
    LC_MONETARY=de_DE.UTF-8
    LC_TIME=de_DE.UTF-8
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/home/mauke/perl5/perlbrew/bin:/home/mauke/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl
    PERLBREW_BASHRC_VERSION=0.73
    PERLBREW_HOME=/home/mauke/.perlbrew
    PERLBREW_ROOT=/home/mauke/perl5/perlbrew
    PERL_BADLANG (unset)
    PERL_UNICODE=SAL
    SHELL=/bin/bash

p5pRT · 2017-09-23T12:28:35Z

From @mauke

On Sat, 23 Sep 2017 04:39:49 -0700, mauke- wrote:

[Please describe your issue here]

Perl parses '...' as 'die("Unimplemented")':

$ perl -MO=Deparse -e '...'
die 'Unimplemented';
-e syntax OK

But it only does this at the beginning of a statement:

$ perl -MO=Deparse -e '+ ...'
syntax error at -e line 1, near "+ ..."
-e had compilation errors.

However, '...' is not by itself a statement:

$ perl -MO=Deparse -e '... sub foo {}'
syntax error at -e line 1, near "... sub foo "
-e had compilation errors.

In fact, it's an expression:

$ perl -MO=Deparse -e '... . "hello"'
die('Unimplemented') . 'hello';
-e syntax OK

$ perl -MO=Deparse -e '... lt "hello"'
die('Unimplemented') lt 'hello';
-e syntax OK

$ perl -MO=Deparse -e '..., foo()'
die('Unimplemented'), foo();
-e syntax OK

However, it's a naughty expression that doesn't tell the parser that
the next
token should be an operator:

$ perl -MO=Deparse -e '... + 0'
syntax error at -e line 1, near "... +"
-e had compilation errors.

Here '+ 0' is parsed as a unary + applied to 0. The syntax error
occurs because
you can't have two terms in a row ('...', '+0') without an operator in
between.

This is especially noticeable with '/':

$ perl -MO=Deparse -e '... / 1'
Search pattern not terminated at -e line 1.

$ perl -MO=Deparse -e '... / 1 /'
syntax error at -e line 1, near "... / 1 /"
-e had compilation errors.

For consistency, either '...' should be made a statement of its own,
or it
should give operator context to the following token.

Patch (to do the latter) attached. Thoughts?

p5pRT · 2017-09-23T12:28:35Z

From @mauke

0001-parse-.-yadayada-as-a-term-not-an-operator-RT-132150.patch

From 02be2fc4ff7e4ede46381ee96b533ffe1aaa50ab Mon Sep 17 00:00:00 2001
From: Lukas Mai <[email protected]>
Date: Sat, 23 Sep 2017 14:25:32 +0200
Subject: [PATCH] parse ... (yadayada) as a term, not an operator (RT #132150)

---
 t/op/yadayada.t | 19 ++++++++++++++++++-
 toke.c          |  2 +-
 2 files changed, 19 insertions(+), 2 deletions(-)

diff --git t/op/yadayada.t t/op/yadayada.t
index 861389f4c5..0013e37046 100644
--- t/op/yadayada.t
+++ t/op/yadayada.t
@@ -8,7 +8,7 @@ BEGIN {
 
 use strict;
 
-plan 9;
+plan 12;
 
 my $err;
 my $err1 = "Unimplemented at $0 line ";
@@ -42,6 +42,23 @@ eval { @transformed = map {;... } @input; };
 is $@, $err, "Disambiguation case 4";
 $@ = '';
 
+
+note("RT #132150: ... is a term, not an operator");
+$err = $err1 . ( __LINE__ + 1 ) . $err2;
+eval { ... + 0 };
+is $@, $err, "... + 0 parses";
+$@ = '';
+
+$err = $err1 . ( __LINE__ + 1 ) . $err2;
+eval { ... % 1 };
+is $@, $err, "... % 1 parses";
+$@ = '';
+
+$err = $err1 . ( __LINE__ + 1 ) . $err2;
+eval { ... / 1 };
+is $@, $err, "... / 1 parses";
+$@ = '';
+
 #
 # Regression tests, making sure ... is still parsable as an operator.
 #
diff --git toke.c toke.c
index a91a4fcfbe..63f8d418ca 100644
--- toke.c
+++ toke.c
@@ -6856,7 +6856,7 @@ Perl_yylex(pTHX)
 	}
 	if (PL_expect == XSTATE && s[1] == '.' && s[2] == '.') {
 	    s += 3;
-	    OPERATOR(YADAYADA);
+	    TERM(YADAYADA);
 	}
 	if (PL_expect == XOPERATOR || !isDIGIT(s[1])) {
 	    char tmp = *s++;
-- 
2.14.1

p5pRT · 2017-09-23T16:54:39Z

From [email protected]

l.mai@web.de wrote:

For consistency, either '...' should be made a statement of its own, or it
should give operator context to the following token.

The current behaviour is indeed inconsistent, but the statement-like
option for making it consistent doesn't require that '...' be a complete
statement. It would be perfectly consistent for '...;' to be the complete
statement (with semicolon implicit before closing brace, as it already
is) with '... sub foo {}' remaining illegal.

-zefram

p5pRT · 2017-09-23T16:54:39Z

The RT System itself - Status changed from 'new' to 'open'

p5pRT · 2017-11-02T19:53:41Z

From [email protected]

l.mai@web.de via RT wrote:

Patch (to do the latter) attached. Thoughts?

perly.y makes clear that yada-yada is intended to be an expression, not
a statement. Patch applied as f5727a1.

-zefram

p5pRT · 2017-11-06T16:51:35Z

From @Smylers

Zefram writes:

perly.y makes clear that yada-yada is intended to be an expression,
not a statement. Patch applied as
f5727a1.

perlsyn documents it as being the ellipsis statement, specifically
saying:

You can only use the elliptical statement to stand in for a complete
statement. … The elliptical statement cannot stand in for an
expression that is part of a larger statement, since the "..." is also
the three-dot version of the flip-flop operator (see "Range Operators"
in perlop).

These examples of attempts to use an ellipsis are syntax errors:

use v5.12;

print ...;
open(my $fh, ">", "/dev/passwd") or ...;
if ($condition && ... ) { say "Howdy" };

Sounds like that is no longer true (and quite possibly that use of the
word “since” there was never true).

Smylers

p5pRT · 2017-11-10T01:06:43Z

From [email protected]

Smylers wrote:

perlsyn documents it as being the ellipsis statement,

Hmm, that's an embuggerance. The bit you quoted about ambiguity with the
flip-flop operator doesn't actually make sense as a rationale, because
we resolve those kinds of ambiguities all the time, for example between
the "+" infix operator and the "+" prefix operator. But it turns out
there's more inconsistency to the parsing code.

Not only was PL_expect being set in a way hostile to treating "..." as a
term, but the "..." token was not recognised in the contexts that would
be required for it to be a term. So after Lukas's patch, although "... +
0" is now accepted, "0 + ..." is still not accepted. It would take more
changes to make it consistently a term.

Taking into account the documentation and these additional implementation
issues, I reckon the old code (prior to Lukas's patch) was closer to
having "...;" be a statement than "..." be an expression. It looks like
the grammar in perly.y was actually the one thing inconsistent with that.
I'll try changing it to a statement, and see how that goes.

-zefram

p5pRT · 2017-11-10T02:27:24Z

From [email protected]

I wrote:

I'll try changing it to a statement, and see how that goes.

Looks like that works just fine. Only takes a small perly.y change
(though I also reverted the toke.c change). The documentation in perlsyn
was a bit ropey: in addition to the incorrect statement about ambiguity
with flip-flop, it had some stuff about block/hash ambiguity in map, in
a situation that since 5.22 has not parsed the way it claims. So I've
removed the incorrect stuff and added a bit more clarification there.
Also some test cases about when "..." should not parse as yada-yada.

Applied to blead as commit 29d69c3.

I wasn't sure how to classify this for perldelta. On the one hand it's
a bugfix; on the other it's making illegal some constructs that made at
least partial sense and were previously accepted. I've conservatively
classified it as an "incompatible change". However, if we think of
it as incompatible, we need to decide whether a deprecation cycle
is required. Implementing a deprecation would be a bit more involved,
but not disproportionately so. The closest precedent I can think of for
this sort of change is the removal of the implicit parens around qw,
for which we did do a deprecation cycle, based on the knowledge that
the thing being deprecated was moderately widespread. The bad uses of
yada-yada are probably a lot less common.

-zefram

p5pRT · 2017-11-10T14:29:07Z

From @cpansprout

On Thu, 09 Nov 2017 17:06:43 -0800, zefram@fysh.org wrote:

The bit you quoted about ambiguity with the
flip-flop operator doesn't actually make sense as a rationale, because
we resolve those kinds of ambiguities all the time, for example between
the "+" infix operator and the "+" prefix operator.

Introducing a new term or prefix operator that looks like an existing infix breaks existing syntax, of the form:

substr $x, 1, . substr $y, 1;

(I write code like that all the time.) If you put ... instead of . then you have an example of code that a ... term will break.

It doesn’t happen the other way round. We can introduce a new infix that looks like an existing prefix or term.

--

Father Chrysostomos

p5pRT · 2017-11-10T14:29:51Z

From @cpansprout

On Thu, 09 Nov 2017 18:27:24 -0800, zefram@fysh.org wrote:

Applied to blead as commit 29d69c3.

I think that means this can be closed. I am doing that.

--

Father Chrysostomos

p5pRT · 2017-11-10T14:29:51Z

@cpansprout - Status changed from 'open' to 'pending release'

p5pRT · 2018-06-23T15:31:37Z

From @khwilliamson

Thank you for filing this report. You have helped make Perl better.

With the release yesterday of Perl 5.28.0, this and 185 other issues have been
resolved.

Perl 5.28.0 may be downloaded via:
https://metacpan.org/release/XSAWYERX/perl-5.28.0

If you find that the problem persists, feel free to reopen this ticket.

p5pRT · 2018-06-23T15:31:37Z

@khwilliamson - Status changed from 'pending release' to 'resolved'

p5pRT closed this as completed Jun 23, 2018

p5pRT added Severity Low type-core labels Oct 19, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

... (yada-yada) parsing is inconsistent #16165

... (yada-yada) parsing is inconsistent #16165

p5pRT commented Sep 23, 2017

p5pRT commented Sep 23, 2017

p5pRT commented Sep 23, 2017

p5pRT commented Sep 23, 2017

p5pRT commented Sep 23, 2017

p5pRT commented Sep 23, 2017

p5pRT commented Nov 2, 2017

p5pRT commented Nov 6, 2017

p5pRT commented Nov 10, 2017

p5pRT commented Nov 10, 2017

p5pRT commented Nov 10, 2017

p5pRT commented Nov 10, 2017

p5pRT commented Nov 10, 2017

p5pRT commented Jun 23, 2018

p5pRT commented Jun 23, 2018

... (yada-yada) parsing is inconsistent #16165

... (yada-yada) parsing is inconsistent #16165

Comments

p5pRT commented Sep 23, 2017

p5pRT commented Sep 23, 2017

From @mauke

Created by @mauke

p5pRT commented Sep 23, 2017

From @mauke

p5pRT commented Sep 23, 2017

From @mauke

p5pRT commented Sep 23, 2017

From [email protected]

p5pRT commented Sep 23, 2017

p5pRT commented Nov 2, 2017

From [email protected]

p5pRT commented Nov 6, 2017

From @Smylers

p5pRT commented Nov 10, 2017

From [email protected]

p5pRT commented Nov 10, 2017

From [email protected]

p5pRT commented Nov 10, 2017

From @cpansprout

p5pRT commented Nov 10, 2017

From @cpansprout

p5pRT commented Nov 10, 2017

p5pRT commented Jun 23, 2018

From @khwilliamson

p5pRT commented Jun 23, 2018