set_formatter() failures introduced in DateTime-1.37

[chanwilson]

I'm seeing this in formerly-working code (heavy with Moose and other modules) after upgrading DateTime. Interestingly enough if the below snippet is placed in a file and run outside the debugger it works as expected, but will always fail when run in the debugger like below.

perl -de 1

Loading DB routines from perl5db.pl version 1.49_04
Editor support available.

Enter h or 'h h' for help, or 'man perldebug' for more help.

main::(-e:1):   1
  DB<1> use DateTime

  DB<2> $d = DateTime->now

  DB<3> p $d
2016-12-21T00:42:44
  DB<4> use DateTime::Format::CLDR

  DB<5> $YMDHMSs_format = DateTime::Format::CLDR->new( pattern => 'yyyy-MM-ddTHH:mm:ss.SSSSS' );

  DB<6> $d->set_formatter($YMDHMSs_format)
Can't call method "can" on unblessed reference at (eval 206)[/home/fms/local/perlbrew//perls/perl-5.24.0/lib/site_perl/5.24.0/Eval/Closure.pm:149] line 34, <DATA> line 1.

  DB<7> $strp_format = DateTime::Format::Strptime->new( pattern => '%FT%R:%S.%N')
Can't locate object method "new" via package "DateTime::Format::Strptime" (perhaps you forgot to load "DateTime::Format::Strptime"?) at (eval 213)[/home/fms/local/perlbrew//perls/perl-5.24.0/lib/5.24.0/perl5db.pl:737] line 2, <DATA> line 1.

  DB<8> use DateTime::Format::Strptime

  DB<9> $strp_format = DateTime::Format::Strptime->new( pattern => '%FT%R:%S.%N')

  DB<10> $d->set_formatter($strp_format)
Can't call method "can" on unblessed reference at (eval 206)[/home/fms/local/perlbrew//perls/perl-5.24.0/lib/site_perl/5.24.0/Eval/Closure.pm:149] line 34, <DATA> line 1.

  DB<11> p $DateTime::VERSION
1.39
  DB<12>

Tracing the set_formatter call gives this:

  DB<5> t
Trace = on
  DB<5> $d->set_formatter($YMDHMSs_format);
main::((eval 211)[/home/fms/local/perlbrew//perls/p524/lib/5.24.0/perl5db.pl:737]:2):
2:      $d->set_formatter($YMDHMSs_format);;
DateTime::set_formatter(/home/fms/local/perlbrew//perls/p524/lib/site_perl/5.24.0/x86_64-linux/DateTime.pm:2055):
2055:           my $self = shift;
DateTime::set_formatter(/home/fms/local/perlbrew//perls/p524/lib/site_perl/5.24.0/x86_64-linux/DateTime.pm:2056):
2056:           my ($formatter) = $validator->(@_);
Eval::Closure::Sandbox_186::CODE(0x39e0cf8)((eval 206)[/home/fms/local/perlbrew//perls/p524/lib/site_perl/5.24.0/Eval/Closure.pm:149]:5):
5:      sub { if ( @_ < 1 ) {
Eval::Closure::Sandbox_186::CODE(0x39e0cf8)((eval 206)[/home/fms/local/perlbrew//perls/p524/lib/site_perl/5.24.0/Eval/Closure.pm:149]:15):
15:     if ( @_ > 1 ) {
Eval::Closure::Sandbox_186::CODE(0x39e0cf8)((eval 206)[/home/fms/local/perlbrew//perls/p524/lib/site_perl/5.24.0/Eval/Closure.pm:149]:34):
34:         List::Util::all { $_[0]->can($_) } "format_datetime"
35:     )
36:      ) )
37:      or Specio::Exception->throw(  message => $_Specio_Constraint_Interface_message_generator59->($_[0]), type    => $_Specio_Constraint_Interface_type59, value   => $_[0] );;
List::Util::all((eval 206)[/home/fms/local/perlbrew//perls/p524/lib/site_perl/5.24.0/Eval/Closure.pm:149]:34):
34:         List::Util::all { $_[0]->can($_) } "format_datetime"
35:     )
36:      ) )
37:      or Specio::Exception->throw(  message => $_Specio_Constraint_Interface_message_generator59->($_[0]), type    => $_Specio_Constraint_Interface_type59, value   => $_[0] );;
Can't call method "can" on unblessed reference at (eval 206)[/home/fms/local/perlbrew//perls/p524/lib/site_perl/5.24.0/Eval/Closure.pm:149] line 34, <DATA> line 1.

Hopefully this is enough information to debug and fix!

[preaction]

I was able to reproduce this on my Perl 5.24, DateTime 1.41, Specio 0.31. I think I tracked this exception down to this block of code: https://metacpan.org/source/DROLSKY/Specio-0.31/lib/Specio/Constraint/AnyCan.pm#L32-40, at first glance there didn't seem a way for this exception to be generated by that code since unblessed refs fail the blessed check and should not make it to the all check.

But I just tested something about all:

use List::Util qw( all );
sub foo { all { warn $_[0] && $_ } qw( 0 1 2 ) }
foo( qw( foo bar baz ) );

0 at test.pl line 2.
1 at test.pl line 2.
2 at test.pl line 2.

The all code block is still a sub call, so $_[0] is resolved in that context. That still doesn't make any sense, because the exception is Can't call method "can" on unblessed reference, and "format_datetime" is not an unblessed reference.

And that's where I left it last night. I'll pick up from here in the next day or so.

[preaction]

After some comments in irc.perl.org #perl, I tried reproducing just the weird all behavior, to no result:

use v5.10;
use List::Util;
sub foo {
  state $sub = eval 'sub { List::Util::all { warn $_[0] && $_ } qw( 0 1 2 ) }';
  $sub->( @_ );
}
foo( qw( foo bar baz ) );

0 at (eval 2) line 1.
1 at (eval 2) line 1.
2 at (eval 2) line 1.

This produces exactly the behavior I'd expect, so the problem can't be isolated to just this level.

[arodland]

@chanwilson could we know what version of List::Util (Scalar-List-Utils) you have installed, please?

[chanwilson]

List::Util::VERSION is 1.46

[wolfsage]

This works fine outside of the debugger, fails under it

use v5.10;
use List::Util;

{
  package cat;
  sub thing { print "hi" }
}

sub foo {
  $sub = eval 'sub { List::Util::all { $_[0]->can($_) } qw("thing") }';
  warn "Calling $sub with Args '@_'\n";
  $sub->( @_ );
}
foo( bless {}, "cat" );

[arodland]

A string eval doesn't seem necessary to get strange results. This program:

use v5.10;
use List::Util;

sub foo {
  $sub = sub { List::Util::all { print "[@_]\n" } qw(foo1 foo2) };
  $sub->( @_ );
}
foo("bar");

prints

[bar]
[bar]

outside the debugger and

[CODE(0x16d1bd0) foo1 foo2]
[CODE(0x16d1bd0) foo1 foo2]

inside it.

[wolfsage]

Simplified further:

sub foo {
  List::Util::all { $_[0]->can($_) } qw(thing);
}

foo( bless {}, "cat" );

[autarch]

I though that maybe List::Util might be coded to do something differently under the debugger, but I don't see anything in its code base to suggest that it is. That suggests a bug in the debugger itself, or possible in List::Util.

Either way, I'm not sure how I might go about fixing this in Specio, which is where the problem really lies.

[arodland]

A workaround would be to replace the all with an equivalent foreach loop in Specio's generated code. But I agree that this has to be either the perl debugger's bug or List::Util's.

[arodland]

Another workaround would be to capture $_[0] in a variable outside of the all sub, and refer to that instead.

[autarch]

@arodland - good points. I don't want to get rid of List::Util because it's highly optimized, and any replace I come up with won't be. Capturing $_[0] would be possible. That said, I'd really love to see this bug fixed at the root.

[autarch]

Fixed in Specio 0.32