Problem with unicode in Getopt::Long HelpMessage() in perl 5.38 #21841

bessarabov · 2024-01-15T19:53:00Z

Description
We have observed that starting from version 5.38.0, there has been a change in behavior.

We are using official docker images of perl.

Here is the test script, I've saved it in file not_working.pl:

use utf8;
use strict;
use warnings;
use feature qw(say);
use open qw(:std :utf8);

use Getopt::Long qw(GetOptions HelpMessage);

say 'привет';

HelpMessage();

__END__

=encoding utf-8

=head1 USAGE

你好

On docker image perl:5.36.3 (the latest image of 5.36) it is working as expected:

$ docker run --rm -it -v `pwd`:/app/ perl:5.36.3 perl /app/not_working.pl
привет
Usage:
    你好

But starting from perl:5.38.0. it does not work. See that instaed of 你好 it output garbage:

$ docker run --rm -it -v `pwd`:/app/ perl:5.38.0 perl /app/not_working.pl
привет
Usage:
    ä½ å¥½

The problem still persist on perl:5.38.2 (the latest 5.38.*):

$ docker run --rm -it -v `pwd`:/app/ perl:5.38.2 perl /app/not_working.pl
привет
Usage:
    ä½ å¥½

Steps to Reproduce

Please see the prvious section for the exact steps.

Expected behavior

I expect that the output of the script running in Perl version 5.38.2 should be identical to that in Perl version 5.36.3. However, the output differs.

Perl configuration

# perl -V output goes here

I'm using the official docker image, but here it is:

$ docker run --rm -it perl:5.38.2 perl -V
Summary of my perl5 (revision 5 version 38 subversion 2) configuration:

  Platform:
    osname=linux
    osvers=6.2.0-1018-azure
    archname=x86_64-linux-gnu
    uname='linux buildkitsandbox 6.2.0-1018-azure #18~22.04.1-ubuntu smp tue nov 21 19:25:02 utc 2023 x86_64 gnulinux '
    config_args='-Darchname=x86_64-linux-gnu -Duse64bitall -Duseshrplib -Dvendorprefix=/usr/local -des'
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=undef
    usemultiplicity=undef
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
  Compiler:
    cc='cc'
    ccflags ='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2'
    optimize='-O2'
    cppflags='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
    ccversion=''
    gccversion='12.2.0'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='cc'
    ldflags =' -fstack-protector-strong -L/usr/local/lib'
    libpth=/usr/local/lib /usr/lib/x86_64-linux-gnu /usr/lib /usr/lib64
    libs=-lpthread -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc
    perllibs=-lpthread -ldl -lm -lcrypt -lutil -lc
    libc=/lib/x86_64-linux-gnu/libc.so.6
    so=so
    useshrplib=true
    libperl=libperl.so
    gnulibc_version='2.36'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=so
    d_dlsymun=undef
    ccdlflags='-Wl,-E -Wl,-rpath,/usr/local/lib/perl5/5.38.2/x86_64-linux-gnu/CORE'
    cccdlflags='-fPIC'
    lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector-strong'


Characteristics of this binary (from libperl):
  Compile-time options:
    HAS_LONG_DOUBLE
    HAS_STRTOLD
    HAS_TIMES
    PERLIO_LAYERS
    PERL_COPY_ON_WRITE
    PERL_DONT_CREATE_GVSV
    PERL_HASH_FUNC_SIPHASH13
    PERL_HASH_USE_SBOX32
    PERL_MALLOC_WRAP
    PERL_OP_PARENT
    PERL_PRESERVE_IVUV
    PERL_USE_SAFE_PUTENV
    USE_64_BIT_ALL
    USE_64_BIT_INT
    USE_LARGE_FILES
    USE_LOCALE
    USE_LOCALE_COLLATE
    USE_LOCALE_CTYPE
    USE_LOCALE_NUMERIC
    USE_LOCALE_TIME
    USE_PERLIO
    USE_PERL_ATOF
  Built under linux
  Compiled at Jan 12 2024 00:32:56
  @INC:
    /usr/local/lib/perl5/site_perl/5.38.2/x86_64-linux-gnu
    /usr/local/lib/perl5/site_perl/5.38.2
    /usr/local/lib/perl5/vendor_perl/5.38.2/x86_64-linux-gnu
    /usr/local/lib/perl5/vendor_perl/5.38.2
    /usr/local/lib/perl5/5.38.2/x86_64-linux-gnu
    /usr/local/lib/perl5/5.38.2
$

The text was updated successfully, but these errors were encountered:

zhmylove · 2024-01-15T19:54:23Z

This bug is annoying, right

xenu · 2024-01-15T21:16:52Z

Downgrading podlators to 4.14 restores the old behaviour.

I think this is the same issue: rra/podlators#25

xenu · 2024-01-15T21:39:50Z

Found a simpler workaround than downgrading: add use PerlIO; to your code.

This is because Pod::Text code does this:

    eval {
        my @options = (output => 1, details => 1);
        my $flag = (PerlIO::get_layers ($$self{output_fh}, @options))[-1];
        if ($flag && ($flag & PerlIO::F_UTF8 ())) {
            $$self{ENCODE} = 0;
        }
    };

which throws a quietly discarded exception: Undefined subroutine &PerlIO::F_UTF8 called at /home/xenu/git/podlators/blib/lib/Pod/Text.pm line 368.

bessarabov · 2024-01-20T14:23:46Z

@xenu Huge thank you for investigating this issue!

You are absolutely right — in perl 5.38 there is a change:

podlators has been upgraded from version 4.14 to 5.01

(this is the quote from https://perldoc.perl.org/5.38.0/perldelta )

My issue is completly solved, I've updated the base images that I use for 5.38. I'm using images based on 2 official perl docker images, here are the fragments of my Dockerfile's with the main things:

FROM perl:5.38.2-bullseye

RUN cpanm Pod::[email protected]

# ... all other docker steps

FROM perl:5.38.2

RUN cpanm Pod::[email protected]

# ... all other docker steps

For me the problem is solved, but I'm not sure that I should do with this GitHub issue.

Probably I can close it right now, or probably it should be closed only when there are official perl docker images that don't have such problem (it can be achieved by using versions 4.14, or waiting for the new Pod::Man version that fix this issue; quite possible that this will be addressed in rra/podlators#25 )

jkeenan · 2024-01-21T17:07:33Z

Bisection confirms that this problem entered the Perl 5 core distribution with this commit in December 2022:

commit 20616d517ad4726a85ee27750db6e24443343601 (HEAD, refs/bisect/bad)
Author:     Russ Allbery <[email protected]>
AuthorDate: Tue Dec 6 22:40:57 2022 +0000
Commit:     James E Keenan <[email protected]>
CommitDate: Fri Dec 9 10:43:28 2022 -0500

    Update podlators to CPAN version 5.00

However, that commit was quite large, so simply identifying the offending commit does not, in and of itself, identify the problem or the remedy. Further research needed (which will first have to be tested upstream).

Note: Perl 5 Porters does not, to my knowledge, maintain any "official Docker images." We only maintain the core distribution, for which we issue monthly developmental releases, an annual production release (around May 20), and maintenance releases as needed. So you'll have to look elsewhere for Docker images.

tonycoz · 2024-01-22T00:07:04Z

Note: Perl 5 Porters does not, to my knowledge, maintain any "official Docker images." We only maintain the core distribution, for which we issue monthly developmental releases, an annual production release (around May 20), and maintenance releases as needed. So you'll have to look elsewhere for Docker images.

They aren't maintained directly by p5p, but docker images are maintained under this organization:

https://github.com/Perl/docker-perl
https://hub.docker.com/_/perl

khwilliamson · 2024-04-18T17:03:30Z

Is this closable?

zhmylove · 2024-04-18T21:34:58Z

I do not think so: cpan/podlators still contains the issue on blead, ref:

perl5/cpan/podlators/lib/Pod/Text.pm

Line 365 in 67164c3

eval {

It probably should be updated

zhmylove · 2024-04-21T21:21:15Z

@reneeb what do you think?
You was the last podlators updater.

This module should be either updated or patched. The cpan version is 5.01, while the mainline in repo does already contain the fix. Maybe we should ask @rra to release next version and upload it to cpan?

rra · 2024-04-21T21:37:53Z

The next release of podlators will be a major version bump with quite a lot of changes, and we're outside the merge window for dual-life modules.

I think the best strategy for fixing this bug in Perl 5.40, if this is possible (I'm not deeply familiar with the core rules about dual-life Perl modules), would be to cherry-pick rra/podlators@1667094 into core. That's a very small, low-impact change that only fixes this bug, and then the divergence can be dropped after the release freeze when the next major version of podlators is imported.

I could also make a patch release if that's necessary, but I will hopefully release podlators v6.0.0 within the next week or two and have so far avoided needing to juggle patch releases.

xenu · 2024-04-22T20:10:49Z

IMO this should be cherry-picked, it's a trivial one-line change and it fixes a real issue.

However, at this stage I think this needs a PSC decision.

I'm marking this as a blocker.

Cherry-picked from rra/podlators#28 Since it's a cherry-pick, podlators was marked as CUSTOMIZED and its version was bumped. Fixes Perl#21841

xenu · 2024-04-22T20:41:14Z

PR with a cherry-pick: #22165

Cherry-picked from rra/podlators#28 Since it's a cherry-pick, podlators was marked as CUSTOMIZED and its version was bumped. Fixes #21841

bessarabov added the Needs Triage label Jan 15, 2024

jkeenan removed the Needs Triage label Jan 15, 2024

jkeenan mentioned this issue Jan 22, 2024

Output is incorrect when I/O layer is specified by binmode() rra/podlators#25

Closed

xenu added the Release Blocker label Apr 22, 2024

xenu added a commit to xenu/perl5 that referenced this issue Apr 22, 2024

podlators: load PerlIO before trying to use its functions

abd4e66

Cherry-picked from rra/podlators#28 Since it's a cherry-pick, podlators was marked as CUSTOMIZED and its version was bumped. Fixes Perl#21841

xenu mentioned this issue Apr 22, 2024

podlators: load PerlIO before trying to use its functions #22165

Merged

haarg closed this as completed in #22165 Apr 25, 2024

haarg pushed a commit that referenced this issue Apr 25, 2024

podlators: load PerlIO before trying to use its functions

7a2df06

Cherry-picked from rra/podlators#28 Since it's a cherry-pick, podlators was marked as CUSTOMIZED and its version was bumped. Fixes #21841

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Problem with unicode in Getopt::Long HelpMessage() in perl 5.38 #21841

Problem with unicode in Getopt::Long HelpMessage() in perl 5.38 #21841

bessarabov commented Jan 15, 2024 •

edited

Loading

zhmylove commented Jan 15, 2024

xenu commented Jan 15, 2024

xenu commented Jan 15, 2024 •

edited

Loading

bessarabov commented Jan 20, 2024 •

edited

Loading

jkeenan commented Jan 21, 2024

tonycoz commented Jan 22, 2024

khwilliamson commented Apr 18, 2024

zhmylove commented Apr 18, 2024 •

edited

Loading

zhmylove commented Apr 21, 2024

rra commented Apr 21, 2024 •

edited

Loading

xenu commented Apr 22, 2024

xenu commented Apr 22, 2024

Problem with unicode in Getopt::Long HelpMessage() in perl 5.38 #21841

Problem with unicode in Getopt::Long HelpMessage() in perl 5.38 #21841

Comments

bessarabov commented Jan 15, 2024 • edited Loading

zhmylove commented Jan 15, 2024

xenu commented Jan 15, 2024

xenu commented Jan 15, 2024 • edited Loading

bessarabov commented Jan 20, 2024 • edited Loading

jkeenan commented Jan 21, 2024

tonycoz commented Jan 22, 2024

khwilliamson commented Apr 18, 2024

zhmylove commented Apr 18, 2024 • edited Loading

zhmylove commented Apr 21, 2024

rra commented Apr 21, 2024 • edited Loading

xenu commented Apr 22, 2024

xenu commented Apr 22, 2024

bessarabov commented Jan 15, 2024 •

edited

Loading

xenu commented Jan 15, 2024 •

edited

Loading

bessarabov commented Jan 20, 2024 •

edited

Loading

zhmylove commented Apr 18, 2024 •

edited

Loading

rra commented Apr 21, 2024 •

edited

Loading