piconv bug of decoding UTF-16 (with fix)

[numa2666]

Hi,

I've found a bug in piconv 2.5 included in perl 5.18.2 distribution.
The problem is, it can not handle UTF-16 correctly, especially
when UTF-16BE or UTF-16LE is specified. It is because that
piconv does not slurp the input (necessary when UTF-16/UTF-32
are read), and try to read line by line, which will lead to failed
conversion. They must be treated the same as UTF-16 and
UTF-32 (non-BE/LE versions). Ah, well, piconv also fails to
determine whether slurp is needed or not, by using $to
encoding instead of $from encoding. I've included a patch
below.

The following patch also includes some minor fixes.

• piconv fails to detect the locale-specific encoding if LANG
  or LC_* environment variables are not set, or even they are set,
  if they don't have any encoding field. It is quite common situation,
  and using Encode::Locale's encoding value instead of environment
  variable is better and portable to (especially) non-Unix systems.
• The help message is fixed to display backslash-escaped string
  correctly.

Hope this helps,

Toshinori Numata
http://blog.livedoor.jp/numa2666/ (in Japanese)

--- c:/strawberry/perl/bin/piconv   Tue Jan  7 19:19:14 2014
+++ piconv  Sun Mar 23 19:30:29 2014
@@ -5,6 +5,7 @@
 use strict;
 use Encode ;
 use Encode::Alias;
+use Encode::Locale;
 my %Scheme =  map {$_ => 1} qw(from_to decode_encode perlio);

 use File::Basename;
@@ -34,7 +35,7 @@

 $Opt{help} and help();
 $Opt{list} and list_encodings();
-my $locale = $ENV{LC_CTYPE} || $ENV{LC_ALL} || $ENV{LANG};
+my $locale = $Encode::Locale::ENCODING_LOCALE;
 defined $Opt{resolve} and resolve_encoding($Opt{resolve});
 $Opt{from} || $Opt{to} || help();
 my $from = $Opt{from} || $locale or help("from_encoding unspecified");
@@ -68,14 +69,14 @@
 EOT
 }

-my %use_bom = map { $_ => 1 } qw/UTF-16 UTF-32/;
+my %use_bom = map { $_ => 1 } qw/UTF-16 UTF-16BE UTF-16LE UTF-32 UTF-32BE UTF-32LE/;

 # we do not use <> (or ARGV) for the sake of binmode()
 @ARGV or push @ARGV, \*STDIN;

 unless ( $scheme eq 'perlio' ) {
     binmode STDOUT;
-    my $need2slurp = $use_bom{ find_encoding($to)->name };
+    my $need2slurp = $use_bom{ find_encoding($from)->name };
     for my $argv (@ARGV) {
         my $ifh = ref $argv ? $argv : undef;
    $ifh or open $ifh, "<", $argv or warn "Can't open $argv: $!" and next;
@@ -169,7 +170,7 @@
   -D,--debug          show debug information
   -S,--scheme scheme  use the scheme for conversion
 Those are handy when you can only see ASCII characters:
-  -p,--perlqq         transliterate characters missing in encoding to \x{HHHH}
+  -p,--perlqq         transliterate characters missing in encoding to \\x{HHHH}
                       where HHHH is the hexadecimal Unicode code point
   --htmlcref          transliterate characters missing in encoding to &#NNN;
                       where NNN is the decimal Unicode code point

[dankogai]

Thank you for your work. And I am sorry I cannot take this patch. Encode is a core module so I cannot depend on non-core module. Encode::Locale is not in core.

I'll come up with other solution…

Dan the Maintainer Thereof

[dankogai]

Encode 2.58 released -- with your fixes except for the Encode::Locale dependency.

Dan the Maintainer Thereof