Encode memory corruption (maybe PerlIO::encoding?) #9966

p5pRT · 2009-11-15T15:38:29Z

Migrated from rt.perl.org#70528 (status was 'resolved')

Searchable as RT70528$

p5pRT · 2009-11-15T15:38:30Z

From @greerga

Created by @greerga

On Ubuntu's stock perl (what perlbug picked up) and blead 5.11.1
(GitLive-blead-3146-g88a6f4f), Encode can be made to corrupt memory:

valgrind (blead):
==31935== Invalid write of size 1
==31935== at 0x689A93F: do_encode (encengine.c:119)
==31935== by 0x68900B2: encode_method (Encode.xs:128)
==31935== by 0x6895938: XS_Encode__XS_encode (Encode.xs:632)
==31935== by 0x54B86B: Perl_pp_entersub (pp_hot.c:2875)
==31935== by 0x4F64DE: Perl_runops_debug (dump.c:2045)
==31935== by 0x449D89: S_run_body (perl.c:2302)
==31935== by 0x449274: perl_run (perl.c:2227)
==31935== by 0x41FA53: main (perlmain.c:117)
==31935== Address 0x644d988 is 0 bytes after a block of size 552 alloc'd
==31935== at 0x4C25153: malloc (vg_replace_malloc.c:195)
==31935== by 0x4F6DA5: Perl_safesysmalloc (util.c:94)
==31935== by 0x55248D: Perl_sv_grow (sv.c:1559)
==31935== by 0x57776E: Perl_newSV (sv.c:4883)
==31935== by 0x688E8BF: encode_method (Encode.xs:105)
==31935== by 0x6895938: XS_Encode__XS_encode (Encode.xs:632)
==31935== by 0x54B86B: Perl_pp_entersub (pp_hot.c:2875)
==31935== by 0x4F64DE: Perl_runops_debug (dump.c:2045)
==31935== by 0x449D89: S_run_body (perl.c:2302)
==31935== by 0x449274: perl_run (perl.c:2227)
==31935== by 0x41FA53: main (perlmain.c:117)

valgrind (ubuntu 5.10.0):
==8012== Conditional jump or move depends on uninitialised value(s)
==8012== at 0x4EAB9EF: Perl_re_compile (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4E8F7C5: Perl_pmruntime (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4E83E15: Perl_yyparse (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4F0762D: ??? (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4F09950: Perl_pp_require (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4ED2F85: Perl_runops_standard (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4ED09EC: Perl_call_sv (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4ED0E8C: Perl_call_list (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4E85DF8: ??? (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4E93BA1: Perl_newATTRSUB (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4E92AD2: Perl_utilize (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4E84CB3: Perl_yyparse (in /usr/lib/libperl.so.5.10.0)
==8012==
==8012== Conditional jump or move depends on uninitialised value(s)
==8012== at 0x4EAB9EF: Perl_re_compile (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4F12DD5: Perl_pp_regcomp (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4ED2F85: Perl_runops_standard (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4ED09EC: Perl_call_sv (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4ED0E8C: Perl_call_list (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4E85DF8: ??? (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4E93BA1: Perl_newATTRSUB (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4E92AD2: Perl_utilize (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4E84CB3: Perl_yyparse (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4F0762D: ??? (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4F09950: Perl_pp_require (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4ED2F85: Perl_runops_standard (in /usr/lib/libperl.so.5.10.0)
==8012==
==8012== Conditional jump or move depends on uninitialised value(s)
==8012== at 0x4EAB9EF: Perl_re_compile (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4E8F7C5: Perl_pmruntime (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4E8FBF0: Perl_ck_split (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4E8FECC: Perl_convert (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4E849AF: Perl_yyparse (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4F0762D: ??? (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4F09950: Perl_pp_require (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4ED2F85: Perl_runops_standard (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4ED09EC: Perl_call_sv (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4ED0E8C: Perl_call_list (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4E85DF8: ??? (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4E93BA1: Perl_newATTRSUB (in /usr/lib/libperl.so.5.10.0)
==8012==
==8012== Conditional jump or move depends on uninitialised value(s)
==8012== at 0x4EAB9EF: Perl_re_compile (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4E8F7C5: Perl_pmruntime (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4E83E15: Perl_yyparse (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4ECF0C8: ??? (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4ED19B2: perl_parse (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x400D0B: main (in /usr/bin/perl)
==8012==
==8012== Conditional jump or move depends on uninitialised value(s)
==8012== at 0x4F3CA41: Perl_bytes_to_utf8 (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4EEF21C: Perl_sv_utf8_upgrade_flags (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4EEF498: Perl_sv_catsv_flags (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x69F0C57: ??? (in /usr/lib/perl/5.10.0/auto/Encode/Encode.so)
==8012== by 0x69F12F9: XS_Encode__XS_encode (in /usr/lib/perl/5.10.0/auto/Encode/Encode.so)
==8012== by 0x4ED4C0E: Perl_pp_entersub (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4ED2F85: Perl_runops_standard (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4ED144B: perl_run (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x400D7B: main (in /usr/bin/perl)
==8012==
==8012== Invalid write of size 1
==8012== at 0x69F3350: do_encode (in /usr/lib/perl/5.10.0/auto/Encode/Encode.so)
==8012== by 0x69F09F3: ??? (in /usr/lib/perl/5.10.0/auto/Encode/Encode.so)
==8012== by 0x69F12F9: XS_Encode__XS_encode (in /usr/lib/perl/5.10.0/auto/Encode/Encode.so)
==8012== by 0x4ED4C0E: Perl_pp_entersub (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4ED2F85: Perl_runops_standard (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4ED144B: perl_run (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x400D7B: main (in /usr/bin/perl)
==8012== Address 0x7ce77b8 is 0 bytes after a block of size 216 alloc'd
==8012== at 0x4C2524D: realloc (vg_replace_malloc.c:476)
==8012== by 0x4EB562E: Perl_safesysrealloc (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4EE9E81: Perl_sv_grow (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4EEB999: Perl_sv_catpvn_flags (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4EEF4AE: Perl_sv_catsv_flags (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x69F0C57: ??? (in /usr/lib/perl/5.10.0/auto/Encode/Encode.so)
==8012== by 0x69F12F9: XS_Encode__XS_encode (in /usr/lib/perl/5.10.0/auto/Encode/Encode.so)
==8012== by 0x4ED4C0E: Perl_pp_entersub (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4ED2F85: Perl_runops_standard (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x4ED144B: perl_run (in /usr/lib/libperl.so.5.10.0)
==8012== by 0x400D7B: main (in /usr/bin/perl)

glibc (ubuntu):
*** glibc detected *** perl: realloc(): invalid next size: 0x00000000029dbee0 ***
======= Backtrace: =========
/lib/libc.so.6[0x7fdafbb5add6]
/lib/libc.so.6[0x7fdafbb60254]
/lib/libc.so.6(realloc+0xf0)[0x7fdafbb605b0]
/usr/lib/libperl.so.5.10(Perl_safesysrealloc+0x3f)[0x7fdafc58362f]
/usr/lib/libperl.so.5.10(Perl_sv_grow+0x72)[0x7fdafc5b7e82]
/usr/lib/perl/5.10/auto/Encode/Encode.so[0x7fdafb29ad64]
/usr/lib/perl/5.10/auto/Encode/Encode.so(XS_Encode__XS_encode+0x14a)[0x7fdafb29b2fa]
/usr/lib/libperl.so.5.10(Perl_pp_entersub+0x53f)[0x7fdafc5a2c0f]
/usr/lib/libperl.so.5.10(Perl_runops_standard+0x16)[0x7fdafc5a0f86]
/usr/lib/libperl.so.5.10(perl_run+0x33c)[0x7fdafc59f44c]
perl(main+0xec)[0x400d7c]
/lib/libc.so.6(__libc_start_main+0xfd)[0x7fdafbb03abd]
perl[0x400bc9]

glibc (blead):
*** glibc detected *** /tmp/perl/bin/perl5.11.1: malloc(): memory corruption: 0x00000000029b34a0 ***
======= Backtrace: =========
/lib/libc.so.6[0x7f77eba29dd6]
/lib/libc.so.6[0x7f77eba2cc0e]
/lib/libc.so.6[0x7f77eba2f0c6]
/lib/libc.so.6(realloc+0xf0)[0x7f77eba2f5b0]
/tmp/perl/bin/perl5.11.1(Perl_safesysrealloc+0x75)[0x4f6ebf]
/tmp/perl/bin/perl5.11.1(Perl_sv_grow+0x243)[0x552479]
/tmp/perl/bin/perl5.11.1(Perl_sv_setsv_flags+0x3e4a)[0x570a1c]
/tmp/perl/bin/perl5.11.1(Perl_pp_sassign+0x11d0)[0x5316f3]
/tmp/perl/bin/perl5.11.1(Perl_runops_debug+0x157)[0x4f64df]
/tmp/perl/bin/perl5.11.1[0x449d8a]
/tmp/perl/bin/perl5.11.1(perl_run+0x100)[0x449275]
/tmp/perl/bin/perl5.11.1(main+0xc0)[0x41fa54]
/lib/libc.so.6(__libc_start_main+0xfd)[0x7f77eb9d2abd]
/tmp/perl/bin/perl5.11.1[0x41f8d9]

I don't have a stock 5.10.1 handy to test, but if blead and 5.10.0 does
it then I expect it would too.

Perl Info


Flags:
    category=library
    severity=high

Site configuration information for perl 5.10.0:

Configured by Debian Project at Thu Oct  1 22:36:47 UTC 2009.

Summary of my perl5 (revision 5 version 10 subversion 0) configuration:
  Platform:
    osname=linux, osvers=2.6.24-23-server, archname=x86_64-linux-gnu-thread-multi
    uname='linux crested 2.6.24-23-server #1 smp wed apr 1 22:14:30 utc 2009 x86_64 gnulinux '
    config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN -Dcccdlflags=-fPIC -Darchname=x86_64-linux-gnu -Dprefix=/usr -Dprivlib=/usr/share/perl/5.10 -Darchlib=/usr/lib/perl/5.10 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.10.0 -Dsitearch=/usr/local/lib/perl/5.10.0 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio -Uusenm -DDEBUGGING=-g -Doptimize=-O2 -Duseshrplib -Dlibperl=libperl.so.5.10.0 -Dd_dosuid -des'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2 -g',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include'
    ccversion='', gccversion='4.4.1', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib /lib64 /usr/lib64
    libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
    perllibs=-ldl -lm -lpthread -lc -lcrypt
    libc=/lib/libc-2.10.1.so, so=so, useshrplib=true, libperl=libperl.so.5.10.0
    gnulibc_version='2.10.1'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -O2 -g -L/usr/local/lib'

Locally applied patches:
    


@INC for perl 5.10.0:
    /etc/perl
    /usr/local/lib/perl/5.10.0
    /usr/local/share/perl/5.10.0
    /usr/lib/perl5
    /usr/share/perl5
    /usr/lib/perl/5.10
    /usr/share/perl/5.10
    /usr/local/lib/site_perl
    .


Environment for perl 5.10.0:
    HOME=/home/tivrusky
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
    PERL_BADLANG (unset)
    SHELL=/bin/bash

p5pRT · 2009-11-16T04:38:35Z

From @greerga

On Sun, 15 Nov 2009, George Greer wrote:

I haven't pared the crashing script down enough to post as a test case but
will do so as soon as I can. I wanted to make sure the bug report was in
before 5.12 escaped. The test is PerlIO::encoding an input file that contains
Latin-1 high characters and then re-encoding them for output, but it is 918
lines at the moment.

Test script:

- - - 8< - - - 8< - - -
use Encode qw[encode];
encode("ISO-8859-1", "\x{b6} \x{b6} \x{b6} \x{b6} \x{b6} \x{b6}
\x{2022}wwwww \x{2022}rrrrr uuu qqqqqqqqq \x{2022}yyyyyyy
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \x{b6} \x{b6} \x{b6} \x{b6} \x{b6}
\x{b6}", sub { "\x{2022}" });
- - - 8< - - - 8< - - -

(That's supposed to be a single line.)

For me, Ubuntu's perl 5.10.0 crashes, blead (GitLive-blead-3146-g88a6f4f)
also crashes, and blead gives this under valgrind:

==17892== Invalid write of size 1
==17892== at 0x608693F: do_encode (encengine.c:119)
==17892== by 0x607C0B2: encode_method (Encode.xs:128)
==17892== by 0x6081938: XS_Encode__XS_encode (Encode.xs:632)
==17892== by 0x54B86B: Perl_pp_entersub (pp_hot.c:2875)
==17892== by 0x4F64DE: Perl_runops_debug (dump.c:2045)
==17892== by 0x449D89: S_run_body (perl.c:2302)
==17892== by 0x449274: perl_run (perl.c:2227)
==17892== by 0x41FA53: main (perlmain.c:117)
==17892== Address 0x5f2fac8 is 0 bytes after a block of size 120 alloc'd
==17892== at 0x4C25153: malloc (vg_replace_malloc.c:195)
==17892== by 0x4F6DA5: Perl_safesysmalloc (util.c:94)
==17892== by 0x55248D: Perl_sv_grow (sv.c:1559)
==17892== by 0x57776E: Perl_newSV (sv.c:4883)
==17892== by 0x607A8BF: encode_method (Encode.xs:105)
==17892== by 0x6081938: XS_Encode__XS_encode (Encode.xs:632)
==17892== by 0x54B86B: Perl_pp_entersub (pp_hot.c:2875)
==17892== by 0x4F64DE: Perl_runops_debug (dump.c:2045)
==17892== by 0x449D89: S_run_body (perl.c:2302)
==17892== by 0x449274: perl_run (perl.c:2227)
==17892== by 0x41FA53: main (perlmain.c:117)

--
George Greer

p5pRT · 2009-11-16T05:33:31Z

From [email protected]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

George,

Thank you for your report.

On 16 Nov 2009, at 13:38, George Greer wrote:

- - - 8< - - - 8< - - -
use Encode qw[encode];
encode("ISO-8859-1", "\x{b6} \x{b6} \x{b6} \x{b6} \x{b6} \x{b6} \x{2022}wwwww \x{2022}rrrrr uuu qqqqqqqqq \x{2022}yyyyyyy xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \x{b6} \x{b6} \x{b6} \x{b6} \x{b6} \x{b6}", sub { "\x{2022}" });

It tries to return DURING string during encoding so the usage is wrong to begin with.
That being said, I successfully reproduced your case with the one-liner below.

perl -MEncode -le 'print encode "ascii", " a\x{b6}\x{2022}a"x8, sub{ "\x{2022}" }'

I also found this does not happen in Perl 5.8.9. So this has something to do with how Perl 5.10 allocates memory.

At any rate, ext/Encode/Encode.xs must be the file to look at.

Dan the Maintainer Thereof
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (Darwin)

iEYEARECAAYFAksA5BYACgkQErJia/WXtBvMiwCdEJ6PbaD8XgC0vXCtL903wu3q
qMUAn1DuSBbgwol6qE5hHyYOxYd6jEGo
=4xqQ
-----END PGP SIGNATURE-----

p5pRT · 2009-11-16T05:33:33Z

The RT System itself - Status changed from 'new' to 'open'

p5pRT · 2009-11-16T05:46:31Z

From @greerga

On Mon, 16 Nov 2009, Dan Kogai wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

George,

Thank you for your report.

And thank you for being so quick to respond.

On 16 Nov 2009, at 13:38, George Greer wrote:

- - - 8< - - - 8< - - -
use Encode qw[encode];
encode("ISO-8859-1", "\x{b6} \x{b6} \x{b6} \x{b6} \x{b6} \x{b6} \x{2022}wwwww \x{2022}rrrrr uuu qqqqqqqqq \x{2022}yyyyyyy xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \x{b6} \x{b6} \x{b6} \x{b6} \x{b6} \x{b6}", sub { "\x{2022}" });

It tries to return DURING string during encoding so the usage is wrong to begin with.
That being said, I successfully reproduced your case with the one-liner below.

perl -MEncode -le 'print encode "ascii", " a\x{b6}\x{2022}a"x8, sub{ "\x{2022}" }'

I also found this does not happen in Perl 5.8.9. So this has something to do with how Perl 5.10 allocates memory.

At any rate, ext/Encode/Encode.xs must be the file to look at.

It might not crash under perl 5.8.9, but making it crash is finicky
anyway since the script doesn't exercise memory much afterward. Valgrind
says 5.8.9 still causes the errant write:

==30569== Command: /home/perl/work/cpan_maint-5.8/perl/bin/perl5.8.9
-MEncode -le print\ encode\ "ascii",\ "\ a\\x{b6}\\x{2022}a"x8,\
sub{"\\x{2022}"}
==30569==
==30569== Invalid write of size 1
==30569== at 0x629D113: do_encode (encengine.c:119)
==30569== by 0x62970B3: encode_method (Encode.xs:128)
==30569== by 0x629920D: XS_Encode__XS_encode (Encode.xs:621)
==30569== by 0x479D0F: Perl_pp_entersub (pp_hot.c:2862)
==30569== by 0x444896: Perl_runops_debug (dump.c:1639)
==30569== by 0x465582: S_run_body (perl.c:2453)
==30569== by 0x464E77: perl_run (perl.c:2368)
==30569== by 0x421CA8: main (perlmain.c:109)
==30569== Address 0x61bb4c0 is 0 bytes after a block of size 48 alloc'd
==30569== at 0x4C2524D: realloc (vg_replace_malloc.c:476)
==30569== by 0x4451F1: Perl_safesysrealloc (util.c:177)
==30569== by 0x47D440: Perl_sv_grow (sv.c:1440)
==30569== by 0x48445A: Perl_sv_catpvn_flags (sv.c:3915)
==30569== by 0x484752: Perl_sv_catsv_flags (sv.c:3975)
==30569== by 0x6296D7A: encode_method (Encode.xs:204)
==30569== by 0x629920D: XS_Encode__XS_encode (Encode.xs:621)
==30569== by 0x479D0F: Perl_pp_entersub (pp_hot.c:2862)
==30569== by 0x444896: Perl_runops_debug (dump.c:1639)
==30569== by 0x465582: S_run_body (perl.c:2453)
==30569== by 0x464E77: perl_run (perl.c:2368)
==30569== by 0x421CA8: main (perlmain.c:109)

perl 5.10.0 crashed a lot more than blead 5.11.1 during my test case
reduction, but the valgrind still showed the write being there even when
blead didn't crash.

- - - 8< - - - 8< - - -
Summary of my perl5 (revision 5 version 8 subversion 9 patch 35104) configuration:
Platform:
osname=linux, osvers=2.6.28-15-generic, archname=x86_64-linux-thread-multi
uname='linux zwei 2.6.28-15-generic #49-ubuntu smp tue aug 18 19:25:34 utc 2009 x86_64 gnulinux '
config_args='-Dusedevel -Dusethreads -Dinstallbin -Duse64bitall -Dprefix=/home/perl/work/cpan_maint-5.8/perl -Doptimize=-g -des'
- - - 8< - - - 8< - - -

--
George Greer

p5pRT · 2009-11-16T08:24:17Z

From [email protected]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

George,

I think I have fixed it now.

On 16 Nov 2009, at 14:46, George Greer wrote:

It might not crash under perl 5.8.9, but making it crash is finicky anyway since the script doesn't exercise memory much afterward. Valgrind says 5.8.9 still causes the errant write:

==30569== Command: /home/perl/work/cpan_maint-5.8/perl/bin/perl5.8.9 -MEncode -le print\ encode\ "ascii",\ "\ a\\x{b6}\\x{2022}a"x8,\ sub{"\\x{2022}"}
==30569==
==30569== Invalid write of size 1
==30569== at 0x629D113: do_encode (encengine.c:119)
==30569== by 0x62970B3: encode_method (Encode.xs:128)
==30569== by 0x629920D: XS_Encode__XS_encode (Encode.xs:621)
==30569== by 0x479D0F: Perl_pp_entersub (pp_hot.c:2862)
==30569== by 0x444896: Perl_runops_debug (dump.c:1639)
==30569== by 0x465582: S_run_body (perl.c:2453)
==30569== by 0x464E77: perl_run (perl.c:2368)
==30569== by 0x421CA8: main (perlmain.c:109)
==30569== Address 0x61bb4c0 is 0 bytes after a block of size 48 alloc'd
==30569== at 0x4C2524D: realloc (vg_replace_malloc.c:476)
==30569== by 0x4451F1: Perl_safesysrealloc (util.c:177)
==30569== by 0x47D440: Perl_sv_grow (sv.c:1440)
==30569== by 0x48445A: Perl_sv_catpvn_flags (sv.c:3915)
==30569== by 0x484752: Perl_sv_catsv_flags (sv.c:3975)
==30569== by 0x6296D7A: encode_method (Encode.xs:204)
==30569== by 0x629920D: XS_Encode__XS_encode (Encode.xs:621)
==30569== by 0x479D0F: Perl_pp_entersub (pp_hot.c:2862)
==30569== by 0x444896: Perl_runops_debug (dump.c:1639)
==30569== by 0x465582: S_run_body (perl.c:2453)
==30569== by 0x464E77: perl_run (perl.c:2368)
==30569== by 0x421CA8: main (perlmain.c:109)

perl 5.10.0 crashed a lot more than blead 5.11.1 during my test case reduction, but the valgrind still showed the write being there even when blead didn't crash.

Would you try the patch below? That fixed the problem on my OS X.

====
% perl -MEncode -le 'print encode "ascii", " a\x{b6}\x{2022}a"x8, sub{ "\x{2022}" }'
Segmentation fault
% perl -Mblib -MEncode -le 'print encode "ascii", " a\x{b6}\x{2022}a"x8, sub{ "\x{2022}" }'
a••a a••a a••a a••a a••a a••a a••a a••a

The patch applies SvUTF8_off when encoding. I also did a little optimization but that does not matter on fixing the problem.

I will VERSION++ after your report. Thank you in advance for testing.

Dan the Maintainer THereof.

===================================================================
RCS file: Encode.xs,v
retrieving revision 2.16
diff -u -r2.16 Encode.xs
- --- Encode.xs 2009/09/06 14:32:21 2.16
+++ Encode.xs 2009/11/16 08:17:11
@@ -68,7 +68,7 @@
{
dSP;
int argc;
- - SV *temp, *retval;
+ SV *retval = newSVpv("",0);
ENTER;
SAVETMPS;
PUSHMARK(sp);
@@ -79,13 +79,10 @@
if (argc != 1){
croak("fallback sub must return scalar!");
}
- - temp = newSVsv(POPs);
+ sv_catsv(retval, POPs);
PUTBACK;
FREETMPS;
LEAVE;
- - retval = newSVpv("",0);
- - sv_catsv(retval, temp);
- - SvREFCNT_dec(temp);
return retval;
}

@@ -199,6 +196,7 @@
: newSVpvf(check & ENCODE_PERLQQ ? "\\x{%04"UVxf"}" :
check & ENCODE_HTMLCREF ? "&#%" UVuf ";" :
"&#x%" UVxf ";", (UV)ch);
+ SvUTF8_off(subchar); /* make sure no decoded string gets in */
sdone += slen + clen;
ddone += dlen + SvCUR(subchar);
sv_catsv(dst, subchar);

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (Darwin)

iEYEARECAAYFAksBDBsACgkQErJia/WXtBuXBgCdEvbSBofXhu+DlP6qm6mo6ZJW
HUwAnjIAj+daYPByCbCd0ST28PDoSpkA
=84SB
-----END PGP SIGNATURE-----

p5pRT · 2009-11-17T00:12:50Z

From @greerga

On Mon, 16 Nov 2009, Dan Kogai wrote:

George,

I think I have fixed it now.

Yes, Valgrind confirms.

I will VERSION++ after your report. Thank you in advance for testing.

Thanks! Obviously trying to give UTF-8 as a fallback character was
unintentional, but it turned out more interesting than I wanted.

--
George Greer

p5pRT · 2009-11-17T14:15:45Z

From @gannett-ggreer

Fixed in Encode 2.38.

p5pRT · 2010-03-14T22:26:35Z

@iabyn - Status changed from 'open' to 'resolved'

p5pRT closed this as completed Mar 14, 2010

p5pRT added Severity High distro-Linux type-Unicode type-library labels Oct 18, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Encode memory corruption (maybe PerlIO::encoding?) #9966

Encode memory corruption (maybe PerlIO::encoding?) #9966

p5pRT commented Nov 15, 2009

p5pRT commented Nov 15, 2009

p5pRT commented Nov 16, 2009

p5pRT commented Nov 16, 2009

p5pRT commented Nov 16, 2009

p5pRT commented Nov 16, 2009

p5pRT commented Nov 16, 2009

p5pRT commented Nov 17, 2009

p5pRT commented Nov 17, 2009

p5pRT commented Mar 14, 2010

Encode memory corruption (maybe PerlIO::encoding?) #9966

Encode memory corruption (maybe PerlIO::encoding?) #9966

Comments

p5pRT commented Nov 15, 2009

p5pRT commented Nov 15, 2009

From @greerga

Created by @greerga

p5pRT commented Nov 16, 2009

From @greerga

p5pRT commented Nov 16, 2009

From [email protected]

p5pRT commented Nov 16, 2009

p5pRT commented Nov 16, 2009

From @greerga

p5pRT commented Nov 16, 2009

From [email protected]

==== % perl -MEncode -le 'print encode "ascii", " a\x{b6}\x{2022}a"x8, sub{ "\x{2022}" }' Segmentation fault % perl -Mblib -MEncode -le 'print encode "ascii", " a\x{b6}\x{2022}a"x8, sub{ "\x{2022}" }' a••a a••a a••a a••a a••a a••a a••a a••a

p5pRT commented Nov 17, 2009

From @greerga

p5pRT commented Nov 17, 2009

From @gannett-ggreer

p5pRT commented Mar 14, 2010

====
% perl -MEncode -le 'print encode "ascii", " a\x{b6}\x{2022}a"x8, sub{ "\x{2022}" }'
Segmentation fault
% perl -Mblib -MEncode -le 'print encode "ascii", " a\x{b6}\x{2022}a"x8, sub{ "\x{2022}" }'
a••a a••a a••a a••a a••a a••a a••a a••a