[apk] "BAD archive" error when installing .apk built using fpm #1227

Daniel15 · 2016-11-14T00:30:30Z

alpinevm:~# apk add --allow-untrusted yarn_0.17.0_noarch.apk
ERROR: yarn_0.17.0-0_noarch.apk: BAD archive

This is the command I'm running to build the package - I removed a bunch of other parameters I was pasing (vendor, url, license, maintainer, etc.) and I'm still seeing the same error:

fpm --input-type dir --chdir tmp/debian_pkg --name yarn --version 0.17.0 --output-type apk --architecture noarch .

cc @Knetic

The text was updated successfully, but these errors were encountered:

peterpme · 2016-12-14T20:19:26Z

👋 any way I can help move this?

Knetic · 2016-12-14T20:35:08Z

Woops, I lost track of this haven't looked into it yet. I'll give it a look this week. What version of Alpine is it occurring on, and Is there a set of files publicly available that can repro this behavior?

Daniel15 · 2016-12-15T01:32:45Z

@Knetic I was using Alpine 3.4.6

Is there a set of files publicly available that can repro this behavior?

I was trying to package Yarn after applying this patch to the repo: Daniel15/yarn@932e90e. To build Yarn, clone its repo then run npm install && ./scripts/build-dist.sh && ./scripts/build-deb.sh

I think I encountered the same issue with a small directory with a single file in it though, so I don't think it's specific to the files being packaged.

wclr · 2017-01-08T12:31:39Z

Still not resolved?

josegonzalez · 2017-01-08T12:40:01Z

@whitecolor apparently not. Do you have something to contribute here to helping us solve the issue?

wclr · 2017-01-08T13:10:19Z

@josegonzalez no, sorry, just wanted to remind.

Knetic · 2017-01-08T18:13:02Z

I sat down this morning to look at it, seems like I can't repro the problem when making .apk from a directory tree, but trying to convert the yarn *.deb to *.apk definitely doesn't work. Doesn't seem to be tied to Alpine version either, 3.1 - 3.5 all seem to act the same way (which is good, means they didn't change their packaging format). Still looking.

Daniel15 · 2017-01-08T22:32:57Z

That's strange, I'm seeing it when trying to create a package from a directory tree. I'll see if I can make a standalone repro. Sent from my phone.

…

On Jan 9, 2017 5:13 AM, "George Lester" ***@***.***> wrote: I sat down this morning to look at it, seems like I can't repro the problem when making .apk from a directory tree, but trying to convert the yarn *.deb to *.apk definitely doesn't work. Doesn't seem to be tied to Alpine version either, 3.1 - 3.5 all seem to act the same way (which is good, means they didn't change their packaging format). Still looking. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1227 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAFnHWOASxNMPVNsRQIJVSn1e8XCOn0uks5rQScygaJpZM4Kw253> .

jordansissel · 2017-01-09T00:30:29Z

Testing from master a simple example works for me:

% ./bin/fpm -s dir -t apk -n example ~/.zshrc
Created package {:path=>"example_1.0_noarch.apk"}
% docker run -it -v $PWD:/fpm alpine /bin/sh

/fpm # apk add *apk
WARNING: Ignoring APKINDEX.5a59b88b.tar.gz: No such file or directory
WARNING: Ignoring APKINDEX.7c1f02d6.tar.gz: No such file or directory
(1/1) Installing example (1.0)
ERROR: Metadata for package example-1.0 is too long.
OK: 5 MiB in 12 packages

This was tested with fpm master and whatever version of Alpine is available with the default docker alpine image as of today.

jordansissel · 2017-01-09T00:32:13Z

Trying the original report also works correctly (it seems?) for me.

% touch tmp/debian_pkg/foo
% fpm --version
1.8.0

% fpm --input-type dir --chdir tmp/debian_pkg --name yarn --version 0.17.0 --output-type apk --architecture noarch .
Created package {:path=>"yarn_0.17.0_noarch.apk"}

% docker run -it -v $PWD:/fpm alpine /bin/sh

/ # apk add /fpm/yarn*apk
WARNING: Ignoring APKINDEX.5a59b88b.tar.gz: No such file or directory
WARNING: Ignoring APKINDEX.7c1f02d6.tar.gz: No such file or directory
(1/1) Installing yarn (0.17.0)
ERROR: Metadata for package yarn-0.17.0 is too long.
OK: 5 MiB in 12 packages

Knetic · 2017-01-09T00:49:10Z

I can repro the second report (after doing the whole git clone yarn and build-deb suggested above)

fpm -n yarn -s deb -t apk yarn_0.20.0-0_all.deb

apk output does not currently sign packages. {:level=>:warn}
It's recommended that your package be installed with '--allow-untrusted' {:level=>:warn}
Created package {:path=>"yarn_0.20.0-0-1_all.apk"}
~/projects/fpm/repl
Sending build context to Docker daemon 5.436 MB
Step 1 : FROM alpine:3.5
 ---> 88e169ea8f46
Step 2 : ADD yarn_0.20.0-0-1_all.apk /
 ---> 957b476b263b
Removing intermediate container f4b01f6da536
Step 3 : RUN apk add --allow-untrusted /yarn_0.20.0-0-1_all.apk
 ---> Running in d2f2d8fc7950
WARNING: Ignoring APKINDEX.c51f8f92.tar.gz: No such file or directory
WARNING: Ignoring APKINDEX.d09172fd.tar.gz: No such file or directory
ERROR: /yarn_0.20.0-0-1_all.apk: BAD archive
The command '/bin/sh -c apk add --allow-untrusted /yarn_0.20.0-0-1_all.apk' returned a non-zero code: 255

And it seems to give me the same thing if i explode the .deb into a temp folder and try to fpm that directory. But I can't repro with any other files - the apk support seems to work fine with any other repo/binaries i run through it.

There's quite a lot of files in the yarn deb (looks like it ships all node_modules required to run it), I suspect one of them must have something that the apk support isn't expecting - not sure what yet though. Trying to narrow down which files are causing it.

Daniel15 · 2017-01-09T00:59:44Z

Here's some (hopefully complete) repro steps with the actual use case:

git clone git://github.com/Daniel15/yarn.git yarn-apk-test
cd yarn-apk-test
git checkout alpinepk
./scripts/bootstrap-env-ubuntu.sh # or just look at the script and ensure all prereqs are installed
mkdir ~/.cache/yarn/

npm install
npm run build-dist
./scripts/build-deb.sh

sudo docker run -it -v $PWD:/fpm alpine /bin/sh
apk add /fpm/artifacts/yarn_0.17.0-0_noarch.apk

Result:

/ # apk add /fpm/artifacts/yarn_0.17.0-0_noarch.apk
WARNING: Ignoring APKINDEX.c51f8f92.tar.gz: No such file or directory
WARNING: Ignoring APKINDEX.d09172fd.tar.gz: No such file or directory
ERROR: /fpm/artifacts/yarn_0.17.0-0_noarch.apk: BAD archive

This was with fpm 1.8.0

Here's a copy of the resulting APK file if you want to take a look: https://nightly.yarnpkg.com/temp/yarn_0.17.0-0_noarch.apk

josegonzalez · 2017-01-09T01:00:24Z

What version of npm and node are you using on the host, and what host are you using?

Daniel15 · 2017-01-09T01:05:32Z

This was on Debian Testing (Stretch) with Node.js 6.9.2 and npm v3.10.9, but I've seen the same thing on my laptop where I'm using Windows Subsystem for Linux (aka "bash on Windows") which currently uses Ubuntu 14.04.

samjarrett · 2017-01-09T01:06:50Z

I've been able to reproduce the original report inside a docker-alpine container for nodejs itself (trying to apk-ize nodejs 7.4.0) using the nodejs example instructions straight out of the manual. nodejs also ships with a significant node_modules dir.

Could it be the large amount of files carried in node_modules directories that's doing it?

Daniel15 · 2017-01-09T01:20:05Z

@samjarrett Yeah I'm guessing it's something to do with node_modules. I tried with a single npm package in node_modules (lodash) and it worked fine so maybe it's the fact that there's a large number of files, or perhaps there's a particular Node.js module that it doesn't like? I don't have time to do it at the moment, but it might be worth trimming down node_modules directories one by one until the package works to see if there's one particular package that causes it to break.

Knetic · 2017-01-09T01:21:52Z

@samjarrett That was a good idea, but it seems alright to me. I generated 5k random files in a directory with this method, packaged with fpm -n rando -s dir -t apk ./rando, and did RUN apk add --allow-untrusted /rando_1.0_noarch.apk in a Dockerfile descending from Alpine - works fine.

Feels like there's yeah, there must be some specific module with something that's choking the packaging. Haven't found anything yet by progressively deleting stuff from the yarn example though.

Knetic · 2017-01-09T01:31:21Z

Yeah it's definitely something in node_modules. Works if you delete only that from the exploded deb.

glittershark · 2017-01-09T15:55:25Z

Just a guess - could it be the symlinks in node_modules/.bin?

Daniel15 · 2017-01-10T00:19:59Z

I just tried that (a package just containing node_modules/.bin/bar symlinked to node_modules/foo) and it worked fine :/ I guess it's something else.

Knetic · 2017-01-10T18:19:07Z

Yeah, even removing all symlinks with find . -type l -exec rm {} \; doesn't do it. It also doesn't seem to be the number of files either, i generated 4k randomly-sized files and tried to package that, which took awhile but worked normally.

I made a couple scripts to step through and remove one module at a time from the exploded yarn deb until it successfully packaged. Alphabetically, it's cool until it reaches babel-runtime, at which point it chokes. But removing just babel-runtime doesn't help (some other module still makes it fail). Babel vendors core, but removing that too didn't help. Babel doesn't seem to contain anything but .js files (i had thought perhaps they had a weird tar or something in there).

On the code side I can't see any obvious places where the contents of files or number or depth of them affect anything. And the package uploaded above (when decompressed) looks legit.

It's bizarre, but there is some combination of modules that work, and some that don't. I'll do a longer run later and see how many of yarn's depedencies can be successfully packaged.

client9 · 2017-07-11T16:17:30Z

not just node, but also making apk from python setup.py

${FPM} \
        --python-bin="$PBIN" \
        --python-package-name-prefix="$PPREFIX" \
         -s python \
        --output-type apk \
        ./setup.py

Host: MacOSX
FPM: running CentOS7 in docker

In this case, the archive has maybe 6 files in it.

looking at APK's source code:

https://github.com/alpinelinux/apk-tools/blob/677d3240a845833dfb5c76491c0fd05efb934b2b/src/archive.c#L349

Unfortunately a number of cases in parsing the format can trigger the error, mostly along the lines of reading a stream and not getting the right number of bytes.

I have another project that always makes APK just fine, so not sure what to make of this.

subnetmarco · 2017-07-24T22:24:54Z

I am experiencing the same problem with the following Kong apk package: https://cl.ly/380j2J1q3q2e/download/kong-next.apk

The package has been generated from a dir source:

$ fpm -f -s dir -t 'apk' -n 'kong' source_folder/

Actual error:

# apk add --allow-untrusted kong-next.apk
ERROR: kong-next.apk: BAD archive

sigil66 · 2017-09-05T17:07:14Z

Running into the same issue when attempting to package the zulu jdk. The modified tar format makes this difficult to debug.

fzgregor · 2017-09-21T12:58:03Z

For me, the error is raised in this loop: https://github.com/alpinelinux/apk-tools/blob/677d3240a845833dfb5c76491c0fd05efb934b2b/src/archive.c#L344.
apk_istream_read successfully returns 512 multiple times and the if doesn't terminate the loop, but then buf.name[0] is not empty and the loop is terminated with r = 512.

So, there seems to be unexpected data at the end of the archive.

nxadm · 2017-10-13T08:07:05Z

I'm also hitting this bug. Through Docker, I can provide a setup where the bug is always triggered.

$ git clone https://github.com/nxadm/rakudo-pkg.git
$ cd rakudo-pkg
$ git checkout alpine
$ sudo bin/pkg_rakudo.pl --arch amd64 --os alpine --os-version 3.6 --moar 2017.09.1 --nqp 2017.09 --rakudo 2017.00 --pkg-rev 01 --dir /var/tmp --id nxadm

If you know you way around Docker, the image is also on the Docker Hub:
nxadm/pkgrakudo-alpine-amd64:3.6

This command will successfully build Rakudo Perl 6 on alpine and then it will package it. It will fail when it tries to install the package as a test:

Created package {:path=>"/staging/build_20171013_074925/perl6-rakudo-moarvm-alpine3.6.2_20170900-01_noarch.apk", :file=>"clamp/command.rb", :line=>"68", :method=>"run"}
Cleaning up staging path {:path=>"/tmp/package-dir-staging-f5baad573ed5a5cec1135f7a87a9de3bca851f8bfa1585c56e3b9f19492c", :level=>:debug, :file=>"fpm/package.rb", :line=>"280", :method=>"cleanup_staging"}
Cleaning up build path {:path=>"/tmp/package-dir-build-05d24faa4cea31dc6007595fadc125f638f139b3d6b55b41fbacb4042e21", :level=>:debug, :file=>"fpm/package.rb", :line=>"287", :method=>"cleanup_build"}
Cleaning up build path {:path=>"/tmp/package-apk-build-4729fa1094f3898e7534434dfe93cba784a5f70aa744c8f570ecd0715dea", :level=>:debug, :file=>"fpm/package.rb", :line=>"287", :method=>"cleanup_build"}

# Test it by installing it
$INSTALL $PKGDIR/perl6-rakudo-moarvm*.$TARGET
+ apk add --allow-untrusted /staging/build_20171013_074925/perl6-rakudo-moarvm-alpine3.6.2_20170900-01_noarch.apk
ERROR: /staging/build_20171013_074925/perl6-rakudo-moarvm-alpine3.6.2_20170900-01_noarch.apk: BAD archive

sarnautov · 2017-11-09T13:13:08Z

In my case commenting out add_paxstring fixed the package that previously failed with "BAD archive" during installation. Seems that adding PaxHeader keyword breaks records with long paths.

nxadm · 2017-11-24T11:10:53Z

I can confirm that @sarnautov's solution works. I just commented out that in fpm's source and installable packages are created without a problem. I don't know what the implication is to remove the add_paxstring call, but my packages work fine without it.

jordansissel · 2017-11-24T17:34:52Z

Thanks for all the testing and research! We should be able to get this fix into fpm.

…

On Fri, Nov 24, 2017 at 3:10 AM nxadm ***@***.***> wrote: I can confirm that @sarnautov <https://github.com/sarnautov>'s solution works. I just commented out that in fpm's source and installable packages are created without a problem. I don't know what the implication is to remove the add_paxstring call, but my packages <https://github.com/nxadm/rakudo-pkg/releases> work fine without it. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1227 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAIC6us18-lA0Sykls7nbCO3DJOvqHLdks5s5qQ-gaJpZM4Kw253> .

subnetmarco · 2018-01-08T22:13:48Z

@jordansissel any update on this issue?

jordansissel · 2018-01-08T23:37:13Z

@thefosk not at this time. I haven't much personal time to work on this, and I dont' think any patch is available for testing?

I can try to make time this week, but I am not sure if I will have time to do so :|

Between the reproduction (#1227 (comment)) and the suggested fix (#1227 (comment)) in this issue, maybe I can make a PR with this and hope it it doesn't break anything?

nxadm · 2018-04-30T18:34:57Z

Sadly, the solution seem only to work partially:

/ # apk add --no-cache rakudo-pkg-Alpine3.6_2018.03-01_x86_64.apk 
fetch http://dl-cdn.alpinelinux.org/alpine/v3.6/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.6/community/x86_64/APKINDEX.tar.gz
(1/1) Installing rakudo-pkg (2018.03-01)
ERROR: Metadata for package rakudo-pkg-2018.03-01 is too long.
OK: 234 MiB in 51 packages
/ # /opt/rakudo-pkg/bin/perl6 -v
This is Rakudo version 2018.03 built on MoarVM version 2018.03
implementing Perl 6.c.
/ # apk del --no-cache rakudo-pkg-Alpine3.6_2018.03-01_x86_64.apk 
fetch http://dl-cdn.alpinelinux.org/alpine/v3.6/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.6/community/x86_64/APKINDEX.tar.gz
/ # /opt/rakudo-pkg/bin/perl6 -v
This is Rakudo version 2018.03 built on MoarVM version 2018.03
implementing Perl 6.c.

The created package is installed despite the warning and the binaries work fine. However, I am not able to delete the package.

nxadm · 2018-04-30T18:47:06Z

I was too fast, it works with the correct apk incantation:

/ # apk add --allow-untrusted rakudo-pkg-Alpine3.6_2018.03-01_x86_64.apk
(1/1) Installing rakudo-pkg (2018.03-01)
OK: 234 MiB in 51 packages
/ # apk add --allow-untrusted *.apk^C
/ # /opt/rakudo-pkg/bin/perl6 -v
This is Rakudo version 2018.03 built on MoarVM version 2018.03
implementing Perl 6.c.
/ # apk del rakudo-pkg
(1/1) Purging rakudo-pkg (2018.03-01)
OK: 234 MiB in 50 packages
/ # /opt/rakudo-pkg/bin/perl6 -v
sh: /opt/rakudo-pkg/bin/perl6: not found

mttradebyte · 2018-07-04T17:15:39Z

Just wanted to drop in and say that the solution listed here worked for me too. Different error but pretty sure that's related to my recipe.

subnetmarco · 2018-07-08T00:19:45Z

Since it seems that some progress has been made on this issue, perhaps somebody familiar with the codebase could submit a PR ?

mttradebyte · 2018-07-09T09:28:25Z

The proposed fix above is extremely simple to do however I'm not familiar enough with the APK package format to know if I'm just papering over the issue rather than actually resolving a bug properly.

khous · 2018-08-17T22:49:44Z

+1 to removing add_paxstring
This has allowed me to move past the BAD ARCHIVE error

[apk] "BAD archive" error when installing .apk built using fpm #1227

[apk] "BAD archive" error when installing .apk built using fpm #1227

Comments

Daniel15 commented Nov 14, 2016

peterpme commented Dec 14, 2016

Knetic commented Dec 14, 2016

Daniel15 commented Dec 15, 2016 • edited Loading

wclr commented Jan 8, 2017

josegonzalez commented Jan 8, 2017

wclr commented Jan 8, 2017 • edited Loading

Knetic commented Jan 8, 2017

Daniel15 commented Jan 8, 2017 via email

jordansissel commented Jan 9, 2017

jordansissel commented Jan 9, 2017 • edited Loading

Knetic commented Jan 9, 2017 • edited Loading

Daniel15 commented Jan 9, 2017

josegonzalez commented Jan 9, 2017

Daniel15 commented Jan 9, 2017

samjarrett commented Jan 9, 2017

Daniel15 commented Jan 9, 2017

Knetic commented Jan 9, 2017 • edited Loading

Knetic commented Jan 9, 2017

glittershark commented Jan 9, 2017

Daniel15 commented Jan 10, 2017

Knetic commented Jan 10, 2017 • edited Loading

client9 commented Jul 11, 2017 • edited Loading

subnetmarco commented Jul 24, 2017 • edited Loading

sigil66 commented Sep 5, 2017

fzgregor commented Sep 21, 2017 • edited Loading

nxadm commented Oct 13, 2017

sarnautov commented Nov 9, 2017

nxadm commented Nov 24, 2017

jordansissel commented Nov 24, 2017 via email

subnetmarco commented Jan 8, 2018

jordansissel commented Jan 8, 2018

nxadm commented Apr 30, 2018

nxadm commented Apr 30, 2018

mttradebyte commented Jul 4, 2018 • edited Loading

subnetmarco commented Jul 8, 2018

mttradebyte commented Jul 9, 2018

khous commented Aug 17, 2018

Daniel15 commented Dec 15, 2016 •

edited

Loading

wclr commented Jan 8, 2017 •

edited

Loading

jordansissel commented Jan 9, 2017 •

edited

Loading

Knetic commented Jan 9, 2017 •

edited

Loading

Knetic commented Jan 9, 2017 •

edited

Loading

Knetic commented Jan 10, 2017 •

edited

Loading

client9 commented Jul 11, 2017 •

edited

Loading

subnetmarco commented Jul 24, 2017 •

edited

Loading

fzgregor commented Sep 21, 2017 •

edited

Loading

mttradebyte commented Jul 4, 2018 •

edited

Loading