Phar problems: crc32 mismatch on file

[coffeeneer]

I'm having some issues with deployer connections. When I try to php ./vendor/bin/dep ssh test it fails 50% of the time. This is both on my local machine running Arch as well on my Gitlab server in CI/CD. Php version is 8.0.7 on Arch, 8.0.5 on the alpine based docker image used for CI/CD in Gitlab.

• Deployer version: 7.0.0-beta.23
• Deployment OS: ubuntu (docker container) / archlinux

In testing I have reduced my deploy.yaml to a minimal setup like this:

config:
  application: APPNAME
  user: USER
  host: HOST
  base_deploy_path: "~/domains"
  repository: "REPONAME"

hosts:
  test:
    remote_user: "{{user}}"
    hostname: "{{host}}"
    deploy_path: "{{base_deploy_path}}/APPLICATION_URL"

This is what I get when the CD fails with this issue:

$ php ./vendor/bin/dep deploy test
In ClassLoader.php line 444:
                                                                               
  include(phar:///builds/APPLICATION_NAME/vendor/deployer/dist/dep/vendor/com  
  poser/../../src/Selector/Selector.php): Failed to open stream: phar error:   
  internal corruption of phar "/builds/APPLICATION_NAME/vendor/deployer/dist/  
  dep" (crc32 mismatch on file "src/Selector/Selector.php")                    
                                                                               
deploy [--tag TAG] [--revision REVISION] [--branch BRANCH] [-o|--option OPTION] [-l|--limit LIMIT] [--no-hooks] [--plan] [--start-from START-FROM] [--log LOG] [--profile PROFILE] [--] [<selector>...]

The application server is running OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017. I could try to have that updated to a newer build as it seems a bit old to me.

[bayer]

I can confirm this behavior on php 7.4, but with a success rate way less than 50%

[Schrank]

Thanks for your report. I never had that problem. Can you please add -vvv to the command and post the output (maybe you need to remove some confidential data), thanks!

[bayer]

I just realized the error message is slightly different, but it is very similar to coffeeneer's error message:

$ dep -V
Deployer 7.0.0-beta.23
$ dep ssh -vvv

In ClassLoader.php line 444:

  [ErrorException]
  include(phar:///usr/local/bin/dep/vendor/composer/../../src/Component/Ssh/Client.php): failed to open stream: phar error: internal corruption of phar "/usr/local/b
  in/dep" (crc32 mismatch on file "src/Component/Ssh/Client.php")


Exception trace:
  at phar:///usr/local/bin/dep/vendor/composer/ClassLoader.php:444
 {closure}() at phar:///usr/local/bin/dep/vendor/composer/ClassLoader.php:444
 include() at phar:///usr/local/bin/dep/vendor/composer/ClassLoader.php:444
 Composer\Autoload\includeFile() at phar:///usr/local/bin/dep/vendor/composer/ClassLoader.php:322
 Composer\Autoload\ClassLoader->loadClass() at n/a:n/a
 spl_autoload_call() at phar:///usr/local/bin/dep/src/Command/SshCommand.php:86
 Deployer\Command\SshCommand->execute() at phar:///usr/local/bin/dep/vendor/symfony/console/Command/Command.php:256
 Symfony\Component\Console\Command\Command->run() at phar:///usr/local/bin/dep/vendor/symfony/console/Application.php:971
 Symfony\Component\Console\Application->doRunCommand() at phar:///usr/local/bin/dep/vendor/symfony/console/Application.php:290
 Symfony\Component\Console\Application->doRun() at phar:///usr/local/bin/dep/vendor/symfony/console/Application.php:166
 Symfony\Component\Console\Application->run() at phar:///usr/local/bin/dep/src/Deployer.php:304
 Deployer\Deployer::run() at phar:///usr/local/bin/dep/bin/dep:104
 require() at /usr/local/bin/dep:4

[coffeeneer]

With verbose output we get the same thing:

➜ php --version                     
PHP 8.0.7 (cli) (built: Jun  2 2021 04:04:16) ( NTS )
Copyright (c) The PHP Group
Zend Engine v4.0.7, Copyright (c) Zend Technologies

➜ php ./vendor/bin/dep --version                                                        
Deployer 7.0.0-beta.23

➜ php ./vendor/bin/dep ssh test -vvv

In ClassLoader.php line 444:
                                                                                                                      
  [ErrorException]                                                                                                    
  include(phar://PROJECTPATH/vendor/deployer/dist/dep/vendor/composer/../../src/Componen  
  t/Ssh/Client.php): Failed to open stream: phar error: internal corruption of phar "PROJECTPATH/vendor/deployer/dist/dep" (crc32 mismatch on file "src/Component/Ssh/Client.php")                          
                                                                                                                      

Exception trace:
  at phar://PROJECTPATH/vendor/deployer/dist/dep/vendor/composer/ClassLoader.php:444
 {closure}() at phar://PROJECTPATH/vendor/deployer/dist/dep/vendor/composer/ClassLoader.php:444
 include() at phar://PROJECTPATH/vendor/deployer/dist/dep/vendor/composer/ClassLoader.php:444
 Composer\Autoload\includeFile() at phar://PROJECTPATH/vendor/deployer/dist/dep/vendor/composer/ClassLoader.php:322
 Composer\Autoload\ClassLoader->loadClass() at phar://PROJECTPATH/vendor/deployer/dist/dep/src/Command/SshCommand.php:86
 Deployer\Command\SshCommand->execute() at phar://PROJECTPATH/vendor/deployer/dist/dep/vendor/symfony/console/Command/Command.php:256
 Symfony\Component\Console\Command\Command->run() at phar://PROJECTPATH/vendor/deployer/dist/dep/vendor/symfony/console/Application.php:971
 Symfony\Component\Console\Application->doRunCommand() at phar://PROJECTPATH/vendor/deployer/dist/dep/vendor/symfony/console/Application.php:290
 Symfony\Component\Console\Application->doRun() at phar://PROJECTPATH/vendor/deployer/dist/dep/vendor/symfony/console/Application.php:166
 Symfony\Component\Console\Application->run() at phar://PROJECTPATH/vendor/deployer/dist/dep/src/Deployer.php:304
 Deployer\Deployer::run() at phar:///PROJECTPATH/vendor/deployer/dist/dep/bin/dep:104
 require() at PROJECTPATH/vendor/deployer/dist/dep:4

ssh [<hostname>]

[coffeeneer]

That's interesting, up until now I hadn't noticed the log in my issue had a different error. I've seen the Ssh/Client.php issue more often on my local machine.

[Schrank]

hmm...

-> % tools/deployer.phar ssh -vvv
Select host:
  [0] stage.example.com
  [1] example.com
 > 0
www-data@stage:~$ exit

This is a downloaded phar file, from https://deployer.org/releases/v7.0.0-beta.23/deployer.phar

[bayer]

as @coffeeneer wrote this is an intermittent error - so it works every 4th or 5th call (for me here on wsl2, ubuntu 20.04).
btw. just today the same error message appeared on dep deploy on the machine of a colleague (ubuntu server 20.04) - but it occurred just once.

[Schrank]

I had this some time in the past, but can't remember where and how. If I interpret the error correctly it says "Your damn PHAR file is broken and it doesn't match the hash anymore".

So the questions is why?

• file changed?
• memory corrupted
• cpu calculates wrong hash

Everything I come up with is broken hardware. Happy for other explanations!

[coffeeneer]

Yeah I don't get why this is intermittent. I'd expect the file is either okay or broken. But for me it's happening intermittently on multiple systems.

My personal system is a really new build with pretty high end hardware. Haven't had any other intermittent issues. To me the hardware has proven itself stable. Only thing I could think of for this system is buggy PHP since Arch is usually pretty bleeding edge.

My build server is a bit older, but running really stable with ECC memory. The build is done with Gitlab runner on a VM. Build image is lorisleiva/laravel:latest docker image, which is alpine based. Alpine uses an alternative C/C++ std lib. So both Arch and Alpine could be suspects, but I noticed @bayer has issues with ubuntu on wsl2.

From both machines I do not expect any issues regarding stability of the systems themselves. If I find some time I'll do some tests with yet another system and maybe try a different target server.

I've tried multiple versions of PHP on my personal rig to no avail. Could try with a live usb to rule out all OS issues.

[coffeeneer]

@Schrank A colleague suggested it could be a race condition in phar loading. Where the crc is checked while it is still loading or multiple processes are loading the phar at the same time and corrupting the unpacked files.

An important piece of information missing on this issue: I'm using the deployer/dist package installed with composer.

I've tried with the source build 'deployer/deployer' and lo and behold, everything works fine. So this issue seems to be only with the deployer/dist package. And it seems to be related to composer class loading. I've found something kind of related maybe:
https://stackoverflow.com/questions/29413013/phar-internal-corruption-crc32-mismatch-during-process-fork

[Schrank]

@antonmedv are the phar files on deployer.org from deployer/dist?

[antonmedv]

Other way around. Deployer/dist has files from deployer.org. Phar built on server during release.

[indrek-k]

Running into the same issue in gitlab CI server (image: thecodingmachine/php:7.4-v4-cli-node10):

    - curl -LO https://deployer.org/releases/v7.0.0-beta.24/deployer.phar
    - sudo mv deployer.phar /usr/local/bin/dep
    - sudo chmod +x /usr/local/bin/dep

    - cd $CI_PROJECT_DIR
    - dep deploy production

Result:

include(phar:///usr/local/bin/dep/vendor/composer/../react/http/src/Server.  
  php): failed to open stream: phar error: internal corruption of phar "/usr/  
  local/bin/dep" (crc32 mismatch on file "vendor/react/http/src/Server.php")

Any ideas what to do, workaround etc?

[antonmedv]

This is really strange. Maybe release phar is broken?

[indrek-k]

I can see that someone above has the same issue with release 23 - I was using 24.
I also tried downloading the source and building the phar myself, but the result was exactly the same. About 50% of the times, random internal corruption error. The specific file that's reported as broken is different every time.

I'm not really familiar with the internals of PHP, but based on the above, it seems like some sort of a deeper issue?

[antonmedv]

Yes, it looks like some deeper issue. Can you test some other releases? Like 17-18?

[indrek-k]

It looks like the problem started from release 21.
Edit: wrong

I tested on my local WSL2 environment (ubuntu 20.04 / php7.4) like this:

r20:

$ curl -LO https://deployer.org/releases/v7.0.0-beta.20/deployer.phar
$ php deployer.phar roll:dice
// this worked properly

r21:

$ curl -LO https://deployer.org/releases/v7.0.0-beta.21/deployer.phar
$ php deployer.phar play:blackjack
// this broke, same error as before:

include(phar:///home/indrek/temp/kalender/deployer.phar/vendor/composer/../symfony/console/Style/SymfonyStyle.php): faile
  d to open stream: phar error: internal corruption of phar "/home/indrek/temp/kalender/deployer.phar" (crc32 mismatch on f
  ile "vendor/symfony/console/Style/SymfonyStyle.php")

[indrek-k]

After some additional testing it seems that r20 also breaks, but quite rarely. Rolling dice didn't seem to trigger the error, but php deployer.phar deploy:info did, about 10% of the times.

[indrek-k]

Managed to reproduce the same issue down to at least r10, until the version differences got big enough that my deploy.php no longer worked.

I also attempted building the phar using build script from 6.x branch and code from r24, but still got the same errors (I don't really know much about building phar archives, so perhaps I did something wrong / there's some internal cache somewhere / etc - so please take this with a grain of salt).

[ansien]

Same problems here! PHP 8.0.7 + Deployer v7.0.0-beta.24

[antonmedv]

And what with older php version?

[Schrank]

https://twitter.com/shyim97/status/1418271370421719041?s=21
I read something in phpunit that they require all classes before executing any code to avoid a bug. Maybe that's help here too?

[bayer]

same on php7.4, using Deployer v7.0.0-beta.23 and -beta.24

[zorca]

Same with php 8.0.7 and last beta. I Will try add phar to repository root, but with no luck. Crash happens randomly on Bitbucket pipeline.

[zorca]

https://twitter.com/shyim97/status/1418271370421719041?s=21
I read something in phpunit that they require all classes before executing any code to avoid a bug. Maybe that's help here too?

Maybe try using unique names to save phar? Current datetime for example. I think its a problem with CI/CD container file structure.

[zorca]

I am trying to do this:

1. Add deployer.phar to repo
2. Then in pipeline: name=$(date '+%Y-%m-%d_%H-%M-%S%z') && cp deployer.phar "deployer_$name.phar" && chmod +x "deployer_$name.phar" && ./"deployer_$name.phar" deploy

[zorca]

The first attempt was successful. If someone can, check with yourself.

[zorca]

the error is repeated (((

[antonmedv]

Sorry to hear. Will try to look more into this issue , but I can't reproduce so it's difficult for me to debug it.

[zahhar]

Error is still there as of today. Tried both PHP 7.4 and PHP 8.0.7 on MacOS 11.5.2 Big Sur — throwths crc errors randomly, using Deployer 7.0.0-beta.24.

[antonmedv]

Can you try to build phar yourself? Will it help? Maybe it’s related to my build machine?

[clue]

Just chiming in here because I've encountered and addressed similar problems for some client project with https://github.com/clue/phar-composer in the past and it looks like this could be related to inherited file descriptors in forked PHP processes. See reactphp/child-process#51 and in particular the linked issues for all the gory details. (Yep, messing with these subtle, low-level details in PHP is what I happen to do for a living and actually spend a lot of time on.)

The gist: Forking is easy – except it's not.

Whenever you spawn a child process or fork a process on Unix-based platforms, all child processes will inherit all open file descriptors by default. On Unix-based platforms, this can be controlled through the FD_CLOEXEC and O_CLOEXEC flags (neither of which are exposed to PHP).

In itself, this isn't a big deal for many applications as child processes would usually disregard any inherited file descriptors and simply open new ones themselves. This is however a major problem when using pcntl_fork() and Phar archives in PHP: The forked process will have a reference to the open Phar archive and the same underlying file descriptor. This means that if either the parent or the child process reads from the Phar archive, it will advance the FD position without the other process being aware. This is why one of the processes may read garbled/invalid data and would report a CRC error in this case.

This can be avoided by making sure each child process has a unique reference to the Phar archive, i.e. by using exec() (and family) instead of pcntl_fork() or not using Phar archives at all in combination with pcntl_fork(). Additionally, you may be able to work around this issue by making sure all classes are loaded into memory before forking, as the child process would no longer have to seek in the Phar archive.

I think https://stackoverflow.com/questions/29413013/phar-internal-corruption-crc32-mismatch-during-process-fork does a pretty good job at explaining what's going on here.

I hope this helps 👍

[antonmedv]

Yes, thanks a lot. I already know how to fix it. Will write pr today and release it.

[zahhar]

Will be happy to support with testing in various configs I run, please notify me when PR is there.

[antonmedv]

Released 7.0.0-beta.25. Please test it out! Should be fixed now!

[Schrank]

🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀

[zahhar]

Tested using PHP 8.0 & PHP 7.4 on Github Actions (ubuntu-latest) and MacOS Big Sur (latest build): everything works perfectly. Tests on Windows build agents will follow next week, but I will keep silence unless any issue found.

Thank you @antonmedv for quick resolution of this nasty bug! Great tool!

[zorca]

Wow!!! Significant!!!