Support zstd for extraction #4137

wstein · 2020-10-13T17:38:11Z

I have defined a new 7zip-zstd helper, which is used in case of *.zst compressed files. 7zip-zstd will be automatically installed, if necessary.

I have also added a new testcase for .tar.zst file.

This patch is needed for PR ScoopInstaller/Main#1494 to fix ScoopInstaller/Main#1492

Closes #3990

Ash258 · 2020-10-13T17:40:31Z

libexec/scoop-checkup.ps1

@@ -18,6 +18,11 @@ if (!(Test-HelperInstalled -Helper 7zip)) {
    $issues++
 }

+if (!(Test-HelperInstalled -Helper 7zip-zstd)) {
+    error "'7zip-zstd' is not installed! It's required for unpacking most programs. Please Run 'scoop install 7zip-zstd'."


This is lie. It is definitely not needed for "most programs"

Ash258 · 2020-10-13T17:42:02Z

lib/decompress.ps1

@@ -104,6 +127,68 @@ function Expand-7zipArchive {
    }
 }

+function Expand-7zip-zstdArchive {


Why did you duplicate whole Expand-7zipArchive function?

Just add -Zstd parameter to Expand-7zipArchive.

Expand-7zip-zstdArchive is horrible name. Never use hyphens in powershell noun part of command.

Ash258 · 2020-10-13T17:43:55Z

lib/decompress.ps1

@@ -20,6 +20,29 @@ function Test-7zipRequirement {
    }
 }

+function Test-7zip-zstdRequirement {


Do not use hyphen in name

Ash258

7zip-zstd is not in main bucket. It is nonsense to make "critical" dependency to versions bucket, which is not added always and should not be added always.

Also there is no reason to believe that zstd fork of 7zip will be synced with master and always switching to it is not acceptable

Rework this to handle extraction with clean zstd

wstein · 2020-10-13T18:52:23Z

Thank for your review! I am changing the implementation.

wstein · 2020-10-13T22:46:20Z

I have finished rewritten the code.

Ash258 · 2020-10-14T05:09:37Z

lib/decompress.ps1

@@ -16,7 +16,25 @@ function Test-7zipRequirement {
            return ($URL | Where-Object { Test-7zipRequirement -File $_ }).Count -gt 0
        }
    } else {
-        return $File -match '\.((gz)|(tar)|(tgz)|(lzma)|(bz)|(bz2)|(7z)|(rar)|(iso)|(xz)|(lzh)|(nupkg))$'
+        return $File -match '\.((gz)|(tar)|(tgz)|(lzma)|(bz)|(bz2)|(7z)|(rar)|(iso)|(xz)|(lzh)|(nupkg)|(tar.zst))$'


Why do you handle this in 7zip when it is not supported?

In general do not touch anything related to 7zip extraction

zstd can not handle tar zrchives. should I add tar as dependency?

So your current flow is this:

Install manifest which has tar.zst as extension

Add ONLY clean 7zip as dependency as it comfort Test-7zipRequirement

Call Expand-ZipArchive (which is not aware that zstd dependency is missing in the most cases)

Throw error to users' face that he is trying to bind null as parameter to Get-HelperPath as he is missing dependency to zstd.

Abort

For me it looks not bad:

>scoop install openssh Installing 'zstd' (1.4.5) [64bit] Loading zstd-v1.4.5-win64.zip from cache Checking hash of zstd-v1.4.5-win64.zip ... ok. Extracting zstd-v1.4.5-win64.zip ... done. Linking D:\scoop\apps\zstd\current => D:\scoop\apps\zstd\1.4.5 Creating shim for 'zstd'. 'zstd' (1.4.5) was installed successfully! Installing 'openssh' (8.3p1-1) [64bit] Loading bash-4.4.023-2-x86_64.pkg.tar.xz from cache Checking hash of bash-4.4.023-2-x86_64.pkg.tar.xz ... ok. Loading gcc-libs-9.3.0-1-x86_64.pkg.tar.xz from cache Checking hash of gcc-libs-9.3.0-1-x86_64.pkg.tar.xz ... ok. Loading heimdal-7.7.0-2-x86_64.pkg.tar.zst from cache Checking hash of heimdal-7.7.0-2-x86_64.pkg.tar.zst ... ok. Loading heimdal-libs-7.7.0-2-x86_64.pkg.tar.zst from cache Checking hash of heimdal-libs-7.7.0-2-x86_64.pkg.tar.zst ... ok. Loading libcrypt-2.1-2-x86_64.pkg.tar.xz from cache Checking hash of libcrypt-2.1-2-x86_64.pkg.tar.xz ... ok. Loading libdb-5.3.28-3-x86_64.pkg.tar.zst from cache Checking hash of libdb-5.3.28-3-x86_64.pkg.tar.zst ... ok. Loading libedit-20191231_3.1-1-x86_64.pkg.tar.xz from cache Checking hash of libedit-20191231_3.1-1-x86_64.pkg.tar.xz ... ok. Loading libopenssl-1.1.1.g-3-x86_64.pkg.tar.zst from cache Checking hash of libopenssl-1.1.1.g-3-x86_64.pkg.tar.zst ... ok. Loading libreadline-8.0.004-1-x86_64.pkg.tar.xz from cache Checking hash of libreadline-8.0.004-1-x86_64.pkg.tar.xz ... ok. Loading libsqlite-3.33.0-2-x86_64.pkg.tar.zst from cache Checking hash of libsqlite-3.33.0-2-x86_64.pkg.tar.zst ... ok. Loading msys2-runtime-3.1.7-2-x86_64.pkg.tar.xz from cache Checking hash of msys2-runtime-3.1.7-2-x86_64.pkg.tar.xz ... ok. Loading ncurses-6.2-1-x86_64.pkg.tar.xz from cache Checking hash of ncurses-6.2-1-x86_64.pkg.tar.xz ... ok. Loading openssh-8.3p1-1-x86_64.pkg.tar.zst from cache Checking hash of openssh-8.3p1-1-x86_64.pkg.tar.zst ... ok. Loading openssl-1.1.1.g-3-x86_64.pkg.tar.zst from cache Checking hash of openssl-1.1.1.g-3-x86_64.pkg.tar.zst ... ok. Loading tcl-8.6.10-1-x86_64.pkg.tar.xz from cache Checking hash of tcl-8.6.10-1-x86_64.pkg.tar.xz ... ok. Loading zlib-1.2.11-1-x86_64.pkg.tar.xz from cache Checking hash of zlib-1.2.11-1-x86_64.pkg.tar.xz ... ok. Extracting bash-4.4.023-2-x86_64.pkg.tar.xz ... done. Extracting gcc-libs-9.3.0-1-x86_64.pkg.tar.xz ... done. Extracting heimdal-7.7.0-2-x86_64.pkg.tar.zst ... done. Extracting heimdal-libs-7.7.0-2-x86_64.pkg.tar.zst ... done. Extracting libcrypt-2.1-2-x86_64.pkg.tar.xz ... done. Extracting libdb-5.3.28-3-x86_64.pkg.tar.zst ... done. Extracting libedit-20191231_3.1-1-x86_64.pkg.tar.xz ... done. Extracting libopenssl-1.1.1.g-3-x86_64.pkg.tar.zst ... done. Extracting libreadline-8.0.004-1-x86_64.pkg.tar.xz ... done. Extracting libsqlite-3.33.0-2-x86_64.pkg.tar.zst ... done. Extracting msys2-runtime-3.1.7-2-x86_64.pkg.tar.xz ... done. Extracting ncurses-6.2-1-x86_64.pkg.tar.xz ... done. Extracting openssh-8.3p1-1-x86_64.pkg.tar.zst ... done. Extracting openssl-1.1.1.g-3-x86_64.pkg.tar.zst ... done. Extracting tcl-8.6.10-1-x86_64.pkg.tar.xz ... done. Extracting zlib-1.2.11-1-x86_64.pkg.tar.xz ... done. Running pre-install script... Directory: D:\scoop\apps\openssh\8.3p1-1 Mode LastWriteTime Length Name ---- ------------- ------ ---- d----- 10/14/2020 5:31 PM tmp d----- 10/14/2020 5:31 PM home Directory: D:\scoop\apps\openssh\8.3p1-1\home Mode LastWriteTime Length Name ---- ------------- ------ ---- d----- 10/14/2020 5:31 PM Werner Directory: C:\Users\Werner Mode LastWriteTime Length Name ---- ------------- ------ ---- d----- 10/3/2020 10:25 PM .ssh Junction created for D:\scoop\apps\openssh\8.3p1-1\home\Werner\.ssh <<===>> C:\Users\Werner\.ssh Linking D:\scoop\apps\openssh\current => D:\scoop\apps\openssh\8.3p1-1 Creating shim for 'findssl.sh'. Creating shim for 'scp'. Creating shim for 'sftp'. Creating shim for 'ssh'. Creating shim for 'ssh-add'. Creating shim for 'ssh-agent'. Creating shim for 'ssh-copy-id'. Creating shim for 'ssh-keygen'. Creating shim for 'ssh-keyscan'. Creating shim for 'sshd'. 'openssh' (8.3p1-1) was installed successfully!

Ash258 · 2020-10-14T05:14:24Z

lib/depends.ps1

@@ -61,6 +61,9 @@ function script_deps($script) {
    if($script -like '*Expand-7zipArchive *' -or $script -like '*extract_7zip *') {
        $deps += '7zip'
    }
+    if($script -like '*Expand-ZstdArchive *' -or $script -like '*extract_zstd *') {


There is no function extract_zstd and no reason to add it

Ash258 · 2020-10-14T05:15:15Z

lib/decompress.ps1

+    }
+    switch ($Overwrite) {
+        "All" { $ArgList += "-f" }
+        "Skip" { $ArgList += "-q" }


-q Only suppress warnings. Nothing about skip overwriting files.

Ok, I will remove this option

Ash258 · 2020-10-14T05:17:46Z

lib/decompress.ps1

+        "Skip" { $ArgList += "-q" }
+    }
+
+    $Status = Invoke-ExternalCommand $ZstdPath $ArgList -LogPath $LogPath


Use try catch and abort in case of [System.Management.Automation.ParameterBindingException]

https://github.com/Ash258/Scoop-Core/blob/96f1407439d2199a244d67b7a02cdf6d7cb36cf6/lib/decompress.ps1#L89-L93

Ok, will do that

Ash258 · 2020-10-14T05:18:03Z

lib/decompress.ps1

+        abort "Failed to extract files from $Path.`nLog file:`n  $(friendly_path $LogPath)`n$(new_issue_msg $app $bucket 'decompress error')"
+    }
+
+    if (Test-Path $LogPath) {


Inline whole if

Ok, let me check

Ash258 · 2020-10-14T05:18:44Z

lib/install.ps1

@@ -555,6 +555,8 @@ function dl_urls($app, $version, $manifest, $bucket, $architecture, $dir, $use_c
            }
        } elseif(Test-7zipRequirement -File $fname) { # 7zip
            $extract_fn = 'Expand-7zipArchive'
+        } elseif($fname -match '\.zst$') {


Why do you create function when you do not use it then?

Why do you create function when you do not use it then?

it is used inside function install_deps($manifest, $arch) {

Should I remove the the function? What should I implement inside install_deps.

What are you talking about?

You have function Test-ZstdRequirement exactly for this reason

Ash258 · 2020-10-14T05:19:25Z

test/Scoop-Decompress.Tests.ps1

@@ -68,6 +69,13 @@ Describe 'Decompression function' -Tag 'Scoop', 'Decompress' {
            (Get-ChildItem $to).Count | Should -Be 1
        }

+        It "extract nested compressed file with different inner name" -Skip:$isUnix {


This test does not make sense for this PR

wstein · 2020-10-14T16:07:54Z

Thanks again for reviewing. I think I got all comments addressed.

TODO: Vanilla 7zip cannot handle the tar.zst - ScoopInstaller#4137

sebma · 2020-12-07T08:02:50Z

@wstein At your comment : discussion_r504775387, what did you do for scoop to point to openssh 8.3p1-1, mine is pointing to openssh 8.2p1-1 (which btw, fails to install ) ?

wstein · 2020-12-08T21:10:52Z

@wstein At your comment : discussion_r504775387, what did you do for scoop to point to openssh 8.3p1-1, mine is pointing to openssh 8.2p1-1 (which btw, fails to install ) ?

@sebma Sorry for the late response, you also need to patch the main bucket: ScoopInstaller/Main@b249cd4

neal2018 · 2021-05-03T12:46:35Z

Any progress on this? I believe we need zst support for gcc and many other programs

niheaven · 2021-11-10T05:08:18Z

Thanks for your job! Zstandard support has been added in #4372.

wstein added 2 commits October 11, 2020 19:55

add .zst to extension list (part to solve #3990)

81f7191

Support zstd for extraction #3990

5d6f217

Ash258 suggested changes Oct 13, 2020

View reviewed changes

wstein added 3 commits October 13, 2020 19:49

new testfile 7ZipTest5.tar.zst

e78be76

indeed 7zip-zstd is not needed in most programs

0165ad5

rename all functions from 7zip-zstd to 7zipZstd

2d97f52

Ash258 suggested changes Oct 13, 2020

View reviewed changes

wstein added 2 commits October 13, 2020 23:56

current

811bffa

Rewrite zstd support

a5989d1

Ash258 suggested changes Oct 14, 2020

View reviewed changes

wstein added 6 commits October 14, 2020 12:33

new tests cases for zstd extraction added

d57e084

zstd: remove skip from overwrite option

b103439

remove $script -like '*extract_zstd *' from condition

fdb09eb

catch ParameterBindingException

4345a36

use Test-ZstdRequirement function

958b5a4

Inline whole if

7776145

rasa mentioned this pull request Oct 21, 2020

openssh: Update to version 8.3p1-1 ScoopInstaller/Main#1494

Closed

Ash258 added a commit to Ash258/Scoop-Core that referenced this pull request Oct 26, 2020

feat(decompress): Support zstd archives

5f99ea5

TODO: Vanilla 7zip cannot handle the tar.zst - ScoopInstaller#4137

rasa mentioned this pull request Oct 26, 2020

Fix 404 Not Found for 4 packages in openssh.json ScoopInstaller/Main#1532

Closed

This was referenced Dec 26, 2020

Backports tracker Ash258/Scoop-Core#27

Open

feat(decompress): Add Expand-ZstdArchive function Ash258/Scoop-Core#97

Merged

chawyehsu mentioned this pull request Jan 17, 2021

gcc: expat package update to 2.2.10-1 and zst compression?!? ScoopInstaller/Main#1761

Closed

yostane mentioned this pull request Jan 24, 2021

Add extraction of zstd archives during install #4240

Closed

yostane mentioned this pull request Feb 16, 2021

Quick fix for gcc and gdb x64 and https ScoopInstaller/Main#1774

Closed

scrouthtv mentioned this pull request Mar 15, 2021

gcc: fixed error by mirror ScoopInstaller/Main#1933

Closed

chawyehsu mentioned this pull request Mar 31, 2021

[email protected]: Download failed ScoopInstaller/Main#1752

Closed

gpetrov mentioned this pull request Apr 10, 2021

lftp: Add version 4.9.2 ScoopInstaller/Main#1806

Merged

fritzmg mentioned this pull request May 10, 2021

Cannot choose Git Bash when installed via scoop microsoft/vscode#123205

Closed

rashil2000 mentioned this pull request Nov 1, 2021

Support zstd for extraction #3990

Closed

niheaven closed this Nov 10, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support zstd for extraction #4137

Support zstd for extraction #4137

wstein commented Oct 13, 2020

Ash258 Oct 13, 2020

Ash258 Oct 13, 2020

Ash258 Oct 13, 2020

Ash258 left a comment

wstein commented Oct 13, 2020

wstein commented Oct 13, 2020

Ash258 Oct 14, 2020

wstein Oct 14, 2020

Ash258 Oct 14, 2020

wstein Oct 14, 2020

Ash258 Oct 14, 2020

wstein Oct 14, 2020

Ash258 Oct 14, 2020

wstein Oct 14, 2020

Ash258 Oct 14, 2020

wstein Oct 14, 2020

Ash258 Oct 14, 2020

wstein Oct 14, 2020

Ash258 Oct 14, 2020

wstein Oct 14, 2020

Ash258 Oct 14, 2020

Ash258 Oct 14, 2020

wstein commented Oct 14, 2020

sebma commented Dec 7, 2020

wstein commented Dec 8, 2020

neal2018 commented May 3, 2021

niheaven commented Nov 10, 2021

Support zstd for extraction #4137

Support zstd for extraction #4137

Conversation

wstein commented Oct 13, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ash258 left a comment

Choose a reason for hiding this comment

wstein commented Oct 13, 2020

wstein commented Oct 13, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wstein commented Oct 14, 2020

sebma commented Dec 7, 2020

wstein commented Dec 8, 2020

neal2018 commented May 3, 2021

niheaven commented Nov 10, 2021