[BUG] Integrity check fails for azure artifacts (and apparently other private registries) NPM packages

[PaulVrugt]

Current Behavior:

We are trying to upgrate to NPM 7, but we run into problems with getting packages from azure artifacts. The following log is shown, when running npm i --verbose:

npm WARN tarball tarball data for https://infolandcode.pkgs.visualstudio.com/_packaging/Shared/npm/registry/xxx/xxx.tgz (sha1-ZTIGEke6SQtyObZXul6THKgWSz0=) seems to be corrupted. Trying again.
npm WARN tarball tarball data for https://infolandcode.pkgs.visualstudio.com/_packaging/Shared/npm/registry/xxx/xxx.tgz (sha1-oYbrH56D5xdd2LzPKGiYWv/1Z9w=) seems to be corrupted. Trying again.
npm WARN tarball tarball data for https://infolandcode.pkgs.visualstudio.com/_packaging/Shared/npm/registry/xxx/xxx.tgz (sha1-ZTIGEke6SQtyObZXul6THKgWSz0=) seems to be corrupted. Trying again.
npm WARN tarball tarball data for https://infolandcode.pkgs.visualstudio.com/_packaging/Shared/npm/registry/xxx/xxx.tgz (sha1-oYbrH56D5xdd2LzPKGiYWv/1Z9w=) seems to be corrupted. Trying again.
npm timing reify:rollback:createSparse Completed in 2ms
npm timing reify:rollback:retireShallow Completed in 1ms
npm timing command:install Completed in 1784ms
npm verb stack Error: sha1-ZTIGEke6SQtyObZXul6THKgWSz0= integrity checksum failed when using sha1: wanted sha1-ZTIGEke6SQtyObZXul6THKgWSz0= but got sha512-1HAETTUJsxZ0ZT/s/kYXiDJgmm4bXD7co77wzewb4t8BBW7Vhk0ONkOs5orSToPCtu41U30n/e50gqTgia8anA== sha1-oCrDJJcm5EvGPkto9uzdg5hsPC0=. (307 bytes)

Expected Behavior:

I expect the packages to be downloaded correctly, just as they did using npm 6

Steps To Reproduce:

Reference packages to azure artifacts in your package.json. Try to run npm install

expected:
packages install

actual:
error as shown above

Environment:

• OS: windows 10
• Node: v12.16.1
• npm: 7@beta10

[leandro-manifesto]

I'm also getting this with Gemfury's private registry, but the signature is a SHA512 in my case.

Both NPM 6.14.6 and Yarn 1.22.5 work properly, but NPM 7.0.0-beta.10 is calculating a different signature and failing the integrity check.

[ValeryVS]

Also getting such error with private registry and [email protected].
Install with [email protected], [email protected] and [email protected] works properly.

[jayphelps]

Also getting this with a custom package installed from the github npm registry https://npm.pkg.github.com. Hopefully this is helpful correlation that it seems to be non-npm official registry related. Downgrading from 7.0.0-beta.12 to v6 worked.

integrity checksum failed when using sha512:
wanted sha512-X99IsgB5ElySriQwf/O4bimqNVHS9Wwij3KZLURUkC3dxO/028WcbiQRO2NtE2YJQgLkoFl8evwf0sR8jmxRFQ==
but got sha512-z4PhNX7vuL3xVChQ1m2AB9Yg5AULVxXcg/SpIdNs6c5H0NE8XYXysP+DGNKHfuwvY7kxvUdBeoGlODJ6+SfaPg==

Since this is a private package, I'm happy to do a Zoom with any team member and give them access if they are unable to reproduce independently.

[PaulVrugt]

@jayphelps
I've tested downgrading to 7.0.0-beta.6 but this didn't help for me. The same issue:

npm ERR! sha1-ZTIGEke6SQtyObZXul6THKgWSz0= integrity checksum failed when using sha1: 
wanted sha1-ZTIGEke6SQtyObZXul6THKgWSz0= 
but got sha512-1HAETTUJsxZ0ZT/s/kYXiDJgmm4bXD7co77wzewb4t8BBW7Vhk0ONkOs5orSToPCtu41U30n/e50gqTgia8anA== sha1-oCrDJJcm5EvGPkto9uzdg5hsPC0=.

[jayphelps]

@PaulVrugt sorry, I meant v6.x.x, the current stable release, not a beta for v7

[darcyclarke]

We've potentially got a fix for this based on some work @nlf is about to land in a point release. I'll let him follow up though to confirm once we've got that out to have you try to reproduce.

[nlf]

We released v7.0.0-rc.2 last Friday and it includes some fixes that may relate to this, can you try updating to that version with npm i -g npm@next-7 and seeing if your issue is fixed?

[leandro-manifesto]

Installed [email protected] and the integrity check still fails.

[jayphelps]

@nlf rc.2 solved the issue for me, or at least the problem "went away". Thank you!

Unclear why it fixed my issue but not for @leandro-manifesto. That's unfortunate.

[leandro-manifesto]

Just to be sure I ran npm i again just like before, with a clean cache, empty node_modules and no package-lock.json or yarn.lock.
Still didn't work.

Here is the log: https://gist.github.com/leandro-manifesto/50bffbfb2af4d433630d388310923af3

[PaulVrugt]

I'll take a look in the morning to see if it is solved for us

[PaulVrugt]

@nlf I've updated to RC2 and I'm afraid the issue remains, just as with @leandro-manifesto

[bonkydog]

Hello, @jayphelps. I'd like to pair with you on diagnosing this problem. Is there a convenient time for you tomorrow (Wed, Oct 7) ? (Please include time zone.)

[PaulVrugt]

@bonkydog jayphelps stated that his problem went away with rc2

[bonkydog]

@PaulVrugt Would it be possible to share the tarball and and package.json files downloaded for the packages being reported as corrupt? This might help us figure out what is going on.

[PaulVrugt]

Cool! I'll give it a go and let you know the results

[leandro-manifesto]

Hey, tested with 7.0.0-rc.4, and now I'm getting 404 when fetching the tarball (the other requests are ok).

I'm just not sure if Gemfury is rejecting the authorization or if it really is a bad url.

[nlf]

If you curl the url (or try to fetch it with a browser) does it work? If so, it sounds like an issue with authentication

[PaulVrugt]

@nlf It seems indeed to be an authentication issue. We get a 401 response while trying to download packages from our private registry. We use the vsts-npm-auth package to authenticate, but it seems not to work with npm 7. Is this an issue with npm 7 or vsts-npm-auth?

the documentation in Azure devops states:
Add a .npmrc to your project, in the same directory as your package.json

registry=https://pkgs.dev.azure.com/xxx_packaging/xxx/npm/registry/ 
always-auth=true

Then, run vsts-npm-auth to get an Azure Artifacts token added to your user-level .npmrc file

vsts-npm-auth -config .npmrc

[nlf]

I'm not familiar with that package at all, and I'm unable to find any code that I could review, so I'm not sure.

Do you know if there are implementation details documented anywhere? How does it modify the .npmrc file when you run it? Can you share the .npmrc file? Please be sure to remove any auth tokens first.

[PaulVrugt]

I can't find any information about it either. I'll post a topic in the microsoft github about it and link it here. The .npmrc doesn't get modified, it simply contains the url to the registry where it should get the packages.

[nlf]

It doesn't add a line like //pkgs.dev.azure.com/:_authToken=YOURTOKENHERE?

If not, and it's not doing something like replacing the registry url with a proxy that performs the authentication for you, then it's likely the auth is failing because we're not able to figure out how to actually do the authentication because it's not configured anywhere.

I think following up with that team is the right way to go. Please do get back to us and we'll see what we can do to help make this work correctly.

[PaulVrugt]

Microsoft issue:
microsoft/azure-pipelines-tasks#13697

[PaulVrugt]

No it doesn't add anything to the .npmrc file, so it probably hooks in somewhere else. I've reverted to npm6, and then the authentication is working as expected, and still no change to the .npmrc file. As you can see, I logged an issue with microsoft. Let's see what they come up with

Edit:
As stated in the post below, it DOES make changes to the .npmrc, but only to the .npmrc file in the user profile

[PaulVrugt]

@nlf
The plot thickens a bit. I've tried to use the .npmrc for authentication without using the vsts-npm-auth package. The contents of the .npmrc is now:

@xxx:registry=https://xxx.pkgs.visualstudio.com/_packaging/Shared/npm/registry/

//xxx.pkgs.visualstudio.com/_packaging/Shared/npm/registry/:username=VssToken
//xxx.pkgs.visualstudio.com/_packaging/Shared/npm/registry/:_password=***
//xxx.pkgs.visualstudio.com/_packaging/Shared/npm/registry/:email=VssEmail
//xxx.pkgs.visualstudio.com/_packaging/Shared/npm/registry/:always-auth=true

and when installing packages, I see that is recognizes the auth, since it lists the same as the above in the log. But the authentication still fails. With the same configuration, npm 6 works like a charm.

963 verbose stack Error: Unable to authenticate, need: Bearer authorization_uri=https://login.windows.net/<guid>, Basic realm="https://pkgsprodsu3weu.app.pkgs.visualstudio.com/", TFS-Federated
963 verbose stack     at C:\npm\prefix\node_modules\npm\node_modules\npm-registry-fetch\check-response.js:113:17
963 verbose stack     at processTicksAndRejections (internal/process/task_queues.js:97:5)
964 verbose statusCode 401
965 verbose pkgid https://xxx.pkgs.visualstudio.com/_packaging/Shared/npm/registry/@xxx/testmonkey-integration-script/-/testmonkey-integration-script-1.0.1.tgz

So even without the vsts-npm-auth the authentication seems to fail if we add authentication to the .npmrc file

Update:
I discovered that the vsts-npm-auth package injects the credentials in the user .npmrc file (located at %userprofile%.npmrc). The content of the user .npmrc file is:

//xxx.pkgs.visualstudio.com/_packaging/Shared/npm/registry/:username=VssSessionToken
; This is an unencrypted authentication token. Treat it like a password. DO NOT share this value with anyone, including Microsoft support.
//xxx.pkgs.visualstudio.com/_packaging/Shared/npm/registry/:_password=xxx
; The npm client won't use username and _password without an email set, but the service doesn't use the email value. The following is an arbitrarily made-up address.
//xxx.pkgs.visualstudio.com/_packaging/Shared/npm/registry/:[email protected]

So I think this means the authentication is set correctly in the .npmrc file, but npm 7 somehow doesn't use it, although when I check the logging, the .npmrc file from the user profile IS loaded during npm install

[nlf]

i think i see a bug in the code.. try removing the path segment of your registry url for the configuration values, but keep it in place for the registry= line and let me know if that helps

so

@xxx:registry=https://xxx.pkgs.visualstudio.com/_packaging/Shared/npm/registry/
//xxx.pkgs.visualstudio.com/_packaging/Shared/npm/registry/:username=VssSessionToken
//xxx.pkgs.visualstudio.com/_packaging/Shared/npm/registry/:_password=xxx
//xxx.pkgs.visualstudio.com/_packaging/Shared/npm/registry/:[email protected]

becomes

@xxx:registry=https://xxx.pkgs.visualstudio.com/_packaging/Shared/npm/registry/
//xxx.pkgs.visualstudio.com/:username=VssSessionToken
//xxx.pkgs.visualstudio.com/:_password=xxx
//xxx.pkgs.visualstudio.com/:[email protected]

[PaulVrugt]

@nlf

I've tried it, but it seems to make no difference at all. The error remain exactly the same :(

[nlf]

Thanks for giving it a try, I'll keep looking and see if I can figure out what's going on

[nlf]

Just to rule out a problem with auth, can I have you try one more thing..

The value after _password should be base64 encoded, can you decode that and try to make the request using curl with the username and password in the configuration and see if you receive the same 401 error?

You can decode the base64 with something like:

node -pe "Buffer.from('xxx', 'base64').toString('utf8')"

which will give you a plain text password, after which something like

curl -svo /dev/null 'https://username:[email protected]/to/the/file'

should print out all the response headers and the status code.

If that is also returning a 401 then the issue is outside of the scope of npm, however if that works then it would seem we're mishandling the auth somewhere.

[PaulVrugt]

I have a few days off work right now, but I'll try to give this a go today or tomorrow if my wife and children let me 😉

[PaulVrugt]

So, I found the time. And following your instructions, see the result of the Curl command below. It returns a 303 with a redirect. When I change the password in the curl command it returns a 401. So the credentials seem to be fine.

Edit: When adding the --location flag, it actually follows the redirect and downloads the package

*   Trying 13.107.42.18...
* TCP_NODELAY set
* Connected to xxx.pkgs.visualstudio.com (13.107.42.18) port 443 (#0)
* schannel: SSL/TLS connection with xxx.pkgs.visualstudio.com port 443 (step 1/3)
* schannel: checking server certificate revocation
* schannel: sending initial handshake data: sending 199 bytes...
* schannel: sent initial handshake data: sent 199 bytes
* schannel: SSL/TLS connection with xxx.pkgs.visualstudio.com port 443 (step 2/3)
* schannel: failed to receive handshake, need more data
* schannel: SSL/TLS connection with xxx.pkgs.visualstudio.com port 443 (step 2/3)
* schannel: encrypted data got 4096
* schannel: encrypted data buffer: offset 4096 length 4096
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with xxx.pkgs.visualstudio.com port 443 (step 2/3)
* schannel: encrypted data got 1024
* schannel: encrypted data buffer: offset 5120 length 5120
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with xxx.pkgs.visualstudio.com port 443 (step 2/3)
* schannel: encrypted data got 1024
* schannel: encrypted data buffer: offset 6144 length 6144
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with xxx.pkgs.visualstudio.com port 443 (step 2/3)
* schannel: encrypted data got 1024
* schannel: encrypted data buffer: offset 7168 length 7168
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with xxx.pkgs.visualstudio.com port 443 (step 2/3)
* schannel: encrypted data got 858
* schannel: encrypted data buffer: offset 8026 length 8192
* schannel: sending next handshake data: sending 158 bytes...
* schannel: SSL/TLS connection with xxx.pkgs.visualstudio.com port 443 (step 2/3)
* schannel: encrypted data got 326
* schannel: encrypted data buffer: offset 326 length 8192
* schannel: SSL/TLS handshake complete
* schannel: SSL/TLS connection with xxx.pkgs.visualstudio.com port 443 (step 3/3)
* schannel: stored credential handle in session cache
* Server auth using Basic with user 'VssSessionToken'
> GET /_packaging/Shared/npm/registry/@xxx/testmonkey-integration-script/-/testmonkey-integration-script-1.0.1.tgz HTTP/1.1
> Host: xxx.pkgs.visualstudio.com
> Authorization: Basic xxx
> User-Agent: curl/7.55.1
> Accept: */*
>
* schannel: client wants to read 102400 bytes
* schannel: encdata_buffer resized 103424
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: encrypted data got 1415
* schannel: encrypted data buffer: offset 1415 length 103424
* schannel: decrypted data length: 1386
* schannel: decrypted data added: 1386
* schannel: decrypted data cached: offset 1386 length 102400
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: decrypted data buffer: offset 1386 length 102400
* schannel: schannel_recv cleanup
* schannel: decrypted data returned 1386
* schannel: decrypted data buffer: offset 0 length 102400
< HTTP/1.1 303 See Other
< Cache-Control: no-cache
< Pragma: no-cache
< Expires: -1
< Location: https://6egvsblobprodsu6weus36.blob.core.windows.net/b-7e5e1ac1118b4eadb4ae3f7165fe853c/1BEED41088483DAD90BC884E56E505D859406194F791A91CF57C09DE12BF9EA600.blob?sv=2019-02-02&sr=b&si=1&sig=cJbL3x91u3lvZnkhnR8d1sv2Umt42e3XnKbZ8FRONk0%3D&spr=https&se=2020-10-15T11%3A58%3A38Z&rscl=x-e2eid-f74f2a7d-d8c74d81-88cc7903-9595d8bd-session-f74f2a7d-d8c74d81-88cc7903-9595d8bd&rscd=attachment%3B%20filename%3D%22testmonkey-integration-script-1.0.1.tgz%22
< P3P: CP="CAO DSP COR ADMa DEV CONo TELo CUR PSA PSD TAI IVDo OUR SAMi BUS DEM NAV STA UNI COM INT PHY ONL FIN PUR LOC CNT"
< X-TFS-ProcessId: d416e68c-4fd0-4669-85eb-03855ef00a73
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< ActivityId: f74f2a7d-d8c7-4d81-88cc-79039595d8bd
< X-TFS-Session: f74f2a7d-d8c7-4d81-88cc-79039595d8bd
< X-VSS-E2EID: f74f2a7d-d8c7-4d81-88cc-79039595d8bd
< X-VSS-UserData: 8b69620c-dc8f-6e39-b800-b63402c40d6b:[email protected]
< X-FRAME-OPTIONS: SAMEORIGIN
< X-Packaging-Migration: NpmBlobMetadata
< Request-Context: appId=cid-v1:f5d75a35-28cc-4e72-8007-1cf59e01402f
< Access-Control-Expose-Headers: Request-Context
< X-Content-Type-Options: nosniff
< X-MSEdge-Ref: Ref A: D211137342504A3090A2E577451F4315 Ref B: BRU30EDGE0320 Ref C: 2020-10-14T11:58:34Z
< Date: Wed, 14 Oct 2020 11:58:34 GMT
< Content-Length: 0
<
* Connection #0 to host xxx.pkgs.visualstudio.com left intact

[nlf]

ok @PaulVrugt I think we've finally got it. we released 7.0.1 last night, which you can install with the usual npm i -g npm@next-7 and should hopefully resolve this for you.

please let me know how it goes!

[PaulVrugt]

@nlf you rock! It works now.

[raghda94]

@nlf I'm facing the same issue with npm v7.0.3 here's an output of what I get
npm http fetch GET 200 https://registry.npmjs.org/aws-sdk/-/aws-sdk-2.778.0.tgz 31307ms npm timing reifyNode:node_modules/serverless/node_modules/aws-sdk Completed in 34351ms npm timing reify:rollback:createSparse Completed in 0ms npm timing reify:rollback:retireShallow Completed in 1ms npm timing command:install Completed in 36218ms npm verb stack Error: TAR_BAD_ARCHIVE: Unrecognized archive format npm verb stack at Unpack.warn (/usr/local/lib/node_modules/npm/node_modules/tar/lib/warn-mixin.js:19:40) npm verb stack at Unpack.warn (/usr/local/lib/node_modules/npm/node_modules/tar/lib/unpack.js:188:18) npm verb stack at Unpack.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/tar/lib/parse.js:82:14) npm verb stack at Unpack.emit (node:events:339:22) npm verb stack at Unpack.[emit] (/usr/local/lib/node_modules/npm/node_modules/tar/lib/parse.js:286:12) npm verb stack at Unpack.[maybeEnd] (/usr/local/lib/node_modules/npm/node_modules/tar/lib/parse.js:402:17) npm verb stack at Unpack.[consumeChunk] (/usr/local/lib/node_modules/npm/node_modules/tar/lib/parse.js:434:21) npm verb stack at Unpack.write (/usr/local/lib/node_modules/npm/node_modules/tar/lib/parse.js:365:25) npm verb stack at Unpack.end (/usr/local/lib/node_modules/npm/node_modules/tar/lib/parse.js:479:14) npm verb stack at Yallist.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/minipass/index.js:396:18)

[jdmarshall]

I think Artifactory and friends occasionally suffer from connection reset issues when talking to clients. Given that the requests in question usually have a Content-Length header, my suspicion is that pacote does not verify Content-Length before attempting to validate the file it was given. We see this same problem both with NPM install operations and with a piece of code that uses pacote directly.

If the wrong number of bytes are sent there is no way the hash is going to validate, but premature end of stream should be the correct error message, not sha errors. Also parts of npm seem to cache these incomplete files, causing the same problem to happen when running a second time.