Private registry: Fetching packages not possible

[ptusch]

Versions

• [email protected]
• [email protected]
• Linux (arch) x86_64

Do you want to request a feature or report a bug?
Bug - or - missing feature

What is the current behavior?
yarn add PRIVATE_PACKAGE does not work - I'm never authenticated
(this happens with a freshly set-up ~/.yarnrc with not pre existing ~/.npmrc)

If the current behavior is a bug, please provide the steps to reproduce.

• Get a private npm repository (I used artifactory)
• Make full access bound to a specific user (note: read only needs user as well - no anonymous!)
• Setup the registry in yarn: yarn config set registry https://my_cool_artifactory/api/npm/cool_npm_repository
• Publish a sample package (this works nicely)
• Attempt to get that package using yarn add PRIVATE_PACKAGE

What is the expected behavior?
I suppose I should get the package.
After I set up the repository and a 4xx error is returned, I expect a CLI query asking for my username, email and password (drop the username and email when those are already entered in ~/.yarnrc)

More descriptive stuff
Okay, we have an artifactory server which serves as both, a proxy and a private repository container.
There is no anonymous access to the server so even read access is bound to a proper log in.

I started from scratch with no pre existing .npmrc.
I created my ~/.yarnrc by doing this:
yarn config set registry https://my_artifactory/and/its/repo
yarn config set always-auth true
yarn config set strict-ssl true
yarn config set cafile /path/to/root/cert
After this set up I invoked yarn login and entered my username and email.

Publishing a package is no problem since I'm asked for my password here.
The workflow for the publishing seems to be

• Query for version
• Query for password
• create a token with HTTP PUT to artifactory
• use? this token to publish my package
• Delete the token from artifactory with HTTP DELETE

After this, I'm left as I started - without a token since it was deleted (and it was probably in the RAM anyway).

When I then invoke yarn add to add my private package, I'm not queried for my password which results is a 403 (Forbidden). Yarn doesn't evaluate this issue and merely reports it back to me.

I didn't find a way to set a proper auth token. I played a bit with postman and was able to create a valid token for myself but not sure how I'd use it after this (maybe there is a hidden CLI flag?).

This seems to be the wrong way anyway. Manually grabbing a token to make the package manager work sounds counter intuitive to me.

Looking into other threads didn't help either. Most "solutions" seem to be work arounds.
The only solution that seemed to work was to put my plain credentials into the ~/.npmrc and work with that. But that only worked for a direct private package with no private packages as dependencies there. I suppose the credentials are stripped from the URL after the first iteration
This also happened another day while testing with a second ~/.npmrc

I guess I'll try to add the token I created with postman in the ~/.npmrc manner and add it like //MY_REGISTRY:TOKEN

Anyway, I hope this is any helpful and maybe someone can help me out.
Cheers

Edit: Adding the token to the ~/.yarnrc Didn't help either.

[aaronjensen]

This also does not work for npm private packages.

This was unrelated, nevermind.

[manuelbieh]

This is a real showstopper for us. We've just changed all our private dependencies from Gitlab ssh links ("our-package": "git+ssh://[email protected]:company/our-package.git#v1.31.1") to private npm packages ("@company/our-package": "^1.31.1") and none of our private scoped npm packages can be found.

I downgraded to 0.28.4. and everything works fine. Seems like .npmrc with our authToken inside is completely ignored in yarn > 1.x.

yarn add @company/our-package

in 0.28.4 works totally fine while in 1.2.x (and 1.3.2) it throws an error:

error An unexpected error occurred: 
"https://registry.yarnpkg.com/@company/our-package/-/our-package-1.31.1.tgz: 
Request failed \"404 Not Found\"".

Is that by intention because support for npmrc authTokens has been dropped or is it a regression bug? Is there a workaround for that?

[Bnaya]

@manuelbieh
My fix that works 100% for this issue was to set the registry to npm
Add .yarnrc to your project with registry "https://registry.npmjs.org"
And re install the private packages

• Over time after yarn upgrades and such, more and more entries in your lock file will be changed to the npm url

[manuelbieh]

Awesome, thanks!

[Qix-]

This is because the new yarn login is trying to be smart and deferring the actual authentication process until you yarn publish. The problem here is that NPM 404's any requests to private packages if you don't supply your token.

Please, maintainers, don't try to be clever here. The mental contract is that when I yarn login I expect that I am authenticated and ready to interact with the NPM registry immediately. There isn't any (obvious) benefit to deferring authentication until npm publish unless you're trying to mitigate token regeneration or something like that, at which point (in my honest opinion) you're creeping the scope of a package manager.

This is, as @manuelbieh mentioned, somewhat of a showstopper for us.

---

EDIT: Went in to make a PR, which unsurprisingly is a lot more involved than I had hoped.

Yarn doesn't appear to be storing the NPM token at all (correct me if I'm wrong) - unless you've used npm login to authorize yourself in the past, at first glance it appears Yarn forces you to authorize each time you want to do something if ~/.npmrc doesn't already exist with a token.

As it's written now, there is no way to use scoped private packages in Yarn without adding a .yarnrc file as #4672 (comment) mentioned or by logging in each time - and even then, Yarn doesn't know the difference between a truly missing package or an attempt to access a private package.

---

If anyone wants to deep-dive into this, here's a great place to start:

diff --git a/src/cli/commands/login.js b/src/cli/commands/login.js
index f252397..8ca5ef0 100644
--- a/src/cli/commands/login.js
+++ b/src/cli/commands/login.js
@@ -113,5 +113,5 @@ export function hasWrapper(commander: Object, args: Array<string>): boolean {
 export function setFlags(commander: Object) {}
 
 export async function run(config: Config, reporter: Reporter, flags: Object, args: Array<string>): Promise<void> {
-  await getCredentials(config, reporter);
+  await getToken(config, reporter);
 }

That will prompt for the username/email, as well as the password. However, it'll do that each time to run yarn login - the token isn't stored anywhere, and what you get back from getToken() is just a function that you can use to revoke it - not the actual token itself.

---

This is indeed a major showstopper for us. At this time, we cannot use Yarn as a replacement for NPM.

[BYK]

This thread is amazing with how much insight you all provided, thank you so much. Especially @Qix- and @ptusch. I'd love to get this fixed since it has been an ongoing issue for some time now. The problem is, it is not easy to reproduce, especially when private registries are involved.

@Qix- from your last message, I understand that your expectation is for Yarn to prompt for a username and a password when it detects a private package. You also mention it is hard to detect this since NPM returns a 404 response for private packages without credentials provided to prevent information leak. I know that Yarn is geared towards minimal user interaction after a command started and I do not think we can pause the resolution process to prompt for user input. I understand that putting credentials in a config file is not ideal but we should also be supporting environment variables for this. Would that be a good solution?

Also, the reason we are trying to be smart is to protect your credentials. We do not want to send your credentials to Yarn's registry or say, to GitHub or to any custom URL that is passed in the version unless you explicitly set always-auth. This helps to prevent accidental credential leaks but we may need to reevaluate that part. I would greatly appreciate your input on this part.

@ptusch - I really appreciate the detailed instructions on how to reproduce the issue. Do you think we can construct a Dockerfile that has all the steps codified so someone can just pul it and run it with the private registry set up as the way you mentioned so the only remaining part is running a command or script that demonstrates the issue? If you think so I'll give that a shot and see how far I can get. Any guidance regarding the actual set up in terms of code or commands would be greatly appreciated to speed up the process.

[Bnaya]

@BYK using the npm registry solves this issue for me, and others, in any yarn version.
registry "https://registry.npmjs.org" in the yarnrc
I think the issue related to the yarn cdn and not necessarily yarn itself

[Qix-]

@BYK: I understand that putting credentials in a config file is not ideal but we should also be supporting environment variables for this. Would that be a good solution?

Sure, that solves some of the problem - namely in the case where we would want to avoid creating an actual .npmrc file.

However, in the case where I want to completely replace npm, I'm curious as to how that workflow would work. yarn login is going to have to authenticate against npm on my behalf in order to generate a token at some point, and subsequent yarn [global] add's are going to need it to pull private packages.

From there, getting it into an environment variable would probably require outputting to stdout, no? Something like export NPM_TOKEN=$(yarn login)? That doesn't seem very user-friendly - especially considering mis-using yarn login would potentially display the token in plain text on the screen, which is a security concern.

Thank you for prioritizing this <3 It's really appreciated.

EDIT: To be clear, it'd be awesome if we could avoid storing the token somehow. I'm not sure how that would look, though, without having to re-authenticate each time.

[ptusch]

@BYK Hey,

Do you think we can construct a Dockerfile that has all the steps codified so someone can just pul it and run it with the private registry set up as the way you mentioned so the only remaining part is running a command or script that demonstrates the issue?

I'm not sure if I can setup an artifactory in a Dockercontainer but I'm sure something like sinopia or verdaccio will do just as fine. Just give me a day or so and I'll prove one (I'm a huge docker noob so please be patient).

I understand that putting credentials in a config file is not ideal but we should also be supporting environment variables for this.

I'd also appreciate this approach. I think npm supports all config elements setable in .npmrc as environment variable if you set a prefix npm_config_MY_NPMRC_ENTRY.
Maybe [this] is a good starting point? Though it might be impossible to set a token that way since it start with double slash and some kind of custom key/value passing.

Thanks for taking care ❤️

[ptusch]

@BYK Alrighty, I think I was able to create a nice docker image for you.
To start the private repo, issue this:
docker run -it --rm -p 4873:4873 ptusch/verdaccio_yarn_testing
It launches verdaccio. You can access the web ui on http://0.0.0.0:4873/
Username and password is t. Email (I think this is not important but why not) is [email protected].

You'll find two packages containing a sample for each, installing and publishing so you can see that publishing is find but installing fails.
Please use yarn and yarn publish respectively to accomplish those feats.

Btw. this isn't a 100% reconstruction. I didn't set up SSL for verdaccio but I'm sure it will do just as well. I can always test against our private repository in case of needed test (say when this one works).

Anyway, I hope this will be of great service.

Edit:
Okay, I can't seem to upload tar/zips in here so I link from my repo:

• Yarn Installing Test
• Yarn Publishing Test

[Qix-]

@ptusch can you paste the dockerfile here? :)

[ptusch]

Woops, totally forgot. Its shamelessly stolen from verdaccio and slightly modified. But please keep in mind you'd still need to manually register someone and publish the package (the register part is interactive so no static fun).

Or do you need a Dockerfile on top on my image to simply execute the thing?
Sorry, I'm such a docker greenhorn :(

The Dockerfile I used to create the image:

FROM node:8.9.0-alpine
LABEL maintainer="https://github.com/verdaccio/verdaccio"

RUN apk --no-cache add openssl && \
    wget -O /usr/local/bin/dumb-init https://github.com/Yelp/dumb-init/releases/download/v1.2.0/dumb-init_1.2.0_amd64 && \
    chmod +x /usr/local/bin/dumb-init && \
    apk del openssl

ENV APPDIR /usr/local/app

WORKDIR $APPDIR

ENV NODE_ENV=production
ENV npm_config_prefix=${APPDIR}

RUN npm config set registry http://registry.npmjs.org/ && \
    npm install [email protected] -g

RUN mkdir -p /verdaccio/storage /verdaccio/conf

ADD config.yaml /verdaccio/conf/config.yaml

RUN addgroup -S verdaccio && adduser -S -G verdaccio verdaccio && \
    chown -R verdaccio:verdaccio "$APPDIR" && \
    chown -R verdaccio:verdaccio /verdaccio

USER verdaccio

ENV PORT 4873
ENV PROTOCOL http

EXPOSE $PORT

ENTRYPOINT ["/usr/local/bin/dumb-init", "--"]

CMD $APPDIR/bin/verdaccio --config /verdaccio/conf/config.yaml --listen $PROTOCOL://0.0.0.0:${PORT}

The modified config (only authenticated can access):

storage: /verdaccio/storage

auth:
  htpasswd:
    file: /verdaccio/conf/htpasswd
    # Maximum amount of users allowed to register, defaults to "+infinity".
    # You can set this to -1 to disable registration.
    #max_users: 1000

# a list of other known repositories we can talk to
uplinks:
  npmjs:
    url: https://registry.npmjs.org/

packages:
  '@*/*':
    # scoped packages
    access: $authenticated
    publish: $authenticated
    proxy: npmjs

  '**':
    # allow all users (including non-authenticated users) to read and
    # publish all packages
    #
    # you can specify usernames/groupnames (depending on your auth plugin)
    # and three keywords: "$all", "$anonymous", "$authenticated"
    access: $authenticated

    # allow all known users to publish packages
    # (anyone can register by default, remember?)
    publish: $authenticated

    # if package is not available locally, proxy requests to 'npmjs' registry
    proxy: npmjs

# log settings
logs:
  - {type: stdout, format: pretty, level: http}
  #- {type: file, path: verdaccio.log, level: info}

[BYK]

This is amazing! Thanks a lot for putting this together in such a short time and sorry for not responding earlier. I'll use these to dig into the issue and write back here about what I find. Thanks again!

PS: If anyone is curious to try and see themselves, don't get discouraged because I said I'll look into this. The aim is to fix this problem ASAP, not specifically me to fix this problem :)

[MeirionHughes]

Probably not related; but I had this 404 issue on windows - even after logging into npm. I suspect the old-credientials where still being used (somehow) by yarn, while npm worked perfectly fine. I cleared my .npmrc and .yarnrc completely from my user-directory. Then did a fresh log-in to npm. yarn started working again with private repositories.

[Qix-]

So just hit a bug the other day where some tokens aren't accepted by NPM even though NPM spit them back out at me - ultimately causing a 404. Not entirely sure how NPM implements token generation but they seem to be inconsistent in how they handle them.

Just a thought.

[BYK]

@MeirionHughes we may wanna start ignoring old auth fields. @arcanis @bestander thoughts?

[stewx]

I am currently experiencing this issue with the latest yarn version. I have my auth token and everything in my .npmrc (I have no .yarnrc) and it worked correctly in [email protected] but it fails now. My private registry is on an Artifactory server. Yarn seems to not use the auth token at all because it gives an HTTP 401 error. Downgrading to 0.27.5 works for me.

.npmrc:

registry=https://artifactory.mycompany.com/artifactory/api/npm/npm-virtual
_auth = myAuthTokenHere
always-auth = true
email = [email protected]

Error:

error An unexpected error occurred: "https://artifactory.mycompany.com/api/npm/npm-virtual/@angular/compiler/-/compiler-4.3.1.tgz: Request failed \"401 Unauthorized\"".

[stewx]

Related: #4451

[valscion]

This issue should have been fixed by #5216. That PR had a description containing this:

Fixes #4157, #4451, #4672, #4119.

and that caused GitHub to only close #4157 automatically when that PR was merged. The other issues should've probably been closed as well.

[ptusch]

@valscion Thanks, a colleague actually told me something similar the other week..
I'll cross-check this evening, just to be sure.

[Qix-]

@valscion GitHub doesn't support commas - you need separate lines now each with "fixes". I think they supported it once but they don't anymore.

[ptusch]

So small update; Seems like it works. It's worth noting though that it requires a working .npmrc with either the password or the token itself which is okay in my opinion but might be something the yarn folks would like to have 100% integrated (not really sure, seems like a design decision).

When the password/token is not present in the .npmrc, the installation will fail with nothing to help:

verbose 0.4 Error: http://localhost:4873/yarn_test_pkg: unregistered users are not allowed to access package yarn_test_pkg

So perhaps, we'd like to have an interactive login, just like npm has.

[valscion]

So perhaps, we'd like to have an interactive login, just like npm has.

Sounds like a new issue to me?

[ptusch]

Sounds like a new issue to me?

I agree. But it's not just the interactive "ness" but also storing that information which might cause discussions since yarn didn't seem to persist that information.

[valscion]

Still, seems like this issue is solved and a new one should be opened in case there's more things to address?

EDIT: Oh wait, sorry, just re-read the issue description. Yeah. This might be a good issue for this workflow problem, but maybe the title should be improved?

[mhumeSF]

Had issues with yarn add ... not resolving from private registries. Appending always-auth = true to my ~/.npmrc resolved the issue.

[wilomgfx]

@mhumeSF thanks... i was bout to give up on yarn.

[stevehawley]

Fetching packages from a private registry is still not possible for me. There are a number of related issues but I can't find a clear description in the documentation of how to setup private registries.

I have the same registry working with npm, but when I try yarn install I get:

error Received malformed response from registry for "@myprivate/package%2fname". The registry may be down.

I've tried to add always-auth = true as suggested but that did not change the behavior.

Could someone document how to configure this?

Update for clarity, I'm not asking how to setup a private registry server, I'm looking for documentation for yarn that says yes private registries are supported and here is how to configure yarn to access your private registry. I'm specifically targeting Gitlab, if that matters.

[ptusch]

@stevehawley There's a Dockerfile a few comments above you: #4672 (comment)

[rhurstdialpad]

Any update here from the yarn team?

[jayarjo]

Having same problem here - yarn doesn't find private npm package. I've upgraded to 1.13.0, but it doesn't help. Neither of above mentioned workarounds helped.

[ptusch]

Hey folks,
I've re-visited this issue and it's working now (worked with 1.15.2) so I think it's good to close.

[unconfident]

Am I understanding it correctly? With that fix in place (#5216) Yarn respects auth token but it is only working when you already have auth token present in the rc file all while you can't get auth token to appear there through normal means by using Yarn? People in the other issue (#6405) suggest using npm for authorization, which is rather ridiculous.

I know that Yarn is geared towards minimal user interaction after a command started and I do not think we can pause the resolution process to prompt for user input.

So perhaps, we'd like to have an interactive login, just like npm has.

How about adding --always-auth flag to yarn login command instead that would change its behavior into asking for password immediately, storing authentication token in the rc file and also setting always-auth true in the same rc file so token in question is later being sent with every request to the registry? I think this is a good compromise and it makes a lot of sense. This will be an explicit opt-in for those who want to be always authorized and those who are not will still have peace of mind being sure that their auth token is not getting leaked accidentally

[wassafr]

As of today, it still doesn't work for us.

In my opinion it's really a bug of yarn add because yarn publish does ask for the password and works perfectly.

Although I full adhere to @BYK's point of view regarding credentials privacy, the only solution that worked for us was to use npm loginthen npm config set always-auth true, which is a bit ridiculous.

[tekhedd]

You can also see the same auth problem with "yarn install" on existing projects. It fails to send (or prompt for) auth in the GET phase for "public" packages accessed via a private caching server. (even if user+password is in the URL) Perhaps this thread chould be linked to an outstanding enhancement issue (I did not find one)?

• yarn 1.22.5
• deleted yarn.lock node_modules
• no preexisting .yarnrc or .npmrc
• verdaccio trivially configured as "all access requires $authenticated"

The only thing that works is what wassafr describes: npm login + always-auth true. Now that I have spent hours figuring it out, I have to agree that /in theory/ this auth process is entirely logical. But, TBH fairly byzantine (obscure? apocryphal?) from a "how do I use this tool in real life" perspective. Which clashes strongly with the rest of yarn's awesomeness, and therefore I think qualifies it as a needed UX enhancement (not a bug). So I mostly that this bug report should stay closed, but possibly a new issue/enhancement is needed as this was not a pleasant experience.

[slhck]

Came here with the same issue on a private Verdaccio registry, none of this solved it.

Found a solution and wrote it up here: #6405 (comment)

Basically, make sure that no auth token for registry.npmjs.org is in your global ~/.npmrc.