Add an opt-out for the “running as root” warning

[hholst80]

What's the problem this feature will solve?

I want to be able to manually remove the warning pip spews out during package installation in root environment:

Running pip as the 'root' user can result in broken permissions and conflicting behaviour ..

Describe the solution you'd like

I want to be able to disable this warning through an environment variable like

env PIP_DISABLE_ROOT_WARNING=1 pip install flask

Alternative Solutions

No in tool workaround known to me.

Additional context

We are all adults here, I know what I am doing and I do not want to see a warning every time I run my build system. Let me disable the warning by setting an environment variable. I do not want my users to think there is anything wrong my system just because of the pip tool spews out indiscriminate warning messages.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pradyunsg]

This was proposed in #6409, was implemented in #9394 and has been discussed in #10028. There's likely a lot more discussions, but I ain't spending more of my time digging those up.

Quoting myself from #10028 (comment):

I don't think there is a way to make it possible for experienced users to not see this warning while also making sure that it serves the purpose of getting inexperienced users to understand that they should not do this in general.

Do read the linked comment above, before responding.

If you have a response that's different from "I don't like warnings." -> "Gimme an escape hatch." (which will become the top-voted Stackoverflow answer, likely without sufficient context to help inexperienced users) -- color me very interested.

[pradyunsg]

Until something like PEP 668 is implemented and generally available to the point that I am comfortable with dropping this warning, I don't think it is a good idea to provide an escape hatch.

I do not want my users to think there is anything wrong my system just because of the pip tool spews out indiscriminate warning messages.

There is a risk though. If you run sudo pip install that modifies $package and your OS depends on $package, you've quite possibly broken your OS.

I'll note that I'm speaking for myself, and not the other pip maintainers.

[pfmoore]

FWIW, I agree. I think that the warning helps more people than it inconveniences. And anyway, the people who know enough to be sure that they are safe are also capable of suppressing the warning (pip install 2>&1 | grep -v "pip as the 'root' user", for example, although IMO anyone who couldn't have constructed that themselves probably doesn't fully understand the risks of running pip as root...).

[hholst80]

Suggestion: If the log warning is made into a warning via the "warnings" module I can solve this within the existing framework of that. Or, if that is not feasible, check for /.dockerenv and if that exists do not spew out a warning because it is a hosted container environment and most likely the user is running a python environment based on pip, knows what they are doing, or a combination of both.

https://docs.python.org/3/library/warnings.html

[pradyunsg]

pip/src/pip/_internal/cli/req_command.py

Lines 179 to 184 in a07bfb3

	logger.warning(
	"Running pip as the 'root' user can result in broken permissions and "
	"conflicting behaviour with the system package manager. "
	"It is recommended to use a virtual environment instead: "
	"https://pip.pypa.io/warnings/venv"
	)

It's done through the logging module, although, we might change how we output things in the future. :)

[potiuk]

Yeah. Checking if you are in container environment would solve vast majority of problems of people who want this warning removed. I think "running root in container" is equally good reason to skip the warning as "running root in cygwin" or "running on windows". Happy to make PR if I know there is a consensus for that one.

[potiuk]

Or maybe even just changing the message to say "If you are in container, it's usualy OK to run pip as root". That woudl be more "factual".

[pfmoore]

Please, can we not have this debate again? Others have already proposed "disable the warning if you're in a container" and we've responded to that (no, the warning is still valid there - I'm quoting others as my personal experience with containers is limited, so don't bother trying to engage me in debate over this). Repeating arguments that you could have found by searching the tracker for previous discussion on this topic isn't likely to change anyone's mind here...

[potiuk]

Well. I do follow that discussion and I have not noticed that. Really sorry I should have checked more carefully.

But I have not seen anyone propose better error description, one that might help people who wonder if the warning is valid for them or not. I think you should be empathetic towards people who have their own users, and have to continue explaining them "Yeah the warning is there, but this is container so this is right". How about explicitly adding explanation that in container it is likely ok ? Still warning, a more reasonable message actually reflecting the reality. What's wrong with that?

[pradyunsg]

It is still possible to modify system-package-manager installed packages, using pip inside a container. That can still break things in weird ways.

PEP 668 will bring in the protections necessary, so if someone really wants to get rid of the warnings, it'd be more impactful to help that effort move forward. You can still try to convince us that the wording should be tweaked or the message should have additional conditionals, but don't be surprised if I'm responding on the PEP 668 discussion and not here. :)

As it stands, there's a risk to running pip as sudo regardless of whether you run it on your local terminal, in a container, or on a remote machine. Outside of mitigating that risk (part of which is done by PEP 668), all that we can do is warn users about it; and that's what this message is doing.

[pfmoore]

I think you should be empathetic towards people who have their own users

That is a valid point, and I'm sorry for not considering it. Do you supply a copy of pip with your application? If so, then you can wrap it to suppress the warning. If you don't, then I'm not sure how you can be so sure your users are using sudo pip safely? I guess if they are following instructions you provide on how to set up the container, you can know they are not doing anything unsafe, but then why not just add to your docs a note that pip issues a warning that doesn't apply for people who are following this particular set of instructions? After all, if they are not reading your docs to see that note, they probably aren't setting up their container the way you advise them to either!

I hope this helps.

[potiuk]

That is a valid point, and I'm sorry for not considering it. Do you supply a copy of pip with your application? If so, then you can wrap it to suppress the warning. If you don't, then I'm not sure how you can be so sure your users are using sudo pip safely.

Just to explain my case.

Yep I am sure it is ok. This is because we have our own Dockerfile https://github.com/apache/airflow/blob/main/Dockerfile which is very versatile and you can build your image using custom docker build . commands providing multiple arguments:

https://airflow.apache.org/docs/docker-stack/build.html#examples-of-image-customizing

For example you can build custom Airflow image like that:

docker build . \
    --build-arg PYTHON_BASE_IMAGE="python:3.8-slim-buster" \
    --build-arg AIRFLOW_VERSION="2.0.2" \
    --build-arg ADDITIONAL_AIRFLOW_EXTRAS="mssql,hdfs" \
    --build-arg ADDITIONAL_PYTHON_DEPS="oauth2client" \
    --tag "my-pypi-extras-and-deps:0.0.1"

But currently while doing it you have quite a few root warnings. This is not a deal-breaker though. If you have an underlying PEP 668 and clear way how to solve it in the future, I am quite ok to wait and explain the users this is fine. But I find it difficult to accept "turning a blind eye" on such use cases.

If the warning is going to stay there forever and there is no solution to solve it in the future, then I'd really appreciate a bit empathy and understanding and at the very least acknowledging and mentioning that there are cases that are valid so that your users do not have to explain their users "yeah ignore that - those guys are just over-protective and the warning really makes no sense, however I have no way to disable it in a reasonable way".

[potiuk]

And surely I could "grep-out" that message in my image.

But the message has already changed several times so I would never be sure it that grep continues working. We are continuously updating to latest versions of PIP as soon as it is released but there are (already were!) cases where PIP version resolver broke dependency resolution (hey this is Airflow with 500 dependencies). So one of the options of the image customisation is also choosing the pip version: https://airflow.apache.org/docs/docker-stack/build-arg-ref.html#basic-arguments - just in case users will have to use previous version of PIP - and there, the message could be different.

Again - this is not HUGE problem - it's annoyance, and we can definitely live with it for a while but I just wanted to explain that we are not just "moaning" - this is a real use case, real problem, real user annoyance and the workarounds suggested (like greping out the message) are band-aid at most.

[uranusjr]

Until PEP 668 goes somewhere I don't thinkwe have a choice. You think this is an annoyance, but distributions also hate people writing to their package store (yes, even in containers, because the distro doesn't realy have knowledge it's in a container), and pip is stuck in the middle of that tension.

[potiuk]

Sure. No worries. I understand the "pressures". As long as there is a long-term plan how to tackle it, this is perfectly fine to continue this route.

[Cougar]

I hit the same warning today (also building a docker image for system use). I see two related problems here. First, it is RED and not yellow or any other less dominant color. Another thing is that it is not a good practice for anyone to get used with errors. One day you'll miss some important error because your brain just ignores it.

[uranusjr]

For what it's worth, we want users to not run pip as root, not get used to the error.

[Cougar]

You have to run it under root if your target is to install the same packages for all users in that particular image or server. Is there any (good) alternative for that? Let every user install the same packages and versions under their venv and every time update all these vnevs together? Or make one global venv for all users which is not much different than install under root in the first place.

[uranusjr]

To be clear, the error only appears when you run pip as root, directly on a Python installation. If the goal is to provision the installation across users, it'd be best to use a virtual environment instead. That is enough to suppress the message. And before we go there, yes, we do think it is still best practice to use virtual environments in a container.

[Cougar]

Do you mean an virtual environment like python -m venv .venv and source .venv/bin/activate?

Alpine based Python 3.10 Docker image is 45.4 MB. If you set up a virtual environment, you just duplicate 1/3 of of data (+15 MB).

What is the rationale behind that? I'm very strong proponent of virtual environments and use it as much as possible but I don't see the point here. It sounds like if all you have is a hammer, everything looks like a nail - like virtual environments, hammer is also very useful tool.

[pfmoore]

What is the rationale behind that?

Ultimately the rationale here is that we have had very strong representations from Linux distribution vendors saying that they do not want people using pip to install Python packages into the area owned by the system package manager. Pip is very much "piggy in the middle" here - we cannot win, as we have conflicting demands from two key parts of our user base.

PEP 668 is the long term solution here. In the short term, it seems pointless to change something in pip just so that the other half of our user base will be yelling at us 🙁

[potiuk]

I actualy sympathise with all 3 parties there:

• similarly as @Cougar I do not want to use virtualenv for Docker image building. Not mentioning the "alpine" image growth, It makes little sense, complicates making stuff like copying installed --user python installation between segments for multi-segmented image to make image even smaller (this is what we do in Apache Airflow for example: https://github.com/apache/airflow/blob/e5422f0233b993acfe7c881dfa72178e662f8e46/Dockerfile#L444 - unlike using --user flag (or other flags to install stuff elsewhere) doing that wit venv is brittle and not guaranteed to contain all required libraries

• I understand the problem of Distro people

• And I also understand PIP people in the middle of that and I perfectly understand PEP 668 is the right solution - we discussed it before above and I am perfectly fine with it.

On the other hand we do not know when PEP 668 is going to land - and I also agree with @Cougar that false negatives which cannot be easily disabled is a wrong thing.

However - looking at the discussion above I think there is one thing that CAN be done that will satisfy everyone here. Sort of win-win-win situation

@pfmoore - you mention that the rational is that Linux distro vendors are saying they do not want people using PIP to install Python packages intto the area owned by the system package manager. Similarly @uranusjr mentions that introducing venv disables the warning. However for building image case introducing venv is NOT a good solution. On the other hand using --user flag or (similarly) using --target is much better and straightforward solution for Docker image building.

Also, coinciendently it happens that both --user (and --target flag if target is not using the system directories) also should not make the distro people angry. Because it does not touch the files they are worried about.

So what I really think is GREAT solution for everyone - if that this warning (in RED) is NOT printed if the --user flag is used or --target flag does not point to any of the distro "sensitive" directories. I think simply that using "DO NOT USE ROOT" as a message in this context is simply wrong. The message should be "DO NOT OVERRIDE SYSTEM PACKAGES". And both --user and --target flag should be tretated as "perfectly OK" when run as root.

Is there any drawback to this proposal? Maybe I have not thought about something, but It seems we have a very easy solution that satisfies everyone in the discussion and we do not have to wait unti PEP 686 materializes.

[pfmoore]

I thought the requirement was "to install the same packages for all users" (see above). --user won't do that. Also I don't know what the root user's home directory is, so I can't say it's OK. --target sucks, because it doesn't support upgrading, and it has a load of weird edge cases. It's not designed for this situation, and we'd probably get the problems reported as bugs if we started recommending it.

And in any case, without PEP 668, we don't know what are "distro-sensitive" locations, so how would we confirm that?

The drawback is the same as always - it doesn't fix the issue, it just changes the group of people who complain at us.

[potiuk]

I thought the requirement was "to install the same packages for all users" (see above). --user won't do that.

Well. Actually this is precisely what --user flag allows when it comes to container images (and we are successfully doing that for more than one and half year in Apache Airflow). The --user case is simply very close to he (recommended by PIP maintainers) venv but better.

It creates a separate, isolated environment where we have not only all packages installed but also all the '.so' and other dependencies installed in one 'folder' that is easy to copy and use. And we can easily make it 'local" for any user running the image, which effectively allows "to install the same packages for all users" (#10556 (comment)). This is precisely that we do in Airflow image - and its not our isolated case - we are simply following OpenShifft recommendations for images (https://docs.openshift.com/container-platform/3.11/creating_images/guidelines.html - look for "Support Arbitrary User IDs". Our image sets the same "HOME" directory for EVERY user. This means that ".local" directory is THE SAME for every user. And it means that literaly "we install the same packages for all users"

--target sucks, because it doesn't support upgrading,

I quite agree with that - that's why we use --user for that (and super-happy with how it works). I am perfectly OK to drop --target from my solution. Leaving only --user flag (i.e. do NOT print the warning when pip --user flag is used. full stop).

I believe (please correct me if I am wrong @pfmoore) that it will not "write to system packages" - so the properties of that solution are :

• Distro people are happy as --user flag excludes writing to system packages
• PIP People are happy. Simply using venv already has this property, so making --user as an exclusion should be ok as well
• Docker-centric users are happy - they can use --user flag to install airflow in non-system place, at the same time making it available for re-use for all users, following the best guidelines out there

Did I miss something @pfmoore ?

[webknjaz]

@potiuk my rant was caused by the sum of behaviors of people interacting here. I didn't intend for it to be targeted at you personally, I'm sorry that it came out like that.

I do understand that you believe that your case is special enough for pip to allow it and I've read your article. And I disagree with this only after trying to understand your frustrations — I still think that this band-aid is not something that pip should provide.
Instead, I'd prefer having some guidance/documentation/helpers to be contributed to help people implement this type of setup without fear of breaking things.
FWIW, in your case I'd go for creating a venv, setting env vars at the beginning of the image creation, and invoking things from that venv directly. Maybe this would need some wrappers to make things work nicely together. I see now that may be annoying. But ultimately, I believe that pip, as a project used on the scale across the board, should enforce generally safe practices letting the power users solve got dangerous adventures with external efforts.

Yes, this flag will probably be implemented but I'd say that being annoyed by the users is not the best motivation for doing such changes. I'm sure, as a fellow maintainer, you can appreciate the point that maintainers normally need to consider such things as the well-being of average users and not narrow use-cases.

As for saying that venv is an antipattern in containers — it's not. There's been a lot of evidence provided in this conversation. As a result, some people claim that it may be, others disagree. It's only an antipattern for a subset of people who think so for various reasons, this doesn't make it true in general. FWIW the provided opinions didn't convince me that venv may be unnecessary in containers but only showed that people do tend not to care about the correctness which I believe that pip should care about (even when some subset of users doesn't).

P.S. If somebody wants to challenge the status of the venv in containers recommendations, it's probably best to invoke a discussion @ https://discuss.python.org/c/packaging and have a conversation there. This is not a question that concerns pip specifically but rather the wider Python packaging community. When some consensus is reached, that could go into https://packaging.python.org.

[potiuk]

FWIW, in your case I'd go for creating a venv, setting env vars at the beginning of the image creation, and invoking things from that venv directly. Maybe this would need some wrappers to make things work nicely together. I see now that may be annoying. But ultimately, I believe that pip, as a project used on the scale across the board, should enforce generally safe practices letting the power users solve got dangerous adventures with external efforts.

This is precisely that I tried and failed in apache/airflow#19189 (and I also mentioned about it several times in this thread). So my frustrations are not only because I think using virtualenv is bad for the image but also because I know it did not work for Apache Airflow. Details of it are in the 53 (!) comments in the conversation (where I - discussing with my fellow commiters) tried different tools and solutions to make it work for us. I gave up after some 2 week of trying and another 2 weeks of thinking if it makes sense to continue or not.

As for saying that venv is an antipattern in containers — it's not. There's been a lot of evidence provided in this conversation. As a result, some people claim that it may be, others disagree. It's only an antipattern for a subset of people who think so for various reasons, this doesn't make it true in general. FWIW the provided opinions didn't convince me that venv may be unnecessary in containers but only showed that people do tend not to care about the correctness which I believe that pip should care about (even when some subset of users doesn't).

I thought a bit about this one and I do agree saying that it is a "generaI" antipattern was wrong. And I am super happy to change my statement to make it clear:

Looking at all the above points, and having experiences with both approaches (and even attempting to convert Airflow to use virtualenv) — I truly belive virtualenv is an anti-pattern for container building in a number of cases. Not always, but there are valid and important cases where it is. The virtualenv is an antipattern especialy in cases where you care for the size of the images produced, and when multi-stage builds are used to achieve this optimisation. Also when you want to create dynamic virtualenvs in the image. There are — of course — cases when size of the image, or dynamic virtualenv execution is not important, then — by all means virtualenv in the image might be a good choice.

This is how I just modified my blog post https://potiuk.com/to-virtualenv-or-not-to-virtualenv-for-docker-this-is-the-question-6f980d753b46 (and I added UPDATE: correction at the top explaining it).

I am not really in a camp of people who say "containers replace virtualenv". I never, ever, said that - simply because I don't believe it is true.

If you find my earlier (over-simpified) statement "virualenv is an antipattern for containers" wrong, then I also hope you will find that "recommended virtualenv as the only solution" where there are valid cases it's not, is equally bad statement.

And what I really would love if that statement (similarly as I just did in my post) is corrected. Ideally both in wording and in the "flag" that allows to disable this statement that - in my opinion - might get people misdirected. And I am telling it from my own experience where this misdirection lead us to dead-end, high-effort PR.

[PatrickDRusk]

At the suggestion of pradyunsg, I heeded this:

Take a Docker image for a Debian-based OS. Run pip list as root. Notice how many things are installed already.

In my case, it was an image for an AWS Lambda. After everything OS-level was installed and all that was left was installing my Python code, I did "pip list":

Package    Version
---------- -------
pip        22.0.3
setuptools 57.5.0
wheel      0.37.1

Not too worrying. I would be +1 for an environment variable to shut off that warning.

[pradyunsg]

I guess you didn't read the whole thing.

If anyone @-mentions me before then, that'll only serve to annoy me by the fact that this person can't respect boundaries (especially since this has happened before, around this topic already) and that will likely push back when I'd come around to addressing this issue.

[PatrickDRusk]

Sorry, I didn't realize the impact of the "@". I apologize.

[jaraco]

This issue is affects pip-run also. Consider:

$ docker run -it jaraco/multipy-tox pip-run -q keyring -- -c pass
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version 21.3.1; however, version 22.0.3 is available.
You should consider upgrading via the '/usr/bin/python3.10 -m pip install --upgrade pip' command.

I want to use that command (with a real exercise instead of pass) to illustrate to a user how to replicate an issue with keyring in a docker image. Nothing is getting installed to system site packages (only a temporary directory) and venv doesn't work in this scenario (one liner). This warning only adds noise to the otherwise clean output. It's adding distraction to the lesson I'm trying to give.

[dmartin-isp]

I know I'm not going to convince anyone, so this is more a vote than anything else.

It's not pip's job to secure my system. It's pip's job to install python packages. I can't think of another piece of software that behaves this way. Having an escape hatch is totally reasonable. Having this error message in every container log from now to the end of time is not reasonable.

[AngellusMortis]

I will admit, I did not read all of the posts, but I read a good chunk of them.

As a few others users pointed out, this is really annoying with using Python docker containers. When you are using a Python docker container, virtualenvs create duplicate files and make the image larger. That is not something anyone ones. There is also zero risk of breaking system packages in the official Python containers as well because the version you are using is in /usr/local/, not /usr/ so it is 100% separate from the system Python. You can pull down the most recent 3.10 Docker image and do a apt install python3 on it and you will have a /usr/local/bin/python3.10 and a /usr/bin/python3.9. Each with their own packages.

And yes, docker containers should not run as root, but no one builds a docker container as the app user. You apt install and pip install all of your reqs as root and then drop down to the app user after your are done to run the app since dependencies should be immutable.

My suggestion is to stop checking for root and start checking for installing into the system package directory. i.e. if the path of the python executable is /bin/python* or /usr/bin/python* and --user is not provided, give the error. But if the python executable is /usr/local/bin/python* (or --user is used as the root user for some odd reason), do not show the error. This will, at the very least, solve the issue most people in this thread have: we do not want to see the error while running pip inside of a docker container designed to run python/pip. We know it is not an issue, stop bothering us.

[virtuald]

My suggestion is to stop checking for root and start checking for installing into the system package directory.

This is probably the first alternative that would accommodate both points of views in this (stopping naive users from messing up their system, and quit annoying advanced users).

I'm not sure if it's straightforward to determine if you're installing into the system package directory, I don't think checking the path of python would be sufficient.

[AngellusMortis]

I'm not sure if it's straightforward to determine if you're installing into the system package directory

If the distro is following the proper folder structure, then yes, it is. Arch, Debian, Ubuntu, CentOS, RedHat, Fedora, and Amazon Linux all installs system python in /usr/bin. That covers just about every major Linux distro family. And as many others have also pointed out this thread, it is not pip's responsibility to maintain the user's system or make decisions for the distro maintainers. The best pip can do is make a reasonable default (check for /usr/bin/ in path of exec) that will handle the official Python docker images as well as the majority of Linux distros and then give distro maintainers a way to customize it for their system if they have a different need.

Some reading stuffs:
https://unix.stackexchange.com/questions/8656/usr-bin-vs-usr-local-bin-on-linux
https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard

[potiuk]

Hey @pradyunsg - as discussed few months ago, I am coming back to the issue. It's been .... a while and seems that end of March - seems to be a good idea to make it happens before the April release.

I prepared the PR: #10990 - It might be a lame atempt to do so, but I tried to follow other commands, I tested it locally and I managed to get a warning-free installation with a root user.

[potiuk]

I tried to follow https://pip.pypa.io/en/stable/development/contributing/ (including news entry etc.). I am not sure if I need to add some more docs (I believe the docs for new options are generated automatically - but if not - I am happy to add more information in the docs as well.

Please let me know if there is anything else I can do to make the change better.

[pradyunsg]

Thanks for picking this up @potiuk! I've reviewed the PR just now, and left some comments. :)