I suggested an enhancement to check for potential conflicts when running !pip install or piplite.install. However, it was rejected with the following explanation.
I’m going to reject this feature request as it’s the wrong solution to avoid conflicts between package installers. Either conda should take ownership of the Python environment using PEP 668 so pip will refuse to install to it, or conda should be following the relevant standards for installed distribution metadata so pip is able to find them and avoid the unnecessary installation.
Pip shouldn’t have to special case for conda to play nice here. If all of the package installers are following the packaging specifications, it should just work
Does this explanation imply that conda is the problem in this situation?
It depends on how you look at it. From the perspective of PyPI and pip, conda is the “problem” because, as mentioned in the comments on that issue, it does not adhere to the PyPA standards for packaging. Conda has its own package repositories, its own package format, its own environments, etc., which form a separate ecosystem from the “standard” PyPI one.
That said, many conda users (myself included) would say that the underlying “problem” is with the PyPA packaging standards, and one of the main reasons conda exists is that those standards are inadequate (although they have improved somewhat in the time since conda was created).[1] In other words the surface-level problem is “conda doesn’t adhere to the standards”, but the reason for that is the deeper problem of “there are many use cases the standards handle poorly or not at all”.
There are ongoing discussions to improve the “standard” packaging system, but those aren’t likely to produce immediate change. For the time being, the upshot is that you shouldn’t blindly run pip install commands in a conda environment, because, as you found, it can create a tangled mix of conda and PyPI packages. Depending on what you’re doing, often the best way is to try to figure out what conda packages provide the Python libraries you need, and install those separately before using your notebook, and/or modify the notebook to use conda install commands instead of pip. @jaimergp has been working on a tool that tries to bridge the gap between pip and conda, which seems like it could be very useful, but it is still in an experimental stage.
As a simple example, conda can install and manage libraries independent of Python (like Qt) or even other languages and programs (like R or nodejs). The Python packaging standards can’t accommodate things like this. ↩︎