Announcing UniDep: Unified Conda and Pip dependency management via pyproject.toml or requirements.yaml

News
1

Dear Conda community, I am beyond excited to share this tool I have been building with you!

UniDep streamlines Python project dependency management by unifying Conda and Pip packages in a single system.

Handling dependencies in Python projects can be challenging, especially when juggling Python and non-Python packages. This often leads to confusion and inefficiency, as developers juggle between multiple dependency files.

  • :memo: Unified Dependency File: Use either requirements.yaml or pyproject.toml to manage both Conda and Pip dependencies in one place.
  • :gear: Build System Integration: Integrates with Setuptools and Hatchling for automatic dependency handling during pip install ./your-package.
  • :computer: One-Command Installation: unidep install handles Conda, Pip, and local dependencies effortlessly.
  • :office: Monorepo-Friendly: Render (multiple) requirements.yaml or pyproject.toml files into one Conda environment.yaml file and maintain fully consistent global and per sub package conda-lock files.
  • :earth_africa: Platform-Specific Support: Specify dependencies for different operating systems or architectures.
  • :wrench: pip-compile Integration: Generate fully pinned requirements.txt files from requirements.yaml or pyproject.toml files using pip-compile.
  • :lock: Integration with conda-lock: Generate fully pinned conda-lock.yml files from (multiple) requirements.yaml or pyproject.toml file(s), leveraging conda-lock.

Example

Example requirements.yaml

Example of a requirements.yaml file:

name: example_environment
channels:
  - conda-forge
dependencies:
  - numpy                   # same name on conda and pip
  - conda: python-graphviz  # When names differ between Conda and Pip
    pip: graphviz
  - pip: slurm-usage >=1.1.0,<2  # pip-only
  - conda: mumps                 # conda-only
  # Use platform selectors
  - conda: cuda-toolkit =11.8    # [linux64]
local_dependencies:
  - ../other-project-using-unidep     # include other projects that use unidep
  - ../common-requirements.yaml       # include other requirements.yaml files
  - ../project-not-managed-by-unidep  # 🚨 Skips its dependencies!
platforms:  # (Optional) specify platforms that are supported (used in conda-lock)
  - linux-64
  - osx-arm64

unidep can process this during pip install and create a Conda installable environment.yaml or conda-lock.yml file, and more!

For a more in-depth example containing multiple installable projects, see the example directory.

Example pyproject.toml

Alternatively, one can fully configure the dependencies in the pyproject.toml file in the [tool.unidep] section:

[tool.unidep]
channels = ["conda-forge"]
dependencies = [
    "numpy",                                         # same name on conda and pip
    { conda = "python-graphviz", pip = "graphviz" }, # When names differ between Conda and Pip
    { pip = "slurm-usage >=1.1.0,<2" },              # pip-only
    { conda = "mumps" },                             # conda-only
    { conda = "cuda-toolkit =11.8:linux64" }         # Use platform selectors by appending `:linux64`
]
local_dependencies = [
    "../other-project-using-unidep",   # include other projects that use unidep
    "../common-requirements.yaml"      # include other requirements.yaml files
    "../project-not-managed-by-unidep" # 🚨 Skips its dependencies!
]
platforms = [ # (Optional) specify platforms that are supported (used in conda-lock)
    "linux-64",
    "osx-arm64"
]

This data structure is identical to the requirements.yaml format, with the exception of the name field and the platform selectors.
In the requirements.yaml file, one can use e.g., # [linux64], which in the pyproject.toml file is :linux64 at the end of the package name.

:question: FAQ

Here is a list of questions we have either been asked by users or potential pitfalls we hope to help users avoid:

Q: When to use UniDep?

A: UniDep is particularly useful for setting up full development environments that require both Python and non-Python dependencies (e.g., CUDA, compilers, etc.) with a single command.

In fields like research, data science, robotics, AI, and ML projects, it is common to work from a locally cloned Git repository.

Setting up a full development environment can be a pain, especially if you need to install non Python dependencies like compilers, low-level numerical libraries, or CUDA (luckily Conda has all of them).

Typically, instructions are different for each OS and their corresponding package managers (apt, brew, yum, winget, etc.).

With UniDep, you can specify all your Pip and Conda dependencies in a single file.

To get set up on a new machine, you just need to install Conda (we recommend micromamba) and run pip install unidep; unidep install-all -e in your project directory, to install all dependencies and local packages in editable mode in the current Conda environment.

For fully reproducible environments, you can run unidep conda-lock to generate a conda-lock.yml file.

Then, run conda env create -f conda-lock.yml -n myenv to create a new Conda environment with all the third-party dependencies.

Finally, run unidep install-all -e --no-dependencies to install all your local packages in editable mode.

For those who prefer not to use Conda, you can simply run pip install -e . on a project using UniDep.

You’ll need to install the non-Python dependencies yourself, but you’ll have a list of them in the requirements.yaml file.

In summary, use UniDep if you:

  • Prefer installing packages with conda but still want your package to be pip installable.

  • Are tired of synchronizing your Pip requirements (requirements.txt) and Conda requirements (environment.yaml).

  • Want a low-effort, comprehensive development environment setup.

Q: How is this different from conda/mamba/pip?

A: UniDep uses pip and conda under the hood to install dependencies, but it is not a replacement for them. UniDep will print the commands it runs, so you can see exactly what it is doing.

Q: I found a project using unidep, now what?

A: You can install it like any other Python package using pip install.

However, to take full advantage of UniDep’s functionality, clone the repository and run unidep install-all -e in the project directory.

This installs all dependencies in editable mode in the current Conda environment.

Q: How to handle local dependencies that do not use UniDep?

A: You can use the local_dependencies field in the requirements.yaml or pyproject.toml file to specify local dependencies.

However, if a local dependency is not managed by UniDep, it will skip installing its dependencies!

To include all its dependencies, either convert the package to use UniDep (:trophy:), or maintain a separate requirements.yaml file, e.g., for a package called foo create, foo-requirements.yaml:

dependencies:
  # List the dependencies of foo here
  - numpy
  - scipy
  - matplotlib
  - bar
local_dependencies:
  - ./path/to/foo  # This is the path to the package

Then, in the requirements.yaml or pyproject.toml file of the package that uses foo, list foo-requirements.yaml as a local dependency:

local_dependencies:
  - ./path/to/foo-requirements.yaml

Q: Can’t Conda already do this?

A: Not quite. Conda can indeed install both Conda and Pip dependencies via an environment.yaml file, however, it does not work the other way around.

Pip cannot install the pip dependencies from an environment.yaml file.

This means, that if you want your package to be installable with pip install -e . and support Conda, you need to maintain two separate files: environment.yaml and requirements.txt (or specify these dependencies in pyproject.toml or setup.py).

Q: What is the difference between conda-lock and unidep conda-lock?

A: conda-lock is a standalone tool that creates a conda-lock.yml file from a environment.yaml file.

On the other hand, unidep conda-lock is a command within the UniDep tool that also generates a conda-lock.yml file (leveraging conda-lock), but it does so from one or more requirements.yaml or pyproject.toml files.

When managing multiple dependent projects (e.g., in a monorepo), a unique feature of unidep conda-lock is its ability to create consistent individual conda-lock.yml files for each requirements.yaml or pyproject.toml file, ensuring consistency with a global conda-lock.yml file.

This feature is not available in the standalone conda-lock tool.

Check out GitHub - basnijholt/unidep: Single source of truth with requirements for pip and conda and please ask me to clarify anything when its unclear!

2 Likes
2

@basnijholt,

You’re always welcome to stop by our community meetings if you would ever like to talk about your project with the community:

1 Like
3

I didn’t understand the flow.

I have a conda environment in which I installed packages using both ‘conda install’ and ‘pip install’. I want to be able to reproduce the same environment on a different computer.
What do I have to do on the source machine? Do I need to install unidep? conda-lock? Do I need to export the conda environment to a yaml file? What else?
What do I have to do on the target machine? Do I need to install unidep and conda-lock on the root environment before creating the target envioronment?

Thanks in advance

4

Reproducing an existing environment is not one of the goals of unidep.

Rather, it allows to define the dependencies of your project in a file such that your project is both

  • pip installable (skips installing deps that are only on conda)
  • installable using conda via unidep install or by compiling an environment.yml file from your unidep configuration in either requirements.yaml or pyproject.toml.

So basically, you would start by

  1. tracking your dependencies in a requirements.yaml or pyproject.toml file (see the docs here).
  2. Then you need to install unidep in e.g., your base environment, conda install -c conda-forge -n base unidep,
  3. then you can either create an empty conda env or use an existing and run unidep install --conda-env-name YOUR_ENV ..

Alternatively, you can generate a conda lock file using unidep conda-lock and then install that one.