Conda-build throwing WARNING: Error: HTTP 404 NOT FOUND for url

Users
1

Vitals:

  • Windows 11
  • conda = 24.5.0
  • Miniconda 3
    I have Miniconda 3 installed in my $HOME (\Users\my-name) directory and a conda environment–intended for JupyterLab dev-- installed (via Miniconda) under a project folder, also located in $HOME. I have conda-build installed in the base Miniconda, but I also have conda-build installed in the conda env (which I did before knowing that it is recommended that it be in the base environment only).

My issue: I want to install the PyPI package jupyterlab-hide-code (version 4.0.0) in my conda env (not base), but that package is only available on PyPI. So, with the conda env activated, I ran conda skeleton (tried grayskull, too) to build a recipe from the PyPI package. That created a .yaml recipe file in a jupyterlab-hide-code folder in my conda env. The top part of the .yaml file is as follows:

{% set name = “jupyterlab-hide-code” %}
{% set version = “4.0.0” %}

package:
name: “{{ name|lower }}”
version: “{{ version }}”

source:
url: “https://pypi.io/packages/source/{{ name[0] }}/{{ name }}/{{ name }}-{{ version }}.tar.gz”
sha256: ec049d886ba8ba77d5471a3dcd5f65b3e7fefc1b460da556d845568d32aebe3e

build:
number: 0
script: “{{ PYTHON }} -m pip install . -vv --no-deps --no-build-isolation”

requirements:
host:
- pip
- python
run:
- python

I then tried to use conda-build (which apparently ran from the base env) to create a conda package from that recipe; however, conda-build throws an error; the following three lines involve the retrieval of the python package from PyPI.

Downloading source to cache: jupyterlab-hide-code-4.0.0_ec049d886b.tar.gz
Downloading https://pypi.io/packages/source/j/jupyterlab-hide-code/jupyterlab-hide-code-4.0.0.tar.gz
WARNING: Error: HTTP 404 NOT FOUND for url https://pypi.io/packages/source/j/jupyterlab-hide-code/jupyterlab-hide-code

Conda skeleton certainly found the package on PyPI and the SHA256 hash matches that given in the file download section on PyPI. The package .tar file name is correct, too; however, something about the URL does not work. I tried pasting the URL from the .yaml file,

https://pypi.io/packages/source/j/jupyterlab-hide-code/jupyterlab-hide-code-4.00.tar.gz

into my browser, and the page is redirected to:

https://files.pythonhosted.org/packages/source/j/jupyterlab-hide-code/jupyterlab-hide-code-4.00.tar.gz

and it also says that the site cannot be reached. (We have been having intermittent network problems at my location.) I have been trying to accomplish this operation multiple times since yesterday morning, though, and always with the same result. So, I have these questions:

  1. Is there something actually wrong with the .yaml file, particularly with the URL?
  2. Might this be the result of running conda-build from the base environment? (I suspect so, to some extent at least.)
  3. If so, how do I force conda to use the conda-build in the conda env (not base)?
  4. Alternatively, can I use the conda-build in the base to build the recipe file in the conda env and put the resulting package in the conda env, say by somehow specifying the full path to the recipe?
  5. How do I safely get rid of the multiple recipe (.yaml) files and enclosing folders that I now have in various places in my directories: can I just delete them?

I am obviously a newbie at all of this, but I am at my wit’s end on this one at the moment. I REALLY need to have the functionality in the jupyter-hide-code package, and I do not yet possess the skills to create such a package for myself. (Working on it.) Thanks, all.

2

Have you tried just using pip to install the package in the conda env? pip still works in conda environments, and if it works to install that package that will likely be much simpler than creating a separate conda package using grayskull and so on.

3

The conda-build error is likely due to filename normalization (dashes and periods get converted to underscore), which was recently added to PyPI. So https://pypi.io/packages/source/j/jupyterlab-hide-code/jupyterlab-hide-code-4.0.0.tar.gz should be https://pypi.io/packages/source/j/jupyterlab-hide-code/jupyterlab_hide_code-4.0.0.tar.gz, I guess.

You can also use whl2conda for this btw, and should be faster. Or, as others mentioned, pip install it directly and hope for the best. We are preparing better PyPI integration via a new plugin too, still alpha, but you can give it a try: conda-pypi. With that you could run conda pip install jupyterlab-hide-code and it should work.

4

Thank you, @BrenBarn, for that reminder. I could do that and very well may still do that. (There does come a point …); however, I would like to have that package (and similar PyPI-only packages) installed as a conda package so that conda knows about it and can manage it. I have tried hard to “keep everything conda” in this little adventure, and I’m not quite ready to concede defeat. If I do go the pip way, I will make sure to come back here and check your answer as the solution :slightly_smiling_face:. Thanks, again.

5

@jaimergp … So, would it work just to hardcode the name of the package with underscores in the meta.yaml recipe file instead of using the { name } parameter? could it be that easy? (Yes, this is all new to me.) … or is the problem that PyPI converts the name dashes to underscores in the download request and then can’t find its own package because the package name actually has dashes in it?

Otherwise, I will give your whl2conda plug-in a whirl. Can’t hurt! Sounds like a much-needed solution. Thank you!

EDIT: I misread your comment; the plug-in you mention is conda-pypi, not whl2conda. I have presently been trying whl2conda for my task and have only been successful in building the conda package. Installing it has been unsuccessful so far. It has something, I’m sure, to do with the fact that the package is a JupyterLab extension.

6

@jaimergp Handy piece of work, conda-pypi! The PyPI-available-only jupyterlab-hide-code package installed without a hitch with conda pip install jupyterlab-hide-code. I know conda-pypi is still in the early stages of development, but please do keep going. It “calms the waters,” for sure. (I have marked your original answer as the Solution.)

However, as I understand it, had jupyterlab-hide-code had dependencies that conflicted with ones already installed, the internal conda-pypi command to pip install would have been run with the --no-deps option, correct? In that case, I have a couple of further questions:

  1. Would a list of the skipped dependencies (of the package being installed)
    have been included in the conda-pypi output displayed in the console?
  2. I would, then, be responsible for making decisions about and installing those individual dependencies myself, correct?
1 Like
7

Yes, that’d work for the packages uploaded with normalization. A little unknown feature is that you can specify different URLs for the same sha256, and they are tried until one succeeds. This is common in the R ecosystem, see example. You could list both with and without underscores.

It’s a bit more advanced than that. See this page in the docs. Quoting that:

The main logic currently works like this:

  1. Collect the PyPI requirements and execute pip install --dry-run to obtain a JSON report of “what would have happened”.
  2. The JSON report is parsed and the resolved dependencies are normalized and mapped to the configured conda channels via different sources (e.g. cf-graph-countyfair, grayskull, parselmouth).
  3. The packages that were found on the configured conda channels are installed with conda. Those not on conda are installed individually with pip install --no-deps.
1 Like
8

Thank you, @jaimergp. All much clearer to me now. Kudos again on conda-pypi. Great stuff!

1 Like