Adding mamba-compatible locking to conda

Conda used to have locking, but it was removed a few years ago. We are planning to add some form of locking back in. One case that I think is interesting is when multiple concurrent container builds, try to use the same cache directory with the cache mount feature in docker’s buildx.

For the purpose of keeping the repodata.json cache consistent, we are working on adding locks for the index cache; we might as well make them compatible with mamba.

My understanding of mamba’s locking implementation is that it uses record locks on the repodata.json files <cache key>.json (and not a separate lock file). This means it prevents (on Windows) or advises (on POSIX) that other processes can access a certain byte range of that file. It locks one byte to indicate that no one else should write to the whole file.

We are working on a compatible implementation for conda. In this implementation it locks byte 21 of the <cache key>.state.json to indicate that neither file should be written by another process. The locking is only there to protect the cache integrity, and not to prevent parallel conda’s from downloading the same repodata.json twice; it tries to hold the lock as briefly as possible, only long enough to write an already-downloaded repodata.json to the cache.

I’ve added the locking scheme to an updated repodata cache CEP.