🎉 conda-package-streaming 0.8.0 and conda-package-handling 2.1.0 released

News
,
1

conda-package-streaming 0.8.0 and conda-package-handling 2.1.0 have been released. These releases are mostly to add the extracted size to the .tar.zst archives that form part of a .conda file, allowing zstd to allocate less memory on decompression when the extracted size is less than the default buffer size for that compression level (most noticeable when using compression levels >= 20).

To accomplish this conda-package-handling iterates over its input files twice when creating a .conda from scratch when conda-build is configured to emit .conda directly, and conda-package-streaming saves uncompressed .tar to a temporary file when doing transmute (also powers the cph transmute command). We expect most users will not notice as the majority of time is spent inside the zstd compressor but transmute will require more disk space.

This will also be the first conda-package-handling release to be on pypi as soon as we are able to claim its package name there.

Changelogs for the releases.

conda-package-handling 2.1.0 (2023-05-04)

Bug fixes

  • Include decompressed size when creating .conda archives with CondaFormat_v2.create(), to reduce memory usage on decompression. (#171) Transmuted archives (converted from .tar.bz2) do not contain the decompressed size. NOTE: Unless using conda-package-streaming >= 0.8.0
  • Include LICENSE, not just LICENSE.txt in info/ section (#172)

Contributors

conda-package-streaming 0.8.0 (2023-05)

  • Update transmute to use SpooledTemporaryFile instead of streaming directly to
    zip (#57). This
    can reduce zstd memory usage during decompression.
  • transmute returns Path to transmuted package instead of None.
1 Like
2

Awesome, congrats on the release. Where is the decompressed size stored? An additional text file in the outer zip archive or something along those lines?

3

It means we tell the zstd compressor how much data we are going to send it, instead of fully streaming. In Python it looks like compressor.stream_writer(target_file_object, size=bytes_to_compress). This adds a header to the compressed object that zstd understands during decompression.