Back: Catalog

Zarr

Zarr is a Python library developed for compressed and fragmented N-dimensional arrays. It allows data to be stored in hierarchical groups on memory, disk or cloud storage units and processed in parallel. This structure, which supports NumPy compatible data types, supports simultaneous read and write operations over multiple threads or processes.

ZA

Zarr

70

zarr-developers · Scientific Research & Computing

STARS2.1K
FORKS477
COMMITS—
CONTRIBUTORS—
VERSIONzarr_metadata-v0.4.0
UPDATEDtoday
n-dimensional-arrayscloud-storagescientific-computingpythonMIT
What can it do?
  • You create N-dimensional arrays with NumPy compatible data types and store them in different storage environments.
  • You optimize it over different dimensions by breaking up the big data cluster into pieces.
  • You read and write data simultaneously using multiple threads or processes.
User reviews
  • ↓ NegativeHacker News

    Loading NumPy arrays from disk: mmap() vs. Zarr/HDF5

    “Given the discussion on Zarr, HDF5 and TileDB, I hope I can get feedback from a problem I have. I want to switch from a format I made (the "FPB" format described at ) to one which scales to more than 250 million entries. (See for my recent publication on the system, which…”

  • ↓ NegativeHacker News

    Loading NumPy arrays from disk: mmap() vs. Zarr/HDF5

    “Otherwise, HDF5 offers every single advantage that zarray has and is much more mature, stable, better documented, and has better support. Absolutely not. HDF5 is an awful format with terrible implementations. For example, try writing a python program with multiple threads where…”

Website GitHub

My note

More in: Scientific Research & Computing