cupynumeric-hdf5

cupynumeric-hdf5 is an NVIDIA-authored developer skill for reading and writing large cuPyNumeric arrays as HDF5 files using Legate's parallel, distributed HDF5 I/O (the legate.io.hdf5 module). It addresses the problem of moving big arrays to and from disk without funneling them through a single process: every rank reads and writes its own tile in parallel, and it can optionally accelerate disk I/O with GPUDirect Storage (GDS). The skill is designed to answer save/load/stream/fence/bridge questions inline from its verified snippets rather than requiring the agent to open the installed Legate source.

It documents three functions imported from legate.io.hdf5: to_file(array, path, dataset_name) writes a cuPyNumeric array to one HDF5 file as a virtual dataset with each rank writing its tile; from_file(path, dataset_name) reads a dataset into a distributed LogicalArray; and from_file_batched(path, dataset_name, chunk_size) reads a dataset in chunks, yielding one LogicalArray plus offsets per chunk. Key guidance includes passing the cuPyNumeric ndarray directly (it implements __legate_data_interface__), bridging results back with cn.asarray for zero-copy, issuing an execution fence before any external reader because Legate I/O is asynchronous, and handling clipped boundary chunks by placing each chunk by its actual shape. It also carries clear routing rules for when NOT to use it (Parquet/Arrow/cuDF, raw binary, sharded/custom layouts, Zarr/S3, .npz/pickle, plain h5py, or pure array compute).

Prerequisites are cuPyNumeric and Legate 26.01+ (the module moved from legate.core.io.hdf5 in 25.03), plus h5py from conda-forge, which is imported at module load. GDS is optional and needs the nv-legate vfd-gds plugin and NVIDIA cuFile. It targets HPC and scientific-computing developers who need single-file HDF5 as a hand-off format for post-processing pipelines.