cupynumeric-parallel-data-load

cupynumeric-parallel-data-load is an NVIDIA-authored developer skill for loading sharded, on-disk datasets into a distributed cuPyNumeric ndarray when no single-call built-in loader fits. cuPyNumeric mirrors NumPy's array API (including cupynumeric.load for a single .npy file), and single-file HDF5 has legate.io.hdf5.from_file, but sharded multi-file datasets, Parquet/Arrow, raw binary, and custom layouts have no built-in loader. This skill fills that gap with a canonical pattern: a manual partition plus a leaf Legate @task launch with CPU/OMP/GPU variants, sized to the machine rather than the files.

The recipe writes a Legate Python task that calls whatever third-party reader the format needs (h5py, pyarrow, np.memmap, and so on) inside the task body, letting Legate distribute the reads across GPUs and nodes. Only axis 0 is sharded while trailing axes ride along inside each tile; per-shard row counts may differ across files as long as dtype and trailing axes match, and the launch fills every available processor regardless of file count. The worked example (assets/examples/parallel_npy_load.py) assumes a directory of contiguous zero-padded shard_NNNN.npy files on a shared filesystem, builds a cumulative row-offset table, and hard-fails with descriptive errors on a wrong layout; to adapt to another layout you change only the glob pattern, the per-file reader, and the metadata discovery, keeping the partition and launch machinery unchanged. The skill card notes it is cleared for commercial and non-commercial use and passed NVSkills-Eval, including a 100% security dimension.

Compatible with linux-x86_64, linux-aarch64, darwin-aarch64, and wsl-x86_64, it targets HPC and ML engineers replacing sequential concatenation of per-file reads with parallel per-GPU reads, or demonstrating how a user-defined Legate task writes into a cuPyNumeric output array.