zarr3 shard user block / offset

[mkitti]

It would nice if for Zarr version 3 shards, one could specify a byte offset for tensorstore to start writing at. This way one could store additional binary data at the beginning of a zarr shard. This is conceptually similar to a HDF5 User Block. The idea would be to allow an arbitrary number of X bytes at the beginning of a shard file that tensorstore will not touch.

A modification to the Zarr v3 standard is not needed since the offset itself is encoded in the Zarr v3 shard index. This is an implementation option.

With this change the user block could contain HDF5 meta data or TIFF IFDs, allowing Zarr shards to also be HDF5 or Zarr v3 files.

[jbms]

In principle this could be supported, but I think it is a pretty niche use case.

[mkitti]

In HDF5, this feature is called a "user-block":

• https://docs.h5py.org/en/stable/high/file.html#user-block
• https://support.hdfgroup.org/documentation/hdf5/latest/_l_b_file_create.html

I see a wide variety of references that people are using it for some kind of purpose, usually including some kind of metadata within the file itself:

• Example for writing and reading user block aldanor/hdf5-rust#41
• Provide option to set size of UserBlock when creating file PyTables/PyTables#123

There's even a dedicated command line tool for it:
https://support.hdfgroup.org/documentation/hdf5/latest/_h5_t_o_o_l__j_a_m__u_g.html