Starting to add authors, and fix some of the lazy wording of the first draft.

Author: Bryan Lawrence

paper.md

@@ -1,5 +1,5 @@
 ---
-title: 'Pyfive: the mother of all HDF5 battles.'
+title: 'Pyfive: A pure-python HDF5 reader'
 tags:
   - Python
   - Atmospheric Science
@@ -8,39 +8,51 @@ tags:
   - Engineering
 authors:
   - name: Bryan Lawrence
-    orcid: 0000-0000-0000-0000
-    equal-contrib: true
-    affiliation: "1, 2" # (Multiple affiliations must be quoted)
+    orcid: 0000-0001-9262-7860
+    affiliation: 1  # (Multiple affiliations must be quoted)
+  - name: Ezequiel Cimadevilla
+    affiliation: 2
   - name: David Hassell
-    equal-contrib: true # (This is how you can denote equal contributions between multiple authors)
+    orcid: 0000-0002-5312-4950
     affiliation: 1
+  - name: Jonathan Helmus
+    affiliation: 3
+  - name: Brian Maranville
+    orcid: 0000-0002-6105-8789
+    affiliation: 4
+  - name: Kai Mühlbauer
+    affiliation: 5
   - name: Valeriu Predoi
-    equal-contrib: true # (This is how you can denote equal contributions between multiple authors)
+    orcid: 0000-0002-9729-657
     affiliation: 1
-  - name: Ezequiel Cimadevilla
-    equal-contrib: true # (This is how you can denote equal contributions between multiple authors)
-    affiliation: 2
 affiliations:
  - name: NCAS-CMS, Meteorology Department, University of Reading, Reading, UK
    index: 1
    ror: 00hx57361
  - name: Institution Name, Spain
    index: 2
-date: 17 September 2025
+ - name: TBD
+   index: 3
+ - name: NIST Center for Neutron Research
+   index: 4
+ - name: Institute for Geophysics, University of Bonn
+   index: 5
+date: 21 September 2025
 bibliography: paper.bib
 
 ---
 
 # Summary
 
-Pyfive (<https://pyfive.readthedocs.io/en/latest/>) is an open-source thread-safe pure Python package for reading data stored in HDF5. While it is not a complete implementation of all the capabilities of HDF5, it includes all the core functionality necessary to read gridded datasets, whether stored contiguously or with chunks, and to carry out the necessary decompression for the standard options (INCLUDE OPTIONS).  All data access is fully lazy, the data is only read from storage when the numpy data arrays are manipulated. Originally developed as a prototype some years ago, the package has recently been upgraded to make it more maintainable and fully featured for handling all the environmental data known to the authors, and hence a realistic option for production data access in environmental science and more widely.  The API is based on that of h5py (which is a python shimmy over the HDF5 c-library, and hence is not thread-safe), with some API extensions to help optimise remote access. 
+Pyfive (<https://pyfive.readthedocs.io/en/latest/>) is an open-source thread-safe pure Python package for reading data stored in HDF5. While it is not a complete implementation of all the capabilities of HDF5, it includes all the core functionality necessary to read gridded datasets, whether stored contiguously or with chunks, and to carry out the necessary decompression for the standard options (INCLUDE OPTIONS).  All data access is fully lazy, the data is only read from storage when the numpy data arrays are manipulated. Originally developed some years ago, the package has recently been upgraded to support
+lazy access, and to add missing features necessary for handling all the environmental data known to the authors. It is now a realistic option for production data access in environmental science and more widely.  The API is based on that of h5py (which is a python shimmy over the HDF5 c-library, and hence is not thread-safe), with some API extensions to help optimise remote access. 
 
 # Statement of need
 
 HDF5 is probably the most important data format in environmental science, particularly given the fact that NetCDF4 is HDF5 under the hood. From satellite missions, to climate models and radar systems, the default binary format has been HDF5 for decades. While newer formats are starting to get mindshare, there are petabytes, if not exabytes of existing HDF5, and there are still many good use-cases for creating new data in HDF5. However, despite the history, there are few libraries for reading HDF5 file data that do not depend on the official HDF5 library maintained by the HDFGroup, and in particular, there are none that can be used with Python. 
 While  the HDF5 c-library is reliable and performant, and battle-tested over decades, there are some caveats to depending upon it: Firstly, it is not thread-safe, and secondly, the code is large and complex, and should anything happen to the financial stability of The HDF5group, it is not obvious the C-code could be maintained. From a long-term curation perspective this last constraint is a concern.
 
-pyfive was developed with the use-case of performant remote access to curated data as the primary motivation, but with additional motivations of having a lightweight HDF5 reader capable of deploying in resource or operating-system constrained environments (such as mobile), and one that could be maintained long-term as a reference reader for curation purposes. The lightweight deployment consequences of a pure-python HDF5 reader need no further introduction, but as additional motivation we now expand on the issues around remote access and curation.
+The original implementation of pyfive (by JH and BM), which included all the low-level functionality to deal with the internals of an HDF5 file was developed with POSIX access in mind. The recent upgrades were developed with the use-case of performant remote access to curated data as the primary motivation, but with additional motivations of having a lightweight HDF5 reader capable of deploying in resource or operating-system constrained environments (such as mobile), and one that could be maintained long-term as a reference reader for curation purposes. The lightweight deployment consequences of a pure-python HDF5 reader need no further introduction, but as additional motivation we now expand on the issues around remote access and curation.
 
 Taking remote access first, one of the reasons for the rapid adoption of pure-python tools like xarray with zarr has been the ability for thread-safe parallelism using dask. Any python solution based on the HDF5 c-library could not meet this requirement, which led to the development of kerchunk mediated direct access to chunked HDF5 data (https://fsspec.github.io/kerchunk/). However, in practice using kerchunk requires the data provider to generate kerchunk indices to support remote users, and it leads to issues of synchronicity between indices and changing datasets. pyfive was developed in such a way to have all the benefits of using kerchunk, but without the need for provider support. Because pyfive can access and cache (in the client) the b-tree (index) on a variable-by-variable basis, most of the benefits of kerchunk are gained without any of the constraints. The one advantage left to kerchunk is that the kerchunk index is always a contiguous object accessible with one get transaction, this is not necessarily the case with the b-tree, unless the source data has been repacked to ensure contiguous metadata using a tool like h5repack. However, in practice, for many use cases, b-tree extraction with pyfive will be comparable in performance to obtaining a kerchunk index, and completely opaque to the user.