Implement file:local_path from the STAC file extension

[sturoscy-personal]

This may be a publisher and/or a client related issue.

• Should this use the existing STAC file:local_path or an ESGF-specific field (as an extension)? Either way, the publisher definitely needs to be modified to populate it, and there will be a schema change somewhere.
• Can the transaction API can automatically populate this based on existing fields?
• Seems like the request is to either provide the DRS or the relevant info to be able to generate the DRS
• Is DRS is intended to specify location of downloads? https://pcmdi.llnl.gov/mips/cmip5/docs/cmip5_data_reference_syntax.pdf section 1.4 use cases 3, 4, and 5 seem to indicate yes

[glevava]

The DRS indeed defines how datasets are identified and organized, independently of the CVs that list the allowed terms for the various data descriptors. The DRS specifies, in particular, the directory structure, file naming, and the dataset identifier template used within ESGF.

Historically, the directory template was included in Solr metadata and, to the best of my knowledge, was primarily (if not exclusively) used to reconstruct the directory structure after download, notably by replication tools.

In the current context, I do not see strong value in exposing this information directly at the STAC index level. That logic is now centralised alongside the CVs and is readily accessible via esgvoc, which can reconstruct or validate directory structures, filenames, and dataset identifiers on demand through its CLI, Python library, or REST API. From this perspective, I would strongly favour keeping esgvoc as the single interface for handling the DRS logic.

This is precisely why we integrated esgvoc into esgpull: to compensate for the fact that STAC no longer carries the DRS template information that used to be available in Solr. In my view, this is good thing!

From a user perspective, local_path provides limited added value. It is mainly relevant for replication centres that manage large pools of replicas for their users or for ESGF republication.

[bstrdsmkr]

That logic is now centralised alongside the CVs and is readily accessible via esgvoc, which can reconstruct or validate directory structures, filenames, and dataset identifiers on demand through its CLI, Python library, or REST API. From this perspective, I would strongly favour keeping esgvoc as the single interface for handling the DRS logic.

I agree that the STAC implementations shouldn't be reconstructing the DRS from scratch, but I also believe exposing a field that contains the DRS (sourced from esgvoc) is a fairly low lift that would ease the transition for existing clients, particularly for web based clients like Metagrid that don't have easy access to esgvoc.

From a user perspective, local_path provides limited added value. It is mainly relevant for replication centres that manage large pools of replicas for their users or for ESGF republication.

I disagree here -- given the specialized nature of the field:

1. It is likely users will use more than one tool to interact with the data (for example, one for discovery, one for acquisition, one for manipulation, etc)
2. It is likely that users will interact with more than single digit numbers of files

Given those, if clients are all sharing the same directory structure, that's going to save users time and resources for not re-downloading the same data for different usages. This also lends itself to places that are maintaining "unofficial" caches of files that multiple different users may have requested. I believe easing that friction is a big benefit to users in general, even if it's not a direct "added capability."

[glevava]

I agree that the STAC implementations shouldn't be reconstructing the DRS from scratch, but I also believe exposing a field that contains the DRS (sourced from esgvoc) is a fairly low lift that would ease the transition for existing clients, particularly for web based clients like Metagrid that don't have easy access to esgvoc.

The CV does not contain the patterns governing the DRS because the logic goes far beyond a simple regular expression. That is precisely what esgvoc is designed to handle. In my view, providing only a pattern in the STAC item is not sufficient to propagate good data management practices on the user side and ultimately leaves room for everyone to interpret and implement things differently. By not exposing such a field, we are not preventing users from organizing downloaded data as they wish; rather, we are indicating a preferred and robust way to do so: through esgvoc.

I disagree here -- given the specialized nature of the field:

The nature of this field is much more complex than a simple pattern. Moreover, tools used to discover, analyze, and manipulate data should not need to understand the DRS at all, that is the point of catalog systems: to abstract away the underlying directory structure. This is even more relevant in the context of object storage, where traditional filesystem hierarchies are more conceptual.

Given those, if clients are all sharing the same directory structure, that's going to save users time and resources for not re-downloading the same data for different usages. This also lends itself to places that are maintaining "unofficial" caches of files that multiple different users may have requested.

Regarding software clients:

• esgpull has already been prepared to rely on esgvoc to reconstruct DRS-compliant paths for downloaded data.
• for Metagrid, I am not entirely sure of the current status (@sashakames can correct me if I'm wrong), but to my knowledge it only supported raw file downloads (similar to basic HTTP downloads) without rebuilding a DRS structure locally. I also do not believe DRS reconstruction was implemented in the wget scripts generated by Metagrid.
• for Globus, I am unsure how this is currently handled, but it would equally benefit from relying on esgvoc to generate or validate correct file paths where needed.

That said, I would not oppose the inclusion of such a field in the STAC item. However, I would find it more useful to provide the fully resolved path (i.e., the concrete DRS path with corresponding values for a given dataset or file) rather than a template that could later be misinterpreted. This information could easily be extracted by the publisher from the mapfiles, without even requiring esgvoc at that stage — assuming, of course, that the mapfiles were generated using esgmapfile, which itself relies on esgvoc.

[sashakames]

On the Metagrid side, we are reducing wget functionality to promote esgpull for more sophisticated use cases. If esgpull supports Globus transfers, I'd opt to enable that to build out directory structures for the transfer destination. We could add to the automated transfer backend but that feature is not immediately planned.

[sashakames]

Removed my assignment as no consensus yet reach to add the property. Is there a specific client requesting this, that information seems to have been lost.