Item level PID required in CMIP7, etc. schemas?

[sashakames]

My suggestion is that it should not be required. The DKRZ process will add it later (could be within seconds, minutes or eventual update). I had a PR to esgvoc for the .yaml.

If I'm in the minority, we could add a placeholder in the publisher and accept a non-formatted string instead of the prefix/UUID format

[glevava]

I would strongly discourage accepting a non-formatted string, as that would not constitute real validation. In the case of PIDs this is even more critical, since the format is well defined and unambiguous.

Just to make sure I understand correctly: does this mean that the dataset-level PID is assigned after the run of ESGF publisher? My understanding was that this was handled at publication time, during the ESGF publisher run and so would be part of the payload send to the STAC Transaction API for validation.

If the workflow is indeed that the publisher payload is validated by the STAC Transaction API first, and the dataset PID is then assigned later by DKRZ, do we actually need this property in the JSON schema at all?

[glevava]

Ok @sashakames I saw you comment on https://docs.google.com/spreadsheets/d/1YgPHT388GulZ4HeBLtWRGmbE7wyEDabHlPxMUe4pNLM/edit?gid=807067613#gid=807067613

[soay]

I think the item-level PID should be created at the time of publication, since it is an important field and should be included directly in the Index/STAC metadata.

For CMIP6, the UUID suffix of a dataset PID was generated from the master_id and the version, which ensures that the PID is reproducible at every step of the workflow. I think we should do the same for CMIP7.

[sashakames]

The Connector as part of the esgfpid package handles that for current projects. As that is coupled with RabbitMQ and we are discontinuing use, (correct?) new PID generation code is needed now.

I falsely assumed it was a random UUID rather than something deterministic. If there is code in esgfpid that could be reused without the connection, that would hopefully be straightforward to integrate.

[glevava]

OK, so it seems necessary that we keep the pid property at the dataset/item level in the JSON schema for validation in the STAC Transaction API.
If we agree on this point, then the current schema is valid and the issue can be closed.

[soay]

We will indeed discontinuing the use of the old RabbitMQ Setup (including esgfpid). Instead we will use Kafka to grab the information so there is no connection needed to the PID generation ion the publisher at all. Basically we only need to generate the dataset PID and add it to the metadata sent to Kafka.

[sashakames]

Hey @soay if you can point me to the code in esgfpid that does the generation, I can add that back to the publisher. Then any project can leverage it, eg. open question if we want pid for CORDEX-CMIP6.

[soay]

Hi @sashakames, It’s a bit scattered throughout the code but the following should work:

import uuid

prefix = "21.14100" #CMIP6
master_id = "CMIP6.ScenarioMIP.DKRZ.MPI-ESM1-2-HR.ssp126.r1i1p1f1.3hr.huss.gn"
dataset_version = "20190710"

hash_basis = master_id+'.v'+str(dataset_version)
ds_uuid = uuid.uuid3(uuid.NAMESPACE_URL, hash_basis)

dataset_pid = 'hdl:'+prefix+'/'+str(ds_uuid)

Thanks.

[sashakames]

@soay Perfect thank you, the UUID+hash line is exactly what I'd need to find!