Extension: artifact upload helper

[soxofaan]

related to:

• feature: artifact-helper openeo-python-client#750

We're working on an experimental feature (in the python client) to create an artifact upload helper, that uploads to existing file storage APIs (e.g. S3). Part of this feature is based on getting backend-specific configuration (buckets, S3 endpoints, ...) from the openEO capabilities document, which requires an API extension.

This ticket is to bootstrap the specification process for that.

[soxofaan]

cc @pvbouwel

[soxofaan]

as initial point of reference, something like this would be added to the openEO capabilities doc (GET /):

 {
       // ...
       "artifacts": {
         "providers": [
           {
             // This id is a logical name
             "id": "s3",
             // The config type of the ArtifactHelper
             "type": "S3STSConfig"
             // The config block its keys can differ for other config types
             "config": {
               // The bucket where the artifacts will be stored
               "bucket": "openeo-artifacts",
               // The role that will be assumed via STS
               "role": "arn:aws:iam::000000000000:role/S3Access",
               // Where S3 API calls are sent
               "s3_endpoint": "https://my.s3.test",
              // Where STS API calls are sent
               "sts_endpoint": "https://my.sts.test"
             },
           }
         ]
       },
       // ...
    }

[m-mohr]

How does this relate to https://github.com/Open-EO/openeo-api/tree/draft/extensions/workspaces ?

Would it make sense to add this to a separate endpoint instead of adding it to the capabilities?

[pvbouwel]

The artifacts capability is meant as a low-entry utility to stage temporary artifacts which you want to use during an OpenEO processing job but which does not need to be stored for an extended amount of time.

For workspaces you still have to manage the workspace and the lifecycle of objects within the workspace so it requires a bit more of investment which does not provide value for throw-away objects.

Adding it to a separate endpoint is possible and would make sense if capabilities is expected to grow over time.

[m-mohr]

I'm not sure whether a third(!) option in-between the /files endpoints and the Workspace API is needed? I think it would need a good justification.

[pvbouwel]

I had a conversation with Jeroen D. because he mentioned workspace API could still be an option if we consider a workspace that is by default available. So this is a workspace that does not need to be created/registered by users but rather is provided by the platform and every user can see this workspace. This would be a workspace without workspace_parameters but which has workspace_properties where the properties will expose the required details from #566 (comment).

[m-mohr]

The "workspace" that is available by default is /files, which can internally be mapped to an external cloud storage and be a proxy on top of it.