Reporting progress of run operation

[niquola]

Let's discuss how to report progress of async operation:

• FHIR Bulk Export API uses x-progress header
• Another option is to use response body with special elements

@smunini

For the in-progress state, we should harmonize with the response headers that Bulk Data
$export uses, notably X-Progress, and Retry-After. See here

@niquola

Respond with FHIR Parameter object:

{
  "resourceType": "Parameters",
  "parameter": [
     { "name": "status", "valueString": "in-progress" },
     { "name": "message", "valueString": "Processing large source (65% complete)" },
     { "name": "progress", "valueInteger": 65 },
     { "name": "time", 
       "part": [ 
       { "name": "start", "valueInstant": "2025-02-25T15:10:23Z" },
         { "name": "end", "valueInstant": "2025-02-25T15:12:23Z" },
         { "name": "duration", "valueInteger": 120 }
      ]
    }
  ]
}

Respond with JSON object with the following fields:

Field	Type	Description
status	string	Status of the export ("in-progress")
progress	number	Progress of the export (0-100)
time	object	Timing information
time.start	datetime	Start time of export
time.end	datetime	Estimated end time of export
time.duration	number	Duration in seconds
message	string	Human readable status message

{
    "status": "in-progress",
    "message": "Processing large source (65% complete)",
    "progress": 65,
    "time": {
        "start": "2025-02-25T15:10:23Z",
        "end": "2025-02-25T15:12:23Z",
        "duration": 120
    }
}

Links

• FHIR Ansync - https://build.fhir.org/async-bundle.html
• Bulk API - https://build.fhir.org/ig/HL7/bulk-data/export.html#endpoint-1

[niquola]

ChatGPT o1

Using headers for progress data is lightweight but limited. It works well for simple use cases, such as indicating just a numeric percentage. However, custom headers may require extra parsing logic on the client side, and expanding beyond a single value (e.g., to include logs or multiple fields) can quickly become unwieldy. Headers also aren’t easily updated multiple times mid-response without more advanced streaming techniques.

Including progress information in the response body, by contrast, is more flexible and aligns with standard RESTful patterns. Structured data formats like JSON allow for richer and more detailed status updates (e.g., sub-steps, estimated completion times), and typical client libraries can easily parse it. In many scenarios, you’ll return 202 Accepted along with a resource or endpoint to poll, and the client retrieves progress from the body of subsequent GET responses or a streaming mechanism like Server-Sent Events.

[niquola]

FHIR Parameter looks more FHIRish solution. @gotdan what do you think? Why do you use Custom JSON in Bulk API?

[gotdan]

My comment from Zulip (tldr is that bulk export pre-dated this looking FHIRish :)

We created $export before the Parameter resource was widely used or integrated into any FHIR tooling, and implementers thought it added a layer of abstraction to the output with no real value in exchange. If we were starting today, I'd probably push for it for the tooling benefit in things like the IG Publisher, but the current manifest format is already widely supported in clients and servers.

We did add Parameter resource support to POST kickoff request in v2 since there are more clear benefits there (e.g., defining how to represent references in a request). The medium term plan is to deprecate GET kickoff requests and just have this, but we're trying to be very careful about breaking backward compatibility when there isn't a clear gain in terms of functionality.

[gotdan]

For the response resource described above, it might be nice for the names of the parameters for end and duration to indicate that they're estimated vs. start which is actual (e.g., endEstimate and durationEstimate)

Also, if the operation is compatible with Bulk Export overall (e.g., by also requiring the progress headers here), existing clients could be used, though maybe this isn't a goal?

[johngrimes]

@niquola Do you want to incorporate this into your Unified Async Operations Pattern?