Support Push based transport mode - #79
Conversation
Co-authored-by: Sebastian Wennemers <sebastian.wennemers@sap.com>
|
|
||
| The `embedded` access strategy indicates that the resource definition content is provided inline within the ORD document itself, rather than being fetched from an external URL. | ||
|
|
||
| This is specifically designed for [push transport](../../spec-v1/index.md#push-transport) scenarios where: |
There was a problem hiding this comment.
Does "designed for" mean "limited to"? Embedded resources seem equally useful in pull mode — a provider might want to serve a self-contained ORD document without requiring the aggregator to make additional HTTP requests for resource definitions. IMO push vs. pull is purely a transport concern and shouldn't dictate what the document can contain.
|
|
||
| ## Description | ||
|
|
||
| The `embedded` access strategy indicates that the resource definition content is provided inline within the ORD document itself, rather than being fetched from an external URL. |
There was a problem hiding this comment.
Why inline instead of a ZIP file?
Embedded resources could easily push the document past the 2MB limit. Also it may greatly increase the memory requirements for parsing the document.
Example:
/
- MANIFEST.MF
- resource-1.ord.json
- resource-M.ord.json
- api-123.openapi.json
- api-XYZ.openapi.json
With a zip file you can push an arbitrary amount of resource definitions (and even ORD documents if it makes sense).
All files should be listed in the manifest. It acts as the document served at the "well-known" endpoint.
Optionally it can contain signatures, hashes, etc.
There was a problem hiding this comment.
Yes, I also thought about it. It would be a different approach, then we don't need embedding. Instead we use relative file links.
Those ZIP files could get quite big though and I'm not sure how ideal it is with regards to HTTP protocol which already does compression as part of the transport.
I see another alternative: POST/PUT all the files as part of a transaction, which then can be committed via separate action on the API. Then there can be a GET on the transaction to also receive feedback.
This would be closer to HTTP conventions and reuse of how the protocol works, also would be more async. It requires some transaction management on aggregator side, though.
| - Pushing ORD documents to an aggregator (push transport) | ||
| - Integrating ORD publishing into CI/CD pipelines | ||
| - The provider cannot or does not want to host resource definitions at accessible URLs | ||
| - You want to ensure atomic updates of metadata and definitions together |
There was a problem hiding this comment.
Should atomic "all-or-nothing" semantics be a hard requirement here? If a push partially fails (e.g. some resource definitions are invalid), should the aggregator reject the entire payload or accept the valid parts?
This needs an explicit statement in the spec — partial updates could leave an aggregator in an inconsistent state, but strict atomicity may be impractical for large payloads.
There was a problem hiding this comment.
Good point, so far we left it to the aggregator as an implementation detail. I think there's some "oppinion" in how strong this is fulfilled.
I'll keep it in mind, but defining this would be probably a general enhancement, not just for the push based appraoch.
| For push transport, the standard [ORD document](#ord-document) format is used with one addition: a `definitions` property that allows inline [resource definitions](#resource-definition). | ||
|
|
||
| When using pull transport, resource definitions are referenced via URLs and fetched separately by the aggregator. | ||
| In push transport, these definitions can be provided inline within the ORD document itself using the `definitions` property. |
There was a problem hiding this comment.
There's no need to limit this to push transport. Most aggregator implementations will have a single document-handling pipeline regardless of how the document arrived — the transport layer should be transparent to the processing logic. Artificially restricting embedded resources to push mode adds complexity for no benefit.
There was a problem hiding this comment.
Agreed, it would also make sense on pull.
Where I'm not sure: It would be a new feature not every consumer would support, so it's an incompatible change if we backport it to pull transport. So it feels a bit dangerous to introduce it to pull without having a major release (which I'd like to avoid).
Conceptually, it would of course work for both transport modes.
| - All URLs in the document MUST be resolvable (e.g. through `baseUrl` or full URLs) | ||
|
|
||
| #### Push Transport | ||
| ### Push Transport |
There was a problem hiding this comment.
This proposal feels underspecified for enabling reusable, interoperable aggregators. For anyone to build a generic aggregator that works across providers, there needs to be a standard push protocol — not just a payload shape. Concretely, that means defining:
- A standard push endpoint (or a standard discovery mechanism for it, e.g. a well-known)
- Transport mode — multipart, streaming, ZIP, plain JSON body? This matters for the 2MB-per-document constraint: embedding resource definitions inline as strings inside a JSON payload doesn't scale, and the choice of transport directly determines whether large payloads are even feasible
- Sync vs. async communication — I'd strongly advocate for async here. A fire-and-forget push with a callback/webhook/polling for validation results would allow providers to push arbitrarily large amounts of data without blocking the CI/CD pipeline on aggregator processing time. Sync push only works for small, fast payloads; async is the only model that scales
Without these being standardized, every aggregator will implement its own variant and the push mechanism won't be reusable — which defeats the purpose.
There was a problem hiding this comment.
Yes, that's on point. Right now it's underdefined, reason is that we would need to build / prototype this along with defining here in the spec and that hasn't happened yet. I agree that those questions need to be clarified.
On the async side, it comes with some drawbacks like not getting errors / problems quick enough. But since the aggregator needs a bit time to validate, maybe it's more realistic to batch more and then also receive validation asynchronously later. It would be a more complicated pattern than just sync push with direct error feedback.
Push Transport Mode for ORD
This PR adds Push Transport as a new transport mode for ORD, enabling providers to push ORD documents directly to aggregators via HTTP POST requests.
Summary
Push transport eliminates the need for providers to implement and host an ORD Provider API. Instead, providers can:
definitionspropertyChangess
definitionsproperty at document level for inline resource definitionsx-introduced-in-version: 1.15.0withbetastatusembeddedaccess strategy type for resource definitionsDesign Decisions
definitionsas dictionary - URL path as key enables direct correlation withresourceDefinitions[].urlembeddedaccess strategy - Explicitly signals that content is inline, not to be fetched.well-knownendpoint will be addressed in a separate PRRelated / Follow Up