-
Notifications
You must be signed in to change notification settings - Fork 5.1k
Description
Page URL
https://docs.airbyte.com/platform/operator-guides/refreshes
Description
Summary
There appears to be a mismatch between the Airbyte refresh documentation and the behavior of the public Jobs API.
The refresh docs state that refreshes can be triggered via the Airbyte API at the stream or connection level.
Both kinds of Refresh Syncs can be triggered from the UI and via the Airbyte API at either the stream or connection levels (although both might not be available depending on your sync mode).
However, the public POST /v1/jobs endpoint appears to support only:
syncreset
And explicitly rejects refresh.
What I found
In airbyte-platform, the public API controller for jobs contains this logic in:
airbyte-server/src/main/kotlin/io/airbyte/server/apis/publicapi/controllers/JobsController.kt
JobTypeEnum.REFRESH is explicitly rejected with:
"Refreshes are not supported in the public API"
The code path appears to be:
SYNC-> allowedRESET-> allowedREFRESH-> throwsUnprocessableEntityProblem
Reproduction
Request
Send a request to the public jobs endpoint:
POST /v1/jobs
with a body using:
connectionIdjobType = REFRESH
Actual result
The request is rejected with an unprocessable entity error.
Expected result
One of these should be true:
REFRESHshould be supported by the public API if refreshes are documented as available via API, or- the docs should clarify that refresh is not supported via the public
POST /v1/jobsendpoint and is only available via another API surface, if applicable.
Why this matters
For large connections, reset is operationally much riskier than refresh.
reset clears destination data and then reloads it, which can leave the destination empty if the reload fails or takes a long time.
refresh is preferable for cases where destination data needs to be rebuilt without exposing that downtime risk.
This is especially relevant when trying to periodically reconcile incremental append tables that accumulate stale rows.
Current behavior appears inconsistent with docs
The public API reference for triggering a job indicates only sync and reset are supported for job creation.
At the same time, the refresh documentation says refreshes can be triggered via API.
From a user perspective, this makes it unclear whether:
- refresh is supposed to be publicly supported but is not yet implemented, or
- refresh is only available through some non-public or different endpoint.
Suggested resolution
Please clarify one of the following:
- Is
REFRESHintentionally unsupported in the publicPOST /v1/jobsAPI? - If yes, which supported API endpoint should be used to trigger refreshes programmatically?
- If no, is support planned for the public jobs endpoint?
- If refresh is not publicly supported, could the docs be updated to make that explicit?
Environment
- Airbyte repository checked:
airbytehq/airbyte-platform - Relevant file:
airbyte-server/src/main/kotlin/io/airbyte/server/apis/publicapi/controllers/JobsController.kt
References
- Public Jobs API controller rejects
REFRESH - Public API docs for job creation show
sync/reset - Refresh docs state refreshes can be triggered via API
Minimal issue statement
The public POST /v1/jobs endpoint rejects jobType=REFRESH with "Refreshes are not supported in the public API", but current refresh documentation suggests refreshes can be triggered via API. Can you clarify whether this is a documentation issue or missing public API support?
Internal Tracking: https://github.com/airbytehq/oncall/issues/11777