ParquetMetaDataPushDecoder API to clear all buffered ranges#9673
ParquetMetaDataPushDecoder API to clear all buffered ranges#9673nathanb9 wants to merge 3 commits intoapache:mainfrom
Conversation
| } | ||
|
|
||
| /// Clear all buffered ranges and their corresponding data | ||
| #[cfg(feature = "arrow")] |
There was a problem hiding this comment.
Not useful i think.
Also, how does the CI work? Does it use the pr label to figure out which feature flags to use?
There was a problem hiding this comment.
Well, the feature gate is necessary without the changes in this PR 😉.
IIRC the CI will do builds with various sets of features enabled. The actual unit tests are then run with default features and all features.
etseidl
left a comment
etseidl
left a comment
There was a problem hiding this comment.
Thanks for the submission @nathanb9. Seems logical to me.
If it's not to much trouble could you also add an issue with a motivating use case? That aids in documentation for releases.
I also think it would be nice to add an example of the use of this API to the documentation at the head of push_decoder.rs.
| } | ||
|
|
||
| /// Clear all buffered ranges and their corresponding data | ||
| #[cfg(feature = "arrow")] |
There was a problem hiding this comment.
Well, the feature gate is necessary without the changes in this PR 😉.
IIRC the CI will do builds with various sets of features enabled. The actual unit tests are then run with default features and all features.
2 participants
This PR is a follow up for this ticket . Implement same API but for the metadata decoder.
Rationale for this change
ParquetMetaDataPushDecoderclears exact requested ranges, but largerspeculative pushed ranges can remain buffered in
PushBuffers. Thisadds a way for callers to explicitly release non exact ranges
What changes are included in this PR?
This adds
clear_all_ranges(), which clears all byte ranges stillstaged in the decoder's internal
PushBuffersAre these changes tested?
yes
Are there any user-facing changes?
Yes, this adds a new public
clear_all_ranges()API onParquetMetaDataPushDecoder