Is your feature request related to a problem? Please describe.
Since crawl data and indexes are now partially also available on the Hugging Face bucket, the cc-downloader should also support downloading from HF.
Describe the solution you'd like
HF provides a S3-compatible gateway to the HF buckets so we can likely reuse the existing S3 tooling: https://huggingface.co/docs/hub/storage-buckets-s3
HF also imposes rate limits (URL resolve) that the downloader needs to be aware of. https://huggingface.co/docs/hub/en/rate-limits
As part of the implementation, we should do throughput benchmarking from different cloud or edge locations to measure the effect of HF's CDN.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
Is your feature request related to a problem? Please describe.
Since crawl data and indexes are now partially also available on the Hugging Face bucket, the cc-downloader should also support downloading from HF.
Describe the solution you'd like
HF provides a S3-compatible gateway to the HF buckets so we can likely reuse the existing S3 tooling: https://huggingface.co/docs/hub/storage-buckets-s3
HF also imposes rate limits (URL resolve) that the downloader needs to be aware of. https://huggingface.co/docs/hub/en/rate-limits
As part of the implementation, we should do throughput benchmarking from different cloud or edge locations to measure the effect of HF's CDN.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.