Download of web graph datasets

[malteos]

Is your feature request related to a problem? Please describe.

I want to use cc-downloader to download the Common Crawl web graph datasets (https://index.commoncrawl.org/web-graphs-index.html).

Describe the solution you'd like

I want to provide the dataset ID like cc-main-2026-mar-apr-may and cc-downloader takes care about fetch the paths and then downloading all the files. Maybe support filtering for subsets (host vs domain level graph etc.)

Describe alternatives you've considered

N/A

Additional context

N/A

[sebastian-nagel]

See also graph_explore_download_webgraph.sh.

Maybe support filtering for subsets (host vs domain level graph etc.)

The webgraph datasets contain the same content in different formats. The graphs are represented either as adjacency lists (edges files, gzipped text) or in the WebGraph format. You would use one of the two formats, but hardly both together.