Repositories list

• cc-index-annotations

  Public

  commoncrawl/cc-index-annotations’s past year of commit activity
  Example code to join an annotation to a host or url index

  Python

  •0•1•0•1•Updated Mar 21, 2026Mar 21, 2026

• cc-index-table

  Public
  Index Common Crawl archives in tabular format

  sqlsparkcolumnar-storage+ 3

  sqlsparkcolumnar-storageaws-athenaapache-parquetcommoncrawl

  Java

  •

  Apache License 2.0

  •15•126•6•1•Updated Mar 20, 2026Mar 20, 2026

• cc-webgraph

  Public
  Tools to construct and process Common Crawl webgraphs

  pagerankwebgraphcommoncrawl+ 3

  pagerankwebgraphcommoncrawlcommon-crawlcentrality-measureswebgraph-framework

  Java

  •

  Apache License 2.0

  •6•105•7•0•Updated Mar 20, 2026Mar 20, 2026

• ipv6-analysis

  Public
  A survey of IPv6 support among the 100k top ranked hosts on the web

  Python

  •

  Apache License 2.0

  •0•1•0•0•Updated Mar 19, 2026Mar 19, 2026

• cc-crawl-statistics

  Public

  commoncrawl/cc-crawl-statistics’s past year of commit activity
  Statistics of Common Crawl monthly archives mined from URL index files

  statisticscommoncrawlcommon-crawl

  statisticscommoncrawlcommon-crawl

  Python

  •

  Apache License 2.0

  •16•212•3•3•Updated Mar 19, 2026Mar 19, 2026

• warcio-s3

  Public

  commoncrawl/warcio-s3’s past year of commit activity
  Streaming WARC/ARC library for fast web archive IO

  Python

  •

  Apache License 2.0

  •68•0•0•0•Updated Mar 19, 2026Mar 19, 2026

• cc-citations

  Public
  Scientific articles using or citing Common Crawl data

  bibtexbibliographyopendata

  bibtexbibliographyopendata

  Jupyter Notebook

  •4•28•0•0•Updated Mar 19, 2026Mar 19, 2026

• cc-nutch-example

  Public
  Apache Nutch example project to archive content in WARC files

  Shell

  •

  Apache License 2.0

  •2•3•0•0•Updated Mar 17, 2026Mar 17, 2026

• awesome-low-resource-languages

  Public

  commoncrawl/awesome-low-resource-languages’s past year of commit activity
  This list provides resources useful for documenting, conserving, developing, preserving, or working with endangered and low resource languages.

  0•0•0•0•Updated Mar 16, 2026Mar 16, 2026

• cc-webgraph-statistics

  Public

  commoncrawl/cc-webgraph-statistics’s past year of commit activity
  Statistics of Common Crawl monthly Web Graphs

  JavaScript

  •

  Apache License 2.0

  •1•5•0•1•Updated Mar 16, 2026Mar 16, 2026

• cc-web-graph-neo4j

  Public

  commoncrawl/cc-web-graph-neo4j’s past year of commit activity
  Instructions and code for using the Common Crawl Web Graph in Neo4j format

  Shell

  •0•2•0•0•Updated Mar 16, 2026Mar 16, 2026

• cc-warc-examples

  Public
  CommonCrawl WARC/WET/WAT examples and processing code for Java + Hadoop

  javahadoopmapreduce+ 1

  javahadoopmapreducecommoncrawl

  Java

  •

  MIT License

  •45•38•0•0•Updated Mar 12, 2026Mar 12, 2026

• nutch

  Public

  commoncrawl/nutch’s past year of commit activity
  Common Crawl fork of Apache Nutch

  javabig-datahadoop+ 2

  javabig-datahadoopweb-crawlercommoncrawl

  Java

  •

  Apache License 2.0

  •1.3k•40•11•3•Updated Mar 12, 2026Mar 12, 2026

• ia-web-commons

  Public
  Web archiving utility library

  cdx-fileswarc-fileswat-files

  cdx-fileswarc-fileswat-files

  Java

  •

  Apache License 2.0

  •76•11•4•1•Updated Mar 11, 2026Mar 11, 2026

• cc-downloader

  Public
  A polite and user-friendly downloader for Common Crawl data

  rustdownloadercommoncrawl

  rustdownloadercommoncrawl

  Rust

  •

  Apache License 2.0

  •4•71•8•1•Updated Mar 3, 2026Mar 3, 2026

• eot2020-host-index

  Public
  Tools to work with the preliminary End of Term Archive host index

  Python

  •0•0•0•0•Updated Mar 2, 2026Mar 2, 2026

• language-detection-cld2

  Public
  Natural language detection, Java bindings for CLD2

  natural-languagelanguage-detectionlanguage-identification

  natural-languagelanguage-detectionlanguage-identification

  Java

  •

  Apache License 2.0

  •2•17•3•1•Updated Feb 26, 2026Feb 26, 2026

• whirlwind-java

  Public
  A whirlwind tour of Common Crawl's data using Java

  Java

  •

  Apache License 2.0

  •0•3•2•0•Updated Feb 26, 2026Feb 26, 2026

• cc-citations-paper-explorer

  Public
  A visual paper explorer based on cc-citations. https://huggingface.co/spaces/commoncrawl/cc-citations

  embeddingsldacommoncrawl+ 1

  embeddingsldacommoncrawlscincl

  Python

  •

  Apache License 2.0

  •0•0•0•0•Updated Feb 19, 2026Feb 19, 2026

• whirlwind-python

  Public
  A whirlwind tour of Common Crawl's data using Python

  pythontutorialarchive+ 1

  pythontutorialarchivewarc

  Python

  •

  Apache License 2.0

  •8•37•0•1•Updated Feb 17, 2026Feb 17, 2026

• presentations

  Public
  A collection of public presentations from the Common Crawl Foundation

  1•11•0•0•Updated Feb 2, 2026Feb 2, 2026

• webarchive-indexing

  Public
  Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.

  cc-index

  cc-index

  Python

  •

  MIT License

  •12•6•0•0•Updated Jan 27, 2026Jan 27, 2026

• cc-mrjob

  Public archive
  Demonstration of using Python to process the Common Crawl dataset with the mrjob framework

  pythonhadoopmap-reduce+ 1

  pythonhadoopmap-reducecommoncrawl

  Python

  •

  MIT License

  •78•168•2•2•Updated Jan 27, 2026Jan 27, 2026

• cdx_toolkit

  Public
  A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine

  pythonwarcweb-archiving+ 4

  pythonwarcweb-archivingcdxweb-archivescommoncrawlcdx-api

  Python

  •

  Apache License 2.0

  •34•201•6•4•Updated Jan 23, 2026Jan 23, 2026

• cc-vec

  Public
  Python

  •

  MIT License

  •2•5•0•1•Updated Jan 20, 2026Jan 20, 2026

• cc-pyspark

  Public
  Process Common Crawl data with Python and Spark

  sparkpysparksparksql+ 5

  sparkpysparksparksqlwetcommoncrawlcommon-crawlwarc-fileswat-files

  Python

  •

  MIT License

  •93•453•4•3•Updated Jan 20, 2026Jan 20, 2026

• robotstxt-experiments

  Public
  How is the Robots Exclusion Protocol (robots.txt) used in the WWW? This projects tries to get some insights mining Common Crawl's robots.txt captures of the yea…

  robots-txtrobotstxt

  robots-txtrobotstxt

  Jupyter Notebook

  •

  MIT License

  •0•0•0•0•Updated Jan 15, 2026Jan 15, 2026

• web-languages

  Public
  Crowd-sourced lists of urls to help Common Crawl crawl under-resourced languages. See https://github.com/commoncrawl/web-languages-code/ for the code

  crawlinglanguage-detectiondataset

  crawlinglanguage-detectiondataset

  87•69•3•1•Updated Jan 7, 2026Jan 7, 2026

• whirlwind-python-notebook

  Public
  A jupyter notebook illistrating the basics of Common Crawl's datasets.

  awss3open-datasets+ 1

  awss3open-datasetssagemaker

  Jupyter Notebook

  •

  Apache License 2.0

  •0•3•0•0•Updated Dec 31, 2025Dec 31, 2025

• crawler-commons

  Public
  A set of reusable Java components that implement functionality common to any web crawler

  Java

  •

  Apache License 2.0

  •91•2•0•2•Updated Dec 4, 2025Dec 4, 2025