DEVOURED
Host- and Domain-Level Web Graphs July, August, and September 2026
Common Crawl released updated web graphs for Q3 2026, indexing 245.8 million hosts and 133.2 million domains for public research use.
What: The July, August, and September 2026 datasets are now available, providing host- and domain-level graph representations of the public web crawl.
Why it matters: Large-scale link graph data remains essential for training search algorithms and ranking models, acting as a crucial open-source counterweight to proprietary datasets maintained by Google or Microsoft.
Deep dive
- The release includes processed graphs at both the host and domain levels.
- Total coverage spans 245.8 million hosts across 133.2 million unique domains.
- Datasets are intended for network analysis, search engine training, and LLM development.
- The data follows previous monthly release cycles provided by Common Crawl.
- Access is provided through public S3 buckets as part of their ongoing commitment to open-access web crawling.
Decoder
- CDXJ: A compressed index format commonly used to store web crawl metadata (e.g., URL, timestamp, status code) enabling efficient lookup in large archive files.
- Web Graph: A data structure representing the hyperlink topology of the web, where nodes are pages or domains and edges are the links between them.
Original article
Full article content is not available for inline reading.
Explore all 53 articles in today's digest
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi