Data and methods
Common Crawl web graph: a guide to dated link data
Read a Common Crawl web graph correctly: nodes, links, host and domain scope, release dates and the limits of product editions.
A Common Crawl web graph represents observed links as connections between hosts or domains. It can support research into linking structure and related websites, but it is a dated dataset with a defined construction method. Reading it correctly requires separating the crawl period, graph release, product edition and date you inspect a result.
The practical question is usually whether a graph can support a particular conclusion. It may show that one domain linked to another within its source material. It cannot, by itself, establish current business ownership, website traffic or a page’s editorial intent.
Understand nodes and directed links
In graph terminology, a node is an entity and an edge is a connection. For a host-level web graph, a node represents a host. An edge from A to B indicates an observed link direction from the source host toward the target host. Incoming and outgoing links answer different questions, so preserve that direction when explaining a result.
Common Crawl publishes both host-level and domain-level graphs. Its official graph documentation, reviewed on 26 September 2026, also says technical links to resources such as images, JavaScript and fonts are included. You cannot assume every edge came from a writer recommending a useful article.
Imagine a fictional local newspaper linking to a community organization and loading a font from a resource host. Both relationships may matter to graph construction. Only one is a plausible lead for researching local organizations, and even that needs page context before it supports an endorsement claim.
Pages, hosts and domains count different things
A page URL identifies a resource. A host identifies a network name such as news.publisher.example. A registrable domain groups names according to registration boundaries, rather than always taking the final two labels. These distinctions determine what has been combined before you see a result.
The Public Suffix List explanation gives the background for boundaries such as co.uk. Common Crawl’s domain graph aggregates hosts at pay-level domain using that list. The host-to-domain grouping worksheet shows why raw hosts and private suffixes need explicit handling before a count is reported. This helps define the graph’s unit; it does not certify that two organizations share an owner or that a domain is currently active.
Aggregation can turn multiple host-to-host observations into a domain-level relationship. Once you are reading that relationship, you cannot recover the number of individual source pages merely by counting its edges. For a worked count reconciliation, see referring domains versus backlinks.
A dated Common Crawl web graph is not a live census
Keep the time boundary attached to the evidence. A website can change after capture. It can remove a link, redirect a domain, switch products or cease operating. A historical relationship remains evidence about the source artifact, but checking today’s website is a separate action.
The official release list contains multiple dated graphs. A new upstream release does not establish that a product using Common Crawl has downloaded, transformed, verified and activated that release. Each of those steps has a separate operational state.
Likewise, the date at the top of this article describes its editorial review. It does not refresh the dataset discussed in the article. Treating publication dates, lookup dates and capture periods as interchangeable makes a historical observation look more current than the evidence allows.
Use a five-stage provenance checklist
This editorial checklist helps you keep source and product transformations visible. You can use it when evaluating a research tool or documenting your own graph analysis. Fill unknown fields with “not established” rather than guessing from a release name.
| Stage | What to record | What it does not establish |
|---|---|---|
| Crawl captures | Collection or crawl period and source scope | The current state of every website |
| Host graph | Release identifier and edge construction rules | Page-level editorial context |
| Domain aggregation | Grouping rules and suffix-list basis | Corporate ownership |
| Product edition or index | Loaded artifact, transformations and retention | That the newest upstream release is active |
| Visible result | Query, filters, ranking mode and lookup date | Unfiltered or complete web coverage |
For example, if a product names an edition but does not expose its full build lineage, you can accurately record the edition while leaving other dates unresolved. That is preferable to inferring the crawl period from the day you opened the product.
Before comparing two exports, check this table for both. A change in retained evidence, filters or aggregation may explain a difference without any underlying website having changed. Comparisons need compatible units as well as different dates. The web-graph edition comparison applies those checks to two dated datasets; it does not imply that the current public UI can select past editions.
What a product can retain and expose
A product may transform the source graph to serve a narrower research task. It can create embeddings for ranking, retain selected incoming links for explanations and limit the number of visible results. Those choices affect what a user can infer from the output.
LinkCensus serves dated editions for related-domain research. Its method page describes the retained evidence boundary: shared linkers (among top 200 retained). That wording is part of the meaning of the count. Omitting it turns a bounded explanation into an unsupported claim of completeness.
Embedding results can exist without overlap in the retained source lists. Conversely, a shared-linker count describes an intersection of sources, not the embedding score itself. The co-citation calculation guide explains why retained overlap and normalized similarity also need separate interpretations.
Result exports and reports are output rows. They should not be assumed to contain the underlying graph, vectors or index. If your research requires reproducing a graph algorithm from source artifacts, check the available artifact and licensing terms directly before choosing a product workflow.
Work directly with the source when the task requires it
Start with Common Crawl’s official graph release page and its cc-webgraph tools repository. The repository documents graph-processing tools and domain-graph construction. Read the instructions for the specific operation and release rather than treating a generic download command as a complete analysis pipeline.
Direct graph work is suitable when you need control over transformations or want to investigate a question that a product’s retained output cannot answer. It also means taking responsibility for storage, processing and reproducibility. Hardware needs vary by artifact and operation; there is no useful universal “runs on any laptop” promise.
Write down the exact input artifacts, script version, filtering decisions and aggregation rules before drawing conclusions. Save the query and output definition with the result. A number without its counting method is difficult for another researcher to check.
Handle missing domains and ambiguous links
A missing domain is not proof that the site has no backlinks. It may be absent from the source material, excluded by construction rules, outside an indexed subset or unresolved by a product’s lookup process. Common Crawl documents exclusions including raw-IP URL hosts and hosts with invalid IANA top-level domains in host-graph construction.
For a present domain, inspect the kinds of sources behind a relationship. Technical services and broad directories can create shared context without making two sites close substitutes. Use the graph to decide what to inspect next, then verify any claim that requires current pages.
If you want candidates rather than raw graph analysis, start with a LinkCensus domain lookup and follow the similar-website validation workflow. Record the edition alongside the shortlist so later readers know which evidence supported it.
Common questions about the web graph
Can you search Common Crawl?
Common Crawl provides several kinds of data and access tools. For graph research, start with its graph release documentation and graph-processing repository. Searching graph relationships is different from retrieving the text of an archived page. Choose the dataset and access method that match the question before interpreting a result.
What does Common Crawl mean?
Common Crawl is the organization behind publicly available web crawl data and related artifacts, including hyperlink graphs. A product using those artifacts adds its own processing and serving choices. Attribute the source to Common Crawl while naming the product edition and limitations separately.
How is the web graph represented in graph theory?
A web graph uses nodes for entities such as hosts or domains and directed edges for observed links between them. The representation depends on the chosen unit and construction rules. A domain-level edge summarizes a relationship at that level; it does not preserve every page-level detail.
