Research guides
Compare web graph snapshots without inventing growth
A decision worksheet for comparing dated web graph editions while accounting for crawl windows, coverage, ranking changes and retained evidence.
To compare web graph snapshots, you need two retained result exports from different editions or independently prepared source datasets. LinkCensus currently serves only its active 2026-01 edition and has no edition selector, so the public UI cannot perform this comparison today. If you have two dated datasets, treat each as a measurement with its own coverage and processing rules before interpreting a candidate's movement.
Common Crawl's web graph release index lists dated host- and domain-level graphs with differing node and edge counts. LinkCensus uses dated, immutable editions; its method page explains the product's ranking and retained evidence. This protocol is for a future pair of saved LinkCensus exports or two independently prepared Common Crawl-derived datasets with comparable fields. It is not a current multi-edition product flow.
Define what changed
Choose a question before exporting results. Did a candidate enter the visible top rows? Did the ordering of the same candidates change? Did a retained shared linker appear or disappear? Each asks for a different comparison. A changed count of returned rows can reflect coverage or limits as well as changes in observed links.
Use neutral status labels: present in both, present only in the earlier export, present only in the later export, and unassessable. “New backlink” is a stronger statement requiring link-level evidence with appropriate crawl timing. “Competitor gained authority” is stronger still and cannot be derived from a rank move in one result list.
The graph provenance guide explains why a web graph is a derived record from crawled pages rather than a live inventory. The graph's own date and the date you ran the query are separate fields.
Align the comparison inputs
Store the query domain exactly as normalized by the product, edition identifier, source crawl window, query date, ranking mode, filters, row cap and export completeness. Then save candidate domain, displayed position, score if shown, and retained shared linkers. Compare like units: a host graph and a grouped-domain graph cannot be treated as the same population.
| Field | Why it matters | Action if different |
|---|---|---|
| Source and crawl window | Each graph samples pages from a particular period. | Describe the time boundary; do not call it a daily change. |
| Host or domain unit | Grouping changes the entities counted. | Rebuild a matched view or stop the count comparison. |
| Index coverage | A domain may be absent from one searchable set. | Mark unassessable until coverage is checked. |
| Ranking mode and filters | Different settings can reorder identical source data. | Rerun with identical settings. |
| Visible row cap | Falling below a cutoff is not necessarily disappearing from the graph. | Say “not in this export” rather than “lost.” |
| Retained source cap | LinkCensus shows shared linkers among top 200 retained. | Keep explanations bounded to the retained set. |
Common Crawl's web graph documentation describes the source graph and its aggregation. Use the product's displayed edition metadata for the specific result you are comparing; a newer Common Crawl release on the index does not mean that LinkCensus has built or loaded a corresponding edition.
Work through a fictional edition comparison
Suppose a fictional outdoor-equipment domain is queried in editions E1 and E2 with the same filters and ranking mode. The numbers below are invented for teaching and are not LinkCensus measurements or evidence of a real site's performance.
| Candidate | E1 position | E2 position | Safe first label |
|---|---|---|---|
summit.example | 1 | 3 | Present in both; moved within visible rows. |
ridge.example | 2 | 1 | Present in both; moved within visible rows. |
valley.example | 3 | Not in top four | Absent from E2 visible top four; deeper status unknown. |
trail.example | Not in top four | 2 | Entered E2 visible top four; earlier deeper status unknown. |
The two stable candidates, summit and ridge, allow a within-list ordering statement. The last two do not support an appearance or disappearance claim across the whole graph because the export stops at four rows. To ask whether a domain remained indexed or had retained shared sources, retrieve a sufficiently complete result for each edition if available. If that is unavailable, keep the status unknown.
Even with complete result rows, a rank change has multiple plausible causes: different observed links, crawl coverage, index eligibility, normalization or algorithm changes. The export alone does not isolate one cause. If the underlying method changed, compare the editions as distinct measurements rather than a clean trend.
Triage a changed candidate
- Confirm that the same normalized query and candidate names were used.
- Check whether both editions cover the candidate and whether the visible row limit hides it.
- Compare the retained shared-linker explanations and write down which exact source names differ.
- Open relevant current pages if the decision needs present-day relevance; record the inspection date.
- Label the conclusion as observed movement, possible explanation or verified external fact.
For a shared-linker difference, remember the top-200 retention rule. A source absent from one displayed list may have fallen outside retention, even if an underlying graph still contains a link. The retained-set explanation shows why overlap counts and ratios have different interpretation limits.
Calculate only matched-set differences
Build a table with one row per candidate domain found in either export, then mark whether each candidate was visible in E1, E2 or both. If both exports are complete for the chosen result scope, you can count the intersection and the candidates unique to each edition. If either export is truncated, the “only in” columns mean only “not visible within the cutoff.” Keep that phrase on charts and tables as well as in the methods note.
For the invented four-row table above, summit and ridge are visible in both: two matched candidates. Valley is visible only in the E1 top four; trail is visible only in the E2 top four. The example does not show a fourth E2 candidate, so even a top-four total is incomplete here. No graph-wide addition or loss count can be calculated from it. This deliberate missing row tests whether the analyst records export completeness before doing arithmetic.
Within the matched pair, one candidate moved from position 1 to 3 and the other from 2 to 1. You can report those ordinal moves exactly. Summing rank changes would be misleading: positions depend on other candidates and any filtered or hidden rows. A percentage increase in position also has no natural interpretation. If scores are shown, compare them only when the scoring method and scale are confirmed stable across editions.
Retained-source comparisons need the same discipline. Suppose an invented E1 explanation lists source A and source B, while E2 lists source A and source C. The visible intersection is A; B and C are edition-specific within the displayed retained sets. That does not prove B removed its link or C created a new one. The source pages may have changed, may have been crawled differently or may have crossed the retention cutoff. A link-level claim requires dated page-level records and a stated observation method.
Keep two output layers: a machine-checkable row table with exact export values, and a short interpretation note that labels coverage, cutoff and method uncertainty. This lets another analyst reproduce the arithmetic and challenge the explanation independently.
Write a claim another analyst can reproduce
Use this format: “For [query], with [mode, filters, row limit], [candidate] was at visible position [x] in [edition and crawl window] and [y or not visible] in [later edition and crawl window]. [Specific retained-source observation]. Coverage or cutoff limits: [state]. Current-site check: [page and date, if relevant].” Attach the two export files or their identifiers.
This is an edition comparison, not a forecast. If a stakeholder needs a current competitor shortlist, validate the candidate's present offer using the competitor research protocol. A graph movement may prioritize that review, but it does not settle the business relationship.
Stop rule: if source unit, ranking mode, filters or visible coverage cannot be aligned, report the two snapshots separately. Do not publish a growth percentage from incomparable denominators.
Does a missing domain mean it lost links?
No. It may be outside the visible result cap, absent from the searchable index, normalized differently or affected by different observed crawl data. Check those possibilities before investigating link-level loss. A dated graph can support a bounded statement about its own recorded relationships, not a complete claim about the current web.
