Clean source hosts before grouping link data

Clean source hosts before grouping link data. Preserve each raw source value, parse a valid hostname, normalize its spelling, and only then apply a declared public-suffix rule. A two-label shortcut can merge different site tenants under github.io or produce co.uk as a false source domain. The worksheet below shows each transformation and keeps rejected rows visible.

This is a data-cleaning procedure for an analyst's source list. It is not a claim that a LinkCensus result export exposes raw page URLs or implements this exact workflow. If you only need the difference between a backlink, a referring host and a referring domain, start with the counting-unit guide. Here the task is to make a mixed input column safe enough to count.

Keep the raw source before cleaning

Create separate raw_value, parsed_host, normalized_host, grouped_domain and status columns. Do not replace the original column in place. A reviewer needs to see whether a change came from URL parsing, capitalization, internationalized-domain conversion, suffix grouping or a manual exception.

Choose one source meaning for the column. It might contain page URLs from a crawl, hostnames from a graph or domains copied from a report. If both URLs and hostnames are allowed, document how each is parsed. Never infer a source host from display text such as an organization name. Keep the input file, dataset or export date, and relevant target scope beside the transformation rule.

For URL input, a source page is still useful after extracting its host: retain the path and original value if someone later needs to inspect the placement. Removing the path for host grouping is a deliberate projection, not a declaration that two pages are the same link. The graph provenance guide explains where page-to-host aggregation changes what can be observed.

Parse URLs into validated hostnames

Use a URL parser for a URL and a hostname validator for a bare hostname. Extract the hostname field, not the last word or the last two dot-separated labels of the entire string. A port, path, query string and fragment are not hostname labels. For example, https://News.Publisher.Example.com.:443/story?ref=mail yields news.publisher.example.com after hostname extraction, lowercasing and removal of the trailing DNS dot.

Write down accepted schemes and failure behavior. LinkCensus's current query normalizer accepts HTTP or HTTPS URLs and hostnames, rejects credentials and malformed ports, lowercases the host, removes a trailing dot and encodes internationalized labels to ASCII before label validation. That is a checked implementation detail of the current local query parser, not a promise that every third-party export uses the same rules. Your own cleaning script should emit a reason for each rejected row instead of silently dropping it.

For internationalized names, choose one stable representation for comparisons. The Public Suffix List format guidance says to canonicalize hostnames and rules to lowercase Punycode before matching. Retain a readable display name separately if needed. Do not group together lookalike Unicode spellings simply because they look alike on screen.

Make private suffix handling explicit

A public suffix is a registration boundary, not simply a top-level domain. The Public Suffix List includes co.uk, so blog.shop.example.co.uk groups to example.co.uk, not co.uk. The list also has ICANN and PRIVATE sections. The current official list file puts github.io in PRIVATE.

That private entry creates a consequential choice. With PRIVATE rules included, alpha.github.io and beta.github.io are separate grouping keys. Without PRIVATE rules, both can collapse to github.io under an ICANN-only interpretation. Neither output should be hidden behind an unlabeled “domain” column. For a study of independently named sites hosted on a shared platform, including private suffixes prevents an obvious platform-wide merge.

Pin the rule set used for the report: PSL retrieval or bundled snapshot date, whether PRIVATE entries were included, and the library or script version. A later list revision can change the result for some names. If you compare two datasets, either process both with the same pinned rule set or explicitly identify changed grouping as a method effect.

Work a before-and-after grouping table

The five rows below are invented input for a cleaning exercise. They are not observed backlinks or LinkCensus results. The example assumes a suffix-aware grouping rule with PRIVATE entries enabled, after a valid hostname has been parsed and normalized. It shows raw-to-normalized transformations rather than another backlink-versus-domain count.

These five validated host values yield four distinct grouping keys under the stated rule: example.com, alpha.github.io, beta.github.io and example.co.uk. An ICANN-only run would merge the two GitHub Pages examples into github.io, leaving three keys. A naive final-two-label rule would also report three keys, but one of them would be the invalid grouping co.uk; a matching count does not make that output correct.

The worksheet intentionally shows a surprise: the first two raw values have different-looking prefixes yet group to example.com, because com is the suffix and example is the next label. If the analyst intended publisher.example.com to remain an independent publication, that is a manual editorial classification after technical grouping. Do not quietly alter the suffix algorithm to force the desired result.

Quarantine rows the rule cannot settle

Add a separate exception table for blank strings, local hostnames without a recognized suffix, malformed URLs, invalid labels, and names the parser cannot classify. Keep the raw value, failure reason and row count. A total that silently omits rejected sources is hard to audit and can create a false difference from another report.

Also separate technical exceptions from ownership judgments. A PSL grouping key answers where a registration boundary lies under the chosen rules. It does not tell you who writes the site, whether two hosts share management or whether a platform tenant is a commercial competitor. For those questions, follow the site-operator evidence checklist or the competitor shortlist method.

If a stored host is unusual, avoid asserting that LinkCensus rejects it solely because the query form would. The local service has a strict user-query normalizer and a more tolerant display and grouping path for stored hosts. This distinction is why this article labels an analyst's own transformation rule and keeps rejected rows visible.

Publish a reproducible measurement note

Attach a short methods note to any grouped result: input source and date; accepted row type; URL parsing and hostname validation rules; IDN representation; PSL snapshot and PRIVATE setting; number of input, accepted and rejected rows; grouped output count; and any manual overrides. Share a small before-and-after sample like the table above. If source privacy or licensing prevents sharing rows, state that limit and keep the method and counts reviewable.

For a comparison, rerun both source lists through the same transformation. Compare raw row counts, accepted host counts and grouping-key counts separately. If a difference appears only after grouping, inspect the keys and exception log before calling it growth or loss. The dated-edition comparison protocol adds the coverage and ranking checks needed when the source itself changes.

Should every subdomain collapse to its parent?

No. A declared PSL rule may group ordinary subdomains under one registrable key while treating names below a private suffix as separate keys. Your decision may require an additional editorial or operating-entity classification. Keep that classification in another column and preserve the normalized host so the two questions are never mistaken for the same one.