Domain Listening: How to Monitor Domain Data at Scale

Most questions about the web are really questions about change. Which TLDs are growing. Whether a CMS is gaining or losing ground. How many domains moved behind a particular CDN this quarter. None of those can be answered by a single lookup, and none of them can be answered by a single file either — they need the same population measured twice. That practice, taking periodic snapshots of domain data and diffing them, is what this guide calls domain listening. It is unglamorous, it is entirely mechanical, and it is the only reliable way to see the web move.

TL;DR / KEY TAKEAWAYS

• Domain listening means monitoring a population of domains over time by comparing snapshots, rather than querying individual names.
• The unit of work is a file, not a query. WebTrackly publishes 1,538 packages — 716 TLD zone files, 79 technology and CMS lists, and 27 curated datasets — as CSV inside a ZIP.
• Scale is the point: the .com zone package contains 157,775,115 domains, the WordPress list 21,639,326, and the all-registered-domains dataset 272,614,863 rows.
• Files are generated at purchase, so each download is a fresh extract rather than a cached snapshot. That is what makes period-over-period comparison meaningful.
• Analysis happens on your side. There is no query builder, no per-domain lookup, and no filter UI. You buy a file and process it with your own tools.
• Domain data only: names, registration dates where published, name servers, MX, IP, and detected technology. No personal contacts, no phone numbers, no intent signals.
• Cost of entry is low: one-time packages from $3.50, Pro at $29/month, Enterprise at $99/month.

Table of Contents

• What Domain Listening Actually Is
• What You Can Measure With Snapshots
• What Is in the Files
• Buying and Processing a Package
• Building a Diff Workflow
• Automating with the Catalog API
• Common Mistakes in Domain Listening and How to Avoid Them
• Frequently Asked Questions
• Related Resources

What Domain Listening Actually Is

Domain listening is the practice of observing a defined population of domains over repeated intervals and recording what changed between them. The population might be an entire TLD, every domain detected as running a given CMS, or every domain resolving to a particular set of name servers. The interval might be weekly, monthly, or quarterly depending on how fast the signal you care about moves.

The mechanics are the same regardless of subject. You take a snapshot, store it, take another snapshot later, and compute three sets: rows present in both, rows that appeared, and rows that disappeared. Everything interesting comes out of those three sets. Growth is the appeared set. Churn is the disappeared set. Migration is a row present in both with a changed field.

This is a different discipline from per-domain lookup. WHOIS and RDAP answer "what does the registry say about this name right now", and they are rate-limited precisely so nobody uses them to enumerate a population. A zone file or a technology list is the population, delivered as a file. The trade-off is directness: you cannot ask a file a question until you have loaded it, but once loaded you can ask it questions no lookup protocol will ever answer.

Two practical consequences follow. First, storage discipline matters more than query cleverness — if you throw away last quarter's file, you cannot recover last quarter's state. Second, the comparison is only valid if both snapshots were produced the same way. Comparing a fresh extract against a vendor's stale cached copy measures the vendor's caching policy, not the web.

What You Can Measure With Snapshots

The following are all answerable from two or more snapshots of the same package. None of them require contact data, and none of them require a query interface.

TLD growth and churn

Diff a zone file against an earlier copy of itself. New rows are registrations that appeared in the zone; missing rows are names that dropped out. Aggregate by month and you have a registration trend for that TLD that does not depend on anyone's published statistics.

Platform market share over time

The technology packages group domains by the platform detected on them. Counting rows in the WordPress package at two points in time gives an absolute movement figure. Counting the same for several platforms and normalising against the zone file for a TLD gives share within that TLD. This is one of the few ways to size a platform's footprint without relying on the platform's own marketing numbers.

Infrastructure migration

Where a dataset carries name server, MX, or IP fields, a row present in both snapshots with a changed value is a migration event. Aggregated, this shows which DNS providers, mail providers, and hosting networks are gaining domains and which are losing them.

Population sizing before a build

Before committing engineering effort to an integration or a niche product, the row count of a technology package is a direct upper bound on the addressable set of sites. That number is a fact about the file, not a projection.

Research and threat-surface work

Security research frequently starts with "which domains resolve into this netblock" or "which domains run this platform". Bulk files answer both as a filter over a column. What they do not tell you is software version or patch level — that requires probing the hosts yourself, and the files are the target list for that work rather than a substitute for it.

Start listening.
1,538 packages — 716 TLD zone files, 79 technology lists, 27 datasets. CSV in a ZIP, generated at purchase, from $3.50.
Browse the catalog → | View pricing →

What Is in the Files

The catalog has three groups:

• Zone files by TLD — 716 packages. .com alone contains 157,775,115 domains.
• Technology and CMS lists — 79 packages of domains grouped by detected platform. WordPress contains 21,639,326 domains.
• Curated datasets — 27 packages, including all registered domains at 272,614,863 rows.

Columns depend on the package. A zone file is narrow; a resolved dataset carries the DNS and IP fields. The table below describes the shape of the data rather than guaranteeing every column in every package — each listing shows its own row count and contents before purchase.

What the files do not contain: names of people, email addresses, phone numbers, firmographics, or behavioural intent signals. WebTrackly publishes domain data. Registrant personal data is redacted at the registry level for most domains under GDPR, and it is not part of the product.

Buying and Processing a Package

1. Pick the package. Browse the catalog, or go directly to /zones/, /domaindata/, or /datasets/. Each listing states its row count.
2. Buy it. One-time purchases start at $3.50. Pro is $29/month (50 packages, 10 datasets, 30,000 API calls); Enterprise is $99/month (200 packages, 50 datasets, 300,000 API calls). See pricing.
3. Download. The ZIP is produced on demand and available immediately after payment.
4. Process it locally. Unzip and work with the CSV in your own environment.

At these row counts, streaming tools beat in-memory ones by a wide margin:

unzip wordpress-domains.zip
wc -l wordpress-domains.csv

# top name servers, straight from the CSV
duckdb -c "SELECT ns, count(*) AS n FROM 'wordpress-domains.csv'
           GROUP BY ns ORDER BY n DESC LIMIT 20"

For anything you intend to query repeatedly, load once into PostgreSQL or ClickHouse and index the columns you filter on. Joining two packages — a technology list against a zone file, for instance — is a join on the domain column.

Building a Diff Workflow

A monitoring setup is three things: a naming convention, a place to keep the snapshots, and a diff step.

Name by package and date. com-zone-2026-07.csv, wordpress-2026-07.csv. The date is the download date, not the analysis date.

Keep the raw files. Compressed CSV is cheap, and a snapshot you deleted cannot be regenerated retroactively — the file reflects the state at the moment it was produced.

Diff on the key column. For an appeared/disappeared analysis, sorted comparison is enough:

cut -d, -f1 wordpress-2026-06.csv | sort -u > old.txt
cut -d, -f1 wordpress-2026-07.csv | sort -u > new.txt

comm -13 old.txt new.txt > appeared.txt   # in new, not in old
comm -23 old.txt new.txt > disappeared.txt # in old, not in new
wc -l appeared.txt disappeared.txt

For field-level change detection — a domain that stayed but moved name servers — do it in SQL:

SELECT n.domain, o.ns AS old_ns, n.ns AS new_ns
FROM new_snapshot n
JOIN old_snapshot o USING (domain)
WHERE n.ns IS DISTINCT FROM o.ns;

Two cautions. Normalise before comparing: lowercase the domain, strip trailing dots, and settle on one encoding for internationalized names, or you will manufacture churn that does not exist. And treat a single interval as noise — the useful signal is the trend across several intervals, not the delta between two arbitrary files.

Automating with the Catalog API

Subscriptions include API access for working with the catalog programmatically — listing packages and datasets and reading their metadata. Pro includes 30,000 calls per month, Enterprise 300,000. The API is a catalog interface; it does not perform per-domain lookups.

curl -H "Authorization: Bearer YOUR_API_KEY" \
  "https://webtrackly.com/api/v1/packages/?type=zone"

curl -H "Authorization: Bearer YOUR_API_KEY" \
  "https://webtrackly.com/api/v1/packages/?type=technology&q=wordpress"

curl -H "Authorization: Bearer YOUR_API_KEY" \
  "https://webtrackly.com/api/v1/datasets/"

The endpoint reference is on the API documentation page. A monthly cron job that reads the catalog, fetches the packages you track, and drops them into dated files is enough to keep a listening pipeline running unattended.

Common Mistakes in Domain Listening and How to Avoid Them

1. Taking one snapshot and calling it monitoring

   • What goes wrong: a single file is downloaded and analysed once.
   • Why: a single snapshot describes a state. Every question about growth, churn, or migration requires at least two.
   • The fix: decide the interval before the first download, and keep every file you take.

2. Comparing snapshots that were not produced the same way

   • What goes wrong: a fresh extract is diffed against an older file from a different source or a different package.
   • Why: the delta then reflects methodology differences rather than real change.
   • The fix: compare the same package against itself, and record the download date with each file.

3. Skipping normalisation before the diff

   • What goes wrong: case differences, trailing dots, or inconsistent IDN encoding produce large phantom appeared/disappeared sets.
   • Why: string comparison is exact; the same domain in two forms is two rows.
   • The fix: lowercase, strip trailing dots, and pick one IDN representation at load time.

4. Loading hundreds of millions of rows into the wrong tool

   • What goes wrong: a multi-gigabyte CSV is opened in a spreadsheet or iterated row by row in a scripting language.
   • Why: the row counts here reach the hundreds of millions. Anything that holds the file in memory will not finish.
   • The fix: stream it — command-line filters, DuckDB over the CSV in place, or a bulk load into a columnar store.

5. Reading a technology detection as a version statement

   • What goes wrong: a technology list is treated as a list of hosts running a specific vulnerable release.
   • Why: the packages group domains by detected platform, not by patch level.
   • The fix: use the list as a target set and determine version or exposure yourself, within whatever authorisation your work requires.

6. Expecting contact data to be somewhere in the file

   • What goes wrong: a workflow is designed around emails or phone numbers attached to domains.
   • Why: that data is not in the packages, and registrant details are redacted at the registry level for most domains under GDPR.
   • The fix: use domain data for what it is — a map of names, infrastructure, and platforms — and handle any contact sourcing through channels with documented consent and provenance.

Frequently Asked Questions

Q: What exactly does WebTrackly sell?
A: Downloadable bulk domain data. 1,538 packages in total: 716 TLD zone files, 79 technology and CMS lists, and 27 curated datasets. Each is delivered as CSV inside a ZIP archive.

Q: How fresh is a download?
A: The file is generated at the moment of purchase rather than served from a pre-built snapshot, so it reflects the state of the source data on the day you buy it.

Q: Is there a search box where I can look up one domain?
A: No. There is no per-domain lookup and no filter interface. You choose a package from the catalog, buy it, download the CSV, and filter it with your own tools.

Q: How big are the files?
A: It varies by package. The .com zone package contains 157,775,115 domains, the WordPress technology list 21,639,326, and the all-registered-domains dataset 272,614,863 rows. Every listing shows its row count before you buy.

Q: What does it cost?
A: One-time package purchases start at $3.50. Pro is $29/month and includes 50 packages, 10 datasets, and 30,000 API calls. Enterprise is $99/month with 200 packages, 50 datasets, and 300,000 API calls. See the pricing page for current terms.

Q: Do the files include contact details?
A: No. No names, emails, phone numbers, or personal registrant data. Under GDPR that data is redacted in public registration records for most domains, and it is outside the scope of what WebTrackly publishes.

Q: Is there an API?
A: Yes, for the catalog. It lists packages and datasets and returns their metadata, authenticated with a bearer token — see the API documentation. It does not perform domain lookups.

Q: How do I detect what changed between two downloads?
A: Normalise the domain column, then compare the two snapshots on it. Rows only in the newer file appeared; rows only in the older file disappeared; rows in both with a changed field are migrations. Command-line comm handles the set comparison, and a SQL join handles field-level change detection.

Conclusion

Domain listening is not a product feature, it is a habit: download the same package on a schedule, keep every copy, and diff them. The tooling is deliberately boring — a ZIP, a CSV, a sorted comparison — and that is what makes it dependable. What you get in return is a measurement of the web that does not depend on anyone's published statistics or on a rate-limited lookup protocol.

Pick the population you care about, take the first snapshot, and start the clock.

Take the first snapshot.
716 TLD zone files, 79 technology lists, 27 datasets — CSV in a ZIP, from $3.50.
Browse packages → | View pricing →

Related Resources

• Package Catalog — 1,538 downloadable domain databases
• Zone Files by TLD — .com, .net, .de and 700+ more
• Domain Data — domain lists by CMS and technology
• Curated Datasets — all registered domains, MX servers, e-commerce
• Pricing — one-time purchases from $3.50