A copy of the .krd web

What counts as a page. Only a document the server actually served, with an HTTP 200. A bot wall answering every request with "Just a moment", or a host serving a 404 at its own root, is a response and not a copy of anything. The first pass of this crawler counted response records instead, and so published 138 names as finished copies of themselves when what was held was one error page each; the stored files were re-read and the counts above are the corrected ones. Those attempts are kept, because a site that refuses an archiver is worth having noticed, but they are counted apart and nothing in them is searchable.

No readable text is a copy that crawled cleanly and holds no words: a single-page application whose content arrives by JavaScript, which a WARC crawl does not run and this archive does not replay. It is counted apart, because calling it a complete copy claims a copy of a site when what is held is a loading spinner.

Complete means no link was left untaken, not that a limit went unreached. A crawl follows links two deep and stops without complaining, so completeness is measured by reading the copy back and looking for addresses it points at and does not hold. There are 82,827 such addresses outstanding; each visit starts from them rather than from the front page again.

These are WARC files, not web pages. WARC is the format web archives use: it holds the original requests and responses rather than a rendering of them, which is what makes a copy worth keeping and what would let this be handed to a real archive. Reading one back needs replay software, which this site does not run. What it can honestly offer is the files themselves, with their checksums.

A name that does not answer is recorded too. A site that has stopped responding is exactly the thing worth having noticed.