How good our own measurement is
We measure vendors on what they publish. This page measures us: how often a finding on this site is right, how often it is not, and which kinds of mistake we know about.
Measured against 461 published findings read by hand on 11 September 2026. Classifier v3-2026-10-20.
Precision and recall
Precision answers the reader's question: when a vendor appears on a value page, how often does it belong there? Recall is the counter-question, and it belongs next to it – a classifier that rejects everything would have perfect precision and no findings at all.
| What we read | Checked by hand | Precision | Recall | Wrongly accepted | Valid findings missed | Before v1-2026-09-04 |
|---|---|---|---|---|---|---|
| Hosting location named | 264 | 99.4 % | 93.3 % | 1 | 13 | 73.1 % |
| Certification named | 197 | 98.6 % | 95.2 % | 2 | 7 | 74.6 % |
The last column is the same measurement against the same 461 findings, run against the classifier as it stood before 11 September 2026. Every fourth published finding was wrong.
The labels and short quotes in this historical test set are unchanged. Missed findings include truncated place names, claims split across sentences or table fields, and badge lists whose meaning depends on page layout. The set also has documented ambiguous labels. These figures are not an estimate of accuracy across every current vendor or every type of detail.
Use the evidence in a buying decision
Start with the source linked from a vendor profile. Check which product, legal entity and plan it describes, and read the measurement date. A named certification is not a verified audit report; a named region does not establish where every backup or support session is handled.
Keep unresolved questions with your shortlist. Select required evidence and export a review brief with the source, date and snapshot for each selected criterion. If a result appears wrong, send a correction.
Which extractors this covers, and which it does not
The hand-read set covers 2 of the 10 kinds of detail we publish: Certification named and Hosting location named. For the others there is no figure here, and we do not print an estimate. An estimated accuracy would be the same offence as an unsourced finding about a vendor.
Error classes that still get through
These are findings we read by hand, judged wrong, and that the classifier still lets past. They are listed because a known gap someone can check beats a clean-looking page.
| Class | Where | In the set | Example |
|---|---|---|---|
| Company location A company location was mistaken for the location of its data. |
Hosting location named | 1 | aMember Membership and subscription management platform USA (assumed) AWS Cloud infrastructure and hosting services Global (notably USA, Europe, Asia- |
| Navigation or topic list A menu item or list of topics was mistaken for a claim. |
Certification named | 2 | on Case Studies Compliance NIS2 NIST CSF 2.0 CIS Controls GDPR CMMC DORA FERPA PCI DSS v4.0.1 HIPAA HITRUST CJIS Controls In the recent MITRE ATT&CK E |
Error classes we now catch
| Class | Caught in the set |
|---|---|
| Location explicitly excluded A statement about data outside a region was read as data inside it. | 17 |
| Postal address An address in the text was mistaken for a data location. | 13 |
| No certification claim The text names a standard without claiming certification. | 13 |
| Company location A company location was mistaken for the location of its data. | 9 |
| Location of people The location of people was mistaken for the location of data. | 9 |
| Definition An explanation of a term was read as a statement about the vendor. | 8 |
| Navigation or topic list A menu item or list of topics was mistaken for a claim. | 8 |
| No hosting statement The quote does not connect the location to data hosting. | 8 |
| Another service provider The statement describes a service provider rather than the vendor being reviewed. | 7 |
| Question A question was read as an affirmative statement. | 5 |
| Legal reference A region mentioned in a legal reference was read as a hosting location. | 5 |
| Future intention A plan or work in progress was read as a current claim. | 4 |
| Alignment, not certification Following a standard is not a statement of certification. | 3 |
| Infrastructure provider A statement about data centres or infrastructure was attributed to the vendor. | 3 |
| Negated statement A negative statement was read as an affirmative claim. | 3 |
| Transfer framework A reference to a transfer framework was read as a hosting location. | 2 |
| Ambiguous statement The text does not clearly support the extracted finding. | 1 |
What this means for the live data
Current stored certification and hosting findings are re-read against their own quotes when the service starts. If the sentence we kept does not carry the finding – a question, a negation, a definition, the opposite – the row stays on the vendor profile with its source and counts in no list, no figure and no requirement page.
This updates quote confidence, not the original measurement. Sources, quotes, subjects, measurement dates, criteria, scores and historical series retain their recorded state. Missing words cannot be recovered from a truncated quote. A later source visit is a separate measurement; it does not rewrite the historical test set.
| What we read | Findings held | Not counted |
|---|---|---|
| Third-party host on first load | 96,588 | 0 |
| HSTS max-age | 5,853 | 0 |
| Certification named | 5,823 | 2,354 |
| HSTS option | 4,529 | 0 |
| Privacy contact | 3,984 | 0 |
| Hosting location named | 3,850 | 9 |
| Subprocessor named | 3,747 | 0 |
| security.txt expiry | 774 | 0 |
| Uptime figure | 280 | 0 |
| Subprocessor notice period | 151 | 0 |
125,579 findings in the current snapshot, 2,363 of them not counted.
How old the measurements are
A fresh figure and an old one look the same. We do not remeasure every vendor every night: a run covers part of the catalogue, so some results stand for a while. The oldest column is the one to read before a procurement decision.
| Criterion | Results | Newest (days) | Oldest (days) |
|---|---|---|---|
| Data location stated | 9,024 | 0 | 59 |
| Data processing agreement | 9,024 | 0 | 59 |
| Public pricing | 9,024 | 0 | 59 |
| security.txt (RFC 9116) | 9,024 | 0 | 59 |
| Uptime SLA with a figure | 9,024 | 0 | 59 |
| Status page with history | 9,024 | 0 | 59 |
| Subprocessor list | 9,024 | 0 | 59 |
| No third-party tracking before consent | 9,024 | 0 | 59 |
| HTTPS enforced with HSTS | 9,024 | 0 | 59 |
| Certifications named | 9,024 | 0 | 59 |
| Named privacy contact | 9,022 | 0 | 58 |
| Notice before subprocessors change | 9,022 | 0 | 58 |
Open corrections
0 corrections submitted and not yet worked through. Tell us about a wrong entry. A correction moves that vendor into our next nightly run. That run reads the same fixed addresses we check for every vendor, not the address you send.
What changed, and when
- v3-2026-10-20 – corrected sentence offsets, explicit storage phrases, repeated certification claims and the distinction between an auditor and a certificate holder. New extracts preserve table-row boundaries and keep the detected words inside the quote budget. HTTP failures remain unmeasured, including failures on sign-in paths. Current quote confidence is re-evaluated without rewriting measurement history.
- v2-2026-09-11 – sentence scope, polarity and subject. Findings contradicted by their own quote stopped contributing to lists.
- v1-2026-09-04 – a certification quote gained a subject: the vendor, their data centres, a named provider, or unclear. Only the first is aggregated.
- Transparency score v1 is frozen and unaffected by any of this. A detail has never entered the score.