About DealerDataOne-Crawler

How Our Crawler Works

Last updated October 26, 2026

Who we are

DealerDataOne-Crawler is operated by Dealer Data One. We measure how well car dealership websites can be read by search engines and AI assistants, and we report those results to dealers. The crawler also collects the vehicle listings dealers advertise on their own websites.

How it identifies itself

Every request from our crawler uses this user agent:

Mozilla/5.0 (compatible; DealerDataOne-Crawler/1.0; +https://dealerdata.one/crawler)

The crawler never presents itself as a web browser or as another company's crawler. If a request claims to be DealerDataOne-Crawler but behaves differently than described on this page, please let us know.

How to verify our requests

Every request is signed with Web Bot Auth (HTTP Message Signatures, Ed25519), so you can confirm it really came from us. Our public key is published at:

https://cc.dealerdata.one/.well-known/http-message-signatures-directory

Signed requests carry this header:

Signature-Agent: "https://cc.dealerdata.one"

robots.txt

The crawler checks your robots.txt before every request and obeys it, both for rules addressed to DealerDataOne-Crawler and for rules addressed to all crawlers (User-agent: *). A disallowed URL is never requested. We keep a copy of your robots.txt for up to a day, so a change takes effect within 24 hours. If robots.txt answers with a server error, we treat the whole site as disallowed, as RFC 9309 requires.

What it reads

Public pages only, including:

• The homepage
• robots.txt and llms.txt
• Inventory search pages and vehicle detail pages
• Service, parts, and specials pages
• The site's own public inventory feed
• Manufacturers' public dealer locator data

What it never does

• It never submits forms, including lead, contact, or credit forms.
• It never logs in or accesses anything behind a password.
• It never attempts to solve a CAPTCHA or bot challenge. A page that blocks it is simply recorded as blocked.
• It never presents itself as a browser or another company's crawler.

Rate and load

Each of our crawls sends its requests to a dealership website one after another, not in parallel. The nightly check reads a handful of public pages (robots.txt, llms.txt, the homepage and a few pages it links to) in one short burst. Inventory, specials and site-audit crawls wait at least one second between requests, or longer if your robots.txt sets a Crawl-delay for DealerDataOne-Crawler or for all crawlers; if the Crawl-delay is over 60 seconds, they skip your site. They also stop at the first 429 (Too Many Requests) response. Across the roughly 19,000 dealership websites we crawled from October 1 to 15, 2026, a site's busiest minute was typically 8 requests, and no site received more than 43 in one minute. When we refresh a brand's dealer list, we query the manufacturer's dealer-locator service at most once a second, or less often if its robots.txt sets a longer Crawl-delay.

Where requests come from

Requests originate from Google Cloud (us-central1). We will list fixed IP addresses here once they are assigned.

Note: Google PageSpeed Insights tests run Lighthouse from Google's own infrastructure with Google's user agent. That traffic comes from Google, not from our crawler.

How to block or limit the crawler

To stop our crawler from accessing your site, add the following to your robots.txt file:

User-agent: DealerDataOne-Crawler
Disallow: /

You can also use Disallow rules to block specific paths while leaving the rest of your site available.

Contact

For rate or access requests, allowlisting, or questions, email crawler@dealerdata.one. We answer rate and access requests within one business day. Including the date, time, and any request or reference IDs from your logs helps us look into it quickly.