DealerDataOne-Crawler is the automated website checker operated by Dealer Data One, an automotive software company that helps dealerships understand how their websites perform. The crawler measures dealership websites for speed and for AI visibility, meaning whether AI assistants and search tools can read a site and its vehicle inventory.

The crawler reads only publicly available pages. On a typical dealership website it may request: The homepage, robots.txt, llms.txt, and llms-full.txt, The XML sitemap, A new vehicle inventory listing page, A single vehicle detail page, Pages used to measure load speed and Core Web Vitals. From these pages it records things like load times, whether vehicle details such as VIN, price, and model appear in the page without JavaScript, what robots.txt says to different crawlers, and whether lead forms are plain HTML or built with JavaScript.

It never submits forms, including lead, contact, or credit forms. Forms are inspected only. It never logs in, creates accounts, or accesses anything behind a password. It never attempts to solve or bypass CAPTCHAs, bot challenges, or firewalls. If a site blocks it, we record that result and move on. It never collects personal information about website visitors or dealership staff.

The crawler typically visits each monitored website once per day, requests only a small number of pages per visit, and spaces out its requests to avoid putting any meaningful load on your server.

How we use the results

Results are used in Dealer Data One reports, dashboards, and industry benchmarks about dealership website performance and AI readiness. We only work with information that anyone visiting your website could see.

About DealerDataOne-Crawler

Q: How it identifies itself

Every request from our crawler uses this user agent string: DealerDataOne-Crawler/1.0 (+https://dealerdata.one/crawler). Our crawler never pretends to be a web browser, a search engine, or another company's bot.

Q: How to block or limit the crawler

You can stop our crawler from accessing your site by adding the following to your robots.txt file. You can also reach out and we will remove your website from our checks.

If you found DealerDataOne-Crawler in your website logs, this page explains who we are, what our crawler checks, how it identifies itself, and how to reach us or opt out.

How Our Crawler Works

Last updated September 29, 2026

Who we are

DealerDataOne-Crawler is operated by Dealer Data One. We measure how well car dealership websites can be read by search engines and AI assistants, and we report those results to dealers. The crawler also collects the vehicle listings dealers advertise on their own websites.

How it identifies itself

Every request from our crawler uses this user agent:

Mozilla/5.0 (compatible; DealerDataOne-Crawler/1.0; +https://dealerdata.one/crawler)

The crawler never presents itself as a web browser or as another company's crawler. If a request claims to be DealerDataOne-Crawler but behaves differently than described on this page, please let us know.

How to verify our requests

Every request is signed with Web Bot Auth (HTTP Message Signatures, Ed25519), so you can confirm it really came from us. Our public key is published at:

https://cc.dealerdata.one/.well-known/http-message-signatures-directory

Signed requests carry this header:

Signature-Agent: "https://cc.dealerdata.one"

robots.txt

The crawler reads your robots.txt before every request and obeys it, both for rules addressed to DealerDataOne-Crawler and for rules addressed to all crawlers (User-agent: *). A disallowed URL is never requested.

What it reads

Public pages only, including:

The homepage
robots.txt and llms.txt
Inventory search pages and vehicle detail pages
Service, parts, and specials pages
The site's own public inventory feed
Manufacturers' public dealer locator data

What it never does

It never submits forms, including lead, contact, or credit forms.
It never logs in or accesses anything behind a password.
It never attempts to solve a CAPTCHA or bot challenge. A page that blocks it is simply recorded as blocked.
It never presents itself as a browser or another company's crawler.

Rate and load

The crawler makes one request at a time per website, with a pause between requests, and about ten requests per minute in total from each crawler host.

Where requests come from

Requests originate from Google Cloud (us-central1). We will list fixed IP addresses here once they are assigned.

Note: Google PageSpeed Insights tests run Lighthouse from Google's own infrastructure with Google's user agent. That traffic comes from Google, not from our crawler.

How to block or limit the crawler

To stop our crawler from accessing your site, add the following to your robots.txt file:

User-agent: DealerDataOne-Crawler
Disallow: /

You can also use Disallow rules to block specific paths while leaving the rest of your site available.

Contact

For rate or access requests, allowlisting, or questions, email crawler@dealerdata.one. We answer rate and access requests within one business day. Including the date, time, and any request or reference IDs from your logs helps us look into it quickly.