We Sent 4 AI Assistants to Shop 24 Dealer Websites. Here's What Actually Worked.

Shoppers are starting to hand the search off. Instead of clicking through five dealer websites, they ask an AI assistant: "Find me the best deal on a new RAV4 at this dealership." So we tested exactly that.

We ran 190 live tests with four AI models against 24 franchise dealership websites. Each assistant was locked to the dealer's own site and asked to find a specific new vehicle, with a price and a link. We measured whether it succeeded, what it cost in tokens and dollars, how long it took, and whether publishing an llms.txt file made any difference.

How we ran the test

• Four models, each using its own web tools: Claude Haiku 4.5, Claude Sonnet, Gemini 3.8 Flash, and gpt-6-luna.
• 24 dealer websites across multiple website platforms, grouped by whether they have no llms.txt, an unoptimized llms.txt, an optimized llms.txt, or llms.txt plus llms-full.txt.
• A strict definition of success: a VIN with a valid check digit, the right model, a price, and a link on the dealer's own website. Every VIN was checked against NHTSA's public vehicle database.

Finding 1: AI assistants don't read llms.txt on their own

Across 136 unprompted runs, a model opened a dealer's llms.txt file just 3 times, all on the same website. On that site, the llms.txt linked to a dedicated inventory feed, and three different models followed it straight to an answer. Everywhere else, the file went unread.

Finding 2: Telling them to read it doesn't improve results

When we told the models to start at /llms.txt, they did. But specific answers dropped slightly, from 27 to 24 out of 48 tests. On well-built files, it cut how much the assistant had to read by about 60%, which lowers cost. It just didn't help the assistant find more cars. Most llms.txt files don't link directly to priced vehicles, so they add a step instead of removing one.

Finding 3: The AI model matters more than anything the dealer controls

• gpt-6-luna found a specific vehicle 82% of the time at a median cost of less than half a cent.
• Claude Haiku 4.5 and Claude Sonnet each succeeded 50% of the time. Sonnet spent a median of 51 cents and over a minute per shopper.
• Gemini 3.8 Flash succeeded 32% of the time and gave up after too many tool calls several times.

The difference comes down to how each tool reads the web. Some pull short search excerpts. Others fetch entire pages, sometimes eight or more per question.

Finding 4: Your website platform matters next

Sites that put priced listings directly in the page text did well. Sites that load inventory with JavaScript, or tuck it inside page code, did poorly regardless of llms.txt. Most AI fetchers strip that code out, so the cars are effectively invisible.

Finding 5: The answers are real, but "best deal" isn't

Every specific answer named a real VIN for the right make and model. But when two models searched the same store, they picked the same vehicle less than half the time. Today's assistants return a matching car, not reliably the best-priced one. That makes the accuracy of every price on your site more important, not less.

What dealers should do now

• Make priced listings readable without JavaScript. Ask your website provider whether vehicle details and prices are in the page itself or loaded afterward.
• Make sure bot protection isn't blocking AI assistants. If the assistant can't get in, your inventory doesn't exist to that shopper.
• Keep your llms.txt, but point it at inventory. It isn't doing the heavy lifting yet, but a file that links straight to priced vehicles is the one case where agents found and used it on their own.
• Check your site's basics. Our free dealership website speed test and dealership SEO audit are a quick place to start.

The bigger point

None of this is hard to measure anymore. We can now put a number on what it costs an AI assistant to answer a shopper on any dealer website, and whether it gets the right car. The hard part is what always gets missed: someone has to act on it. That means talking to the website provider, fixing the inventory setup, and checking again next month when the AI tools change. That execution gap is exactly what Astra is built to close.

A few caveats

Samples were small, five or six sites per group, so read the direction of these results rather than small percentage differences. Website platform and llms.txt overlap in our sample. Tests were run September 28 and 29, 2026, and AI tools change quickly. We'll rerun this as they do.