Visa's test: the AI agent buys the wrong thing. And pays anyway

Picture a salesperson who recommends the wrong product in almost every second conversation. And still rings up the sale more than half the time.

That's where AI shopping agents are today. According to Visa.

What Visa tested

On November 5, Visa published the Visa Agentic Commerce Benchmark. In round one, 22 AI models from 12 labs had to handle 15 shopping scenarios, like buying a TV, a laptop or a personal care appliance. Three attempts per scenario. A run only passed if the agent uncovered every customer requirement and made the right call: buy, or deliberately don't.

• 42.8% of runs failed.
• In about 54% of failed runs, the agent bought anyway.
• The best models passed 82.2%, the weakest 28.9%.
• All three attempts passed only 34.2% of the time.

Visa doesn't name the models. The benchmark is public on Visa Payments Frontier, with replays and a whitepaper.

Payment isn't the problem. The questions are.

What I find interesting: the payments industry has spent months talking about tokens, protocols and secure checkout for agents. Visa now shows the agent fails earlier. At understanding.

The data is clear. The five models that uncovered the most hidden buying criteria passed 70.2% of runs. The bottom five only 44.4%. Which means: ask the right follow-up questions and you buy the right thing. In the TV example: budget, formats, HDR10+ for family videos, gaming. A good salesperson covers that in two minutes. Many agents don't.

And money only solves half of it. The five most expensive models per run passed 67.1%, the five cheapest 50.2%.

What it means for shops: returns on schedule

The second-order point: an agent that buys wrong and pays anyway doesn't create revenue. It creates a return. Plus a customer who trusts the next agent purchase a little less. Visa's own Trust Index from September found only 23% of US consumers trust AI to handle payments anyway.

For merchants, the work shifts. Away from checkout, toward product data an agent can check against what the customer wants. Compatibility, exclusion criteria, "not suitable for…". Whatever hides in prose descriptions today needs to be a clear attribute.

The catch

It's Visa's own benchmark with 15 scenarios. Checkout and payment aren't tested yet; Visa plans to add them. And models improve fast. The direction is still clear.

What I'd do now

• Test your category. Look at the public benchmark and check whether your category, or a similar one, is in it.
• Put exclusions in the data. For each top product, add three attributes saying who it's not for.
• Track returns. Once agent traffic shows up, measure its return rate separately.

Bottom line: Agentic commerce isn't failing at the register. It's failing at the advice. And the advice lives in your product data.