BlogTechnical SEOB.000c

Agent SEO — when AI agents crawl your site, what to actually serve

ChatGPT Agents, Claude Computer Use, and Gemini's browsing are hitting sites at scale. The robots.txt, rate-limit, and content rules I'm using to stay both visible and protected.

SHSeyam Hossen May 16, 2026 10 min read Technical SEO
Reading progress10 min

In April my biggest e-commerce client saw a 340% increase in non-Googlebot traffic. The culprit wasn't a scraper. It was ChatGPT Agents executing user tasks — checking prices, comparing specs, sometimes adding items to carts on behalf of humans who'd never see the page.

The new traffic class

Agent traffic behaves nothing like classic crawler traffic. It's transactional, it converts (sometimes), and it doesn't respect crawl-delay the way Googlebot does. Most sites are either blocking it entirely or letting it through unmanaged. Both are mistakes.

  • Identify the user agents — OpenAI's GPTBot and ChatGPT-User are different things.
  • Decide per-bot what to allow, not blanket allow or blanket deny.
  • Rate-limit agent traffic on transactional endpoints, not on content pages.
  • Serve the same HTML to agents as to humans — cloaking signals distrust.

The conversion question

When an agent converts on behalf of a user, who owns the customer relationship? I don't know yet. Neither does anyone else. But the sites that show up as the source the agent chose will be the ones that get to find out.

Treat AI agents like very motivated, very fast affiliate partners. Optimise the page for both the human watching and the model deciding.
Found this useful? Share it.
[ Work with me ]

Readenough?Let'sapplyittoyoursite.

If a pattern here resonates, the fastest way to act on it is a 4-week Sprint engagement. I take on two clients per quarter.