Half Your Visitors Are Now Bots. Here’s How to Feed Them Without Ruining Your Site.

As of June 2026, more than half the measured traffic on the web comes from agents — AI browsers, chatbot crawlers, and bots acting for humans. Your analytics undercount your real audience, because part of your audience now reads through a machine. Here’s how to serve that half without ruining your site for the human half.

Yesterday we covered getting cited by ChatGPT and Perplexity — the editorial half. This is the plumbing half: what agent traffic actually is, and the site decisions that determine whether it turns into readers.

Know your bots — they’re not one thing

Three kinds of machine visitors, three different stakes:

1. Training crawlers ingest your content to train future models. Slow, bulk, no attribution. This is the one people mean when they’re angry.
2. Answer-engine crawlers (Perplexity, ChatGPT search, Google’s AI results) fetch pages to answer a question right now — with a citation link. This is the one that sends you readers.
3. Personal agents — Comet, Claude for Chrome and their kin — visit on behalf of one specific human who asked. Blocking these blocks a reader who chose you.

The biggest mistake of 2026 site management is treating all three as one enemy. A blanket “block AI bots” toggle — often a security plugin’s default — throws away categories 2 and 3 to spite category 1.

The checklist: agent-readable without being human-hostile

• Audit your robots.txt today. Know which crawlers you block and why. A reasonable stance for most blogs: allow answer engines and agents, decide deliberately about training crawlers. Whatever you choose, choose it — don’t inherit it from a plugin.
• Keep content in clean HTML. Agents are impatient renderers. Text in real paragraphs and headings wins; text locked in JavaScript widgets, image embeds, or PDF-only downloads loses. (This is also just accessibility — the machines and the screen readers want the same things.)
• Facts in text, every time. Prices, dates, steps, and names must exist as words on the page, not pixels in a screenshot.
• Answer at the top. Agents quote lead paragraphs and section openers. Structure that serves skimming humans serves them identically.
• Keep feeds and sitemaps healthy. RSS is having a quiet renaissance — it’s the cheapest machine-readable version of your site, and agents use it.
• Watch server logs, not just analytics. JavaScript analytics miss most agents. Your logs show the real ratio — and which pages the machines love, which is surprisingly good editorial signal.

The mindset shift

None of this is about pleasing our new robot overlords. An agent is a lens a human chose — someone asked their AI a question, and your page either answers it through the lens or it doesn’t. The sites hurt worst this year are the ones optimized for ad-impression pageviews, because agents don’t see ads. The sites doing fine are the ones where the content itself is the product. Write specific answers, publish them in clean HTML, keep your feeds alive, and the bot half of the web becomes a distribution channel instead of a tax.

Companion post: How to Get Your Blog Cited by ChatGPT and Perplexity. Related: Omarchy for marketers.