A client shows a report: impressions up 40% for the quarter, clicks down 18%. Nothing broke — an AI Overviews block appeared above the organic results and answered the question instead of sending anyone to the site. ChatGPT and Perplexity do the same. Below: how an AI answer is assembled, which format gets extracted, which crawlers to let through, and how to measure results once rankings explain nothing.

How an AI answer is actually assembled

The common mistake is imagining AI search as a normal results page, only prettier. The mechanics differ, which is why old optimisation habits only half work.

1. 1

   The query is fanned out into sub-questions

   The system does not search your phrase whole. It generates 5–15 clarifying queries and answers each separately, which is why a page often surfaces for a question you never optimised for.

2. 2

   A passage is retrieved, not a document

   The unit of retrieval is a 40–120 word paragraph. If your answer spans three screens and leans on phrases like "as we mentioned above", there is nothing to extract.

3. 3

   Candidates are cross-checked against each other

   When three sources give one number and yours differs with no explanation, consensus wins. The source that gets picked frames its number: which period, which market, which assumptions.

4. 4

   The citation attaches to a claim, not to a site

   The link binds to a single sentence in the generated answer. Domain authority matters less here than the clarity of one paragraph — genuinely good news for smaller sites.

The consequence matters: AI Overviews are built from the same index ordinary Googlebot fills. There is no separate "AI optimisation" divorced from classic SEO — a page that is not indexed, or takes six seconds to load, becomes nobody’s source. A solid SEO checklist stays the foundation rather than the alternative.

What content actually ends up quoted

What has to change is delivery, not topics. Text a human reads pleasantly top to bottom and text you can cut a self-contained paragraph out of are different things. The second reads drier and performs better.

The biggest gain comes from something banal: deleting the two warm-up sentences at the start of every section. On projects where we did only that and pushed specifics into the opening paragraph, citations appeared within a month or two without a single new backlink. Building that structure systematically is covered in content marketing and SEO.

The technical layer: who you let in and what you serve

This is where most of the confusion lives. Different bots from one company do entirely different jobs, so you cannot block "AI" with a single line — it is easy to switch yourself off from search while leaving training access wide open.

A minimal working configuration: search agents fully allowed, training bots restricted if you choose, utility sections closed to everyone.

# robots.txt — search agents of AI systems
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Training bots: a content decision, not a traffic one
User-agent: GPTBot
Disallow: /admin/
Disallow: /cart/

# Everyone else
User-agent: *
Disallow: /admin/
Disallow: /*?sort=
Disallow: /*?utm_

Sitemap: https://example.com/sitemap.xml

There is a lot of noise around llms.txt. As of 2026 no major vendor has confirmed it reads the file as a binding directive, so treat it as a cheap bet: twenty minutes, zero risk, unknown payoff. Correct structured data pays far better — Article, FAQPage, Organization, Product — because it names the entities outright instead of making a model guess. Detail per type is in our structured data guide.

Why you get cited: entities and off-site mentions

A model does not pick sources on page text alone. It checks whether your claim agrees with what the rest of the web says about you — signals that barely appear in a standard SEO report.

• Unlinked mentions count too. Your brand name in a review, a forum thread or a vendor shortlist is already a signal. A link amplifies it but is not a precondition.
• One spelling of your name, everywhere. "WebEngine", "Web Engine" and "web engine" read as three entities. Pick one and lock it into Organization markup and every external profile.
• Aggregated sources punch above their weight. Directories and "10 studios in Kyiv" roundups get cited far more than individual sites, because they read as a survey rather than a pitch.
• An author profile with verifiable experience. Name, role, links to actual work. Where being wrong is expensive, this is a first-pass filter.
• Freshness, honestly done. A dateModified two years old loses to an equally good but updated page. Auto-stamping today’s date nightly gets noticed and discounted.

How to tell whether it is working

No tool shows "your position in AI", and most likely none will: the answer is regenerated every time and differs between two consecutive users. Measurement gets assembled from four ordinary sources instead.

1. 1Split Search Console into branded and non-branded queries. Falling CTR alongside rising impressions on non-branded informational queries is the sharpest signal that an AI block is eating your traffic.
2. 2Write down 30–50 questions from your niche and run them manually through ChatGPT, Perplexity and Google once a month. Record who gets cited. Tedious, and the only reliable method.
3. 3Check server logs for OAI-SearchBot, PerplexityBot and ClaudeBot. None at all means a technical problem: a CDN rule, a firewall, or a forgotten robots.txt line.
4. 4Watch referral traffic from chatgpt.com, perplexity.ai and claude.ai. Volume is low single digits, but conversion to enquiry beats average organic in our projects — the visitor arrives already holding an answer.

Frequently asked questions about AI search optimisation

In short

• An AI answer is assembled from paragraphs, not pages: optimise the self-contained 40–120 word block.
• The technical foundation has not changed — no indexation and no speed means no citations.
• Let AI search agents through; the call on training bots is separate and is not about traffic.
• Measure manually against a fixed question list monthly. There is no "AI position", and tools selling one measure something else.