New Page Again

The question: why does ChatGPT answer without naming us?

Because, as of October 2026, an assistant can only name what it can reach. Before your business can appear in an answer, your page has to pass four gates in order: a crawler must be allowed in, it must fetch the page, it must find words in the raw HTML, and the page must say something specific enough to be worth citing. Most sites fail at one of the first three and never find out.

This page walks the gates in the order a bot meets them. Fix the first one that fails. Nothing after it matters until you do.

Gate 1 — Is the crawler allowed in?

Each assistant reads the web with its own named crawler, and your robots.txt either lets each one in or does not. The tokens that decide an answer today are the search crawlers, not the training ones.

• OAI-SearchBot — what ChatGPT search reads
• Claude-SearchBot — what Claude reads when it searches
• PerplexityBot — Perplexity's index
• Bingbot — what Copilot answers from
• Googlebot — what AI Overviews draws on

Blocking GPTBot or ClaudeBot stops training, not citation. Many sites block those on purpose and lose nothing in answers. Blocking the search tokens is the mistake.

A rule that keeps the answers open and the training closed looks like this:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

One line to check: a Disallow: / under User-agent: * closes every door at once, including the five above.

Gate 2 — Does the page answer?

A crawler asks for the page and expects a 200. Three things break this more often than an outage does.

1. An old URL that now returns 404 but is still in your sitemap or linked from a page.
2. A redirect chain — /pricing sending to /pricing/, then to /plans. Each hop is a chance to stop.
3. A page behind a login or a cookie wall. The bot sees the wall, not the page.

The fix is the same each time: one address per page, answering 200 on the first request.

Gate 3 — Are there words in the raw HTML?

Most AI crawlers do not run JavaScript. They read the HTML your server sends, and if the text arrives later through a script, the bot reads an empty shell.

Test it yourself. Save the page with "View source", not "Inspect", and search for a sentence from your own copy. If you cannot find it there, neither can the crawler.

If the words are not in the source, the page does not exist to the bot.

Two tags also take a page out of the answers without an error: noindex in the head, and nosnippet, which lets the page be found but forbids quoting it.

Gate 4 — Is there something worth citing?

Once a bot can read the page, it chooses between you and every other page that passed. Two things move that choice, and both can be checked before you publish.

Specific over general

A section that names real things — a product, a price, a place, a date, a person — gets cited. A section that says "we offer a range of solutions" does not, because there is nothing in it to attribute. Read each section and ask: what fact would an assistant repeat from this?

Current over stale

Assistants prefer pages that say when they were true. State a visible "last updated" date on the page, keep dateModified in your structured data, and refresh the page inside 90 days. A date written as "14 October 2026" is read correctly everywhere; "10/14/26" is one day in Perth and another in Seattle.

---

What to do this week

• Open yoursite.com/robots.txt and confirm none of the five search tokens is disallowed.
• Pick your three most important pages and fetch each one with curl -I. Every one should answer 200 with no redirect.
• View source on each and search for a sentence from the first paragraph.
• Add or update a visible date and the dateModified field on each page.

Then watch. A page that newly passes all four gates can take days to weeks to show up in answers, because the crawler has to come back, and it comes back on its own schedule.

Last checked: 16 October 2026.