You’ve probably noindex-ed pages you don’t want showing up in search. It’s a solid practice for product variants or paid destination pages, and it works for Google rankings.
But AI crawlers don’t care about noindex tags, and as of yet there is no corresponding version of noindex for LLMs. robots.txt is an option, but our own research shows mixed results.
So what does that mean for the pages you thought were hidden from AI?
Whether you should spend time on this issue depends on if you’re seeing an impact across attribution or brand perception. Let’s dive in.
noindex Doesn't Matter to LLMs
The ‘noindex’ meta tag was not built for this problem.
A crawler has to fetch the page before it can read the meta noindex tag. By the time the bot sees that signal, it has already ingested the content.
Traditional search engines have an established trust relationship and agree to respect bot instruction. LLMs haven't made those same commitments. At this point, noindex feels like a legacy mechanism without a strong, widely adopted replacement.
AI-specific meta tags (like ‘noai’ or ‘noimageai’) exist but haven't seen wide adoption. Until a major player formally signals which tag matters, they aren't a reliable control layer.
The same goes for llms.txt, a proposed standard for giving AI crawlers a curated view of your site. Although it’s gaining traction, no major LLM provider has committed to actually honoring it. So for now, llms.txt belongs in the same 'not yet reliable' bucket as the meta tags above.
robots.txt Is Stronger, But Not Airtight
robots.txt operates upstream. A “well-behaved” crawler checks robots.txt before fetching a page, which means the content never gets read in the first place. This is meaningfully better than noindex. But "well-behaved" is doing a lot of work in that sentence.
Here's what we actually know about compliance, based on our own research and direct audit work with clients:
If you want a more detailed breakdown of how to think about bot access across your entire site, we updated our original GBTbot guide with everything we’re learned since 2023.
What This Means
Training crawlers and AI search crawlers will generally stay out if you add explicit Disallow directives. That covers a meaningful portion of LLM-related crawl traffic and is low-effort to implement. If your paid landing pages live on a subdomain, you can scope these directives there without touching your main site's robots.txt at all.
User-initiated retrieval is a different story.
When a real person asks ChatGPT or Perplexity a question and the bot fetches live content in real time, robots.txt won't reliably stop it.
OpenAI removed ChatGPT-User compliance language from their documentation in December 2025, and Perplexity has stated that Perplexity-User is an agent and therefore exempt from robots.txt.
The only reliable control for that tier is server-level (WAF or Cloudflare-level blocking), which is a heavier lift and should only be scoped if monitoring shows you actually have a problem.
In practical terms: you can block most LLM crawlers, but not the ones doing live retrieval at the moment a user asks a question.
Is This Worth Your Time to Solve?
There's a more fundamental question sitting underneath all of this: should you even bother trying to address the LLM crawler issue?
The large language models powering AI search have largely scraped the internet without meaningful regard for copyright law or crawl restrictions. Whether a robots.txt directive will stop a model that has already ingested a significant portion of the web is genuinely open. And the concern about two similar pages competing for the same prompt may be a legacy way of thinking that doesn't map cleanly onto how these systems actually work.
Everything is derivative, including LLMs and language.
So here's the practical decision framework:
Skip the problem-solving if: Your campaign pages are thin, conversion-focused, and you haven't seen them surface in LLM responses. The marketing-heavy copy on a paid landing page is unlikely to be what an LLM reaches for when answering an informational question. Our overall read is that the risk is low for most sites.
Take action if: Your campaign pages closely mirror core product pages in depth and content (same information, different formatting or CTA), or if you start actively seeing campaign page URLs cited in AI-generated responses. That is the signal to implement robots.txt directives and start the test. Escalate to server-level controls only if monitoring shows meaningful LLM-driven traffic to those pages even after robots.txt directives are in place.
When Page Indexing Matters: Attribution and Brand Representation
If LLMs are surfacing your noindex pages in a way that impacts attribution or brand representation, that’s when it moves beyond a technical SEO debate and becomes important to marketing leadership.
We’re beginning to see scenarios where LLM citations disrupt attribution workflows.
As AI-driven referral traffic grows, the line between organic, paid, and conversational is going to get blurry in ways that weren't a real concern two years ago. The confidence you've built into your attribution model is at risk if this goes unmonitored.
The Duplicate Page Dilemma
Brands with page variations that are optimized for paid campaigns (form-first, light on educational depth) also face a new challenge: because LLMs ignore noindex-ed pages, these duplicate pages can end up being cited in AI responses.
While it makes sense to create stripped-down pages to improve quality scores and lower bid costs, your underlying site architecture or duplicate-content pages could end up hurting your brand.
If you’re seeing LLMs cite these pages, it’s worth clarifying what role each version is playing and asking whether the variations are needed.
When to consolidate: If the pages are duplicates with no differentiated content, consider deleting or consolidating. Sometimes the short-term tactics aren’t worth the long-term technical SEO (or GEO) headache.
When to keep: If the pages serve genuinely different purposes (such as one for conversion, one for organic visibility and authority), they need to be differentiated enough for both users and LLMs. The pages you want cited by LLMs should surface product, solution, and resource content that showcases your brand. This is also where you should revisit your robots.txt file and/or block AI crawlers on the pages you want to keep out of LLM citations.
Monitoring for Content Governance
Keeping paid landing pages out of LLM corpora protects how the brand shows up in AI-generated answers, but that depends on someone actually tracking which pages exist and how they're being controlled.
In a recent audit of a client's paid subdomain, we found roughly 559 pages indexed in Google that weren't supposed to be; the noindex tag had simply never been applied. That's a Google indexation problem, not an LLM one, but it's a useful proxy.
A team that isn't tracking noindex closely enough to catch 559 mis-indexed pages is very unlikely to have scoped robots.txt directives correctly either. The content governance question isn't hypothetical, it shows up in Google's index today, and it's the same discipline gap that determines what ends up in an LLM's corpus tomorrow.
If you have similar or duplicate pages, you should be actively tracking whether they’re being cited by LLMs the same way your SEO team tracks noindex pages.
What to Do Next
If you're an SEO or technical practitioner:
- Don’t rely on noindex as a meaningful control for LLMs. Confirm noindex tags are actually present on all paid landing pages. In the audit referenced above, a sample page was missing the tag entirely.
- Add explicit Disallow directives in robots.txt for training and AI search crawlers: GPTBot, ClaudeBot, Google-Extended, CCBot, OAI-SearchBot, Claude-SearchBot, PerplexityBot.
- If paid pages live on a subdomain, scope the directives there so your main site is unaffected.
- Add separate allowlist rules for any paid platforms (like Google Ads) that need to crawl those pages for quality score evaluation.
- Monitor server logs and Cloudflare AI Audit (if available) over 30 to 90 days to assess whether crawl traffic drops.
If you're a marketing or paid media lead:
- Ask whether your paid landing page infrastructure has been audited for LLM exposure. Many teams assume noindex should do the trick (but it won’t).
- Ask your attribution team to segment or flag LLM referral traffic to paid landing pages specifically, rather than assuming it's absorbed cleanly into existing channel buckets. That segmentation is what protects the reporting model's confidence, not just awareness that the risk exists.
- If you're seeing campaign pages appear in LLM-generated responses, that's your signal to escalate. Start by analyzing why the LLM preferred that version over the organic page: was it simply the page the crawler fetched first, or is the content structured in a way (clearer Q&A format, tighter schema, more scannable lists) that made it easier to extract and cite? If it's the latter, that's a finding worth handing to your organic team; whatever made the paid page more citable is worth testing on the organic page too. Either way, test page changes (Q&A, lists, schema, etc.) that direct LLMs to cite the version you want them to.
The tools we've historically used to control crawl behavior were built for a different era. LLM crawler compliance is fragmented, the standards bodies are publishing guidance slower than the landscape is moving, and anyone who tells you they have a definitive answer right now is getting ahead of the evidence.
We suggest implementing the robots.txt directives as a first-pass test, monitoring for 30 to 90 days, and escalating only if monitoring shows a real problem.
Looking to revamp your SEO strategy to account for AI and LLMs but don’t know where to start? Let’s chat about how the Seer team can help.
.png)