I Built a Tool to Watch ChatGPT Search, and It Predicted My Next Question

De-Mystifying the Black Box of AI Search

AI search is a black box. We still don’t know a lot about how it searches, where it pulls information from, or why it cites some sources but not others.

After Suganthan Mohanadasan's blog on ChatGPT search language made the rounds in our internal Slack, I was inspired to figure out how to reverse-engineer ChatGPT’s AI search.

I wanted a diagnostic engine that would simulate realistic user conversations and capture the complete retrieval layer, so we could monitor the AI prompts and conversations our clients care about. So I built one.

Instead of just knowing whether a brand shows up in a response, I can see exactly what ChatGPT searches for. Our team can use this information to see which sites ChatGPT queries to find information about a client’s vertical, analyze why a competitor was cited instead of a client, and even run tactical tests on client content.

Let’s dive into what I built and my initial findings.

I Built A Script to Watch ChatGPT Search in Real Time

Suganathan’s blog looked at a single query, but I wanted the backend data. Basically, the AI has its own language it speaks when fetching real-time data, and you can see it if you know where to look.

While ChatGPT builds an answer, the website streams data to your browser, including:

• What searches it runs

• The results that come back

• Which of those become citations

• Product records

I'll call that information the stream.

Working with AI coding tools, I built a script that types prompts into ChatGPT's website the way a person would. It saves the answer and a screenshot, and it keeps the full stream for every turn.

The other challenge I had to address was Suganthan’s post looked at a single query, but real buyer journeys are a sequence. Most people prompt and then have follow-up questions based on the AI’s answer.

So my script runs an opening question and its follow-ups in the same chat, saving the data at each step. While a single prompt is still a good way to check whether a brand shows up, the follow-ups show what ChatGPT goes looking for once someone starts deciding.

Now my script was ready for testing.

My Test: Three Conversations, Three Turns Each

On October 1, I ran three conversations that I wrote for this test (I chose random topics). Each one happened in a single chat, so every follow-up built on the answer before it. The follow-ups are my guesses at what someone would ask next.

All nine answers came from one run on the free plan, and the stream showed ChatGPT's smaller "mini" model on every turn.

ChatGPT’s Initial Queries Were More Specific Than My Prompts and Even Listed Brands

ChatGPT rewrites your question into its own search queries, and the stream shows them. Here's the first query in each conversation, alongside my original ask and what I was planning to ask for turn number two.

In the first turn, ChatGPT got more specific in its queries than any of the prompts I used, even including specific brands. It also started querying topics and features that I was planning to ask in the second turn.

The luggage query surprised me most. I never mentioned wheels or handles in the first question, yet "durable wheels handles" was in the very first search, one turn before I planned to ask about those features. The cloud backup query did something similar, adding "pricing" a turn before I asked about cost.

I wrote those follow-ups because they seemed like the obvious next questions, and ChatGPT apparently made a similar guess. I can’t tell where that comes from (training data, my wording, or something else) based on the query fan-outs. But the query does give me a written list of what ChatGPT thinks this buyer cares about.

How Did ChatGPT Get to Its Answer?

There was one question I kept coming back to: how did ChatGPT decide which product to recommend?

I used the suitcase conversation as the example here. I wanted to understand why ChatGPT picked the Travelpro Maxlite 5 (its final recommendation), and how it decided that was the most durable option.

Turn 1: ChatGPT Recommends Its Preferred Product

ChatGPT first brought up the Travelpro Maxlite 5 in its response to turn one:

And the initial reasons it gave (turn one) cited Pack & Porter and mentioned price, weight, and value:

The stream doesn't show ChatGPT's reasoning, but it does show what was on the table. ChatGPT ranked the Maxlite 5 first and backed it with two Pack & Porter pages. I read the Pack & Porter's "Best Carry-On Luggage Under $200" page, which ranked the Maxlite 5 first and carried the specs from ChatGPT’s answer:

• 5.4 lb

• A lifetime warranty

• Published dimensions of 23 × 14.5 × 9 inches, which is where ChatGPT's warning about strict 22 × 14 × 9 airline limits came from

• For the Samsonite Freeform: 6.5 lb, a polypropylene shell, and a 10-year warranty

So the recommendation lines up with the top-cited site. That doesn’t mean Pack & Porter caused the pick: the brand list was already in the query before any page came back.

Turns 2 and 3: ChatGPT Looks to Validate Its Choice

Turn two is where things got interesting. When I asked which suitcase it would choose for durable wheels and handles, ChatGPT ran exactly one search: Travelpro Maxlite 5 wheels handle warranty durable review.

Instead of looking up which carry-on is most durable, ChatGPT looked for support for the product it had already picked. Whatever drove the choice happened before that search.

The sources cited in turn two both have reviews that describe hands-on testing. GearLab ran drop tests (2022). Pack Hacker logged a two-week trip and wrote that the handle "does wiggle substantially" when extended, which shows up in the answer as "can feel somewhat wobbly." The supporting "+1" source, NewCarryOn's Platinum Elite review, relied on OutdoorGearLab's tests rather than its own.

There were two 2026 reviews from the same source set that didn’t get cited, and both included on-page disclaimers that they never tested the bag. A third 2026 review that claimed a year of testing wasn't cited either, so I'm treating this as one data point.

By the third turn on warranty and repair, the search covered a single site. All 13 results came from travelpro.com, and the summary line read "Searched 1 website."

It’s also worth noting that in the data, the first turn's sources already included a Pack & Porter page that ranked carry-ons by warranty.

A Carry-On Question Returned a Checked-Bag Record

One odd thing I noted from the carry-on convo: a product recommended by ChatGPT showed pricing/product disparities. The American Tourister product card showed $174.99, while the answer’s table said about $72. The stream included offers at both prices, but the $174.99 brand offer linked to a large checked bag - not a carry-on. For ecommerce brands, I’d check that the product records match the variant being recommended.

How Early in the Conversation Did ChatGPT Start Citing a Brand Directly?

Turn one of the luggage conversation cited a review site. By the third turn, every citation was Travelpro's own site:

Only the luggage conversation shifted from third-party reviews to the brand. The cloud backup and Spanish conversations went to brand sites from the first turn (and featured a lot of help-center and policy pages):

• All 36 citations across those two chats were on the brands' own sites

• Half of all the queries in the run (12 of 24) included the word "official"

• Two queries used site: to search a single domain

The opening Spanish question did retrieve ten third-party "best Spanish apps" roundups, but all ten appeared only in the second set of results.

If I think about it as a funnel, it makes sense that ChatGPT often turned directly to the brands to answer product-spec questions (such as warranty, cost, and cancellation policies).

Even if your brand isn't cited early in a conversation, you should have help articles, warranty pages, and pricing pages ready for the questions people ask right before they buy.

Every Cited URL Appeared in the First Result Set

The stream recorded an initial set of search results and additional results later. Across all eight turns that searched, every cited URL appeared in the first set. None of the results that appeared only later were cited. I don’t know why, but it gives us another thing to investigate: did our page appear in the set containing the cited sources, or only among the additional results?

Here's the first set for the opening luggage question, in the order the stream numbered it:

Interestingly, five of the nine citations came from one travel gear review site: packandporter.com. This implies that ChatGPT places a high degree of trust in content from Pack & Porter, perhaps because the site explicitly states who writes the reviews, how they rank products, and even how they earn money.

The page titles caught my eye too. The two cited pages are the closest match to what ChatGPT searched for:

• Carry-on

• Under $200

• Travelpro vs Samsonite

A checked-luggage list and a general "luggage under $200" list came back in the same batch and weren't cited. Considering this is a small data set, I'm treating title match as something to test further.

If you track AI visibility, this is worth checking for your own pages. A page can be retrieved in either set, and in this run only one set produced citations.

Where ChatGPT's Answers Diverged from Sources

It Rated Duolingo Without a Corresponding Citation

Duolingo was in the first Spanish query, but ChatGPT never ran a Duolingo-specific search, and no duolingo.com page came back.

The answer still gave Duolingo two stars for speaking practice, and it’s unclear where that rating came from. It could be training data, the conversation so far, or a third-party page that mentions Duolingo. Duolingo does document a speaking feature on its blog, Video Call, which is part of its Max plan, but that page didn't appear in the results.

Surprisingly, ChatGPT's suggested follow-up dropped Duolingo.

On the next turn (turn three), no new search was recorded at all. It picked Pimsleur and started answering in under two seconds.

ChatGPT Ignored Prices That Loaded with JavaScript

The most interesting finding that I want to run further tests on involved how pricing information was listed on a brand’s site.

In the Spanish chat's pricing turn (turn three), the brand with a price in their page's HTML was quoted exactly, while two others had "pricing varies" language.

I checked the source of each pricing page to see what each brand’s pages said and how they were formatted. I did this on October 2, so this information is not necessarily what ChatGPT received:

• Pimsleur: The page's HTML includes structured subscription data, "Monthly 7-day free trial" at 19.95, even though the visible text rounds to "$20." The answer quoted "$19.95/month" and the 7-day trial.

• Babbel: The prices page loads with almost nothing in it until JavaScript runs ("JavaScript required to see this page"). The answer still quoted Babbel's annual plan ($8.95/month equivalent, $107.40 billed upfront), but said its monthly pricing "varies by offer." Monthly pricing was what I'd asked about.

• Busuu: The plans page had no prices in its HTML. The answer said pricing "varies by plan/platform."

There's outside support for why this could happen. Vercel's 2024 study of AI crawlers found that OpenAI's crawlers download JavaScript files but don't run them, so content that only appears after JavaScript runs may never reach them.

Babbel's annual price still got through from somewhere, so JavaScript content isn't simply invisible. And ChatGPT had already picked Pimsleur a turn earlier without searching, so I can't credit the pricing page for the pick. We've been testing server-rendered HTML against JavaScript-rendered content for clients, and this gives us a clean side-by-side to build on.

CrashPlan's Own Pages Disagreed

Cloud backup had a pricing problem of a different kind. When I asked about cost, ChatGPT searched site:crashplan.com personal backup discontinued individual 2026, checking whether CrashPlan still sells a personal plan.

CrashPlan's help center says its Essential plan is a legacy plan that's no longer available for new purchase. Meanwhile, a CrashPlan Essential page is still live, offering $2.99 a month and a "Start Free Trial" button. ChatGPT quoted my theoretical user, a solo photographer, with the business rate of $7.33 per user per month, and told me to ignore "the older $2.99/month Essential price you may see elsewhere."

For me, this raises a straightforward question: are your pricing pages and help center describing the same offer? If a plan is no longer available to new customers, its old landing page shouldn’t still make it look available. I’d check those pages together, clear up any conflicting information, and rerun the question to see whether the answer changes.

What I'd Test Based on These Findings

None of these have been run yet. Each one comes straight from something in the data above.

You don’t need a script to start digging. Open ChatGPT in Chrome, open DevTools on the Network tab, and ask your category’s main question. Find the response for that conversation and search it for the query ChatGPT wrote. Then ask your two most likely follow-ups in the same chat and note how the queries and cited sites change.

1. What did ChatGPT add to the first query? Note what it added on its own, such as “durable wheels handles.” Check whether your page answers that concern in plain text. If it doesn’t, add it. Then rerun the same conversation over several weeks and track which set of results your page lands in, as well as whether it gets cited.

2. Do your help pages answer the follow-up questions? Write down the likely next question in your category, like a warranty claim, a cancellation, or recovering a deleted file. Check that a help article answers it directly, then compare what ChatGPT says against the actual policy.

3. Does putting the price in HTML change the answer? Take a page where pricing only appears after JavaScript runs. Add plain-text pricing and cancellation terms to the HTML, and leave a similar page unchanged. Then measure whether ChatGPT quotes the exact price or says pricing “varies.”

4. Which brands does ChatGPT search for? Ask the opening question in fresh chats several times. Record which brands appear in the first query, which get a follow-up query of their own, and which get described without any of their own pages in the results.

5. Do the product records match the recommendation? For shopping categories, compare the product records ChatGPT carries against the product the answer is recommending. A carry-on question carrying a checked-bag record is the kind of mismatch to look for.

6. Do cited page titles match the query? For the first set in your category, compare each page’s title to ChatGPT’s query. Over repeated runs, check whether the cited pages are consistently the closest title match. If they are, test retitling a page to match the query wording and see whether it starts getting cited.

Answers vary from run to run, so these tests need repeated captures before and after a change before I’d call something an improvement. Save what you capture so you can check it again later.

Things I Still Don't Know

• Whether this repeats. This was one run per conversation, on the mini model, in my own account. The sidebar showed earlier chats on these topics from a first attempt, so I can't rule out account history.

• What the model had while writing. The stream shows when results reached the browser, not what was available to ChatGPT behind the scenes.

• Which search engine supplies the results. OpenAI says ChatGPT search "sometimes partners with other search providers." Peec's research maps several providers, including Google results gathered by third parties and Microsoft's grounding service. I ran four of the captured queries in Google and Bing myself a day later, from my own browser, so treat this as a rough check: only 3 of 32 Google results and 2 of 19 Bing results matched ChatGPT's first set. Ranking on Google wasn't a good stand-in for being in that set here.

• Why a page lands in the first set or the second, and whether being in the first set means the page was read in full.

• Whether Pack & Porter's ranking drove the recommendation, or whether both reflect what ChatGPT already believed about these bags.

• Where the brand shortlist comes from.

I built this tool to better support the questions our clients ask. Knowing whether a brand shows up in the first answer is useful. Seeing what ChatGPT goes looking for when someone pushes back on that answer gives us much more to work with.

If you want to talk through what this could mean for your AI search strategy, chat with the Seer team.

Closing note: This is a tracker I'm building as I go, and what ChatGPT exposes keeps shifting. I keep records of the raw streams so I can go back and recheck patterns like the citation split against older captures. I can also monitor the parts we don't understand yet, and rerun fixed prompts to see what changes.

Methodology

I captured three conversations on October 1, 2026, each with three prewritten prompts asked in the same chat, for nine prompts in total. They ran on the ChatGPT free plan in a desktop Chrome browser, and every turn was answered by the mini model. For each turn, we saved the answer, a screenshot and the full stream. Counts come from the stream itself. Pages were fetched on October 2, the day after the capture, and may differ from what was live during the run. None of the proposed tests have been run.