Same Prompt, Different Data Structure: ChatGPT's Search Data Keeps Changing

Skip to the methodology →

Single-Run AI Prompts Don’t Tell the Full Story

I ran the exact same prompt through ChatGPT, two weeks apart (once on September 3rd and again on September 16th), and found something odd: some of the search details I could see in the first run were no longer visible.

In the first run, ChatGPT exposed four search queries under the hood. The second run? Those queries were still there, but the pipe-style data structure beneath them had disappeared. Zero recognized calls, zero timeframe values. I even ran a full structural scan in case I’d missed something.

The two prompts were byte-for-byte identical. I hashed it to be sure. Same 148 characters, no date, nothing that would nudge it toward anything time-sensitive.

So I ran it again a third time, 24 hours later. Same issue. And again between two captures taken about eight minutes apart on the same day, using the same browser, and the same signed-in profile.

Here's what that means for your reporting: if you're checking your brand's AI visibility by running a prompt once, you're missing the full picture, because it’s constantly changing.

How I Accidentally Discovered This While Mapping ChatGPT Search Calls

 This whole discovery came about by accident, while I was building a script to capture ChatGPT’s hidden tool calls across multi-prompt conversations. The goal was to automatically track the backend language behind prompts that clients would care about while accounting for conversations typical to the buyer journey. 

For example, a real buyer might search “what’s the best X?” followed by “how much does X cost?” and then “how do I switch from Y to X?”

For every prompt, my tool captured three things:

• A screenshot of what actually rendered
  

• The raw network payloads showing the hidden tool calls
  

• A parsed summary of the search queries, URLs recorded in search results, and which URLs appeared in citation records
  

So now that you know what I was building, let’s jump back to the interesting part (what I found).

Same Prompt, Different Data Structure

For all analysis in this blog, the prompt we used was a variation of the following (we've anonymized it to protect client information): 

"What's the best [industry] platform for features X, Y, and Z?"

Here’s what I found when I ran the same prompt two weeks apart in September:

Study: ChatGPT Search Data Stability

Same prompt, two weeks apart, less visibility

Observable evidence	Sept 3	Sept 16
Search-query records	4	4
Pipe-language lines	4	0
Recognized fast calls	3	0
Unparsed slow calls	1	0
Numeric timeframe-like value	30 on all four	absent from stream
Unique retrieved URLs	14	40
Unique citation-context URLs	5	6
Unique retrieved-only URLs	9	34

Based on 1 identical prompt (148 characters, ChatGPT, 2 captures, Sept 3 and 16)

seerinteractive

Same prompt. Same everything on my end. But the exposed data structure just changed underneath me.

Our team was most interested in the numeric value of 30, because if it referred to days, it would say something about how ChatGPT treats freshness. Suganthan's article says reading it as a recency window in days is an assumption rather than a documented spec, and I agree with him. I saw the number. I don't know what it means. 

In the second complete capture, the field did not appear in the complete selected answer stream. That doesn't mean ChatGPT stopped using a time window, or that OpenAI removed anything. It means it wasn't exposed to me in that response.

I didn't believe it at first, and thought I'd broken my own parser. I spent a good chunk of time digging through the raw stream before I was willing to say the field was gone. And I only say it now because the stream completed clean, integrity checks passed, and no unfamiliar structure was sitting there that could have hidden it.

There's a separate thing I had to untangle here.

On September 16 I also had to patch the browser attachment process so the script would run reliably. That patch added a focus wait after permissions, but it didn't touch stream collection or parsing, so the browser maintenance and the change in what the stream exposed are two different issues.

What Happened When I Ran the Prompt a Few Minutes Apart

What surprised me the most is when I ran the identical prompt a third time, about eight minutes after the second run. Again, byte-for-byte.

Every query wording that ChatGPT generated differed between the two runs. Zero exact query strings shared. And the retrieved and cited URL sets moved too.

So now I had two separate observations stacked on top of each other. In the first set of runs I compared (two weeks apart), the data structure changed. In the second set (eight minutes apart), the search behaved differently.

Study: ChatGPT Search Data Stability

Same prompt, two different ChatGPT behaviors

Observation 1

 

Prompts run two weeks apart, on 9/3 and 9/16

Something changed between the runs, with the pipe-language disappearing

Observation 2

 

Prompts run eight minutes apart, on 9/17

The exact same prompt doesn’t produce the exact same search behavior

Based on 1 prompt (ChatGPT, runs on Sept 3, Sept 16, and Sept 17, signed-in Chrome)

seerinteractive

What the Data Tells Us (and What It Doesn't)

I want to be really direct here, because it’s easy to cross the line from useful evidence into overreach without meaning to.

The evidence can tell us what changed across prompts, turns, and captures for:

• Queries
• Retrieval-related URL observations
• Citation contexts
• Rendered citations
• Model identifiers (when they’re available)
• Answer content
• Other recorded structures

What it can’t tell us is ChatGPT’s complete internal reasoning, its ranking system, or anything happening outside of what’s exposed in that stream.

A URL showing up in a retrieval record doesn't prove ChatGPT read the whole page, or trusted it, or that it influenced the answer. And retrieved-but-not-cited does not mean ignored. I can't see the ranking, I can't see the reasoning, I only see what the browser was given.

But it does let you ask much better questions about your own pages. Was my page ever retrieved at all? Was it retrieved and then not rendered as a citation? Did ChatGPT write a query my page was never going to match?

Things I Still Don't Know

There’s one question I really struggled to answer: how do I tell whether ChatGPT stopped exposing something or if my parser was the issue?

I couldn’t answer that by staring at the final report. A zero in a field could have three possible reasons:

1. The thing didn't happen

2. The stream didn't expose it

3. My parser didn't recognize a new format
   

So I keep a category called unknown_shape. That's my system's warning label, not something ChatGPT sends. In the six-prompt suite I ran, four uncatalogued structures showed up across three turns, including one envelope named beacon_ui_response. I don't know what it is. It could be search evidence, it could be UI state, it could be telemetry.

Because those structures could affect how I read the queries and citations for those turns, I mark those families as an observed subset instead of calling them a total. That way an unfamiliar shape never quietly turns into a fake zero.

Same suite, another thing I can't explain: the stream exposed gpt-5-6 for prompts 1 and 2, and gpt-5-6-mini for prompts 3 through 6. A within-run model change. I don't know what caused it or whether it changed anything about the queries or sources.

I keep every raw stream after parsing for exactly this reason. If I figure out what beacon_ui_response is next month, I can go back through everything I've already captured and re-read it. Something meaningless to us today might be the clue we need in a month, once we’ve run it enough times to see a pattern.

If you’re wondering what to glean from all of this, here are my top takeaways:

1. ChatGPT is already shrinking what it reveals about its own search process. This is happening in real time, without warning. The exact same prompt exposed an entire layer of data that had disappeared two weeks later. You have to be continuously capturing raw streams to notice and react to these shifts.

2. You should be paying attention to more than the rendered answer. We’ve done enough studies to know that ChatGPT retrieves far more than it cites. You need a way to capture the retrieval layer, not just the citations. That’s your opportunity to find gaps and patterns that citation-counting tools will miss.

3. A single-run “insight” is a sample, not a reliable baseline. The same prompt, run minutes apart, can produce zero overlapping search queries. Anyone selling a one-off AI visibility audit or a simple “here’s why you’re not cited” answer is skipping over real, measured noise that exists within the system.

Instead of counting AI citations, look at what ChatGPT is retrieving and searching for. And run your prompts continuously. You can’t tell noise from a real pattern without running prompts over and over.

What to Do Next

This is a rare (and likely temporary) window where ChatGPT is showing its work, but that window is already closing. Here’s how to take advantage of it before ChatGPT starts gatekeeping all of that information.

If you're the one running the reports:

• Before your next reporting cycle, pick your single highest-priority prompt and run it five times. Log the queries, not just whether you got mentioned. If the queries move the way mine did, you now know the variance you're reporting inside.

• Freeze the prompt wording, using the same characters every time. If the prompt drifts, you can't tell whether the system changed or you did.

• Pull the list of URLs that got retrieved but weren't rendered as citations. That list is your closest thing to a gap list, as long as you treat each one as a question and not a verdict.

• Track which of the three zeroes you're looking at when a field comes back empty: didn't happen, wasn't exposed, or wasn't recognized.

If you're approving the budget for this:

• Ask whoever sells you AI visibility numbers how many runs sit behind a single score. If the answer is one, the number is a sample, and you should be treating position as a range.

• Fund repeat measurement rather than a one-time audit. The surface I'm measuring changed observably inside two weeks, and a snapshot from six weeks ago may be describing a stream shape that no longer exists.

• Separate the two kinds of measurement when you set expectations. Share of voice tells you whether the work produced a result. Diagnostic capture tells you what to try next. They answer different questions and you need both.

I'm building a tracker now that runs the same fixed prompt and the same canonical suite on a schedule so changes get caught as they happen. I haven't settled on the cadence yet - daily, twice a week, I'm still working that out.

Anyway, that's the gist. If you want to try this on your own prompts I'm more than happy to walk through how the capture works. And if anyone figures out what beacon_ui_response is before I do, please tell me.

If you want help building an effective AI search strategy, Seer tracks all of these platform and model changes so you don’t have to. Chat with us to learn about what we can do for your brand.

Methodology

Prompts came from a mid-market ecommerce SaaS client's commercial research set.

All captures used a real signed-in Chrome profile with the regular installed browser. Prompt text was SHA-256 hashed to confirm byte-identical reuse across runs. A 30-second delay was adopted between prompts as a precaution - the captures do not establish a causal blocking threshold.

Counts are per-turn records, not cross-run deduplicated domains. Different evidence families are counted at different grains and should not be converted into ratios.

Terms used above:

• Selected answer stream: The streamed response bound to the submitted prompt and rendered answer.
• Retrieved-URL record: A URL observed in a recognized retrieval-related structure. Does not prove full-page reading, trust, or influence.
• Retrieved-only: Retrieved under the current parser classification, not normalized as cited. Not the same as ignored.
• Absent from stream: The complete captured stream did not expose the recognized evidence, and no relevant unknown structure blocks that call. Does not mean backend non-use.
• Unknown shape: An unfamiliar structure may affect the evidence family, so recognized results are an observed subset.