Insights

Use Google Analytics to Understand if Your Customers are Using Bard & Other AI-Search Tools

Facts over Frenzy

It's easy to get swept up in the latest buzzwords and trends in the forever evolving world of digital marketing. But when it comes to the impact of Artificial Intelligence (AI) on search marketing, it's time to cut through the noise and focus on the facts.

Welcome to our 'Facts over Frenzy' series, where we use readily available data to understand how AI is truly reshaping the search marketing landscape.

Table of Contents

AI-driven search traffic overview

7 brands analyzed for ai-driven search traffic trends

Tracking AI-Driven Search Traffic

404 Increases are an AI-driven search signal

Opportunities Next Gen Marketers have during this shift

So, whether you're a seasoned pro or a curious newcomer, read on to discover the real story of AI's role in search marketing.

AI-Driven Search Traffic in Google Analytics

For AI-driven search tools like chat-gpt.org, Microsoft Bing Chat, Google Bard, Neeva (before shutting down), and Writesonic; we can now see traffic showing up in our Google Analytics - or analytics tool of choice - alongside our other traffic channel groupings:

ai-powered search in google analytics

How a user gets from an AI tool to your website

Use cases for AI chat are growing daily.

Sometimes folks are trying to figure out where to start on their newfound love of baking...

openai-chat-links-1

or trying to track down the best bakeries to go to for inspiration:
links in bardEither way, links are slowly beginning to trickle into results in platforms like Chat GPT, Bard, and Bing Co-Pilot.

And our analytics team at Seer Interactive figured out a way to see some of that data in Google Analytics.

Read on to see what we found when we looked in aggregate across a handful of clients.

 

7 brands analyzed for ai-driven search traffic

Monthly sessions peaked at 177 sessions in aggregate

While we believe it's worth setting up the tracking to make sure you can understand if and when AI-driven search traffic starts to ramp for your business.

But, from initial findings, we aren't finding any brands at this time where AI-driven traffic is becoming a significant part of their traffic.

ai-driven search traffic 7 brands analysis

Neeva drove 64% of ai-driven search traffic

Across these 7 brands - Neeva drove the majority of the ai-driven search traffic that we're able to identify at this time.

The major drop from May to June 2023 is the deprecation of Neeva.

neeva is driving the most traffic

404s increased across all brands

All of the brands seem to be experiencing a mixture of outdated URLs and hallucinations as seen by the "null" and "(not set)" landing pages across these 7 brands.

'Login' also showed up for one of our SAAS clients.

landing page 404s increase-1

 

How to track AI-driven search traffic

Copy Seer's 'AI Signal' Looker Dashboards (2-Minutes)

As AI continues to influence search marketing, it's important to understand how your digital presence is affected.

[Pro Tip] Traffic to your site from AI tools is not the only data input you need to understand AI's impact on how users find your business. (jump to additional AI analyses)

But it is a quick and easy start to understanding if there's instances of your brand being linked out from those tools.

This is where our free Looker Dashboard(s) come into play.

The views will allow you to track AI-search traffic landing on your website directly in Google Analytics.

ai-driven search traffic looker dashboardd

Create a copy of Looker Dashboard for Google Analytics 4 Accounts:

Create a copy of Looker Dashboard for Google Analytics 'Universal Analytics' Accounts:

Dashboards use referrer traffic data to capture trends

What is referral traffic?

Referral traffic comes from another website, social media platforms, or any other assets that link to your site.

Another way to think of referral traffic is that it is traffic that comes to your site from sources other than search engines and paid media.

Tracking varies by analytics tools and websites.

What AI-Tools are driving referral traffic?

Our Analytics Development team surveyed several AI tools and found:

  • 3 platforms that suppress referral information on outbound links
  • 5 platforms that provide referral information on outbound links
  • 1 platform that was deprecated (but previously providing referral info)

Perplexity

❌ Does not provide referrer

Will be dark traffic

chat.openai.com

❌ Does not provide referrer

Will be dark traffic

Google's Search Generative Experience (SGE)

❌ Does not provide referrer

Traffic will come in as Organic traffic from Google.

chat-gpt.org

✅ Does provide referrer

 

Microsoft Bing Chat

✳️ Does provide referrer ; not available in Google Analytics

Available in Microsoft Edge. Bing chat traffic comes through just like organic search traffic, so in order to get insights from Bing chat, you’ll want to consider getting that data directly from Bing Webmaster Tools.

Google Bard

✅ Does provide referrer

 

Neeva

✅ Does provide referrer

Launched: June 2021

Deprecated: June 2022

Writesonic

✅ Does provide referrer

 

As this is a rapidly-moving part of the industry, sites and domain names may change, so a channel grouping will require regular maintenance.

Withheld referrer data came in two ways:

1. Same-origin Referrer Policy:

    • What it is: A policy set by websites to control the referrer information sent when users navigate from one page to another.
    • Effect: When a user clicks a link on a website with this policy, the referrer information (i.e., the originating page's URL) is only sent if the destination page is on the same website. If the user is directed to a different website, no referrer information is sent.
    • Example: If you're on chat.openai.com and click a link to another page on the same site, the referrer information is sent. But if you click a link leading to a different website, no referrer data is shared.
  1. rel="noreferrer":

    • What it is: An attribute added to specific links in HTML code.
    • Effect: When a user clicks a link with this attribute, no referrer information is sent to the destination page, regardless of whether it's on the same website or a different one. Additionally, it behaves as if another attribute, 'noopener', is also specified, which means the new page won't have access to the previous page's window object.
    • Example: If a website uses "rel=noreferrer" on a link, and you click it, the destination page won't know which page you came from.

In essence, while both methods prevent the sharing of referrer data in certain situations, the "Same-origin Referrer Policy" is a broader website-wide setting, whereas "rel=noreferrer" is applied to individual links.

Making decisions with this data

We have 3 considerations for you before using this data to make decisions.

1. This is not holistic data - don't rely on raw numbers

As noted above, not all AI-search tools pass referrer information to your analytics tools, so this is not a holistic look of all ai-driven search traffic.

Instead, look at this as a percentage of total traffic and set some baselines. As that percentage rises, create a milestone where you'll start to investigate it more heavily and create strategy around how to handle and harness this traffic.

[Pro Tip] Set-up a AI-Powered Search channel in your analytics account. Jump to setion on how to create a Channel.

2. Run additional AI-impact analyses

Zero-click searches will become more common in AI-driven search, as it removes friction for the user in providing the answer they need without additional action(s) the user needs to take (i.e. a click and often website friction like pop-ups and un-needed supplemental information.)

We've created two analyses that attempt to understand more at the impression level, what will be impact by AI-driven search first:

In the following sections, we'll dive deeper into how AI-driven search traffic is tracked and what it means for your website and business.

404 increases are another signal to watch

We identified some nuances about ai-search driven traffic through our research that can result in spikes in your 404 pages. Here's the quick hitter list:

  • Some traffic to ChatGPT traffic will be to non-existent pages 
  • Large Language Models (LLMs) may have old URLs in them
  • LLMs may also generate URLs from educated guesses.
[Pro Tip] Monitor 404s tab available in Looker dashboards.

404 increases led us to "chat.openai.com" referrer

Recently, we worked with a client to better understand a significant jump in 404 “page not found” errors that occurred on January 29th, 2023. 

What we saw was an increase of traffic coming from chat.openai.com. This must be the day this brand's links started appearing in the chatbot's results.

We also saw that traffic abruptly drop off on February 18th, 2023. If you're reading this post top to bottom, you'll recall from above that "chat.openai.com" is one of the tools that now blocks a referrer from being passed to Google Analytics. Our best guess is that is around the time chat.openai.com implemented their same-origin referrer policy, blocking the referrer string.

Here’s a screenshot of that in Google Analytics:

chatgpt-ga4

Non-existent URLs from AI tools were increasing 404s

We discovered two main reasons for non-existent URLs to be showing up in AI chat:

1. Outdated URLs

Outdated URLs in Large Language Model (LLM) training data are a natural consequence of how these algorithms work.

ChatGPT 3 was trained on books, Wikipedia and other web crawls coming from June 2020, while ChatGPT 3.5 and  ChatGPT 4 are only aware of historical phenomena prior to September 2021.

ChatGPT cannot access updates to URLs prior to those time frames.

2. The Extrapolation Effect (AKA “Hallucinations”)

Multiple reports have come out about ChatGPT chat responses that are factually incorrect.

In one example, the British dailymail.com reported multiple requests for articles that never existed. They found out that some article citations from ChatGPT had been made up.

This effect has been called “hallucination”, and stems from the foundational mechanics of AI algorithms: they are great at choosing “which word comes next”. However, this sometimes leads to wrong choices, and wrong answers, including URLs.

Use 404 page increases as an opportunity to delight

No matter the reason for the traffic, it presents an opportunity to capture these users’ attention, delight them, and convert them.

If you are experiencing a rise in 404 “page not found” traffic, we believe there are some ways to turn a “sorry, we don’t have what you want” message into a “you came to the right place!” message.

Below are some ideas for re-capturing this traffic.

Let Your 302 Redirects Live a Little Longer

Since LLMs have crawl data from History Past, links in answers have a chance of being old.

If you were considering removing redirects from recent content migrations, consider leaving them in place for a bit longer to ensure that users will get to the content you want them to see.

Improve your 404 page User Experience

It’s not often that you encounter a 404 page “in the wild”, but while working on this article, we encountered several.

And we can definitely say that a 404 page is like running into a brick wall from a user experience perspective.

Provide Search Functionality on 404 Pages

Make it obvious to the user that they can easily find what they’re looking for.

Want to take it to the next level? Do some of the work for them: extract significant words from the URL by removing stop words and preemptively search your own site, displaying your best content to users looking for more.

Provide other content-finding or product-finding tools

Showcase your Featured Articles, Top Content, Frequently Asked Questions…whatever it takes to help reorient someone who may be lost, and engage them!

Set up AI-search traffic channel in your analytics account (<10 minutes)

It's going to feel good to get ahead on something for once, won't it? Take the 10-minutes. Your boss will thank you later.

Google Analytics 4

  1. Open Google Analytics 4
  2. Navigate: Admin > Data Settings > Channel Groups
    chat-gpt-ga360
  3. Enter 'Group Name' - we used "Channels 2023"
  4. Enter 'Channel Name' - we used "AI-Driven Search"
  5. For conditions use "matches regex" and copy / paste this regex formula:

    (.*gpt.*|(.*google.*bard.*|.*bard.*google.*)|.*neeva.*|.*writesonic.*)

6. Save channel

ga4 channel directions

PiwikPRO:

  1. Open PiwickPRO
  2. Navigate: Settings > Custom channel grouping > Add a custom channel grouping

channel-grouping-piwik

Adobe Analytics

  1. Open Adobe Analytics
  2. Navigate: Admin > Report Suite Manager > Marketing Channels > Marketing Channel Manager

chat-gpt-adobe-analytics

 

We love helping marketers like you.

Sign up for our newsletter to receive updates and more:

Erin Simmons
Erin Simmons
Associate Director, Marketing
Jessica Propst
Jessica Propst
Sr. Manager, Google Analytics Program
Chris Baker
Chris Baker
Lead, Digital Measurement Solutions
John Varghese
John Varghese
Sr. Manager, Technical SEO
Roderick Brownridge
Roderick Brownridge
Sr. Associate, Digital Measurement Solutions