People will use ChatGPT to find answers in 2024 are your search KPIs ready?

With generative AI predicted to drive a 50%+ drop in organic search by 2028, clinging to 2023’s (outdated) success metrics will lead to a lot of missed projections in 2024/2025.

A recent prediction from Gartner states, "By 2028, brands’ organic search traffic will decrease by 50% or more as consumers embrace generative AI-powered search." This rapid adoption of generative AI (GenAI) in search engines will significantly disrupt CMOs' ability to harness organic search to drive sales.

They could be right or wrong, I’m not here to debate that. I do think as search marketers, we need to start preparing our organizations for a new set of KPIs. Are you going to continue your search strategy optimizing for 2023’s metrics in 2024. The search landscape is shifting dramatically, and we need forward-looking strategies grounded in that data we have today, not the data we wish to have.

The 1 minute version:

No one knows how much Generative AI will change your traffic, but there are ways to try to back into the impact.

The 1 question every CMO should ask their search agencies:

Lets say chat search becomes the standard tomorrow – how would you determine how big the market is, my visibility in that market and impact of that visibility on leads and sales??

Forward looking marketers need to respond to that question

Remember: Current chat search engines don’t give marketers any official data

The issue is that we have a lack of data from any providers around usage of generative AI assisted search. In today's world, you can look at a keyword and determine market size, competition, and your visibility in a plethora of tools ranging from Google to Bing, to SEMrush to Ahrefs to Moz.

Where is the set of metrics that answers the question:

“Are my customers shifting to chat search engines and are my competitors more visible on those?”

Question we must answer in 2024:

How many of my customers are moving to chat search?

It is still critical to know audience interest and demand - whether 10 or 10M people seeking a solution. Whether it's Google Ads, Bing Ads, TikTok, Ahrefs, Moz, SEMrush, all of these tools can give you some version of market sizing.

If we look specifically at search, historically the tooling has been very robust on reporting monthly search volume for keywords (giving you audience sizing). Now people are starting to shift how they search, embracing new generative AI search experiences.

We are completely lacking data and metrics to understand what percentages of our customers are migrating over to AI assisted search. Knowing that is critical to determining how much to invest.

In August 2023, Bing claimed they would release data on AI assisted search volume. But so far, neither Bing nor Google have delivered any transparent data or metrics in their webmaster tools and it’s not expected that they will anytime soon.

As search marketers, we can’t wait for this data, we must start exploring creative proxy solutions in the interim. The 50% organic search drop predicted by Gartner makes this an important thing to start working on.

Hypothesis: We believe words per query will increase as people get used to chatting with AI search assistants vs shouting orders at a search box.

Step 1: Get paid search query data (sorry performance max peeps)

Step 2: Count up the number of words per query & trend that out over time.

We're doing this for every client, automatically every month, using paid search query data with us, enabling us to see how much growth are we seeing in longer queries.

Check out supernova, this is just 1 of about 50 innovative insights we have automated.

While proxy approaches will have limitations, we believe some visibility tracking is better than none.

I hope I don't have to use this metric for long but in the absence of any metrics from any of these platforms this is better than using Gartner 's "50% of all SEO traffic is wiped out in 5 years" prediction unchecked.

Question we must answer in 2024:

What is my competitor chat search market share?

A critical component of managing any search marketing program is understanding your competition - who you are up against in both organic and paid search results. Tools have given us rich competitor data, enabling analysis of factors like their rankings, spending levels, and visibility. We use this data to determine the likelihood that we can compete and get visibility with reasonable effort and investment.

As later sections of this post will cover, while the engines themselves currently provide no transparency, we believe clever workarounds with existing data can help fill gaps and get competitor tracking, don’t hold your breath assuming that these search engines are going to share this data!

The techniques we used to measure competitor visibility and rankings we use to determine our own visibility! So let’s move to that part below.

traffic is wiped out in 5 years" prediction unchecked.

Question we must answer in 2024:

How visible is my brand in chat search?

Tracking rankings has been a core SEO report since I started in 1999, remember webposition gold?.

Historically we could answer questions like:

Do we rank in the top 10 organic positions?
What % of keywords rank in the top 3 organic?
How has our average position changed over time?

Ok, in a world where chat assisted search rules the roost, and we aren’t given data from search engines or existing tools today, how do you help answer these questions? How do you know how much to invest in winning at chat search if you don’t even know whether you are already winning or not?

You could do this by manually copy and pasting questions into ChatGPT and see if your brand shows up, like we are doing below. It is a start!

Doing this manually on millions of keywords, sucks – so lets scale it.

Scale this with sheets & Claude 2 for Google Sheets or ChatGPT for Sheets.

Here is the Seer post on how to do this and the video too.

Both of these tools will let you run questions at scale and retrieve the chat answer in a sheet. That then will allow you to see how often your brand shows up. This will let you get your own visibility score and also allow you to track your competitors visibility at scale. Both of which are critical metrics for any search professional doing search.

In 2024 track visibility -
In 2025 track impact too

Visibility is just the start, as performance marketers, tracking traffic and conversions is in our blood too, but that is going to get even harder. Knowing how many leads are coming in overall will help you a ton.

Now that you are tracking visibility both in Traditional and Chat Search you can start to determine your market share in both, if you have higher market share in Traditional search

If your visibility in traditional search and chat search is high & leads dip, your dips in SEO traffic on non brand may have been offset by chat where people are searching for brand.

Imagine if your leads / sales are down and your traditional SEO visibility is the same, maybe your customers are moving more to chat.

If your chat visibility is poor, you might need to be super serious about optimizing for chat fast.

Click through rate data in traditional search is available at the search term level in paid and organic. (Performance Max Campaigns wont have this data).

Click through rates in this generative AI assisted search world will be crushed, use Wikipedia traffic data to determine if you are getting hit too.

Did you know that Wikipedia traffic data is publicly available?

I see Wikipedia ranking for a lot of top of funnel queries on page 1 and a lot of informational queries. If your traffic is declining and you wonder if it is chat search, you can take all the queries where Wikipedia and you rank on page 1, and run their traffic drops against yours – it could be seasonal.

In a Bing study, you can see their take on which queries where chat created better user experiences vs search and vice versa, thankful for Fabrice Canel at Bing who shares a lot with the community.

There's no analytics reporting for chat search, we tried.

You are completely out of luck if you are trying to get referral traffic from chat search engines. Our team hunted down the referral strings from chat search engines to try to understand how those users behave when they get to our site. But alas, that was taken away.

How are 2024’s marketers going to determine the throughput (Traffic) coming from that visibility in chat search?

I don't have good answers here. But that's our job - to try to figure out the next best answers that keep us from relying on best practices and hunches.

Determining traffic with Google’s SGE (search generative experience)

Google continues to play around with layouts of their search generative experience, which means knowing if you are visible in the search generative experience is going to be quite important.

Monitoring Google's SGE impact on traffic & conversions in 2024

• Manually screen grab SGE results
• Run OCR on the screen grabs (using ChatGPT)
• Find out if your site or your competitors sites are showing up in the links provided
• Monitor how the combination of your visibility + SGE layouts impact on your traffic & conversions to the pages. This will help you make better predictions on searchers moving to SGE will impact click through rates, traffic, and conversions on your existing visible links.

I think the layout will be in a state of flux for years as Google has top manage getting people great answers without destroying its ad revenue models.

Mike King has done some amazing work on running queries against Google's search generative experience at scale (we were doing it manually, so he's the guy to follow on this), so peep his read.

I’m looking forward to seeing how our friends at companies like Traject, Conductor, Nozzle, Stat, and SEMrush will scale these, because doing them manually is too much work, and Mike’s solution might not be feasible for most companies.

Monitoring ChatGPT's impact on traffic & conversions in 2024

If retrieval-augmented generation continues to be a major way that ChatGPT surfaces fresher answers, then you can determine which n-grams trigger retrieval augmented search in ChatGPT from your paid search queries (sorry Pmax), to prepare for any threats coming.

Once you figure out those n-grams, you can then isolate them and say for instance that when someone adds a location or "2023" at the end of their query, that ChatGPT is going to go out and get results to make the answer better.

Use your log files to find ChatGPT visiting your site from queries that trigger RAG

Jan-Willem shared that really blew my mind. When ChatGPT goes out to scan the web for answers using Bing, they leave behind a user agent that you can find in your log files – here is how you can use a wordpress plugin to track this for you over time, which again helps you determine how many of your customers are doing searches where a page on your site gets triggered as an answer, you can compare that triggering to traffic.

We can't take this stuff lying down and pouting about the lack of visibility. We must play with the hands we're dealt and come up with the next best answer until these things improve.

This week it was reported that Bing Chat is now placing favicons next to links in chat search, potentially in an effort to try to get more people to click on those results.

People don't have fewer questions they just aren't using Google as much to answer them. Go where they are going.

Step 1: Admitting the old metrics (MSV, Rank, CTR, Traffic) don't apply to chat search. Be honest about that, you must keep tracking them, just add some new metrics as well to see where customers are going.

Step 2: Educate your execs on why traffic numbers from search are going to miss a big part of the "are our customers finding us" story. Then take a few of the metrics above and apply them, create a "customer search to Gen AI index" internally. (I have other ideas on this for another post).

Step 3: Innovate. The search engines are not giving us data to determine customer shifts that impact our businesses. The tool providers do not seem to have solutions in place yet. You have the opportunity to build new ways, and I hope we showed you some of them above.

Your traditional SEO metrics will not tell the complete story of the work you've been doing over the years, so get in front of that!