Google just announced the rollout of AI Overviews (AIOs) to non-logged-in users in the US last week, along with some brand-new mobile and desktop layouts. These changes are set to shake up the SEO world in the coming days, weeks, and months. Since this is all so new, we're just starting to dive into these layouts and run studies to assess the impact.
A few days ago a headline on Search Engine Land caught my eye: “Google AI Overviews, organic results overlap jumps to 99%, analysis finds.”
It made me pause because, in the early days of AI Overviews (and back when it was still called Search Generative Experience), Google often pulled sources from way beyond the top 10 organic results - well at least according to our studies of ecommerce keywords and AIO results and branded keywords and AIO results.
A casual reader might take this headline at face value and not seek to understand the nuances of the research study. I shared my thoughts on LinkedIn, which sparked an interesting exchange with Mark Traphagen, who conducted the study.
I didn't have fresh data to underpin my opinion, so, I decided to run a study of 6,599 keywords on google.com in the US to go into further depth.
I’m not saying his study was incorrect, in fact my study corroborates Mark’s findings.
Mark’s findings:
- 99.5% of the AIO results contained at least one top 10 organic result.
My findings:
- 97.3% of the AIO results contained at least one match from the top 10 organic results
However, I felt it was important to provide some additional data and context around this - because I feel it’s going to be essential for the SEO community to get to grips with the new AI Overviews and how they relate to their existing organic ranking positions and pages.
So, when you break this down further, you can see that this tails off to close to zero.
The chances of finding a SERP result where the AIO results exactly match all the top organic results is very slim!
But the real headline findings from my study are:
- The Authoritas GOA Score™ tells us that across all keywords nearly 4 out of every 10 generative URLs will have matching organic URLs in the same SERP.
- The Authoritas OGA Score™ tells us that across all keywords just over 4 out of every 10 organic URLs will be ranked in the generative URLs in the same SERP.
- So, when you compare whole sets of results together for each keyword on average around 6 out of 10 results do not match.
The more detailed findings surround how beneficial it is to have the top #1 ranking organic position.
- There is a 66.5% chance of the number 1 organic URL ranking anywhere in the AI Overviews (oga_score_top_1) and a 38.5% chance of the number one organic ranking URL also ranking number 1 in AI Overviews (oga_score_top_1_1).
- There’s a 59.2% chance of the number 1 organic ranking page ranking in the top 2 AIO results (oga_score_top_1) and a 55.2% chance of it ranking in the top 3 AIO positions (oga_score_top_3).
However, Google is definitely grounding its AIO results more in organic results than it did in the early days.
Google is still experimenting, and one thing we can all agree on is that this will continue to evolve and we need to keep up.
These findings are from a study of 6,599 keywords across a range of categories. 1,159 of these keywords generated SERPs with AI Overviews, an AIO penetration rate of 16.9%.
I have shared the AIO to Organic Summary metrics on Google Drive - these metrics match the references to the formulae that appear in brackets throughout this article.
1. The Importance of understanding how well the new AI Overview listings and organic listings align
It’s really going to be very important to get a good comprehension of the relationship between the AI Overview listings, the organic rankings and the AIO Citations. In my opinion, evaluating and understanding this relationship is going to become as fundamental as understanding search intent is today.
Here’s just a couple of other reasons why understanding the alignment between generative and organic results is crucial:
- Visibility and Traffic: High alignment ensures that your top-performing organic pages are also featured in generative results, maximizing visibility meaning your hard SEO graft to date should still pay off.
- Competitive Positioning: In markets where generative results differ significantly from organic rankings, new competitors might emerge, altering the competitive landscape.
- Strategic Adjustments: Measuring alignment helps identify when you need to optimize existing content, create new content, or explore alternative strategies like paid search to maintain visibility.
- Search Intent: You can see whether the SERP and AIO results in particular are related to just the seed keyword or the seed keyword and a set of reformulated queries. This will help you optimise your content accordingly to improve relevance by measuring cosine similarity for the correct key phrases.
In many ways, understanding whether there’s a strong or weak alignment will help you better interpret the SERP results for the keywords you care about and lead to a more profound understanding of search intent (or at least how Google interprets this).
For example, a strong alignment may speak to Google being very satisfied with the quality and diversity of documents returned for your seed keyword, whereas a weak alignment may indicate otherwise and show that Google has reformulated your original seed query to show the user additional quality documents for adjacent user needs. I’m not going to delve into this any further here, if you are interested please read Rich Sanger’s excellent piece on AI Overview ranking studies and understanding AI Overview’s Source Selection.
So, this relationship is going to vary by category and within categories, e.g. I would expect the data to differ from YMYL topics like Health and Finance to say Food, Travel and Entertainment. In the SGE beta we have already seen many different implementations of AI Overviews, so it’s likely that this relationship will change for search phrases across the buyer journey with Google choosing to ground (or not) its results more or less firmly in the existing organic results.
Please, understand also that you can look at this question from both sides.
For example, these questions seem very similar but actually yield slightly different results:
- What percentage of organic results match the generative AI results?
- What percentage of generative AI results match the organic results?
And these questions aren’t quite deep enough either.
This is looking at ‘any’ rankings.
We really need to look at top organic ranking positions and how they relate to the top AIO positions (or vice-versa). E.g. How useful is a number 1 organic ranking in a world of AI Overviews?
This is what I have endeavoured to do below.
2. The Research Data
In this analysis, I have been specifically looking at the relationship between the generative results appearing in the AIO List on the right-hand side rail on desktop and the top organic results.
I’ve only studied this for desktop rankings so far, but I would be surprised to see a big difference on mobile. But that is a study for another day.
I’ve only been comparing page1 of Google, so there’s generally 8 to 10 organic listings (the average was 8.58).
The new AIO layout on the desktop devices typically has between 7 to 10 unique AIO listings in the right-hand rail (the average was 9.57).
This is from a study of 6,599 keywords across a range of categories. 1,159 of these keywords generated SERPs with AI Overviews, an AIO penetration rate of 16.9%.
3. Introducing the Authoritas GOA Score™ and OGA Score™
I mentioned you can look at alignment from both the organic and the AIO perspective, so I have done both and you can too using these formulae, just calculate these metrics at the keyword level and then aggregate it at category level and take the average scores.
GOA Score™ Formula:
The GOA Score™ (Generative to Organic Alignment Score) measures how well generative search results align with organic search rankings. Specifically, it looks at the overlap between URLs that appear in both the generative and organic result sets for a given keyword.
GOA Score™ = No. of unique generative URLs with a match in the organic set / No. of unique generative URLs
For this study, we get a GOA Score™ of 42.2%.
OGA Score™ Formula:
The OGA Score™ (Organic to Generative Alignment Score) does the reverse, measuring how well organic search results align with generative search rankings.
OGA Score™ = No. of unique organic URLs with a match in the generative set / No. of organic URLs
This is a simple analysis which compares sets of results together. E.g. You can visualise this in Venn diagrams.
For this study, we get an OGA Score™44.4%
The GOA Score™ tells us that across all keywords on average 4 out of every 10 generative URLs will have matching organic URLs in the same SERP.
The OGA Score™ tells us that across all keywords on average just over 4 out of every 10 organic URLs will be ranked in the generative URLs in the same SERP.
As you might expect, the numbers are in-line and the minor differences are likely down to small differences between the number of results for each type.
The flip side of this is that around 6 out of 10 results do not match!
But we need to dig deeper as this stat doesn’t tell you the whole picture you need to know.
High Alignment Example:
Consider this keyword phrase, “can’t pee after surgery”.
Here the GOA Score™ is 1.0 and the OGA Score™ is 0.33.
This means that all of the generative URLs also appear in the organic results, but not all of the organic results appear in the generative results.
This is evident when you extract the URLs from each SERP feature:
3 Generative AIO Results
There are only 3 AI Overview results in the list in this example and all are listed in organic in different positions, so the GOA Score™ is 1.0. 100% of the generative results for this keyword also rank in organic.
9 Organic Results
But there are 9 organic listings, the maximum number of matches is limited by the number of generative results, so the max OGA Score™ is 0.33. 33% of the organic results rank in the AIO results.
Medium Alignment Example:
Conversely, a keyword with a medium GOA Score™ of 0.5 and OGA Score™ of 0.5, indicates a 50:50 overlap between organic and generative results.
This scenario suggests that some different content is being favoured and provided by Google’s AI Overview. In the example, for ‘best email app for iphone’ below, it appears that Google is showing several useful links from the same site.
10 Generative AIO Results
10 Organic Results
Low Alignment Example:
Conversely, a keyword like ‘gifts for employees’ with a low GOA Score™ of 0.23 and OGA Score™ of 0.38, indicates a low overlap between organic and generative results and a much greater diversity of results being shown in AIOs.
13 Generative URLs
Only 3 of 13 generative URLs match organic, which results in a GOA Score™ of 0.23.
8 Organic URLs
Only 3 of 8 organic results match the generative giving an OGA Score™ of 0.38.
No Alignment Example:
For the 'blackout tattoo’ search term, the AI Overview results were completely different to the organic rankings.
8 Generative URLs
None of which match the organic ranking URLs, which results in a GOA Score™ of 0.0.
9 Organic URLs
None of which match the generative ranking URLs, which results in a OGA Score™ of 0.0.
What do the different tiers of alignment score tell us?
High Alignment: Focus on optimising your current top-ranking pages and securing multiple listings and citations in generative results. This ensures that your visibility remains strong across both organic and generative searches.
Medium Alignment: You’ve got a roughly 50:50 chance of your organic ranking being ranked in AIOs. You also could have an opportunity to achieve multiple different ranking pages if you have produced enough high quality related content. Whether or not you also need decent domain authority I cannot say.
Low Alignment: When alignment is weak, it’s essential to explore whether new competitors are taking the top generative positions, whether the ranking pages are offering additional context or covering adjacent user needs to the original keywords. Then you can re-examine your content, evaluate new content formats, engage in PR efforts and consider paid search options to cover gaps in visibility.
No Alignment: Google is ranking completely different types of sites and content and/or is broadening the scope of the original query to return additional content which users might be interested in. Study the different types of sites and content to determine whether you have what it takes to rank in the AIOs for this term or whether you need completely different content or a completely different type of site. If the latter, then partnering with these top-ranking sites may be a sensible approach.
3. Breaking Down the GOA Score™ & OGA Score™ by Ranking Position
This previous data looks at sets of results regardless of ranking position and clearly some ranking positions are better than others 🙂
After all, as SEOs we’ve been slaving away for years trying to get our sites or clients’ sites to the first organic ranking position and we all want to know how far this effort will take us in a new world of SERPs with AI Overviews.
These are the questions I immediately had and have endeavoured to answer below:
- How likely is it that your #1 organic ranking page also appears in the #1 AIO ranking position?
- How likely is it that your #1 organic ranking page also appears in the top 2 AIO ranking positions?
- How likely is it that your #1 organic ranking page also appears in the top 3 AIO ranking positions?
- How likely is it that your #1 organic ranking page appears in any AIO ranking position?
To provide a more insightful analysis, we can break down the GOA Score™ by specific positions:
So, we can calculate the GOA Score™ as before, but this time we limit the set of generative results to the Top 1, Top 2, Top 3, etc, and compare each page(s) with all the organic pages for each keyword.
- If a page ranks in 1st position in the AIO results, there is an 83.7% probability that it was picked from the top 10 organic results. (goa_score_top_1)
- If a page ranks in top 2 positions in the AIO results, there is a 79.4% probability that it was picked from the top 10 organic results. (goa_score_top_2)
- If a page ranks in top 3 positions in the AIO results, there is a 73.5% probability that it was picked from the top 10 organic results. (goa_score_top_3)
- Finally, as we’ve already established (see table above), if a page ranks in any position in the AIO results, there is a 42.2% probability that it was picked from the top 10 organic results. (goa_score)
Then, we can calculate the OGA Score™ as before, but this time we limit the set of organic results to the Top 1, Top 2, Top 3, etc, and compare each page(s) with all the generative pages for each keyword.
- If a page ranks in the 1st place in the organic results, there is a 66.5% chance that it will rank in some position in the AIO results. (oga_score_top_1)
- If a page ranks in one of the top 2 places in the organic results, there is a 59.2% chance that it will rank in some position in the AIO results. (oga_score_top_2)
- If a page ranks in one of the top 3 places in the organic results, there is a 55.2% chance that it will rank in some position in the AIO results. (oga_score_top_3)
- Finally, as we’ve already established (see table above), if a page ranks in any position in the top 10 places in the organic results, there is a 44.4% chance that it will rank in some position in the AIO results. (oga_score)
But going back to my questions. What’s the chance that my number 1 organic ranking also ranks #1 in AI Overviews?
Like marthastewart.com is for ‘How to kill wasps” - This is about as good as it can get from an SEO perspective - ranking number 1 with a Featured Snippet and ranking first in the AIO List on the right-rail (visible without clicking above the fold).
Well, according to our research, there is a 38.5% chance of the number one organic ranking page also ranking in the first most prominent pre-click right-rail position (oga_score_top_1_1).
There’s a 50.8% chance of it ranking in top 2 AIO results and a 56.9% chance of it ranking in the top 3 positions.
So, getting 1 in every 3 or so of your #1 ranking organic keywords into the first AIO List result is probably the most you can target right now.
Comparing 'Top' vs. 'Any' Scores
‘Top Scores’ focus on the exact positional match between organic and generative results. They are crucial for understanding how well your top organic content is performing in generative results.
‘Any Scores’ provide a broader view, showing overall content overlap regardless of position. This helps identify cases where generative results are derived from organic content, even if not in the same order.
In previous studies we looked at exact URL matches and matches at a domain level where a different page from the same site is ranking.
I’ve run it again for this analysis, as you can see there is a 66.5% chance of the number 1 organic page ranking anywhere in the AI Overviews (oga_score_top_1) - see the first green exact match bar. But on occasions Google might choose a different URL to show from the same domain ranking first in organic search - so overall this will bring the chance of your domain ranking anywhere in AI Overviews if you have a number 1 organic ranking to around the 70% mark.
Please remember, it’s still early days.
I’ve even got questions about how we define which is the number 1 ranking position with these new AI layouts as they do break the traditional rank tracking model which selects the topmost listing as the number 1 rank according to its x/y position on the page.
That being said, it’s worth looking at this data for the keywords that matter to you, because these metrics will change depending upon which sector you are in and the type of keywords that you are using.
4. Generative to Organic Alignment by Category and Ranking Position
As I mentioned previously, the alignment varies considerably by category and will vary for different types of queries within each category. This will become abundantly clear if Google eventually rolls out the multiple different types of AIO layouts we have seen in the SGE beta.
The GOA Scores™ show you that there is a much stronger grounding in high performing organic results in categories like Health, Finance and Travel then there is in Home and Food & Drink.
This shows you the importance of getting in-depth SERP data with fully expanded AI Overview results for all the keywords you care about. But I would say that wouldn’t I? And ‘Yes’ we can help you with our AIO-enabled Keyword Rank Tracker.
(You can download this table on Google Drive).
5. Practical Applications of GOA Score™ and OGA Score™
High Alignment Strategies
- Content Optimization: Focus on refining the content that’s already performing well organically. Ensure it remains relevant for generative results.
- Generative Listings: Aim to secure multiple listings in the generative results and citations in the AI Overview summary text by optimising various aspects of your content.
- SEO Visibility Tactics: Explore ways to enhance the visibility of your content that may be hidden in generative carousels.
Low Alignment Strategies
- New Content Creation: Develop new content tailored to the formats or topics favoured in generative results.
- PR and Outreach: Collaborate with sites that are ranking well in generative results to boost your brand’s visibility.
- Paid Search Considerations: If organic visibility is lacking in generative results, consider investing in paid search or shopping ads.
Vertical-Specific Insights
Different industries and keyword categories show varying alignment patterns. Understanding these nuances will help you tailor your strategy:
High Alignment Verticals: Focus on maintaining and optimizing your existing content.
Low Alignment Verticals: Explore new content strategies and consider a mixed approach of organic and paid efforts.
6. Further Study: Exploring GOA Scores with Reformulated Keywords
As we continue to explore the alignment between generative and organic results, our upcoming research will focus on analysing GOA Scores™ using reformulated keywords. This approach is inspired by Google’s own methods, as detailed in their patent on query reformulation. By examining how AIO results align not just with the original query but with related, reformulated queries, we can gain a more comprehensive understanding of how Google’s AI generates these results.
In fact, Rich Sanger recently critiqued previous AI Overview ranking studies from AWR, Authoritas and SE Ranking in his post, “Google AI Overviews: Do Ranking Studies Tell the Whole Story?”. Rich highlights that looking at AIOs through the lens of query reformulation reveals a more complex and nuanced picture of how these AI-generated results are constructed.
In my previous AIO research, I found that only around 50% of the AIO results were matching the top 10 organic results for the original seed keyword.
However, this number increased to about 70% when including the top 10 organic results for the ‘Related Searches’ that Google showed for the original query. It increased further still to circa 74% when considering the top 10 organic results for five reformulated queries derived by our algorithm.
We’re about to do a study with Rich which goes into even greater depth than this study, to decode the impact query reformulation will have on our optimisation strategies for AI Overviews.
Stay tuned for insights that could significantly impact how you approach SEO in this evolving landscape coming in the next couple of weeks.
7. AI Overview Optimisation Strategy
I will write more about this in due course after we’ve conducted more experiments but suffice to say having an understanding of whether your existing organic ranking pages are going to cut it in a world of AI Overviews is essential. To do this you’ll need accurate SERP data for your keywords and domains.
Keyword Ranking Tracking for AI Overviews
In-depth keyword rank tracking of AI Overviews is crucial because these new layouts significantly alter how search results are displayed, making it challenging to determine your true ranking position.
But you must be able to check the SERPs without a user interaction that expands the AI Overview and with a user action that expands the AI Overview by clicking the ‘Show More’ and/or the ‘Show All’ link.
You must also ensure that the rank tracking tool captures every ranking in the AIO List or AIO citations from the AIO Summary text (Oh, and you’ll need that too!). The AI Overview is like a SERP within a SERP!
By having the ability to track rankings with AI Overviews both on and off, you can compare side by side to see the direct impact on your visibility and performance. This comparison helps you understand how your content fares in the traditional SERP versus the new AI-enhanced landscape, allowing you to refine your SEO strategy and ensure that your content remains competitive in both scenarios.
A closer look at the SERP features reveals for which keywords Google is returning AIOs in the SERPs and whether you are ranking in them (icon is highlighted in green) or not.
You can then dive into the SERP to see in detail which pages are ranking in the AI Overview and how far down the page your original organic ranking or featured snippet is driven.
Optimising for AI Overviews
Now you have data, you can start to think about how you derive the insights from the data to give you a framework for determining what you need to prioritise.
If you look at this example table, you can see that you can easily build an optimisation prioritisation framework by bucketing your keywords into the following categories and subcategories, based on whether you have:
- A ranking page in organic
- A ranking page in AI Overviews
- Whether your AI Overview ranking is visible without a user interaction.
Be sure to only include keywords where AI Overviews appear irrespective of whether you are ranking in organic or not.
Now, we can group our keywords into 4 sets using the YES/NO values.
Obviously, you can make this more sophisticated by looking at exact ranking positions and whether you are visible or not in the AIOs without a user having to click and you can further break your sets into subsets by using the AIO Visibility column.
E.g. ‘aardvark’ and ‘a different aardvark’ both rank in Set A (both rank in organic and AIOs) but only ‘aardvark’ has a visible AIO ranking - so it may be prudent to focus on improving your ranking for ‘a different aardvark’ first.
Finally, of course, it would also be prudent to consider how closely the generative and organic results overlap by adding in the GOA Score™.
This way you can break all your keywords down into different sets and subsets and prioritise keywords where you need to:
- Improve AIO rankings where you are ranked but not visible.
- Break into AIOs where you are ranking well in organic and there is a high GOA Score™.
- Improve organic rankings where there is a high GOA™ Score and an improvement in organic rankings could lead to a position in AIOs.
- Review content where you are not ranking in either organic or AIOs.
8. Conclusion
Key Takeaways
Our research shows that nearly 6 out of every 10 generative URLs will have NOT have matching organic URLs in the same SERP.
Additionally, there is a 38.5% chance that the number one organic ranking page also ranks number one in AI Overviews. These findings highlight that while there is some alignment, a significant portion of AI Overview results come from sources beyond the top organic rankings.
GOA Score™ and OGA Score™ are essential for understanding the alignment between organic and generative results and adjusting your SEO strategy accordingly.
Start measuring your GOA Score™ and OGA Score™ today to stay ahead of the competition!
We can help you use the tools and insights discussed in this post to optimise your strategy for the evolving search landscape.
And if you like, please do book a meeting with me - even if you don’t yet plan on signing up to use our enhanced AI Overview rank tracker, I’d love to get your thoughts on the new AI Overview layouts and what it means for optimising your site(s) in 2024 and beyond.
