SERP Organic and AI Overview Volatility Research

I’m pleased to be able to release our latest in a series of studies into the impact of generative AI in the SERPs.

This time we looked at how volatile different aspects of Google’s search results were to try and understand whether measuring the rate of change of the SERP, and the pages ranking in organic and in AI Overviews would be helpful in determining how and when to optimise your pages to improve your overall SEO visibility.

The SERP reflects the ever changing world around us and we should expect a level of change that is consistent with this changing world, as companies and products launch and fail, world events unfold and new stuff happens!  This is of course just what every well informed consumer wants when using a traditional or AI search engine, but there’s an obvious downside of this flux in the SERPs for businesses trying to stay relevant as illustrated below.

‍

TL;Dr

• This study found that AI Overviews are very volatile and that over 2 to 3 months you could reasonably expect 70% of the pages ranking in AI Overviews to change.
• There is a greater rate of change in pages ranking in AI Overviews than there is in organic ranking pages.
• The Pages ranking in AI Overviews change independently of the organic ranking pages.
• The ranking pages in the AI Overviews change independently of the generative AI Overview text snippet.
• There is a weak correlation (0.19) between a SERP Layout change and a change in organic rankings changing.
• There is a very weak correlation (0.04) between a SERP Layout change and a change in the AI Overview ranking pages and a slightly less weak correlation (0.14) with the number of generative results appearing.
• Just because the SERP layout has been modified, it does not necessarily follow that there will be significant changes in organic or AIO rankings.

Methodology – Data Collection

Our research was based on a categorised keyword list of 11,203 keywords and SERPS collected from Google.com in the US on three separate occasions (23^rd August 2024, 17^th October 2024 and 17^th January 2025).  This allowed us to compare various aspects of the SERP data we collected to observe and analyse what has changed.

First observation window: This looked at an ~8 Week period between 23^rd August 2024 to 17^th October 2024. 

Second observation window: This looked at an ~13 Week period between 17^th October 2024 to 17^th January 2025.

We chose the same keyword list that we have been regularly using for our collaborative AIO research with Rich Sanger.  Data was collected using desktop devices.  There is a small chance of some differences appearing were we to re-do this study using mobile devices.

Methodology – How to measure SERP Volatility

There are a number of SEO tools that already measure fluctuations in the SERP results to help SEOs understand if there is an algorithm update happening.  I’ve looked at how each tool calculates the level of change in the SERP but not found too much evidence outlining their respective methodologies.

Most brands will explain what low, medium and high means, but they don’t actually tell you how it’s calculated, which is why if you check them all you will get slightly different perspectives on a day-to-day basis.

For this reason, if you want to know,  “Has Google changed or is it just me?”  you need to understand what is being counted and how it is calculated to understand whether it’s truly representative of the data you care about.

Here’s just a selection of SEO tools that calculate Google algorithm volatility.  I’d wager a small bet, that at the time of writing this at least, that not a single tool yet takes into account the AIO results into their volatility scoring models.

‍

If you don’t understand how a metric is calculated, it’s difficult to put it to good use and also difficult to determine how much faith to place in it when making decisions.  So, with that in mind, I have shared below how we have measured the following different aspects of change in the SERP.

Levenshtein Distance

Firstly, if you want to get into the nuts-and-bolts of it all then our starting point was the Levenshtein Distance algorithm which tells you how different two strings or sequences are.  It does this calculating by how many edits are needed to change one list into the other list – here is a good explanation for beginners if this is new to you.

I’ll get into how we have applied this formula below, but first you need to know that we were interested in measuring the rate of change of these four SEO related factors. 

Example of how we calculate SERP Volatility

For SERP volatility we look at two aspects of the SERP features that appear and the final volatility score is the weighted sum of these two scores:

• Order Volatility (Weight 70%): How much has the order of the SERP features changed between the two sequences. It uses weighted Levenshtein distance giving more weight to changes at the top of the SERP.
  
  
• Presence Volatility (Weight 30%): How much the presence of specific SERP feature result types has changed between the two sequences. Some SERP features will have a bigger impact than others, e.g. AI Overviews and Featured Snippets always appear above the organic results, so within this score there are more important and less important SERP Feature types as set out in the table below:

(For the data scientists reading this) we chose exponentially decaying weights with a base of 1 and decay rate of 0.7.

We can interpret the SERP Volatility as a measure of how great a percentage of the maximum possible change has happened in a period. 

Example of a SERP with very high SERP Volatility – “how to repair scratched wood floor”

‍

‍

Example of a SERP with very low SERP Volatility – ‘loan a kindle book to a friend’

You can see that the layout is identical and the ranking pages look identical too (although as our correlation analysis shows these two factors do not always go hand-in-hand).

‍

‍

How to calculate Organic Rankings Volatility

We can interpret the Rank Volatility (whether we’re looking at organic or generative URLs) as how great a percentage of the maximum possible change has happened (with higher weighting for lower/better ranks). To get more technical, we are calculating Weighted Levenshtein distance, which is the total cost of transforming one list to another list with positional weights applied.

Example of a SERP with very high Organic Rankings Volatility – ‘chain management salary’

‍

‍

You can see in the table below how many ranking positions have changed:

Both these results have an AI Overview (not pictured) and a Salary Estimates feature.  Below this are the organic results and as you can see there is a complete shake-up in top ranking positions over the period.  Keyword rank tracking alone may not paint the full picture of this shake-up if your rank has only changed by 1 or 2 positions. This is particularly helpful if you are looking at whether internal or external factors have caused the change in ranks.

Example of a SERP with very low Organic Rankings Volatility – ‘chevy suburban height in feet’

‍

‍

There are far fewer ranking positions changes for this keyword between these dates:

How to calculate AI Overview Rankings Volatility

This is calculated the same way as Organic Ranking Volatility, so the scores are broadly comparable.

Example of a SERP with very high AI Overview Rankings Volatility – “How to protect your trademark’.

‍

‍

Example of a SERP with very low AI Overview Rankings Volatility – ‘sink aerator purpose’

‍

‍

How to calculate AI Overview Text Snippet Volatility

It was actually quite difficult to find examples where the AIO text snippet had not changed by much.  In every comparison the wording changed.  In some examples, it was slight grammatical changes, in others like the example above, the general meaning theme and sub-themes covered were the same with one or two minor differences.

Trying to track these types of minor changes would be a fool’s errand.  What is much more important to track is the overall scope of the answer and whether that has changed at all and whether it has coincided with a change in the generative results. 

The premise is simple enough: If the generative answer Google returns has changed notably in content, format and importantly meaning, then this might reflect a shift in Google’s understanding of what users really find useful.  If SEOs can identify when these shifts happen and what’s changed, then they could optimise their content accordingly and potentially improve their rankings in AI Overviews. (If you are interested, I have some examples of using an LLM to do this later).

So, for measuring how much the AI Overview AI generated text has changed from one date to another, we measured the cosine distance between the embeddings of both pieces of text. 

You can compare this score with itself day-over-day to see how much it is changing and you can also compare the rate of change across keyword groups or categories. The score ranges from 0 (no change) to 2 (complete change).

We used the Ollama nomic-embed-text model for the embeddings.  It calculates by how much the meaning of the text has changed, rather than specific word changes.  Since the texts are answering the same question, it would be generally be less than 0.3.

This metric will not have a standalone meaning and cannot readily be compared to other metrics.  You might reasonably expect the AIO text snippet to change if the AIO sources have changed, but there is only a slightly weak positive correlation of 0.08 between the metrics.

Example of a SERP with very high AI Overview Text Volatility – ‘What’s the best way to store onions’

‍

I've extracted and cleaned up the AIO text snippet from each of the three collection dates and it's apparent that it has changed considerably. On this occasion the ranking generative URLs also change considerably.

Example of a SERP with very low AI Overview Text Volatility – ‘How to conduct a raffle’

You can see that in this case there are only minor cosmetic changes to the AI Overview text snippet which means there's been no big shift in user intent.

‍

‍

How to use SEO Volatility Data

Finally, before we get into the study results, a small word of warning about making direct comparisons.  Some metrics are relative metrics and cannot be directly compared as they are calculated differently – but you can compare the rate of change of ranking pages within organic and AI Overviews.

Study Data

As always, we like to share our research data as well as a summary of our findings.

In this Google Drive folder you will find a series of spreadsheets containing the keywords and the exact organic URLs, generative URLs and AI Overview text snippets we captured on each date.  It also contains some summarised data by category and tag, as well as copies of the screenshots and charts from this article.

If you do anything interesting with it, then please do share.

Research Findings

SERP Volatility: How much does Google’s SERP change over time?

It is natural to see some variation in the SERP results from day-to-day and week-to-week – after all the world is changing, consumer habits and behaviour is changing, and websites are updating or producing new content continuously.   There was around 8 weeks between the two data collection exercises and if you look at the chart below you can see that there are enough changes in the layout, makeup and order of the SERP features presented to be detected by our algorithm.

Please note, we’re not looking at the changes in the organic ranking pages, this is to come.  You can see that the SERP hasn’t changed markedly, there are some expected changes but no single category stands out from the others.

Of course, SERP volatility is to be expected and just because in this study one particular category of keywords and SERPs was more volatile than another doesn’t mean of course that this will always be the case.  

Should Google choose to introduce a new SERP feature that is specific to a vertical, say a new travel widget for example, then we would expect this to feature on the chart below.  But no such major changes were detected for the keywords and categories covered in our study.

‍

‍

We also looked the user intent of these keywords but did not find anything remarkable.

‍

‍

In summary, the SERP changes;  It will always change and the Authoritas SERP Volatility tool will pick this up when it happens – but you need to use this metric in conjunction with our other volatility measures before determining what course of action is necessary.

Organic Volatility: How much do the top organic ranking pages fluctuate over time?

Just as the SERP changes, we would naturally expect some changes in the organic rankings over a 2-to-3-month period.  

Just because you see high volatility over a short period of time, don’t just assume a Google core update has happened.  More often than not this may well be the case, but also consider other factors such as; user intent shifts reflecting seasonal changes; competitors may well have been penalised; or, one-off events may have hit the news headlines which may cause Google to pick an alternative meaning for a phrase for short period of time.

The Organic Volatility by Category chart below shows that organic rankings in the ‘Beauty and Style’ and ‘Travel’ categories changed at a greater rate than other categories.  It’s difficult to deduce what’s driving this change in this study, but if you were looking at traffic loses from organic across your site, then understanding that particular verticals have been worse hit by a rumoured (or confirmed) algorithm update may save you some time in getting to the root cause of your decline.

‍

‍

As we move into a world of generative AI and AI search in general, it’s interesting to look at organic volatility and how it flexes in relation to AI Overviews.

‍

AI Overview Volatility: How much do the top-ranking pages in AI Overviews fluctuate over time?

The chart below looks at the same categories, but this time we’ve plotted the volatility scores side by side by category for organic ranking pages and generative AI Overview ranking pages.  This is a smaller sample as it only contains keywords where Google showed an AI Overview (18.8% AIO penetration - 2,104 keywords of 11,203 keywords had AIOs) . 

‍

‍

This chart clearly shows that generative results are changing at a faster rate than organic across the board.  Google is still testing and tweaking the layouts, so this is not a surprise, but it will be interesting to see if this trend continues or not.

‍

AI Overview Text Volatility: How much does the AI generated text summary in the AI Overview SERP feature change over time?

Here we were looking to understand how frequently AIO text snippets changed and whether the change was more significant for certain categories or for certain user intents.

‍

‍

‍

As far as I can tell, the generative text changed for every single query.  However, our measure was looking for meaningful changes in the generative text which might reflect that Google had shifted the intent of the original query slightly to return different generative ranking pages.  I hoped that by spotting where the text changes became meaningful, we could derive a signal that would tell us it was the right time to update the content on our ranking pages to reflect this shift in intent.

Take this example below: ‘What’s the best way to store onions?’

I copied the AIO snippet from the three different dates and asked several LLMs for an opinion on the type of changes in the snippets, what was omitted, added, emphasised, etc and whether the changes in the content of the AIO snippet over time reflected a shift in user intent or in the kind of information Google was looking to give users.

The URLs had also changed considerably in both observation windows, but I didn’t specifically include them in the prompt unless they were mentioned as part of the AIO text snippet.

I tried ChatGPT 4o, DeepSeek and Claude 3.5 Sonnet and they gave me a couple of relevant insights and recommendations.  DeepSeek was interesting because it searched multiple sources only and so considered factors like E-EAT in its answer which I had not mentioned in my question.  But I found ChatGPT and Claude to give me the most succinct and practical insights. I've not included full screenshots here as they are too long, but these are edited sectons of the LLMs' responses:

‍
LLM Insights (Claude 3.5 Sonnet) into what the changes in AIO Text Snippet tells us about what’s important to users (in Google’s eyes) for this query:

• Earlier content (Aug) includes more detailed alternative methods like the jar method and canning 
• Later versions (Oct/Jan) focus more on basic storage principles and immediate practical solutions 
• The canning method discussion was completely removed in later versions, suggesting less emphasis on long-term preservation
• More emphasis on what NOT to do (stronger focus on avoiding common mistakes)
• Earlier content seems aimed at both gardeners and general consumers (mentions harvesting, curing) 
• Later versions focus more on everyday kitchen storage for general consumers 
• Increased emphasis on quick, practical solutions over long-term storage

LLM Recommendations (Claude 3.5 Sonnet) as to how to update our content in response

1. Structure:

• Use clear, action-oriented headers
• Lead with the most common/practical solutions
• Group information by immediate vs. long-term storage needs

2. Focus Areas:

• Emphasise practical, everyday storage solutions
• Include common mistakes to avoid
• Provide specific container recommendations
• Keep temperature guidelines general rather than specific

3. Content Types:

• Focus on short-term storage solutions for regular consumers
• Include quick tips for common scenarios
• Reduce emphasis on long-term preservation methods unless specifically relevant

Snapshot of ChatGPT 4o Insights:

• Google may be optimising for simplicity over exhaustive guidance.
• If you want to rank in AIO snippets, focusing on everyday usability rather than long-term storage techniques may be more effective.
• AI favours practical, immediately useful content over comprehensive preservation guides.
• Prioritise clear, structured formatting (bullet points, step-by-step methods) to match AI’s preferred answer structure.

What does Correlation Analysis tell us about SERP, Organic and AIO Volatility?

Here is the correlation matrix for the volatility metrics we calculated.

‍

‍

Weak or No Strong Correlations:

• All correlation values are low (≤ 0.19), suggesting little to no strong relationship between the different volatility metrics.
• The highest correlation is between Organic Volatility & SERP Volatility (0.19), indicating that organic fluctuations slightly contribute to overall SERP volatility. 
• Other values are mostly close to zero, meaning each volatility type behaves independently.

Organic & Generative Volatility (0.09):

• There is a very weak positive correlation, suggesting that organic ranking fluctuations are not significantly influenced by AI Overview volatility.
• This aligns with the idea that Google’s AI Overviews are not necessarily causing major organic disruptions—at least not in a predictable way.

SERP Volatility & Generative Volatility (0.04):

• Another weak correlation suggests that changes in AI-generated content do not strongly impact overall SERP stability.
• This could mean that Google's generative responses are relatively independent of traditional ranking shifts.

Generative Text Volatility & Generative Volatility (0.13):

• This is slightly higher than other correlations, implying that when generative volatility increases, the textual changes within AI Overviews may also fluctuate.
• However, this is still a weak relationship, suggesting that generative text updates do not always correspond with larger AI-generated ranking shifts.

I also repeated the analysis again, but this time I only including the rows where there was generative data on both dates. As expected, the rows and columns involving generative/generative text were the same as before, but the overall insights were the same.

‍

‍

These correlation charts appear to align with my previous studies indicating that around 40% of the top 10 organic ranking pages were not cited in AI Overviews (AIOs).  This is why I feel that measuring Generative to Organic Alignment Score (GOA Score™) is important for optimising for AI Overviews.

‍

‍

• The red section (No Match Percentage) represents the proportion of organic ranking pages that are not cited in the AI Overview.
• Across organic ranking positions 1 to 10, the red section consistently hovers around or slightly above 40%
• The green section (Exact Match Percentage) is higher for top organic rankings (positions 1-3) but gradually decreases as rankings go down.
• The blue section (Domain Match Percentage) remains relatively small, meaning domain-level matches exist but aren't dominant.  

What are the implications for SEOs?

• Since AI Overview volatility does not appear strongly correlated with organic ranking volatility, AI-driven changes might be operating separately from traditional SERP movements.
• The lack of strong SERP-wide volatility correlations means that AI Overviews could be replacing featured snippets or enhancing search results without necessarily shaking up core rankings.
• This suggests that SEOs need to consider AI Overviews as a distinct layer of analysis, rather than assuming they directly affect organic results.

One final point on this analysis.  My gut feel when pouring over the data was there seems to be a relationship between high SERP Volatility and a low number or even zero generative results.   

‍

‍

However, when analysed properly, there was a weak positive correlation of 0.14 showing that more volatile SERPs are likely to have a slightly greater number of AI Overviews rankings, and my ‘gut feel’ is not as reliable as I thought! (Although, in my defence the appearance of AIOs is a factor in SERP Volatility score – so perhaps this is a bit circular).

Although, this was not a sector covered in this study, ‘News’ is a good example of a niche that I look at regularly and where the SERPs change very frequently and there are next to no AI Overviews right now.  So perhaps this was colouring my perspective.

Just goes to show – it’s always worth spending the time to run the data!

How your knowledge of SEO Volatility metrics can be put to good use in your SEO projects

Understanding SERP volatility provides crucial context for SEO decision-making and helps distinguish between normal fluctuations and significant changes requiring action. When analysing volatility data, consider several key aspects:

‍

‍

Assessing the Scope of Changes

Start by examining whether the ranking changes you're seeing are isolated to specific pages or keywords, or if they represent a broader pattern across your site. Compare your volatility data against industry benchmarks and historical patterns for your website. This historical context helps determine if current fluctuations fall within normal ranges or signal unusual activity.

For example, if you notice a 20% ranking drop for your primary keywords, but industry volatility metrics show similar movements across competitors, this suggests a broader algorithmic shift rather than a site-specific issue. Conversely, if your rankings change dramatically while industry metrics remain stable, this often indicates a local problem requiring immediate attention.

SERP Feature Analysis

Monitor how SERP features evolve over time, particularly those relevant to your target keywords. Pay special attention to the introduction of new features or significant changes in existing ones. Track which competitors appear in these features and analyse their content strategy to understand why they might be earning these positions.

When examining SERP features, consider:

• New feature types appearing in your target SERPs
• Changes in feature placement and prominence
• Design and layout changes which may make a feature more or less prominent and impact organic CTR%
• Shifts in competitor presence across different feature types
• Requirements for qualifying for specific SERP features

Content Freshness Comparison

Develop a systematic approach to tracking content updates across your competitive landscape. Create a content freshness index that compares your update frequency against competitors. This might include:

• The average time between substantial content revisions
• Frequency of minor updates and optimisations
• Correlation between update frequency and ranking stability
• Types of updates that seem to impact rankings most significantly

Compare AIO Text Snippet Changes using an LLM to understand shifts in User Intent

I’ve experimented with feeding the AIO text snippets from volatile AIO results into an LLM and asking it to compare the text and to describe how it’s changed, what’s now included and what’s now omitted and give me advice on how to consider updating my content accordingly.  I’ve already covered one example in this article, but here’s another abbreviated example:

Example: ‘How much does a heart ablation cost?’

‍

‍

Here's the response from Claude 3.5 Sonnet comparing the AI Overview snippet from August 2024 with October 2024 to identify shifts in content structure and user intent.

Claude 3.5 Sonnet’s Perspective on what’s changed and what this means for your content

Key Changes from August to October:

1. Organization & Structure:

- August: More narrative flow, paragraph-based

- October: More structured with clear headers ("Average cost", "Insurance coverage", "Medicare Part A")

- October adds explicit section breaks and categorization

2. Cost Information Presentation:

August:

- Embeds cost in paragraph: "according to Fair Health, the average cost is around $32,000"

- More informal presentation of range

October:

- Presents cost information in separate, dedicated section

- More direct presentation of figures

- Same numbers but cleaner formatting

3. Insurance Coverage:

August:

- More tentative language ("may cover all or most")

- Focuses on private insurance first

- Mentions excess/co-payment

October:

- Broader approach to insurance coverage

- Added section on cost reduction strategies

- More actionable advice about consulting providers

4. New Content Additions (October):

- Added section on cost reduction strategies

- Specifically mentions clinical trials

- Includes drugmaker discount programs

- More emphasis on proactive cost management

5. Content Strategy Evolution:

- Shift from informational to more action-oriented content

- More emphasis on cost reduction strategies

- Better organized for scanning and quick reference

- More comprehensive coverage of payment options

6. Source Citation:

- Both versions rely heavily on Resolve Medical Bills

- October version includes more diverse sources

- Better integration of source material into structured format

This evolution suggests:

1. Increasing user need for clear, actionable cost information

2. Greater emphasis on cost reduction strategies

3. Shift toward more structured, scannable content

4. Focus on comprehensive coverage of payment options

‍

I feel that this could become a very valuable way of determining whether the user intent for a keyword is shifting and whether a content update would be prudent sooner rather than later.

I will be continuing to experiment with this and plan to implement some of these recommendations to see if it helps me gain new AIOs, promote ranking pages that are hidden in the AIO until a user clicks and regain lost AIO rankings.

Algorithm Update Impact Assessment

I have not dwelled on Google algorithm updates, but needless to say they shake-up the SERP and this will be reflected in your tracked SEO volatility metrics. You can use this with other SEO metrics like keyword ranking changes and measure the impact of these changes using Google Search Console clicks and impressions or potential impact using AdWords keyword search volumes.

1. Track your rankings/volatility metrics against known algorithm update dates

2. Compare volatility patterns with previously (un)confirmed Google updates

3. Analyse which types of pages or content are most affected then dig into the detail to work out what's changed

4. Document your findings so you can revisit them to see whether you can repeat effective remedies next time there's major flux in the SERPs

Site Health Indicators

When you observe ranking changes without corresponding SERP volatility, conduct a thorough technical and competitive audit focusing on whether your own house is in order:

Technical Health Check:

• Server response times
• Core Web Vitals metrics
• Mobile usability issues
• Crawl efficiency

Backlink Profile Health:

• Recent link acquisition patterns vs competitors
• Changes in referring domain quality
• Competitor link profile comparisons

Action Framework

Here’s one way to think about developing a structured framework based on volatility analysis.  Consider the scale of the volatility (using whichever metric you find most impactful SERP volatility, organic volatility, AIO ranking or text volatility) against the significance of the ranking changes for your domain.  You could look at all changes or just ranking losses or gains.

‍

‍

1. Low Volatility + Minor Ranking Gains/Losses: Business as usual - continue regular optimisation and monitoring as minimal attention is needed.

2. Low Volatility + Significant Ranking Gains/Losses: Ask yourself, “Is it just me?” and investigate site/page specific issues closely.

3. High Volatility + Minor Ranking Gains/Losses: Ask yourself, “Why aren’t I affected?” and document why you think your rankings have proved to be resilient.

4. High Volatility + Significant Ranking Gains/Losses: Immediately ask yourself, “What’s changed?” and assess whether the user intent has shifted by comparing previous AIO snippet text with current AIO snippet text.

By systematically analysing these factors, you can better understand the root causes of ranking changes and develop more targeted, effective responses to SERP volatility. This structured approach helps prioritise resources and focus optimisation efforts where they'll have the most impact.

If you have any further thoughts as to how you put SEO volatility data to good use for SEO, then by all means please drop me a line and I’ll update the article accordingly.

We're conducting a series of research studies into AI Search engines this year and are always open to collaborative studies - so if you've got this far and have an idea for how we can collaborate on in-depth practical studies then please get in touch.