BERT, MUM, and SGE: How LLMs Changed Ranking Forever

When Google announced BERT in October 2019, the SEO community's initial response was predictable: keyword density adjustments, entity stuffing, and "write naturally" platitudes. Four years later, with Multitask Unified Model (MUM), Search Generative Experience (SGE, now AI Overviews), and the integration of Gemini into core search, it is clear those reactions missed the fundamental shift entirely. This article is not about "what BERT means for content." It is about the architectural changes in how Google's ranking pipeline processes text, and the specific technical signals that determine whether your content is understood, trusted, and selected by these systems.

BERT Architecture: What the Transformer Actually Does to Your Content

BERT (Bidirectional Encoder Representations from Transformers, Devlin et al. 2018) is a pre-trained language model using the Transformer encoder architecture. The critical architectural property from an SEO standpoint: bidirectionality. Unlike earlier NLP models (word2vec, GloVe) that represent words in isolation, BERT's self-attention mechanism means every token is contextualized by every other token in the sequence simultaneously.

This has a concrete technical implication: the same word in different sentence contexts produces different vector representations. "Apple released a new phone" and "Apple makes great pie" produce different embeddings for "Apple." Google can distinguish these uses without keyword matching—it is reading semantic context, not frequency distributions.

The BERT model Google uses for search is not the original 110M-parameter BERT-Base. Google's patent filings and research publications (most relevantly, the 2020 paper "Language-Agnostic BERT Sentence Embedding" and subsequent work from the Google Brain team) indicate use of larger, fine-tuned models specifically trained on search queries and web documents. The fine-tuning objective: given a query embedding, maximize the similarity to the correct document embedding while minimizing similarity to irrelevant documents.

# Demonstrating BERT contextual embedding for SEO content analysis
# pip install transformers torch

from transformers import BertTokenizer, BertModel
import torch
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')
model.eval()

def get_sentence_embedding(text, layer=-2):
    """
    Extract sentence-level BERT embedding using second-to-last layer.
    Last layer is most task-specific; second-to-last often generalizes better.
    """
    inputs = tokenizer(
        text,
        return_tensors='pt',
        truncation=True,
        max_length=512,
        padding=True
    )

    with torch.no_grad():
        outputs = model(**inputs, output_hidden_states=True)

    # Average token embeddings from specified layer (excluding [CLS] and [SEP])
    hidden_state = outputs.hidden_states[layer]
    attention_mask = inputs['attention_mask']

    # Masked mean pooling
    mask_expanded = attention_mask.unsqueeze(-1).expand(hidden_state.size()).float()
    sum_embeddings = torch.sum(hidden_state * mask_expanded, 1)
    sum_mask = torch.clamp(mask_expanded.sum(1), min=1e-9)
    return (sum_embeddings / sum_mask).squeeze().numpy()


def analyze_query_document_alignment(query, document_sections):
    """
    Measure how well each document section aligns with query intent.
    Uses cosine similarity as a proxy for semantic relevance.
    """
    query_embedding = get_sentence_embedding(query)

    results = []
    for section_title, section_text in document_sections.items():
        section_embedding = get_sentence_embedding(section_text[:500])
        similarity = cosine_similarity(
            query_embedding.reshape(1, -1),
            section_embedding.reshape(1, -1)
        )[0][0]
        results.append({
            'section': section_title,
            'similarity': float(similarity),
            'interpretation': (
                'high alignment' if similarity > 0.75 else
                'medium alignment' if similarity > 0.55 else
                'low alignment'
            )
        })

    return sorted(results, key=lambda x: x['similarity'], reverse=True)


# Usage: audit your content against target queries
query = "how to fix crawl budget waste"
sections = {
    "intro": "Crawl budget refers to the number of URLs Googlebot crawls on your site...",
    "faceted_nav": "Faceted navigation creates URL explosions that consume crawl budget...",
    "log_analysis": "Server log files reveal exactly which URLs Googlebot requests...",
    "xml_sitemap": "XML sitemaps help Googlebot discover important URLs more efficiently...",
}

results = analyze_query_document_alignment(query, sections)
for r in results:
    print(f"{r['section']}: {r['similarity']:.3f} ({r['interpretation']})")

How BERT Integrates into the Ranking Pipeline

Google's ranking pipeline is a multi-stage retrieval and re-ranking system. BERT does not operate on every query against every document at retrieval time—the computational cost would be prohibitive. Instead, it operates at two key stages:

Stage 1 (Query understanding): BERT processes the raw query to produce a query representation that captures intent, entities, and semantic nuance. This happens before any document retrieval. The "prepositional phrase" example Google used in the BERT announcement—where "2019 brazil traveler to usa need a visa" changes meaning based on the prepositions—illustrates this stage. The query representation feeds into the retrieval system to pull candidate documents.

Stage 2 (Re-ranking/passage extraction): For a subset of results, BERT is used to re-rank by computing query-document relevance scores at the passage level. This is where the structure of your content matters: BERT processes passages, not whole documents. A well-structured article with clear H2/H3 sections provides natural passage boundaries; a wall of text does not.

MUM: Multimodal Understanding and Cross-Lingual Transfer

MUM (Multitask Unified Model, Ni et al., Google Research 2021) extends BERT's architecture in two significant directions: multimodality (text + images + video in a unified embedding space) and cross-lingual transfer (trained on 75+ languages simultaneously, enabling knowledge transfer across languages without explicit translation).

The cross-lingual property has a direct SEO implication: a well-documented factual claim in English can influence how Google understands related queries in Japanese or Portuguese. The model transfers factual associations across languages at the embedding level. For international SEO, this means that the quality of your English content has a non-zero effect on your topical authority across all MUM-supported languages—not as a ranking factor for those language queries, but as part of the entity association model Google builds about your site.

The multimodal aspect means that Google can now compare image content against text content and detect misalignment. An article titled "best hiking boots 2026" with images of dress shoes creates a semantic inconsistency that MUM-class models can detect. Alt text is not just for accessibility—it is a semantic bridge that should accurately describe image content in context.

# Cross-lingual semantic similarity using multilingual BERT
# Demonstrates why English authority affects multilingual topical understanding
from transformers import AutoTokenizer, AutoModel
import torch

# mBERT: Multilingual BERT, trained on 104 languages simultaneously
tokenizer = AutoTokenizer.from_pretrained('bert-base-multilingual-cased')
model = AutoModel.from_pretrained('bert-base-multilingual-cased')

def cross_lingual_similarity(text_en, text_ja):
    """
    Measure semantic similarity between English and Japanese text.
    High similarity confirms the cross-lingual transfer property.
    """
    def embed(text):
        inputs = tokenizer(text, return_tensors='pt', truncation=True,
                          max_length=512, padding=True)
        with torch.no_grad():
            output = model(**inputs)
        # [CLS] token embedding as sentence representation
        return output.last_hidden_state[:, 0, :].squeeze().numpy()

    emb_en = embed(text_en)
    emb_ja = embed(text_ja)

    from sklearn.metrics.pairwise import cosine_similarity
    sim = cosine_similarity(emb_en.reshape(1,-1), emb_ja.reshape(1,-1))[0][0]
    return float(sim)

# Example: same concept across languages
en = "Technical SEO involves optimizing website crawlability and indexing."
ja = "テクニカルSEOとは、ウェブサイトのクロール可能性とインデックス化を最適化することです。"

similarity = cross_lingual_similarity(en, ja)
print(f"Cross-lingual similarity: {similarity:.3f}")
# Expected: ~0.85-0.92 for semantically equivalent text

SGE/AI Overviews: Selection Criteria for Citation

AI Overviews (formerly SGE) synthesizes answers from multiple sources and cites them inline. Understanding the selection criteria—which is not fully published but can be reverse-engineered from citation patterns—is critical for any site that wants to capture the growing share of SERP real estate these boxes occupy.

From large-scale analysis of AI Overview citations, several patterns emerge:

Entity clarity: Cited pages have unambiguous entity associations. The page is clearly about a specific named entity (person, organization, product, concept), and that entity is mentioned in the first paragraph, title, and structured data. Vague or hedging introductions reduce citation probability.

Factual density with citation support: Pages that make specific, verifiable claims with clear attribution (to studies, dates, statistics) are cited more frequently than pages with qualitative assertions. "Structured data increases click-through rates by 20–30% in some verticals (source)" outperforms "structured data can help improve visibility."

Passage-level answerability: The cited content contains a passage that directly answers the query without requiring the full page context. This reinforces the passage indexing architecture—each H2 section should be able to stand alone as a self-contained answer to a facet of the topic.

# Analyze your content for AI Overview citation readiness
import re
import spacy
from collections import Counter

# pip install spacy && python -m spacy download en_core_web_trf
nlp = spacy.load('en_core_web_trf')

def analyze_citation_readiness(page_text, target_query):
    """
    Score content against signals associated with AI Overview citation.
    """
    doc = nlp(page_text)
    scores = {}

    # 1. Entity density — named entities per 100 words
    entities = [(ent.text, ent.label_) for ent in doc.ents]
    words = len(page_text.split())
    entity_density = len(entities) / (words / 100)
    scores['entity_density'] = min(entity_density / 10, 1.0)  # Normalize to [0,1]

    # 2. Factual claim indicators (statistics, dates, specific figures)
    stat_pattern = r'\b\d+(?:\.\d+)?(?:\s*%|\s*percent|\s*million|\s*billion)\b'
    stats = re.findall(stat_pattern, page_text, re.IGNORECASE)
    scores['factual_density'] = min(len(stats) / 10, 1.0)

    # 3. Direct answer patterns (phrases that immediately answer questions)
    answer_patterns = [
        r'\b(?:is|are|was|were)\s+(?:the|a|an)?\s*\w+(?:\s+\w+){1,5}\b',
        r'\baccording to\b',
        r'\bresearch (?:shows?|suggests?|indicates?|found)\b',
        r'\bstudies? (?:show|suggest|indicate|found)\b',
    ]
    answer_signals = sum(
        len(re.findall(p, page_text, re.IGNORECASE))
        for p in answer_patterns
    )
    scores['answer_signal_density'] = min(answer_signals / 20, 1.0)

    # 4. Passage self-containment — check if any paragraph answers the query
    paragraphs = [p.text for p in doc.sents if len(p.text.split()) > 15]
    query_words = set(target_query.lower().split())
    max_overlap = 0
    for para in paragraphs:
        para_words = set(para.lower().split())
        overlap = len(query_words & para_words) / len(query_words)
        max_overlap = max(max_overlap, overlap)
    scores['passage_relevance'] = max_overlap

    # Composite score
    scores['composite'] = sum(scores.values()) / len(scores)

    return scores


# Audit a piece of content
page_text = """Your page content here..."""
query = "how does structured data affect search rankings"
results = analyze_citation_readiness(page_text, query)
for signal, score in results.items():
    print(f"{signal}: {score:.2f}")

Entity Salience and NLP-Based Content Analysis

Google's Natural Language API (which reflects the NLP capabilities used internally, though not identically) exposes entity salience—a measure of how central an entity is to the document's meaning. Salience is computed from: frequency of mention, position of first mention, co-occurrence patterns with other entities, and whether the entity is the subject of key predicates in the text.

# Use Google's Natural Language API to analyze entity salience in your content
from google.cloud import language_v1
import json

def analyze_entity_salience(text):
    """
    Analyze entity salience using Google Natural Language API.
    Reveals how Google's NLP layer understands your content's central entities.
    """
    client = language_v1.LanguageServiceClient()
    document = language_v1.Document(
        content=text,
        type_=language_v1.Document.Type.PLAIN_TEXT
    )

    response = client.analyze_entities(
        document=document,
        encoding_type=language_v1.EncodingType.UTF8
    )

    entities = []
    for entity in response.entities:
        entities.append({
            'name': entity.name,
            'type': language_v1.Entity.Type(entity.type_).name,
            'salience': round(entity.salience, 4),
            'metadata': dict(entity.metadata),
            'mentions': len(entity.mentions),
            'wikipedia_url': entity.metadata.get('wikipedia_url', ''),
        })

    # Sort by salience
    entities.sort(key=lambda x: x['salience'], reverse=True)

    # Identify primary entity (highest salience)
    primary = entities[0] if entities else None
    if primary:
        print(f"Primary entity: {primary['name']} ({primary['type']})")
        print(f"Salience: {primary['salience']:.3f}")
        print(f"Wikipedia: {primary['wikipedia_url']}")

    return entities


def check_entity_alignment(text, target_entity, target_type):
    """
    Verify that your target entity is the primary salient entity.
    Critical for content intended to rank for entity-based queries.
    """
    entities = analyze_entity_salience(text)
    if not entities:
        return {'aligned': False, 'reason': 'no entities detected'}

    primary = entities[0]
    is_aligned = (
        primary['name'].lower() == target_entity.lower() and
        primary['type'] == target_type
    )

    if not is_aligned:
        return {
            'aligned': False,
            'primary_entity': primary['name'],
            'primary_type': primary['type'],
            'primary_salience': primary['salience'],
            'target_rank': next(
                (i+1 for i, e in enumerate(entities)
                 if e['name'].lower() == target_entity.lower()),
                None
            )
        }

    return {
        'aligned': True,
        'salience': primary['salience'],
        'entity_type': primary['type'],
        'has_knowledge_graph': bool(primary.get('wikipedia_url')),
    }

Query Understanding: From Keywords to Intent Vectors

Pre-BERT query understanding was largely lexical: tokenize, expand with synonyms, weight by IDF, match against inverted index. Post-BERT, queries are embedded into dense vector spaces where proximity encodes semantic similarity. This has a specific implication for keyword research: intent cluster analysis matters more than individual keyword variations.

# Intent cluster analysis using sentence embeddings
# Groups keyword variants by semantic intent, not surface form

from sentence_transformers import SentenceTransformer
from sklearn.cluster import DBSCAN
import numpy as np

model = SentenceTransformer('all-mpnet-base-v2')

def cluster_keywords_by_intent(keywords, eps=0.25, min_samples=2):
    """
    Cluster keywords by semantic intent using dense embeddings + DBSCAN.
    eps: maximum distance for same cluster (lower = tighter clusters)
    Returns: {cluster_id: [keywords]} where -1 is noise/outliers
    """
    embeddings = model.encode(keywords, convert_to_numpy=True, normalize_embeddings=True)

    # DBSCAN with cosine distance (1 - cosine_similarity)
    # Cosine distance: 0 = identical, 1 = orthogonal, 2 = opposite
    from sklearn.metrics.pairwise import cosine_distances
    distance_matrix = cosine_distances(embeddings)

    clustering = DBSCAN(
        eps=eps,
        min_samples=min_samples,
        metric='precomputed'
    ).fit(distance_matrix)

    labels = clustering.labels_
    clusters = {}
    for keyword, label in zip(keywords, labels):
        clusters.setdefault(label, []).append(keyword)

    # Find cluster centroid keywords (closest to cluster mean)
    cluster_centroids = {}
    for label, kw_list in clusters.items():
        if label == -1:
            continue
        kw_indices = [keywords.index(kw) for kw in kw_list]
        cluster_embeddings = embeddings[kw_indices]
        centroid = cluster_embeddings.mean(axis=0)
        distances_to_centroid = np.linalg.norm(cluster_embeddings - centroid, axis=1)
        centroid_kw = kw_list[np.argmin(distances_to_centroid)]
        cluster_centroids[label] = centroid_kw

    return clusters, cluster_centroids


# Usage: group 200 keyword variants into topical clusters
keywords = [
    "how to fix crawl budget",
    "crawl budget optimization",
    "googlebot crawl frequency",
    "reduce crawl waste",
    "server log analysis seo",
    "log file analysis for seo",
    "analyze googlebot crawls",
    # ... more keywords
]

clusters, centroids = cluster_keywords_by_intent(keywords)
for cluster_id, kws in clusters.items():
    centroid = centroids.get(cluster_id, 'noise')
    print(f"\nCluster {cluster_id} (centroid: {centroid}):")
    for kw in kws:
        print(f"  - {kw}")

Technical Content Signals That Matter Post-BERT

The post-BERT content optimization paradigm operates on different axes than keyword density optimization. The signals that matter technically:

Semantic completeness: Does the content cover the entity space expected for the topic? Google's Knowledge Graph defines what entities are associated with a concept. Content about "technical SEO" that omits entities like "crawl budget," "canonical tags," and "Core Web Vitals" is semantically incomplete relative to what the Knowledge Graph expects. The Google NLP API's entity classification is a useful proxy for checking completeness.

Predicate clarity: Natural Language Processing extracts Subject-Predicate-Object triples. "Technical SEO improves crawlability" generates the triple (technical SEO, improves, crawlability). Ambiguous, passive, or nominalized constructions reduce predicate extraction confidence. Active voice with clear subject-predicate-object structure improves NLP parsing accuracy.

Content-schema alignment: If your JSON-LD declares the page as an Article with author X and topic Y, but the NLP analysis of the page text identifies different primary entities, there is a schema-content misalignment. Google resolves these conflicts using the text as the ground truth—the schema declaration is a claim that must be validated against page content. See the schema.org advanced article for the reconciliation mechanics.

Testing Semantic Understanding: Practical Methodologies

# Systematic semantic gap analysis using GSC data + embeddings
import pandas as pd
from google.oauth2 import service_account
from googleapiclient.discovery import build
from sentence_transformers import SentenceTransformer

def fetch_gsc_queries(site_url, start_date, end_date, credentials_path):
    """Fetch query data from GSC API."""
    credentials = service_account.Credentials.from_service_account_file(
        credentials_path,
        scopes=['https://www.googleapis.com/auth/webmasters.readonly']
    )
    service = build('searchconsole', 'v1', credentials=credentials)

    response = service.searchanalytics().query(
        siteUrl=site_url,
        body={
            'startDate': start_date,
            'endDate': end_date,
            'dimensions': ['query', 'page'],
            'rowLimit': 25000,
            'dimensionFilterGroups': [{
                'filters': [{
                    'dimension': 'position',
                    'operator': 'lessThan',
                    'expression': '20'
                }]
            }]
        }
    ).execute()

    rows = response.get('rows', [])
    return pd.DataFrame([{
        'query': r['keys'][0],
        'page': r['keys'][1],
        'clicks': r['clicks'],
        'impressions': r['impressions'],
        'position': r['position'],
    } for r in rows])


def identify_semantic_gaps(gsc_df, page_contents, model):
    """
    Find queries where semantic gap between query intent and page content is high.
    These are pages Google shows for queries the content doesn't optimally answer.
    """
    gaps = []

    for page_url in gsc_df['page'].unique():
        page_queries = gsc_df[gsc_df['page'] == page_url]
        content = page_contents.get(page_url, '')
        if not content:
            continue

        # Embed page content (use first 1000 chars as proxy)
        content_embedding = model.encode([content[:1000]], normalize_embeddings=True)

        for _, row in page_queries.iterrows():
            query_embedding = model.encode([row['query']], normalize_embeddings=True)

            from sklearn.metrics.pairwise import cosine_similarity
            sim = cosine_similarity(query_embedding, content_embedding)[0][0]

            if sim < 0.6 and row['impressions'] > 100:
                gaps.append({
                    'page': page_url,
                    'query': row['query'],
                    'semantic_similarity': round(float(sim), 3),
                    'impressions': int(row['impressions']),
                    'avg_position': round(row['position'], 1),
                    'gap_severity': 'high' if sim < 0.4 else 'medium',
                })

    return sorted(gaps, key=lambda x: x['impressions'], reverse=True)

For monitoring semantic alignment changes over time, run this analysis monthly against rolling GSC windows and track gap reduction as a KPI. Connect the output to the Looker Studio dashboard for stakeholder visibility. Reference the Google Search Central documentation on how AI models process search queries for the official framing.

FAQ

Does BERT replace TF-IDF entirely in Google's ranking system?

No. Google's ranking is an ensemble of signals. BERT-based semantic matching operates at the re-ranking stage on top of an initial candidate set retrieved via traditional inverted index (which still relies on term frequency signals). BM25-style signals identify the candidate pool; BERT re-ranks within it. Both operate simultaneously, not sequentially in a replacement relationship. Exact keyword matching still matters for head queries where lexical match is the most reliable relevance signal.

How does passage indexing interact with BERT-based ranking?

Passage indexing (announced 2020) allows Google to rank a specific passage of a page for a query even if the overall page topic is different. BERT evaluates query-passage relevance at the passage level. The combined effect: your H2 sections are independently rankable units, not just navigation aids. Each section should be written to fully answer its specific question without requiring surrounding context—the same constraint as writing effective FAQ answer text.

What does MUM mean for multilingual site strategy?

MUM enables Google to answer queries in one language using information from sources in another language. For multilingual sites: the quality of your English (or highest-resource-language) content affects entity associations globally. For hreflang strategy, this does not replace proper localization—MUM operates at the entity/knowledge level, not the SERP targeting level. Local language content is still necessary for local-language ranking; MUM improves the depth of entity understanding across languages.

How do I get my content cited in AI Overviews?

From reverse-engineered citation patterns: be an established entity in Google's Knowledge Graph, write passage-level answers that are self-contained within 2–3 sentences, include specific verifiable claims with clear sourcing, and ensure the page has strong E-E-A-T signals (author credentials in structured data, external citations to the page). There is no deterministic path to citation—the model selects dynamically per query.

Does keyword optimization still matter after BERT?

Yes, but the optimization target changes. Optimizing for the exact keyword phrase matters less; optimizing for the semantic intent cluster matters more. Practically: include primary entity names explicitly (BERT still uses them as anchors), cover the entity space expected for the topic (semantic completeness), and write clear predicate relationships. Do not keyword-stuff, but do not write around entities rather than about them.

How does Google handle content that ranks for queries it wasn't written for?

BERT-based dense retrieval surfaces semantic overlaps that lexical matching misses. A page about "crawl budget optimization" may surface for "how to reduce server load from bots" because the semantic overlap is high even without exact keyword match. This is normal and exploitable: audit your GSC impression data for these semantic drift queries, and if they represent significant volume, consider whether to expand the existing page or create targeted content for that sub-intent.

Key Takeaways

• BERT processes context bidirectionally via self-attention—the same word produces different embeddings in different contexts, enabling intent understanding beyond keyword matching.
• BERT operates at two pipeline stages: query understanding (always) and passage-level re-ranking (for a subset of results)—structure your content to create clear passage boundaries.
• MUM enables cross-lingual entity association transfer—the quality of your primary-language content affects entity understanding across all 75+ MUM languages.
• AI Overview citation correlates with: entity clarity, factual density with attribution, passage-level answerability, and Knowledge Graph entity recognition.
• Entity salience analysis via Google's NLP API reveals whether your target entity is the primary salient entity in your content—misalignment explains ranking underperformance.
• Semantic gap analysis (GSC queries × page content cosine similarity) identifies pages serving queries for which the content is semantically misaligned—a priority optimization queue.

Conclusion

BERT, MUM, and the Gemini-powered AI Overviews represent a fundamental shift from statistical keyword matching to semantic understanding. The SEO practitioner who understands the architecture—not as a black box to "game," but as a system with specific input expectations—has a substantial advantage. The inputs the system expects: unambiguous entity identity, complete semantic coverage of the entity space, passage-level answerability, cross-lingual consistency for multilingual sites, and factual density with attribution. The technical toolkit to audit these signals—BERT embeddings, spaCy entity analysis, Google NLP API, semantic similarity clustering—is available and practical to deploy. The practitioners running this analysis systematically are operating on a different plane from those still counting keyword density.