Key Takeaways

  • Treat robots.txt as a crawler-access convention rather than security, since blocked URLs can still be indexed without content when other signals point to them 8.
  • Audit JavaScript rendering for crawlable href links and parity between server-rendered HTML and client-rendered DOM to avoid canonical ambiguity Google resolves on its own 7.
  • Keep sitemaps limited to indexable, 200-status canonical URLs so Search Console coverage reports isolate issues to specific templates rather than oversized files 9.
  • Consolidate duplicates with 301 redirects, rel=canonical, consistent internal links, and sitemap entries, not robots.txt, since the canonical tag is a hint Google may override 8.
  • Enforce LCP ≤2.5s, INP ≤200ms, and CLS ≤0.1 at the 75th percentile of field data per device, since lab tests miss real mid-range mobile failures 4, 5.
  • Fix INP by auditing event handlers, third-party tags, hydration, and long main-thread tasks, since initial load tuning alone won't resolve mid-session interaction delays 6.
  • Produce unique, non-commodity content tied to client data, operator experience, or proprietary process, because commodity drafts won't surface in classic results or AI Overviews 1, 2.
  • Keep a named editorial reviewer on AI-assisted drafts to verify accuracy and sources, separating legitimate assistance from the scaled content abuse Google penalizes 3.
  • Ensure titles, descriptions, and canonical tags remain unique and consistent between raw HTML and rendered DOM, since hydration frequently overwrites server-set tags across templates 7.
  • Remove Book Actions, Course Info, Claim Review, Estimated Salary, Learning Video, Special Announcement, and Vehicle Listing markup, since Google phased them out on June 12, 2025 11.
  • Scope schema work to types in Google's Search Gallery and validate with the Rich Results Test, since speculative markup is overhead rather than optimization 10.
  • Use Search Console as the first-party source for queries, pages, and coverage buckets, triggering specialist work only when numeric alert lines are crossed 12.

What Google Actually Confirmed About AI Search in 2025

Google's May 2025 guidance for AI Overviews reiterated that AI experiences draw from the same Search index, ranking systems, and retrieval pipelines that classic results use. Performance in these features still depends on unique, non-commodity content, strong page experience, accessible and indexable pages, accurate structured data, and conversion-based measurement 1. The follow-up optimization guide explicitly states:

"SEO is still relevant" for generative AI features, as these features are rooted in core Search quality systems 2.

For agencies, this means the production checklist has not changed shape, only emphasis. Fundamentals like crawlability, canonical hygiene, Core Web Vitals, validated schema, editorial quality control, and Search Console monitoring remain crucial for both blue-link rankings and AI Overview inclusion. Google has also explicitly rejected several tactics circulating as "AEO" or "GEO" best practices, including llms.txt files, mandatory content chunking, and special AI-only markup for Google Search 2.

A Head of SEO should standardize these fundamentals across all client accounts and treat novel AI-search hacks as unverified until Google's own documentation confirms them. The 15 tips that follow are structured as audit criteria a team can enforce at scale.

Crawlability and Indexing: The Non-Negotiable Floor

Tip 1 — Treat robots.txt as a Precise Access Policy, Not a Security Layer

RFC 9309 defines robots.txt as the mechanism site owners use to control how automatic clients may access content. It is a crawler-access convention, not an access control, meaning blocked URLs can still be indexed without their content if other signals point to them 8.

Auditors should confirm the robots.txt file returns a clean 200 status, resolves without redirect chains, uses UTF-8 encoding, and scopes User-agent and Disallow rules to the paths actually intended. Confidential staging environments, admin paths, and PII endpoints should be behind authentication, not merely disallowed. Agencies managing multiple client robots.txt files should version-control each one and alert on unexpected changes, as a single stray wildcard can quietly deindex a template section across a client portfolio.

Tip 2 — Audit JavaScript Rendering Against Crawlable href and Canonical Parity

Googlebot discovers URLs by parsing HTML links and then queues them for crawl and rendering 7. Links that only resolve through onclick handlers, router events, or post-render JavaScript injection are not reliably discovered. All primary navigation items, pagination controls, and internal cross-links important for ranking should be within an anchor tag with a real href.

The second check is parity: the server-rendered HTML and the client-rendered DOM should agree on titles, meta descriptions, canonical tags, hreflang, and primary content. While Googlebot can execute JavaScript, divergence between these layers complicates debugging indexing issues and introduces canonical ambiguity that Google will resolve on its own terms 7. Agency auditors should run a rendered-vs-raw diff on templated page types for each client stack (e.g., product detail, category, location page, blog article). Templates that pass once with a stable framework version usually remain consistent, unlike those built on headless frameworks shipped mid-sprint.

Tip 3 — Keep Sitemaps Canonical-Only and Treat Them as Discovery Signals

Google attempts to crawl URLs exactly as listed in a sitemap and generally shows canonical URLs in search results, making the sitemap a discovery and canonicalization signal rather than an indexing guarantee 9. Submitting a URL does not force its inclusion.

Agency sitemap hygiene requires listing only indexable, 200-status, canonical URLs. Exclude parameterized duplicates, paginated series beyond the primary, noindexed pages, and redirect targets' sources. Split large sitemaps by content type so Search Console coverage reports can isolate problems to specific templates rather than an oversized file. For multi-location clients, a per-location sitemap indexed under a sitemap index file can pinpoint indexing issues to a specific location folder for quick inspection.

Tip 4 — Consolidate Duplicates Without Relying on robots.txt

Google's canonicalization guidance explicitly states not to use robots.txt for canonical selection, as blocked URLs can still be indexed without their content, leading to unpredictable duplicate cluster resolution 8. Consolidation relies on 301 redirects, rel=canonical, consistent internal linking, and sitemap entries all pointing to the same chosen URL.

Common offenders in agency audits include faceted navigation, tracking parameters, HTTP/HTTPS and www/non-www variants, trailing-slash inconsistency, syndicated or franchise-shared content, and printer-friendly or AMP legacy versions. The canonical tag is a hint, not a command; Google may still select a different URL if other signals conflict 8. The audit standard is signal alignment across all four surfaces for every templated URL type, verified by URL Inspection in Search Console. Catching misaligned canonicals in a quarterly sweep prevents the slow bleed of split equity across near-duplicate URLs.

Page Experience: Hitting the Thresholds That Field Data Actually Measures

Tip 5 — Enforce LCP ≤2.5s, INP ≤200ms, CLS ≤0.1 at the 75th Percentile

The Core Web Vitals pass line is defined by three numbers:

  • Largest Contentful Paint (LCP) at 2,500 milliseconds or less
  • Interaction to Next Paint (INP) at 200 milliseconds or less
  • Cumulative Layout Shift (CLS) at 0.1 or less

All three are evaluated at the 75th percentile of real-user field data, segmented by device 4, 5. An INP above 500 milliseconds is classified as poor and requires immediate attention 6.

These thresholds are crucial because they are measured in the field, not in a synthetic lab. A page loading quickly on a specialist's laptop might still fail LCP at the 75th percentile across a client's actual mid-range mobile traffic. Agency auditors should pull Chrome User Experience Report data per template, not per URL, and report pass/fail by device class in client dashboards. A single scorecard row per client indicating "mobile LCP 75p: 3.1s — fail" is more actionable than a lab waterfall chart.

The operational standard across a portfolio is that no template ships, and no CMS theme update merges, without a before/after check against these three numbers on both mobile and desktop field data. Templates that pass once with a given framework version tend to remain in range until a dependency update or a third-party tag changes the budget.

Tip 6 — Chase INP Through Event Handlers and Long Tasks, Not Initial Load

INP is measured across the full lifespan of a visit and reflects the slowest meaningful interaction a user has with the page, meaning initial load tuning alone won't fix it 6. A page can comfortably hit LCP but still fail INP if a search filter, menu expansion, or form field triggers a 600-millisecond main-thread block halfway through the session.

Audit targets include:

  • Oversized event handlers
  • Synchronous third-party tags firing on interaction
  • Hydration code that re-runs on click
  • Long JavaScript tasks that block input processing

Chrome DevTools' Performance panel and the web-vitals library can identify the specific interaction and script causing the worst INP sample. Teams auditing INP across clients should log the top three offending interactions per template, assign them to the development queue based on field-data impact, and re-measure at the 75th percentile after each fix. Tag managers, chat widgets, and A/B testing scripts are frequent culprits across client stacks.

Visualize the three Core Web Vitals thresholds cited in the section (LCP, INP, CLS) as a reference scorecard that matches the numbers in nearby proseVisualize the three Core Web Vitals thresholds cited in the section (LCP, INP, CLS) as a reference scorecard that matches the numbers in nearby prose

Test advanced SEO workflows with real outputs

Experience live SEO execution and measure impact across client sites before making a commitment.

Start Free Trial

Content That Earns Placement in AI Overviews and Classic Results

Tip 7 — Produce Unique, Non-Commodity Content That Satisfies the Query

Google's May 2025 AI search guidance prioritizes "unique, non-commodity content" for performance in AI Overviews and classic results 1. This phrase is deliberately specific. Rewrites of competitor outlines, templated location pages that only swap a city name, and summary pieces aggregating existing articles do not meet this standard.

For an agency, the standard functions as a pre-publish question: what in this draft could only have been written from this client's data, operator experience, or proprietary process? If the answer is nothing, the piece is commodity output regardless of word count or heading structure. Teams auditing content pipelines should log the source of originality for every piece shipped—case data, practitioner interview, first-party research, or operator decision framework—and reject drafts where that field is empty. Since AI features draw from the same index, content that cannot differentiate for blue links will not surface in the generative layer either 2.

Tip 8 — Keep Editorial Review on AI-Assisted Drafts to Avoid Scaled Content Abuse

Google's stance on AI-assisted content is nuanced. Using generative tools to help produce content is not a policy violation. However, publishing large volumes of pages primarily to manipulate rankings, with little added value, is considered scaled content abuse 3.

The operational distinction lies in editorial review. AI-assisted drafts must pass through a named reviewer who checks factual accuracy, verifies cited sources, confirms the piece answers the user's query, and signs off on publication. For agencies scaling production, this review cannot be superficial. Teams should track the reviewer-of-record per piece, time spent on substantive edits versus formatting, and a reject-and-rewrite rate. A review queue where every draft ships untouched is a pattern Google's spam systems are designed to detect. The policy targets scale without quality control, not AI as a tool 3.

Tip 9 — Write Unique Titles and Descriptions That Survive Rendering

Google's JavaScript SEO documentation requires unique, descriptive title elements and meta descriptions on every indexable page, and these tags must remain consistent between the server-rendered HTML and the client-rendered DOM 7. Duplicate titles across templated page sets and framework hydration overwriting server-set tags are common failure modes agency auditors encounter.

The audit is straightforward: crawl the client site twice, once as raw HTML and once as rendered DOM, then compare titles, descriptions, and canonical tags on every template. Any divergence should be routed to the development queue before the next content push.

Structured Data After the June 2025 Deprecations

Tip 10 — Retire Book Actions, Course Info, Claim Review, Estimated Salary, Learning Video, Special Announcement, and Vehicle Listing Markup

On June 12, 2025, Google announced the phase-out of seven structured-data features in search results: Book Actions, Course Info, Claim Review, Estimated Salary, Learning Video, Special Announcement, and Vehicle Listing 11. While the markup remains valid Schema.org vocabulary, it no longer produces a supported rich result in Google Search. This means the engineering and QA cost of maintaining it on client sites yields no visible benefit where agencies justify their work.

The audit action is portfolio-wide. Teams should query each client's schema library, flag any JSON-LD blocks that still emit these seven types, and schedule them for removal or demotion in the next release cycle. Education clients using Course Info, legal and media clients using Claim Review, recruiting clients using Estimated Salary, and automotive dealers using Vehicle Listing are particularly exposed. The visual below pairs these seven retired types against the schemas still producing rich results in Google's Search Gallery 10, allowing a schema librarian to categorize each client's inventory as "keep," "deprecate," or "remove" in one pass. Treating a schema portfolio as permanent is a pattern this deprecation penalizes 11.

Tip 11 — Validate Only the Schemas That Still Produce Supported Rich Results

Google's Search Gallery is the authoritative list of structured-data types eligible for supported rich results. Eligibility does not guarantee a rich appearance, a ranking lift, or inclusion in AI-generated features 10. Therefore, schema work should be scoped to types Google currently documents as producing a visible surface, validated against the Rich Results Test, and monitored in Search Console's Enhancements reports.

The agency standard is to review a client's schema inventory against the Search Gallery quarterly. Any markup outside that list should either be justified by a non-Google use case or removed. Speculative schema added for perceived ranking benefit is overhead, not optimization 10.

Comparison table visualizing the seven deprecated schema types versus the standard of validating only Search Gallery types still producing rich results, directly supporting the schema audit action described in the sectionComparison table visualizing the seven deprecated schema types versus the standard of validating only Search Gallery types still producing rich results, directly supporting the schema audit action described in the section

Measurement: Replace Rank Reports with Conversion and Coverage Loops

Tip 12 — Make Search Console the First-Party System for Queries, Pages, and Coverage

Search Console's Search performance report categorizes traffic into queries, pages, and countries, with impression and click trends sliceable by date, device, and search appearance 12. This makes it Google's only first-party measurement surface, and third-party rank trackers should be downstream of it, not replace it.

The operating loop is concise:

  1. Pull coverage weekly to identify new "not-indexed," "crawled-not-indexed," and "discovered-not-indexed" buckets per client.
  2. Pull the Search performance report monthly to identify queries losing impressions and pages losing clicks against their trailing baseline.
  3. Inspect affected URLs individually before escalating.

Agency teams managing many accounts should standardize exports and thresholds that trigger a ticket, ensuring specialists engage with a client only when a specific coverage bucket or page cluster crosses a numeric alert line, not on a fixed calendar cadence 12.

Tip 13 — Report Conversions, Not Clicks, Per Google's May 2025 Guidance

Google's May 2025 AI search guidance advises creators to measure conversions rather than clicks alone, because AI features alter how visitors discover pages and how often a click is required to resolve a query 1. Dashboards still leading with ranking position and session count will understate value on informational queries resolved in the Overview and overstate value on commercial queries where click volume held but intent weakened.

Agency reporting should prioritize conversion events, qualified calls, bookings, and pipeline contribution at the top of each client scorecard, with Search Console impressions and clicks serving as diagnostic inputs below. Clients retained on blended organic outcomes are less likely to churn over ranking fluctuations if they can see it's not impacting revenue 1.

Tip 14 — Diagnose Traffic Drops Across Technical, Algorithmic, Seasonal, and Demand Causes Before Rewriting

Google's own debugging guide instructs teams to separate four causes before modifying a page:

  1. Technical problems
  2. Algorithmic changes
  3. Seasonality
  4. Demand shifts

...with reporting artifacts as a fifth confounder 13. Server availability, robots.txt retrieval failures, and spikes in not-found pages are technical examples Google highlights as capable of blocking crawling, indexing, or serving 13.

The decision infographic below serves as a triage sheet a specialist can follow quickly. Start with Search Console's Crawl stats and Pages report to rule out a 5xx spike, a robots.txt fetch error, or a surge in 404s. If the technical layer is clean, overlay the drop window against Google's Search Status Dashboard entries and known update windows to test the algorithmic hypothesis. If neither explains the decline, pull year-over-year comparisons to isolate seasonality, then check Google Trends for the top affected queries to measure demand change.

Content rewrites should be the last step in this sequence, not the first. Teams that reflexively rewrite on every drop waste production hours that could have been resolved with a crawler access fix or a demand-trend explanation to the client 13.

Process/decision flow infographic that mirrors Google's recommended traffic-drop triage sequence described in the paragraph, so a specialist can follow it as a diagnostic sheetProcess/decision flow infographic that mirrors Google's recommended traffic-drop triage sequence described in the paragraph, so a specialist can follow it as a diagnostic sheet

See How Leading Agencies Are Scaling SEO Execution with AI-Driven Precision

Discover the workflow and automation strategies top agencies use to deliver technical and content SEO at scale—without expanding headcount. Request a tailored demo for your agency or enterprise team.

Contact Sales

Local Reputation Under the FTC Reviews Rule

Tip 15 — Collect Authentic Reviews and Disclose Material Relationships Under the October 21, 2024 Rule

The FTC's Consumer Reviews and Testimonials Rule, effective October 21, 2024, targets deceptive or unfair conduct involving consumer reviews and testimonials. This includes fake reviews, undisclosed material connections, suppression of negative reviews, and incentivized ratings presented as organic 15. Local SEO workflows that still treat review generation as a volume play now carry regulatory exposure in addition to platform-policy risks.

The agency standard has four control points:

  • Review requests must go to actual customers after a verified transaction, not to purchased lists or staff.
  • Any incentive tied to a review must be disclosed and cannot be conditioned on sentiment.
  • Employee, family, and vendor reviews require clear disclosure of the material relationship.
  • Suppression of unfavorable reviews on a client's owned properties is prohibited 15.

Teams managing reputation across regulated verticals should document the approval path for every testimonial published on a client site, logging reviewer identity, consent, and disclosure language alongside the asset.

If You Manage Multiple Client Accounts: Portfolio Economics of a Standardized Checklist

This section is for Heads of SEO managing teams across 15 to 150 client accounts, where margin is determined by specialist hours per client per month, not by individual expertise.

A standardized checklist transforms this dynamic. The preceding tips form a fixed audit grid covering robots.txt diffs, rendered-vs-raw parity, canonical-only sitemaps, Core Web Vitals field data at the 75th percentile 4, 5, INP interaction logs 6, schema inventory against the Search Gallery and the June 12, 2025 deprecation list 10, 11, Search Console coverage and performance pulls 12, traffic-drop triage sheets 13, and FTC review-workflow controls 15. Each item has a defined input, a pass/fail threshold, and a standardized ticket format.

The portfolio consequence is scope control. Let H be specialist hours per full audit, A audits per quarter per client, and C clients per team. A team's quarterly audit load is H × A × C. When H is fixed by a checklist and audit output is standardized, two things occur: new client onboarding stops expanding to fill available hours, and incident work routes to the specific checklist row that failed rather than to an open-ended investigation.

VariableMeaningAgency lever
HSpecialist hours per auditShrinks as checklist tooling matures
AAudits per client per quarterSet by contract tier
CClients per podExpands as H falls

Margin per account increases when H falls faster than C grows. Platforms that enforce the same checklist across every client account—with approval gates on each ticket—make this compression repeatable without sacrificing the oversight clients expect 2, 3.

What Google Has Publicly Rejected: AEO, GEO, llms.txt, and Mandatory Chunking

Google's optimization guide for generative AI features is direct about what it does not require. The document rejects the need for special "AEO/GEO hacks," mandatory content chunking, or AI-only markup, and lists llms.txt files as an unsupported optimization tactic for Google Search 2. The guide reiterates that generative AI features run on the existing Search index, ranking systems, and retrieval pipelines, which is why "SEO is still relevant" and why novel AI-specific files or formats do not confer an advantage within Google's surfaces 2.

For a Head of SEO allocating hours across clients, this implies a scope reduction. Time budgeted for llms.txt rollouts, forced paragraph chunking, FAQ-stuffing for AI answers, or hidden AI-only markup should be reallocated to the audit grid that actually drives performance: crawlability, canonical hygiene, Core Web Vitals field data, validated schema from the Search Gallery, editorially reviewed content, and Search Console-based measurement 1, 2. If a vendor pitch relies on tactics Google's own documentation names as unnecessary, the burden of proof lies with the vendor, not the team.

Frequently Asked Questions