Google Updates Crawl Budget Best Practices via @sejournal, @MattGSouthern

Google has updated its Search Central documentation with new best practices for large websites that maintain separate mobile and desktop versions.

This update specifically addresses link parity issues that may affect crawl efficiency.

Context

Crawl budget refers to the number of pages Google will crawl and index on a website within a given timeframe.

This resource allocation is important for SEO, as it determines how quickly Google discovers, crawls, and indexes new or updated content.

For large websites, efficient use of crawl budget can mean the difference between discovering new pages quickly or languishing in obscurity for weeks.

Key Changes

Google now explicitly recommends that large websites provide the same links on their mobile version as on their desktop version.

Google has added the following specific instruction to its crawl budget documentation:

“If your site uses separate HTML for mobile and desktop versions, provide the same set of links on the mobile version as you have on the desktop version. If it’s not possible to provide the same set of links on the mobile version, ensure that they’re included in a sitemap file. Google only indexes the mobile version of pages, and limiting the links shown there can slow down discovery of new pages.”

This addition provides two clear paths for managing links across mobile and desktop versions:

• Include all desktop version links on the mobile version (preferred method)
• Alternatively, ensure all links are included in the website’s sitemap file

This practice helps prevent delays in the discovery of new pages.

Impact

This update primarily affects:

• Sites with one million+ pages that update content weekly
• Medium-sized sites (10,000+ pages) with daily content changes
• Websites with significant portions of URLs marked as “Discovered – currently not indexed” in Search Console

The new guidelines reflect Google’s focus on mobile-first indexing and efficient crawling practices.

Google notes that having fewer links on mobile versions can greatly hinder the discovery of new pages, which may negatively affect a site’s visibility in search results.

Recommendations

To ensure optimal crawling efficiency, you should:

1. Audit your website’s mobile and desktop link structures
2. Update sitemaps to include all important URLs
3. Verify that critical navigation paths are accessible on mobile versions
4. Monitor crawl stats for potential discovery issues

Consider this update a reminder that crawl budget optimization remains a top consideration for large websites.

Featured Image: Mameraman/Shutterstock

SEJ STAFF Matt G. Southern Senior News Writer at Search Engine Journal

Matt G. Southern, Senior News Writer, has been with Search Engine Journal since 2013. With a bachelor’s degree in communications, ...