Google Search Console is a powerful tool for website owners. It’s like a direct line from Google telling you about the health of your website from their perspective. One of the most important things it reports on is crawl errors. These happen when Google’s automated program (called Googlebot) tries to visit a page on your site but can’t access it for some reason. Seeing a chart full of red error lines in Search Console can be a bit alarming at first – I know I felt a jolt of panic seeing those early reports – but understanding what they mean and how to fix them is key to a healthy website and good performance in Google Search.
Fixing crawl errors ensures that Googlebot can access and index your important pages, which is fundamental for SEO (Search Engine Optimization). It also helps provide a better experience for users who might encounter broken links. This guide will walk you through how to find these errors in Google Search Console and fix the most common types.
What Are Crawl Errors and Why Fix Them?
A crawl error simply means Googlebot tried to reach a specific URL on your website and failed. This failure can happen for many reasons (which we’ll cover).
Why is fixing them important?
- Indexing & Ranking: If Googlebot can’t crawl a page, it can’t index it. If it’s not indexed, it can’t rank in search results.
- Crawl Budget: Google has a limited amount of time/resources to spend “crawling” your site (your crawl budget). Errors waste this budget on broken pages instead of discovering new or updated content.
- User Experience: Many crawl errors correspond to broken links (pages returning a “Not Found” message), which frustrates visitors.
Let’s dive into Search Console to find these issues.
Step 1: Find Crawl Errors in Google Search Console
Crawl errors related to Googlebot being unable to access pages are primarily reported in the Pages section of Google Search Console.
- Log In: Go to Google Search Console and log in to your account.
- Select Your Property: Choose the website you want to check from the property dropdown in the top left.
- Go to the Pages Report: In the left-hand menu, click on Indexing, then click on Pages.
- Review Error Details: This report shows you how many pages on your site are indexed and, more importantly for this task, how many pages aren’t indexed and the reason why. Look for reasons listed under the “Why pages aren’t indexed” chart that indicate crawling or access problems, such as:
- Not found (404)
- Server error (5xx)
- Soft 404
- Blocked by robots.txt
- Blocked by authorization request (401)
- Redirect error
- Check Sitemaps Report (Optional but Recommended): Also check the Sitemaps report (under Indexing). If your sitemap file submitted to Google contains URLs that are causing errors, this report might flag issues related to those specific URLs.
Focus on the errors listed in the Pages report, as these represent pages Google is actively trying to crawl but failing.
Step 2: Understand the Error Types
Before you fix, you need to know what each error type generally means:
- Not found (404): The server responded that the page at that URL does not exist. This is often due to a broken link or a page that was deleted or moved without a redirect.
- Server error (5xx): Googlebot tried to access the page, but your website’s server had a problem. This could be due to server overload, maintenance, or errors in your website’s code interacting with the server.
- Soft 404: The server responded that the page exists (returned a 200 OK status), but the content is missing, empty, or appears to be an error page to Google. This can confuse search engines.
- Blocked by robots.txt: Your robots.txt file (a file on your server that tells search engine bots where they can and can’t go on your site) is preventing Googlebot from crawling this specific URL.
- Blocked by authorization request (401): The page requires a login or authentication to access it. Googlebot cannot log in.
- Redirect error: Googlebot encountered a problem while following a redirect from one URL to another (e.g., a redirect chain that’s too long, a redirect loop where URLs redirect back to each other, or a redirect to a bad URL).
Step 3: Investigate Specific URLs Using the URL Inspection Tool
Once you’ve identified an error type, you need to look at specific examples to understand why it’s happening for that particular URL. The URL Inspection tool is your best friend here.
- In the Pages report, click on the specific error type you want to investigate (e.g., “Not found (404)”).
- You’ll see a table listing example URLs experiencing this error. Click on a specific URL from the list.
- Click the Inspect URL button or icon (it looks like a magnifying glass). This opens the URL Inspection tool for that specific page.
- The tool will perform a live test and show you details about Google’s last crawl attempt, index status, and the exact error Googlebot encountered. This information is vital for figuring out the root cause. It might show if the URL was found in your sitemap or linked from another page on your site.
How to do it: Use this tool on several examples for each error type to see if there’s a pattern (e.g., all 404s are from an old section of the site, all 5xx errors happen at certain times).
Step 4: Fix the Most Common Types of Crawl Errors (Implement Solutions)
Based on the error type and your investigation with the URL Inspection tool, here’s how to fix the most common issues:
- Fixing 404 “Not found” Errors:
- Diagnosis: GSC reports 404, URL Inspection confirms it. The URL doesn’t load in a browser. Find out where the bad link is coming from (GSC might indicate “Referring page”).
- Solution A (Broken Link): If the page exists but the link is wrong, or if the page is gone but other pages link to it, the fix is to find the links pointing to the incorrect or missing page and update them to the correct URL or remove them. Check internal links on your own site first. If the link is external, contact the other website owner if possible.
- Solution B (Page Moved or Deleted with Replacement): If the content has been moved to a new URL or replaced by a similar, relevant page, implement a 301 Redirect (permanent redirect) from the old 404 URL to the new relevant URL. This tells Google and users where the page moved.
- Solution C (Page Permanently Gone with No Replacement): If the page is removed and there is no relevant replacement page, simply let the 404 error stand. Return a proper 404 status code (which web servers do by default if a file isn’t found). Google will eventually stop trying to crawl it. Make sure you fix any internal links on your site that point to this old 404 page (Solution A). Do NOT redirect all 404 pages to your website’s homepage, as this is bad for user experience and can be seen as a soft 404 by Google if the homepage isn’t truly a replacement for the missing content.
- Fixing Server Error (5xx) Errors:
- Diagnosis: GSC reports 5xx, URL Inspection confirms a server error. Your website is inaccessible due to a problem on the server side.
- Solution: This requires investigating your web hosting environment. Check your server logs for error messages. See if your hosting plan has resource limits you might be hitting (like CPU or memory). If you’re unsure, contact your web hosting provider immediately. They can help diagnose and fix server-side issues.
- Fixing Soft 404 Errors:
- Diagnosis: GSC reports Soft 404, URL Inspection confirms the page returns a 200 status code but looks empty or like an error page.
- Solution: If the page should have content, add unique, substantial content to it. If the page should not exist or is intentionally empty (like an old product page that’s gone), ensure it returns a proper 404 Not Found or 410 Gone status code instead of 200 OK, or implement a 301 Redirect to a relevant category or product page. The goal is to accurately signal to Google that the original content isn’t there.
- Fixing “Blocked by robots.txt” Errors:
- Diagnosis: GSC reports this, URL Inspection confirms robots.txt is disallowing the page. Your robots.txt file is telling Googlebot not to crawl this URL.
- Solution: Access your website’s robots.txt file (usually at yourwebsite.com/robots.txt). Find the Disallow rule that is blocking the specific URL or directory. Edit or remove that specific Disallow rule to allow Googlebot access. Be very careful when editing robots.txt, as mistakes can accidentally block your entire site. Note: robots.txt only prevents crawling. If the page is linked from elsewhere, Google might still index the URL without crawling the content. To prevent both crawling AND indexing, use a noindex meta tag on the page itself.
- Fixing “Blocked by authorization request (401)” Errors:
- Diagnosis: GSC reports this, the page requires a login.
- Solution: If the page should be publicly accessible, remove the login requirement for that specific URL. If the page is intentionally behind a login (e.g., a user profile page), and this URL is not linked from public pages in a way that suggests it should be crawlable, then this is often an expected error for Googlebot and might not require action unless it’s hindering access to public content linked from behind the login.
- Fixing Redirect Errors:
- Diagnosis: GSC reports redirect error, URL Inspection shows problems following redirects (loop, chain too long, etc.).
- Solution: Trace the path of the redirect chain for the problematic URL. Use online redirect checker tools if needed. Identify where the chain is broken or causing a loop (e.g., URL A redirects to URL B, which redirects back to URL A; or URL A redirects to URL B, which redirects to URL C, which is a 404). Correct the redirects to eliminate loops, shorten long chains (ideally one 301 redirect directly to the final destination), and ensure the final destination URL is valid and loads correctly.
Step 5: Validate Your Fix
Once you’ve implemented the fixes for an error type, tell Google Search Console to re-check the affected URLs.
- Go back to the Pages report in Google Search Console.
- Click on the specific error type for which you’ve made fixes (e.g., “Not found (404)”).
- Click the Validate Fix button.
- Search Console will initiate a validation process. It will test a sample of the affected URLs to see if the error is resolved. This process can take anywhere from a few days to several weeks, depending on the number of URLs and Google’s crawling speed.
- Monitor the Report: Check back on the Pages report periodically. The number of URLs affected by the error should decrease as Google re-crawls and finds the issues are resolved. Search Console will notify you if the validation is successful or if some URLs still show the error.
Proactive Measures: Preventing Future Errors
- Regularly Check Internal Links: Use a website crawler tool to scan your site for broken internal links.
- Maintain Your Sitemap: Ensure your XML sitemap only includes valid, existing URLs you want indexed. Update it regularly.
- Test Changes: When moving or deleting pages, always implement proper 301 redirects. Test them after deployment.
- Monitor Search Console: Check your Pages report regularly (at least weekly) to catch new errors early.
Fixing crawl errors is ongoing website maintenance. It’s a bit like being a detective for your website, identifying problems and implementing solutions. By using Google Search Console and following these steps, you can keep your site accessible to Googlebot, improve its SEO health, and ensure a better experience for your visitors. Seeing that error count go down after your fixes is genuinely satisfying!
