Site Crawl Overview

In this guide you’ll learn about the Site Crawl Overview page in your Moz Pro Campaign. If you are looking for more in-depth information about a specific section Site Crawl, please see the associated guides listed in the left hand navigation menu.

Quick Links

• Moz Pro Site Crawl Overview (Video Available)
• Why Audit Your Site?
• Weekly Updates
• Recrawling Your Site
• Total Issues & Total Pages Crawled
• New Issues
• Why Am I Seeing “New” Issues That Are Not New?
• All Issues
• Moz Recommends Fixing
• Your First Moz Crawl

Running regular site audits is essential for keeping on top of any major issues that could be damaging your site's visibility. When you set up your Campaign, our friendly bot, Rogerbot, will crawl your site every week seeking out any issues that may impact your rankings, or the ability of search engines to crawl and index your site, and will prioritize them by Issue Type and frequency.

Critical Crawler Issues like 5xx, 4xx status codes and redirects to 4xx pages are prioritized so you can quickly identify and fix these issues. Site Crawl will also identify any Crawler Warnings, Redirect Issues, Metadata Issues, and Content Issues that may be affecting your site's visibility and traffic.

To get started, open your Moz Pro Campaign and head to the Site Crawl section from the left navigation.

The first thing you’ll see is a breakdown of the number of Pages Crawled, your New Issues count, Issues By Category, and Total Issues.

You can click on the number of Pages Crawled to head to your All Crawled Pages or click on the issue category name to drill deeper into those issues.

Weekly monitoring is great for the “set it and forget it” style of working. We know you do it, and there’s nothing wrong with that! We’ll crawl your site every week and flag up any issues so you don’t have to worry about it.

You can check when your next crawl will run on the top right-hand side.

Made updates to your site? Super! If you’re on the Medium plan or higher, you can force a recrawl of your site.

The Recrawl allowance is for your whole account and is shared between all your Campaigns and seated users. You can see the date your Recrawl limit will replenish just below your Recrawl allowance count.

If you launch a recrawl between Campaign updates, your Latest Crawl date will reflect when that recrawl was done. Your Next Crawl date will still reflect the next scheduled weekly update as recrawls do not stop the automatic weekly update from occurring.

If a crawl is already in progress, the Recrawl my site button will be greyed out until that crawl completes.

Check out the Total Pages Crawled and Total Issues over time for your site in the graphs provided. Hover over any point on the graph for more information including issue counts by category.

We’ve got a positive trend here: our blue graph is fairly stable, so we’re crawling a similar amount of pages and our orange graph is diving, showing the number of new issues going down over time. Show these shiny graphs to your clients to demonstrate the fruits of all your labors. Go on, brush your shoulders off!

Every time we crawl your site, we’ll tag new issues discovered since your previous crawl as New. We’ll alert you to these in an email and flag them in your Campaign.

Click See all new issues to see more issues discovered in your latest crawl or click the magnifying glass under Analyze to see pages associated with that issue type.

Could these new issues be an indication of a larger issue? Is your server down? Maybe you’ve made some changes to your site structure and some pages are now returning 404 errors. Or perhaps you’ve created new content that hasn’t been correctly optimized — does this mean your team needs a gentle nudge towards basic on-page SEO methods? Keeping on top of these issues as they come up will help you ensure your content is accessible to search engines and you’re doing all your can to help your rankings.

If your site is larger than your allotted crawl limit, then we will be unable to crawl every page every week. As a result, we may see a page for the first time and flag a New issue when in fact, that page has been around for some time.

Here are some ways to solve this:

• You may adjust your crawl limit to allow for a full crawl of your site
• You may upgrade your subscription to get higher crawl limits
• You may adjust your robots.txt to tell Moz to crawl fewer pages

For more information and tips on how to investigate spikes in issues, please see our guide.

“So what did you fix?” your savvy client may ask. Right below the overview we have the Issue Type ordered by Number of Issues and the change value icon.

Issues marked with an exclamation point are Critical Crawler Issues. Click on any of the Issue Type titles to get more information about the issue and associated pages.

Click See All Issue Types to view issues we did not find on your site. They’ll be grayed out, and you may also see a change value here if we previously saw these errors but you’ve fixed them all.

Direct from Moz HQ, we’ve got our SEO experts to guide you towards the best way to fix these issues.

You can click the title of the issue or Review issues of this type to learn more about the issue and to see what pages are being flagged.

If your site is more than 250 pages, we will do a rush crawl when your Campaign is first set up so you’re able to get started right away. This mini crawl typically covers about 250-300 pages of your site. Once this first crawl is complete, you’ll receive an email letting you know that there is data in your Campaign, ready for analyzing.

If your site is less than 250 pages, we will still do the first crawl as noted above but it will encompass your whole site instead of only a portion of it.

As soon as your initial crawl is complete, a new crawl will start to crawl your entire site right away. There is no need to launch a recrawl as this will occur automatically. Once that full crawl is complete, you will see the Pages Crawled count update along with the rest of your Site Crawl data.