Skip to main content

Crawler Migration

Felix Petsovits avatar


What is the Siteimprove crawler?

Siteimprove uses a crawler to evaluate the contents of our customers' websites. The crawler has a starting point (index URL), which is typically the home page. Following a list of instructions (for example domain matches, robots.txt, and exclusions), the crawler is able to regularly index websites and then evaluate for various issues (e.g. broken links and misspellings).

Why is Siteimprove introducing a new crawler?

The Siteimprove crawler has been updated, improved and tweaked over the course of the years, and in the past couple of years, we have been working on introducing a new crawler utilizing the newest technologies available. 

We are making this update primarily for the following reasons:

  • To accommodate new web technologies
  • To be able to crawl websites that rely heavily on Javascript
  • To enable our NextGen Accessibility product

What will change?

Siteimprove will migrate all accounts that are not already using the new crawler. 

In November 2019, we will start to migrate accounts in batches. The migration will happen throughout the year 2020.

How does this migration affect Siteimprove customers?

No action is required by our users, and we do not anticipate any issues with the migration. If we identify any fluctuations in pages, links, or in other areas, our Technical Support Team will handle these on a case by case basis. 

When will the change happen?

When your account has been selected for migration, Account Owners and Administrators will receive a notification email two weeks before the migration itself. There will also be a notification within the platform when the migration is taking place. 

Please contact Siteimprove Technical support if you have any questions regarding this. 

Was this article helpful?

6 out of 10 found this helpful