Skip to main content

Why is my website crawl taking a long time?

Sean Needham avatar
By Sean Needham

There can be a number of reasons preventing a website crawl from ending.
Typical reasons and possible solutions are provided in the table below. 

Reason for Issue Possible Solutions

Your website is responding slowly, therefore, our crawler has automatically slowed down so as not to overload the website.

Exclude any areas of the website that you do not want to be checked.

See: How to add and remove content from a crawl

The login credentials provided to Siteimprove, for a website behind a login, have expired.

Provide Siteimprove Technical Support with the new login credentials for the website.

The crawler has found a new section on the website and needs to be re-configured accordingly, e.g. a new forum, a web-shop with duplicate pages, etc.

Exclude any areas of the website that you do not want to be checked.

See: How to add and remove content from a crawl

Open a ticket to ask Technical Support to review the crawler settings and configure the crawler to reduce duplicate pages or exclude unwanted areas of the website from the crawl.

The IP address for the crawler has been blocked by your firewall.

Ensure that Siteimprove Crawler IP addresses can access the website.

Contact Siteimprove Technical Support if you continue to have issues with a website crawl not ending. 

Was this article helpful?
0 out of 0 found this helpful