Why is my website crawl taking a long time?
By Sean Needham
There can be a number of reasons preventing a website crawl from ending.
Typical reasons and possible solutions are provided in the table below.
| Reason for Issue | Possible Solutions |
|---|---|
|
Your website is responding slowly, therefore, our crawler has automatically slowed down so as not to overload the website. |
Exclude any areas of the website that you do not want to be checked. |
|
The login credentials provided to Siteimprove, for a website behind a login, have expired. |
Provide Siteimprove Technical Support with the new login credentials for the website. |
|
The crawler has found a new section on the website and needs to be re-configured accordingly, e.g. a new forum, a web-shop with duplicate pages, etc. |
Exclude any areas of the website that you do not want to be checked. See: How to add and remove content from a crawl Open a ticket to ask Technical Support to review the crawler settings and configure the crawler to reduce duplicate pages or exclude unwanted areas of the website from the crawl. |
|
The IP address for the crawler has been blocked by your firewall. |
Ensure that Siteimprove Crawler IP addresses can access the website. |
Contact Siteimprove Technical Support if you continue to have issues with a website crawl not ending.