How does crawling and indexing work?

Answer

Our crawler visits the webpages of the domains you add to your collection. Read “How the crawler works” for more information.

Once the collection is indexed, all the webpages in the index are then re-visited by the crawler periodically between 3-7 days.

If you have our Instant Indexing ping-back code installed on your website, any new or updated webpage is updated in the collection when any of the following fields are updated and the page is visited within 30 minutes of being updated.

  1. title

  2. description

  3. canonical

  4. robots

  5. og:title

  6. og:image

  7. og:description

Instant indexing does not remove records from the index if a webpage’s status code is changed to a 404, 403, 301, or a 302. However, the regular crawl cycle does take the status code into account and will remove the page from the index if the page returns a 404, 403, 301, or a 302.