This can occur when the crawler has not re-visted the webpage to classify it as a 404/403 and remove it.


Webpages that respond with a 404/403 are removed from a collections index when any one of the following occur:

  1. The webpage fires our instant indexing tracker. See how to set up and conditions that apply to instant indexing

  2. The crawler removes it when it automatically visits the webpage on its next scheduled crawl cycle

  3. The webpage is manually removed from the index using the Diagnose tool in the Domains menu

  4. A manual recrawl of the entire site occurs by Re-index all domains in the Domains menu. Note as this will recrawl the entire site it may take some time before the 404 is removed

  5. Our API Delete endpoint is manually called to remove the webpage from the index