Can index password protected sites?


Yes! It is possible to index login-protected pages.

You can whitelist our crawler (via user agent) so that we can index your login-protected website pages. You can do this by allowing requests that have “Sajaribot -“ at the start of the user-agent.

Alternatively, you can also whitelist our crawler using IP addresses. Here is a list of the current IP addresses. These IP’s will change over time albeit not frequently. To programmatically obtain the current IP’s we recommend using this tool.