What is the best way to index and test search in a password-protected staging environment?

Answer

If your staging environment is not publicly accessible, then you will need to allow our crawler access to it.

There are a number of ways to achieve this if you want:

  • The easiest way to do this is to look for Sajaribot - at the start of the user-agent in HTTP requests, and allow these requests.

  • It is also possible to whitelist a range of IP addresses used by our crawling infrastructure. Here is a list of the current IP addresses.

  • If this is difficult, you can always index your production site instead, and then test new search interface integration on your staging site using your production data. This presents no performance issues and will not change the search functionality of your production site. This method allows the UI to be developed without the need for us to index your staging site.