What is the best way to index and test search in a password-protected staging environment?

Answer

If your staging environment is not publicly accessible, then you will need to allow our crawler access to it.

There are a number of ways to achieve this if you want:

  • The easiest way to do this is to look for Sajaribot - at the start of the user-agent in HTTP requests, and allow these requests.

  • It is also possible to whitelist a range of IP addresses used by our crawling infrastructure. We generally recommend that you check these often as they are likely to change. Our primary crawling system runs within Google Cloud and has a very large and dynamic address range. Raise a request at via the Service Desk and we will get back to you with the current IP list.

  • If this is difficult, you can always index your production site instead, and then test new search interface integration on your staging site using your production data. This presents no performance issues and will not change the search functionality of your production site. This method allows the UI to be developed without the need for us to index your staging site.