How to instruct the crawler to visit all subdomains or prevent subdomains being crawled?
Do I need to add www subdomain?
In the Domains menu you define which domains you want the crawler to visit. Adding a www-prefix instructs the crawler to only crawl the apex domain (www.example.com
), whilst adding a domain without the prefix will instruct the crawler to additionally visit all subdomains i.e. blog.example.com
, shop.example.com
, etc.
Alternatively you could define the domain without the www-prefix i.e. example.com
and then create rules in the Crawler rules menu to select specific subdomains for exclusions.