Does Swiftype respect robots.txt files?


#1

Absolutely! You can use a robots.txt file to limit what parts of your site the Swiftype search agent will crawl for engine indexing.

For additional details about robots.txt support, you can refer to our documentation.