We have a staging site that is protected by HTTP Basic Authentication. Is there any way to get Swiftype to index a password protected site?
Unfortunately, Swiftype’s crawler does not automatically support indexing content protected by HTTP authentication at this time.
It’s possible to work around this limitation, by configuring your site to allow access to our crawler agent (“Swiftbot”).
This is typically achieved by configuring your server’s .htaccess file to allow access to Swiftbot, or more securely, your account specific user agent string.
We can even provide you with your account specific User Agent ID (upon request), or the range of IPs our crawler will originate from if that form of whitelisting is preferred.
Also, another option to consider is using the Swiftype API to send us your content directly instead of utilizing the Swiftype crawler.