I do not want my Swiftype to index part of my site. How can I exclude certain pages from being indexed?
Swiftype gives you a number of ways to control exactly want you want indexed.
1. To delete a single page result from your engine index:
Look up the page using the Content tab (within the Manage section) in the Swiftype Dashboard. You can search/filter by title or URL. Next, click the Details button, to view the full result document. Under the Options dropdown in the upper right, you’ll see an option to delete the current page result. Once you’ve deleted a page, it will not be reindexed.
2. To exclude content on a page from being indexed:
If there are parts of your template or page that are being indexed which you do not want, you can use content exclusion to remove them.
<div data-swiftype-index="false">This will not be indexed.</div>
3. To exclude sections of your site from the crawl:
If there are sections of your site you do not want crawled by Swiftype, use a robots.txt file. Swiftype’s robots.txt support will prevent our web crawler from visiting those sections of your site.
User-agent: Swiftbot
Disallow: /blog/
Disallow: /photos/
If you cannot edit your site’s robots.txt, or you have more complicated rules, you can also take advantage of Swiftype’s path-based whitelists and blacklists.