# Setup & Indexing

**URL:** https://community.swiftype.com/c/faq/indexing/5.md

[Latest](https://community.swiftype.com/latest.md) · [Categories](https://community.swiftype.com/categories.md) · [Tags](https://community.swiftype.com/tags.md)

---

## [About the Setup & Indexing category](https://community.swiftype.com/t/about-the-setup-indexing-category/20)

<div class="topic-metadata">

**Author:** [@system](https://community.swiftype.com/u/system)\
**Replies:** 0\
**Last updated:** [December 18, 2015, 12:37am UTC](https://community.swiftype.com/t/about-the-setup-indexing-category/20 "2015-12-18T00:37:35Z")

</div>

Home base for general discussions about getting started with Swiftype! This channel is also broken up into sub-categories with regard to: meta tags The Swiftype API WordPress Plug-ins & Extensions

---

## [My search index contains sensitive data. How can I prevent it from being exposed?](https://community.swiftype.com/t/my-search-index-contains-sensitive-data-how-can-i-prevent-it-from-being-exposed/74)

<div class="topic-metadata">

**Author:** [@system](https://community.swiftype.com/u/system)\
**Replies:** 0\
**Last updated:** [December 21, 2015, 10:56pm UTC](https://community.swiftype.com/t/my-search-index-contains-sensitive-data-how-can-i-prevent-it-from-being-exposed/74 "2015-12-21T22:56:52Z")

</div>

I’m indexing content from multiple users and I want to make sure a user can only see their own content. How can I prevent sensitive data from being exposed with Swiftype? Swiftype can handle user-specific content usin…

---

## [PDF published\_at metadata](https://community.swiftype.com/t/pdf-published-at-metadata/862)

<div class="topic-metadata">

**Author:** [@sdhardesty](https://community.swiftype.com/u/sdhardesty)\
**Replies:** 1\
**Last updated:** [September 19, 2018, 2:34pm UTC](https://community.swiftype.com/t/pdf-published-at-metadata/862 "2018-09-19T14:34:43Z")

</div>

How do you assign a published\_at value to .pdf files? I tried setting it as a custom metadata field within the PDF’s properties, but that failed. Just deriving it by the last time the pdf was indexed is wildly inaccurate…

---

## [Why does Swiftype crawl/index fewer pages than are on my site?](https://community.swiftype.com/t/why-does-swiftype-crawl-index-fewer-pages-than-are-on-my-site/36)

<div class="topic-metadata">

**Author:** [@system](https://community.swiftype.com/u/system)\
**Replies:** 0\
**Last updated:** [December 21, 2015, 6:04pm UTC](https://community.swiftype.com/t/why-does-swiftype-crawl-index-fewer-pages-than-are-on-my-site/36 "2015-12-21T18:04:21Z")

</div>

We have noticed that the number of crawled results in the Contents tab is fewer than the actual number pages on the site. How can we crawl all of our content? Here are some of the most common reasons Swiftype might no…

---

## [Can Swiftype's crawler index dynamically generated (AJAX) page content?](https://community.swiftype.com/t/can-swiftypes-crawler-index-dynamically-generated-ajax-page-content/141)

<div class="topic-metadata">

**Author:** [@system](https://community.swiftype.com/u/system)\
**Replies:** 0\
**Last updated:** [March 17, 2016, 6:47pm UTC](https://community.swiftype.com/t/can-swiftypes-crawler-index-dynamically-generated-ajax-page-content/141 "2016-03-17T18:47:52Z")

</div>

Is the Swiftype content crawler able to handle asynchronously served content (AJAX content)? Swiftbot is unable to process JavaScript directly, however, it can index dynamic AJAX or Single-Page Application content as …

---

## [How to associate meta tag field in search result in configuration?](https://community.swiftype.com/t/how-to-associate-meta-tag-field-in-search-result-in-configuration/646)

<div class="topic-metadata">

**Author:** [@cfu](https://community.swiftype.com/u/cfu)\
**Replies:** 14\
**Last updated:** [July 9, 2018, 1:24pm UTC](https://community.swiftype.com/t/how-to-associate-meta-tag-field-in-search-result-in-configuration/646 "2018-07-09T13:24:14Z")

</div>

With out-of-box customize appearance search result content on Swiftype account dashboard, there is only a list of fields to select from. But we’d like to use the page description, which is tagged by meta tag name=“descri…

---

## [How to index dynamic pages with URL parameters](https://community.swiftype.com/t/how-to-index-dynamic-pages-with-url-parameters/700)

<div class="topic-metadata">

**Author:** [@hakanu](https://community.swiftype.com/u/hakanu)\
**Replies:** 0\
**Last updated:** [May 16, 2018, 2:42am UTC](https://community.swiftype.com/t/how-to-index-dynamic-pages-with-url-parameters/700 "2018-05-16T02:42:26Z")

</div>

Unfortunately, our website setup includes lots of URLs with dynamic content through URL parameters. Seems like swiftype is not crawling any of those, and couldn’t find any information in documentation.

---

## [Crawler stops under 30k](https://community.swiftype.com/t/crawler-stops-under-30k/695)

<div class="topic-metadata">

**Author:** [@peranziani-team](https://community.swiftype.com/u/peranziani-team)\
**Replies:** 0\
**Last updated:** [May 7, 2018, 7:21am UTC](https://community.swiftype.com/t/crawler-stops-under-30k/695 "2018-05-07T07:21:29Z")

</div>

Hi, i have a sitemap.xml referencing more than 40k pages. But your crawler stops at 29k pages. Tried deleting engine and re-istantiating crawling: same result. Do you have a limit?

---

## [AND logic in search results](https://community.swiftype.com/t/and-logic-in-search-results/291)

<div class="topic-metadata">

**Author:** [@communications](https://community.swiftype.com/u/communications)\
**Replies:** 4\
**Last updated:** [May 3, 2018, 8:39pm UTC](https://community.swiftype.com/t/and-logic-in-search-results/291 "2018-05-03T20:39:26Z")

</div>

Hi, Looking for advice on how to adjust the way in which Swiftype handles free-text phrases. At the moment the engine is defaulting to OR logic which can often throw up some unwanted results across common phrases. I wa…

---

## [Will Custom Results Ranking be Lost After Domain Move?](https://community.swiftype.com/t/will-custom-results-ranking-be-lost-after-domain-move/687)

<div class="topic-metadata">

**Author:** [@rfindustries](https://community.swiftype.com/u/rfindustries)\
**Replies:** 1\
**Last updated:** [May 1, 2018, 11:59pm UTC](https://community.swiftype.com/t/will-custom-results-ranking-be-lost-after-domain-move/687 "2018-05-01T23:59:39Z")

</div>

right now my site search is installed on a dev website. when I go live with my site and move from dev.mywebsite.com to www.mywebsite.com, will I have to redo all my customization for search results? I’ve been customizi…

---

## [Site crawler HEAD](https://community.swiftype.com/t/site-crawler-head/691)

<div class="topic-metadata">

**Author:** [@peranziani-team](https://community.swiftype.com/u/peranziani-team)\
**Replies:** 0\
**Last updated:** [April 30, 2018, 2:40pm UTC](https://community.swiftype.com/t/site-crawler-head/691 "2018-04-30T14:40:08Z")

</div>

Hi, can we know if your crawler makes an HTTP HEAD hit before indexing a page? Thank you.

---

## [Excluding internal IP addresses](https://community.swiftype.com/t/excluding-internal-ip-addresses/684)

<div class="topic-metadata">

**Author:** [@customerservice](https://community.swiftype.com/u/customerservice)\
**Replies:** 0\
**Last updated:** [April 25, 2018, 4:37pm UTC](https://community.swiftype.com/t/excluding-internal-ip-addresses/684 "2018-04-25T16:37:27Z")

</div>

Any update on being able to exclude certain IP addresses from showing up in search analytics?

---

## [Using an enum in a Body-embedded data-attribute tag?](https://community.swiftype.com/t/using-an-enum-in-a-body-embedded-data-attribute-tag/676)

<div class="topic-metadata">

**Author:** [@brian.cama](https://community.swiftype.com/u/brian.cama)\
**Replies:** 1\
**Last updated:** [April 18, 2018, 7:04pm UTC](https://community.swiftype.com/t/using-an-enum-in-a-body-embedded-data-attribute-tag/676 "2018-04-18T19:04:18Z")

</div>

In attempting to index our web pages I’ve found the data-attribute tags to be invaluable as in our current CMS we lack some basic access to the backend and specifically can’t create dynamic meta tags within the head of o…

---

## [How to control crawler indexing to get a page fall in a certain document\_type](https://community.swiftype.com/t/how-to-control-crawler-indexing-to-get-a-page-fall-in-a-certain-document-type/678)

<div class="topic-metadata">

**Author:** [@peranziani-team](https://community.swiftype.com/u/peranziani-team)\
**Replies:** 1\
**Last updated:** [April 18, 2018, 1:39pm UTC](https://community.swiftype.com/t/how-to-control-crawler-indexing-to-get-a-page-fall-in-a-certain-document-type/678 "2018-04-18T13:39:02Z")

</div>

Hi, your documentation states: “The crawler creates a DocumentType called page with the following schema \[…\]” There is a way, using meta tags, to make the crawler index the page in a specific document type which I pre…

---

## [Does Swiftype work with Magento?](https://community.swiftype.com/t/does-swiftype-work-with-magento/94)

<div class="topic-metadata">

**Author:** [@system](https://community.swiftype.com/u/system)\
**Replies:** 0\
**Last updated:** [December 21, 2015, 11:41pm UTC](https://community.swiftype.com/t/does-swiftype-work-with-magento/94 "2015-12-21T23:41:34Z")

</div>

Can Swiftype be used with the Magento eCommerce platform? Yes! Similar to any other CMS platform that allows for customization of templates and functionality, it’s possible to configure your Magento instance to work w…

---

## [Swiftype search on sites that require authentication](https://community.swiftype.com/t/swiftype-search-on-sites-that-require-authentication/636)

<div class="topic-metadata">

**Author:** [@milan\_milosavljevic](https://community.swiftype.com/u/milan_milosavljevic)\
**Replies:** 3\
**Last updated:** [March 28, 2018, 2:16pm UTC](https://community.swiftype.com/t/swiftype-search-on-sites-that-require-authentication/636 "2018-03-28T14:16:57Z")

</div>

Hi, Can we use Swiftype search on sites that require authentication? We are using IIS with Basic authentication enabled. Regards, Milan

---

## [Crawling www.mydomain.com (Drupal 7 site) but links are indexed as http://mydomain.com (without the www). Why?](https://community.swiftype.com/t/crawling-www-mydomain-com-drupal-7-site-but-links-are-indexed-as-http-mydomain-com-without-the-www-why/624)

<div class="topic-metadata">

**Author:** [@yanick.girouard](https://community.swiftype.com/u/yanick.girouard)\
**Replies:** 0\
**Last updated:** [March 20, 2018, 2:14pm UTC](https://community.swiftype.com/t/crawling-www-mydomain-com-drupal-7-site-but-links-are-indexed-as-http-mydomain-com-without-the-www-why/624 "2018-03-20T14:14:29Z")

</div>

I’m testing Swiftype for a Drupal 7 site, and I’m having a problem where even though I’m asking to crawl https://www.mydomain.com, some of the pages being indexed are showing http://mydomain.com/the/page/ as their URL in…

---

## [Special characters not displaying correctly](https://community.swiftype.com/t/special-characters-not-displaying-correctly/622)

<div class="topic-metadata">

**Author:** [@rvega](https://community.swiftype.com/u/rvega)\
**Replies:** 0\
**Last updated:** [March 20, 2018, 12:38am UTC](https://community.swiftype.com/t/special-characters-not-displaying-correctly/622 "2018-03-20T00:38:52Z")

</div>

I am getting broken special characters in my search results, is there a way to either hide them or not index them?

---

## [How to correctly add an image to a search result using the meta tag on a page](https://community.swiftype.com/t/how-to-correctly-add-an-image-to-a-search-result-using-the-meta-tag-on-a-page/106)

<div class="topic-metadata">

**Author:** [@system](https://community.swiftype.com/u/system)\
**Replies:** 0\
**Last updated:** [December 22, 2015, 5:50pm UTC](https://community.swiftype.com/t/how-to-correctly-add-an-image-to-a-search-result-using-the-meta-tag-on-a-page/106 "2015-12-22T17:50:45Z")

</div>

When I add the meta tag to a page to have a thumbnail image display on the search results page, should I use the full link to the image such as: \<meta class='swiftype' name='image' data-type='enum' content='http://www.…

---

## [How can I include images in my search results?](https://community.swiftype.com/t/how-can-i-include-images-in-my-search-results/27)

<div class="topic-metadata">

**Author:** [@system](https://community.swiftype.com/u/system)\
**Replies:** 0\
**Last updated:** [December 21, 2015, 4:55pm UTC](https://community.swiftype.com/t/how-can-i-include-images-in-my-search-results/27 "2015-12-21T16:55:22Z")

</div>

I have a lot of different images on my website and I want my site search box to return results with those images. Will Swiftype’s custom site search engine be able to show the images in results like I want? Swiftype d…

---

## [Bulk\_create\_or\_update\_verbose returns "Unexpected error saving document." everytime](https://community.swiftype.com/t/bulk-create-or-update-verbose-returns-unexpected-error-saving-document-everytime/598)

<div class="topic-metadata">

**Author:** [@tim1](https://community.swiftype.com/u/tim1)\
**Replies:** 0\
**Last updated:** [February 26, 2018, 12:34pm UTC](https://community.swiftype.com/t/bulk-create-or-update-verbose-returns-unexpected-error-saving-document-everytime/598 "2018-02-26T12:34:29Z")

</div>

We are using the API to add documents and when I use the end point document\_types/products/documents.json we manage to get success. However anytime I try to use document\_types/pages/documents/bulk\_create\_or\_update\_verbos…

---

## [Using the JavaScript embed with ASP.net Forms](https://community.swiftype.com/t/using-the-javascript-embed-with-asp-net-forms/553)

<div class="topic-metadata">

**Author:** [@cpatton](https://community.swiftype.com/u/cpatton)\
**Replies:** 5\
**Last updated:** [February 23, 2018, 4:05pm UTC](https://community.swiftype.com/t/using-the-javascript-embed-with-asp-net-forms/553 "2018-02-23T16:05:20Z")

</div>

Some ASP.net pages are wrapped in tags, which can cause conflicts with our JavaScript embed/widget code. The most common issue that arises is that our embed will take over the form and provide search results instead of…

---

## [Can the crawler "go through" blacklisted pages?](https://community.swiftype.com/t/can-the-crawler-go-through-blacklisted-pages/556)

<div class="topic-metadata">

**Author:** [@valentin.crettaz](https://community.swiftype.com/u/valentin.crettaz)\
**Replies:** 2\
**Last updated:** [February 3, 2018, 10:06am UTC](https://community.swiftype.com/t/can-the-crawler-go-through-blacklisted-pages/556 "2018-02-03T10:06:46Z")

</div>

I wanted to know whether the crawler would still pass through a blacklisted page, even though the blacklisted page itself is not indexed. The reason behind this is when you have some kind of summary page that lists othe…

---

## [Autocomplete returns a different url value than Search](https://community.swiftype.com/t/autocomplete-returns-a-different-url-value-than-search/531)

<div class="topic-metadata">

**Author:** [@rkeairns](https://community.swiftype.com/u/rkeairns)\
**Replies:** 0\
**Last updated:** [December 14, 2017, 4:42pm UTC](https://community.swiftype.com/t/autocomplete-returns-a-different-url-value-than-search/531 "2017-12-14T16:42:01Z")

</div>

I understand that autocomplete is optimized for a quicker search, but why would the results return different urls? Specifically, when configuring your install, the value of the url is different between Autocomplete and S…

---

## [Does Swiftype index anchor tag URLs when crawling a page?](https://community.swiftype.com/t/does-swiftype-index-anchor-tag-urls-when-crawling-a-page/265)

<div class="topic-metadata">

**Author:** [@jpearson](https://community.swiftype.com/u/jpearson)\
**Replies:** 0\
**Last updated:** [August 22, 2016, 6:33pm UTC](https://community.swiftype.com/t/does-swiftype-index-anchor-tag-urls-when-crawling-a-page/265 "2016-08-22T18:33:08Z")

</div>

While the crawler based engine doesn’t crawl existing anchor tag URLs and associate them with the parent page result document, by default a crawled page will make note of any section headings wrapped in h tags. These sec…

---

## [Finding a Swiftype tuning expert / freelancer](https://community.swiftype.com/t/finding-a-swiftype-tuning-expert-freelancer/521)

<div class="topic-metadata">

**Author:** [@nate](https://community.swiftype.com/u/nate)\
**Replies:** 0\
**Last updated:** [December 5, 2017, 11:03pm UTC](https://community.swiftype.com/t/finding-a-swiftype-tuning-expert-freelancer/521 "2017-12-05T23:03:19Z")

</div>

Dear Swiftype experts and Swiftype aficionados, Where is the best place to find a swiftype tuning / configuration expert for hire who would feel comfortable adjusting the weighting and relevancy of search results based …

---

## [Multiple search fields (different engines) on the same page without jQuery](https://community.swiftype.com/t/multiple-search-fields-different-engines-on-the-same-page-without-jquery/494)

<div class="topic-metadata">

**Author:** [@notify](https://community.swiftype.com/u/notify)\
**Replies:** 0\
**Last updated:** [November 15, 2017, 6:19pm UTC](https://community.swiftype.com/t/multiple-search-fields-different-engines-on-the-same-page-without-jquery/494 "2017-11-15T18:19:17Z")

</div>

The typical installation from the admin panel is done by adding an engine via: (function (w, d, t, u, n, s, e) { w\['SwiftypeObject'\] = n; w\[n\] = w\[n\] || function () { …

---

## [Prevent results from linking to "sections"](https://community.swiftype.com/t/prevent-results-from-linking-to-sections/414)

<div class="topic-metadata">

**Author:** [@jcash](https://community.swiftype.com/u/jcash)\
**Replies:** 2\
**Last updated:** [August 11, 2017, 12:26am UTC](https://community.swiftype.com/t/prevent-results-from-linking-to-sections/414 "2017-08-11T00:26:46Z")

</div>

We have a website that is set up in Adobe Muse. Unfortunately, we’re experiencing a problem with the search results where they are automatically linking to certain sections of a page instead of simply opening the page it…

---

## [Can Swiftype index password protected sites?](https://community.swiftype.com/t/can-swiftype-index-password-protected-sites/58)

<div class="topic-metadata">

**Author:** [@system](https://community.swiftype.com/u/system)\
**Replies:** 0\
**Last updated:** [December 21, 2015, 10:01pm UTC](https://community.swiftype.com/t/can-swiftype-index-password-protected-sites/58 "2015-12-21T22:01:01Z")

</div>

We have a staging site that is protected by HTTP Basic Authentication. Is there any way to get Swiftype to index a password protected site? Unfortunately, Swiftype’s crawler does not automatically support indexing con…

---

## [Can I use the crawler to index my YouTube videos?](https://community.swiftype.com/t/can-i-use-the-crawler-to-index-my-youtube-videos/331)

<div class="topic-metadata">

**Author:** [@system](https://community.swiftype.com/u/system)\
**Replies:** 0\
**Last updated:** [March 16, 2017, 6:49pm UTC](https://community.swiftype.com/t/can-i-use-the-crawler-to-index-my-youtube-videos/331 "2017-03-16T18:49:47Z")

</div>

Yes! The best method for indexing your YouTube videos (or other streaming video content) is to have each video embedded within their own pages on your site. Swiftype will not crawl and index the actual video file embedd…

[Next page](https://community.swiftype.com/c/faq/indexing/5.md?page=1)