Swiftype Community Forums
Site crawler HEAD
Frequently Asked Questions
Setup & Indexing
crawler
peranziani-team
April 30, 2018, 2:40pm
1
Hi,
can we know if your crawler makes an HTTP HEAD hit before indexing a page?
Thank you.
Related topics
Topic
Replies
Views
Activity
I need to know the crawler ip range
Helpline
crawler
0
7490
November 21, 2018
Further control over crawler, sections field
crawler
2
4105
July 6, 2018
App Search Document Indexing API rate limit
API
2
4149
June 7, 2018
Can Swiftype's crawler index dynamically generated (AJAX) page content?
Setup & Indexing
ajax
,
api
0
8383
March 17, 2016
Does Swiftype index anchor tag URLs when crawling a page?
Setup & Indexing
crawler
0
5383
August 22, 2016