Swiftype Community Forums
Site crawler HEAD
Frequently Asked Questions
Setup & Indexing
crawler
peranziani-team
April 30, 2018, 2:40pm
1
Hi,
can we know if your crawler makes an HTTP HEAD hit before indexing a page?
Thank you.
Related topics
Topic
Replies
Views
Activity
I need to know the crawler ip range
Helpline
crawler
0
7520
November 21, 2018
Further control over crawler, sections field
crawler
2
4124
July 6, 2018
App Search Document Indexing API rate limit
API
2
4165
June 7, 2018
Can Swiftype's crawler index dynamically generated (AJAX) page content?
Setup & Indexing
ajax
,
api
0
8399
March 17, 2016
Does Swiftype index anchor tag URLs when crawling a page?
Setup & Indexing
crawler
0
5396
August 22, 2016