Swiftype Community Forums
Site crawler HEAD
Frequently Asked Questions
Setup & Indexing
crawler
peranziani-team
April 30, 2018, 2:40pm
1
Hi,
can we know if your crawler makes an HTTP HEAD hit before indexing a page?
Thank you.
Related Topics
Topic
Replies
Views
Activity
Crawler stops under 30k
Setup & Indexing
0
2731
May 7, 2018
I need to know the crawler ip range
Helpline
crawler
0
6862
November 21, 2018
Can the crawler "go through" blacklisted pages?
Setup & Indexing
crawler
2
3601
February 3, 2018
App Search Document Indexing API rate limit
API
2
3834
June 7, 2018
Can Swiftype index an intranet site?
Setup & Indexing
0
5574
December 21, 2015