Crawler running for 3 days



Is it possible it is in a loop or following too many redirects?

Our website is developed in Domino / Lotus Notes. It replicates every 30 minutes. Would that process make it restart or confuse it?


Hey there,

I can’t yet speak to the replication being a factor, but it does appear that the crawl for your engine is spidering through a lot of URL variants including various URL parameters.

A couple of options to help optimize the crawl process in these cases is to limit the crawler to fetching only the URLs listed within a current sitemap, or to blacklist any URL parameters that are specific to calling/rendering content within Domino/Lotus but don’t need to be followed by the search agent.

If you require additional assistance, please reach out to us via the Support form in your Swiftype dashboard.