# Crawler running for 3 days
**URL:** https://community.swiftype.com/t/crawler-running-for-3-days/560
**Category:** General
**Tags:** crawler
**Created:** [February 5, 2018, 3:16pm UTC](https://community.swiftype.com/t/crawler-running-for-3-days/560 "2018-02-05T15:16:47Z")
**Posts on this page:** 2
**Page:** 1
### Author:  [@sysadmin](https://community.swiftype.com/u/sysadmin)
#### Post date: [February 5, 2018, 3:16pm UTC](https://community.swiftype.com/t/crawler-running-for-3-days/560/1 "2018-02-05T15:16:47Z")
Is it possible it is in a loop or following too many redirects?
Our website is developed in Domino / Lotus Notes. It replicates every 30 minutes. Would that process make it restart or confuse it?
---
### Author:  [@mike](https://community.swiftype.com/u/mike)
#### Post date: [February 6, 2018, 12:52am UTC](https://community.swiftype.com/t/crawler-running-for-3-days/560/2 "2018-02-06T00:52:04Z")
Hey there,
I can’t yet speak to the replication being a factor, but it does appear that the crawl for your engine is spidering through a lot of URL variants including various URL parameters.
A couple of options to help optimize the crawl process in these cases is to limit the crawler to fetching only the URLs listed within a current sitemap, or to [blacklist](https://swiftype.com/documentation/path_rules) any URL parameters that are specific to calling/rendering content within Domino/Lotus but don’t need to be followed by the search agent.
If you require additional assistance, please reach out to us via the Support form in your Swiftype dashboard.