Hi,
in general we do not write crawler modules that crawl e.g.:
- Full websites
- Search URLs
- Other similar functionality which basically results in a crawler doing endless requests
However - you have the following options:
1. If the website is simple, you might be able to get it to work using multiple
LinkCrawler Rules.
2.
We're open source - you can write your own plugins or modify existing ones.
-psp-