Quote:
Originally Posted by StefanM
Adding (random) delays between requests and/or downloads
|
A. Only possible with downloads or while feeding origin links.
B. Eventscripter does not have access to crawling process.
C. Can use sleep. (Will post example script when @Jiaz adds the new method)
One more option which can help with rate limiting is reduce the linkchecker/linkcrawler threads in advanced settings. Default is 4 and 12. (I use 4 for both)
The best approach is to control it at the plugin level, as suggested by @Jiaz/@psp.
Quote:
Originally Posted by Jiaz
B.) @mgpai: you could use packagaizer hook here to add sleep/wait in linkcrawler
|
Does it help with rate limit? I thought that is triggered only after crawling/collectiing process is completed.