@mgpai
Do you see any chance to create a script (delays at least for LinkGrabber) now that Jiaz mentioned
use of packagaizer hook here to add sleep/wait in linkcrawler.?
Quote:
Originally Posted by mgpai
A. Only possible with downloads or while feeding origin links.
B. Eventscripter does not have access to crawling process.
C. Can use sleep. (Will post example script when @Jiaz adds the new method)
One more option which can help with rate limiting is reduce the linkchecker/linkcrawler threads in advanced settings. Default is 4 and 12. (I use 4 for both)
Quote:
Originally Posted by Jiaz View Post
B.) @mgpai: you could use packagaizer hook here to add sleep/wait in linkcrawler
|
The best approach is to control it at the plugin level, as suggested by @Jiaz/@psp.
Does it help with rate limit? I thought that is triggered only after crawling/collectiing process is completed. |