Thread: [User feedback required] LinkGrabber: save and resume crawl state
View Single Post
  #2  
Old 03.10.2019, 12:27
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,349
Default

At the moment that's not possible and lot of work/changes would be required to make this possible
because the whole linkcrawling process is heavily multithreaded and using multiple different queues.

The YT plugin already supports *resume* by bypassing processing links that are already in linkgrabber. That means you just have to reuse the linkgrabber list and JD won't have to crawl through all already processed links again.
Enable Settings-Advanced Settings-Youtube.crawldupecheckenabled
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 03.10.2019 at 12:32.
Reply With Quote