#1
|
|||
|
|||
LinkGrabber: save and resume crawl state
Some links take a day or more to crawl (for example huge YouTube channels with over ten thousand videos).
Sometimes the computer crashes, or has to be rebooted, or the internet disconnects, or I have to switch to a different computer with JDownloader Portable, etc., and then the crawling has to start all over again. The crawling attempt can waste many weeks. Is it possible to save the state of a running Linkgrabber and resume later? |
#2
|
||||
|
||||
At the moment that's not possible and lot of work/changes would be required to make this possible
because the whole linkcrawling process is heavily multithreaded and using multiple different queues. The YT plugin already supports *resume* by bypassing processing links that are already in linkgrabber. That means you just have to reuse the linkgrabber list and JD won't have to crawl through all already processed links again. Enable Settings-Advanced Settings-Youtube.crawldupecheckenabled
__________________
JD-Dev & Server-Admin Last edited by Jiaz; 03.10.2019 at 11:32. |
Thread Tools | |
Display Modes | |
|
|