JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 03.10.2019, 11:50
SMS SMS is offline
Vacuum Cleaner
 
Join Date: Jun 2019
Posts: 15
Default LinkGrabber: save and resume crawl state

Some links take a day or more to crawl (for example huge YouTube channels with over ten thousand videos).

Sometimes the computer crashes, or has to be rebooted, or the internet disconnects, or I have to switch to a different computer with JDownloader Portable, etc., and then the crawling has to start all over again. The crawling attempt can waste many weeks.

Is it possible to save the state of a running Linkgrabber and resume later?
Reply With Quote
  #2  
Old 03.10.2019, 12:27
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 66,134
Default

At the moment that's not possible and lot of work/changes would be required to make this possible
because the whole linkcrawling process is heavily multithreaded and using multiple different queues.

The YT plugin already supports *resume* by bypassing processing links that are already in linkgrabber. That means you just have to reuse the linkgrabber list and JD won't have to crawl through all already processed links again.
Enable Settings-Advanced Settings-Youtube.crawldupecheckenabled
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 03.10.2019 at 12:32.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 15:12.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.