JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #41  
Old 21.03.2017, 18:51
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 63,945
Default

It makes absolute NO sense to export/import

You have about 2,8 mio links!?! Then I suggest to provide more memory
-Xmx12g
__________________
JD-Dev & Server-Admin
Reply With Quote
  #42  
Old 22.03.2017, 01:05
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 16,143
Default

it comes to a point when you can't just keep adding more memory
you need to remove/cleanup links that have been downloaded to free up memory.
each link requires specific amount of memory.

The other solution is to only add links when you have (near on) finished the previous group.

my 2 cents

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #43  
Old 22.03.2017, 02:45
verheiratet1952 verheiratet1952 is offline
Bandwidth Beast
 
Join Date: Jan 2016
Posts: 148
Default

Quote:
Originally Posted by raztoki View Post
it comes to a point when you can't just keep adding more memory
you need to remove/cleanup links that have been downloaded to free up memory.
each link requires specific amount of memory.

The other solution is to only add links when you have (near on) finished the previous group.

my 2 cents

raztoki

well, that makes me believe I can create DLC from added links, remove them and add them again with DLC...

by the way: -Xmx12g seems to work with almost 2.9 million links
Reply With Quote
  #44  
Old 22.03.2017, 02:57
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 16,143
Default

dlc is just another container like csv with more downside than upside, its crypted and would require more resources to add links.

As stated earlier I would remove old links when downloaded, I would add new batch of csv when required. This is the best and really the only solution when talking about sheer volume of links you are dealing with.

Also I would recommend that you disable linkchecking in linkgrabber, to speed up adding of links, settings > advanced > LinkCollector.dolinkcheck
This will reduce a lot of overheads with the required browser requests just to check filesize (if your links are directlink you should already know the filename) and online availability. linkchecking will then take place within download routine.

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]

Last edited by raztoki; 22.03.2017 at 03:11. Reason: downloadside/downside
Reply With Quote
  #45  
Old 22.03.2017, 09:11
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 63,945
Default

You can copy/backup the linkcollector or downloadList zips from cfg folder. They contain the complete list. For example copy the file, then empty the list and later you can import the file as normal container (similiar to dlc). But it keeps all settings and is MUCH MUCH faster as JDownloader doesn't has to process all links again
__________________
JD-Dev & Server-Admin
Reply With Quote
  #46  
Old 22.03.2017, 14:32
verheiratet1952 verheiratet1952 is offline
Bandwidth Beast
 
Join Date: Jan 2016
Posts: 148
Default

Quote:
Originally Posted by raztoki View Post
dlc is just another container like csv with more downside than upside, its crypted and would require more resources to add links.

As stated earlier I would remove old links when downloaded, I would add new batch of csv when required. This is the best and really the only solution when talking about sheer volume of links you are dealing with.

Also I would recommend that you disable linkchecking in linkgrabber, to speed up adding of links, settings > advanced > LinkCollector.dolinkcheck
This will reduce a lot of overheads with the required browser requests just to check filesize (if your links are directlink you should already know the filename) and online availability. linkchecking will then take place within download routine.

raztoki

disable linkchecking is a good idea, definitely when doing those 1 million and more links jobs...


-Xmx12g or -Xmx8g I think was best for some massive performance push...

but it's really wicked to see 7,33gb 11,86gb 11,86gb
(screenshot)

what does that mean exactly? why three different specs?

I am just trying to get the most out of it, so I can handle most issues...
Attached Thumbnails
jd-about-12gb-2nd.JPG  
Reply With Quote
  #47  
Old 22.03.2017, 14:40
verheiratet1952 verheiratet1952 is offline
Bandwidth Beast
 
Join Date: Jan 2016
Posts: 148
Default

Quote:
Originally Posted by Jiaz View Post
You can copy/backup the linkcollector or downloadList zips from cfg folder. They contain the complete list. For example copy the file, then empty the list and later you can import the file as normal container (similiar to dlc). But it keeps all settings and is MUCH MUCH faster as JDownloader doesn't has to process all links again
can you show me more precisely please?
cfg folder?
copy/backup the linkcollector or downloadList zips?

(I was looking for another backup over the years to export/import links than DLC, but I was sure there is no other...)
Reply With Quote
  #48  
Old 22.03.2017, 14:53
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 63,945
Default

Quote:
Originally Posted by verheiratet1952 View Post
but it's really wicked to see 7,33gb 11,86gb 11,86gb
(screenshot)
what does that mean exactly? why three different specs?
(a)Current heap memory in use - (b)Current allocated heap memory - (c)Max possible heap memory

b should be about the same as taskmanager shows

in case b is near c and a near b, that means that heap memory is nearly full and java will spend more time/cpu on memory management
__________________
JD-Dev & Server-Admin
Reply With Quote
  #49  
Old 22.03.2017, 14:55
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 63,945
Default

Quote:
Originally Posted by verheiratet1952 View Post
can you show me more precisely please?
cfg folder?
copy/backup the linkcollector or downloadList zips?

(I was looking for another backup over the years to export/import links than DLC, but I was sure there is no other...)
linkcollector zips contain the current complete linkgrabber list
downloadlist zips contain the current complete download list

you can easy copy the files somewhere else and later import them back via drag/drop or load container.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #50  
Old 22.03.2017, 15:08
verheiratet1952 verheiratet1952 is offline
Bandwidth Beast
 
Join Date: Jan 2016
Posts: 148
Default

Quote:
Originally Posted by Jiaz View Post
(a)Current heap memory in use - (b)Current allocated heap memory - (c)Max possible heap memory

b should be about the same as taskmanager shows

in case b is near c and a near b, that means that heap memory is nearly full and java will spend more time/cpu on memory management

upgrading to 32gb could be an option
Reply With Quote
  #51  
Old 22.03.2017, 15:19
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 63,945
Default

More memory is always an option but no solution
__________________
JD-Dev & Server-Admin
Reply With Quote
  #52  
Old 22.03.2017, 15:39
verheiratet1952 verheiratet1952 is offline
Bandwidth Beast
 
Join Date: Jan 2016
Posts: 148
Default

Quote:
Originally Posted by Jiaz View Post
linkcollector zips contain the current complete linkgrabber list
downloadlist zips contain the current complete download list

you can easy copy the files somewhere else and later import them back via drag/drop or load container.
now I got it
...\JDownloader v2.0\cfg

better to close JD first and then copy those ZIPs?

there is more than just one file each? how do I know what files exactly?

there are also .backup files


changing cfg folder to another fast drive does not bring better performance?
Reply With Quote
  #53  
Old 22.03.2017, 15:52
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 63,945
Default

You can copy the zip without closing JDownloader. JDownloader uses CopyOnWrite.
The highest number is the current one.
.backup files are auto created in case something went wrong when trying to load the zip
putting jd on faster drive/ssd has no real influence on performance
__________________
JD-Dev & Server-Admin
Reply With Quote
  #54  
Old 23.03.2017, 12:02
verheiratet1952 verheiratet1952 is offline
Bandwidth Beast
 
Join Date: Jan 2016
Posts: 148
Default

Quote:
Originally Posted by Jiaz View Post
You can copy the zip without closing JDownloader. JDownloader uses CopyOnWrite.
The highest number is the current one.
.backup files are auto created in case something went wrong when trying to load the zip
putting jd on faster drive/ssd has no real influence on performance

if I collect links with my older desktop computer having 8gb RAM, I can use -Xmx7g ?
Reply With Quote
  #55  
Old 23.03.2017, 12:10
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 63,945
Default

Try it and you will see if it works
__________________
JD-Dev & Server-Admin
Reply With Quote
  #56  
Old 24.03.2017, 19:58
verheiratet1952 verheiratet1952 is offline
Bandwidth Beast
 
Join Date: Jan 2016
Posts: 148
Default

Quote:
Originally Posted by Jiaz View Post
Try it and you will see if it works
it is telling me "java heap size might be too large"
java dev kit 64bit must be installed?

I am using 64-bit Java for Windows recommended Version 8 Update 121.

(I was trying with -Xmx7g, -Xmx6g, -Xmx4g)
Reply With Quote
  #57  
Old 24.03.2017, 20:15
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 63,945
Default

Of course your system must have enough free memory available!
Check about dialog for used java version. JDownloader uses its bundled java version by default.
32Bit JVM only supports up to 2G heap.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #58  
Old 25.03.2017, 13:56
verheiratet1952 verheiratet1952 is offline
Bandwidth Beast
 
Join Date: Jan 2016
Posts: 148
Default

Quote:
Originally Posted by Jiaz View Post
It makes absolute NO sense to export/import

You have about 2,8 mio links!?! Then I suggest to provide more memory
-Xmx12g
now it is working fine... with linkcrawler...

something strange is about my csv file that has at about 45'000 lines...
(I cleaned with Excel, removing duplicates, ...)

but JD only detects 37'000 (+3'000 offline), what csv file coding is best?



another issue: 90% from JD shown as offline are not offline !?




by the way: I was re-doing linkcrawler... but there was no fill-up...
Attached Thumbnails
jd-csv-scan.JPG  
Reply With Quote
  #59  
Old 26.03.2017, 00:44
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 16,143
Default

Quote:
Originally Posted by raztoki View Post
Also I would recommend that you disable linkchecking in linkgrabber, to speed up adding of links, settings > advanced > LinkCollector.dolinkcheck
This will reduce a lot of overheads with the required browser requests just to check filesize (if your links are directlink you should already know the filename) and online availability. linkchecking will then take place within download routine.

raztoki
seems you haven't done this?
you could be getting blocked accessing too often/fast. linkchecking basically does a full download just to test that its downloadable and get the filesize from response header.

If you are also downloading at the time, this could be making it even worse, or lagging out your internet connection which might create socket issues.

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #60  
Old 27.03.2017, 16:20
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 63,945
Default

Quote:
Originally Posted by verheiratet1952 View Post
but JD only detects 37'000 (+3'000 offline), what csv file coding is best?
Hard to tell without having access to such a csv file.
JDownloader does dupe checks. So adding the same url will result in only 1 link in list.
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 04:52.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.