JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #21  
Old 20.03.2017, 19:11
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 62,452
Default

add new line with
-Xmx8g
__________________
JD-Dev & Server-Admin
Reply With Quote
  #22  
Old 20.03.2017, 19:12
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 62,452
Default

Quote:
Originally Posted by Jiaz View Post
How many links/packages do you have in both list? What kind of links are you adding? What host?
Memory usage per link highly depends on the link type!
Any feedback about those questions?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #23  
Old 21.03.2017, 09:15
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 62,452
Default

Any Feedback ?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #24  
Old 21.03.2017, 12:25
verheiratet1952 verheiratet1952 is offline
Bandwidth Beast
 
Join Date: Jan 2016
Posts: 145
Default

Quote:
Originally Posted by Jiaz View Post
It's not the size of CSV, the number of links/type if links. To me it looks like you have LOTS of links/packages in JDownloader or the links you are trying to add require much memory.

"LOTS of links" mean how many for you?

"links require much memory" - it also depends on host? it is normal websites... for that special case it is ranked round about 9000th with alexa.com
Reply With Quote
  #25  
Old 21.03.2017, 12:34
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 62,452
Default

Is it a direct http link or a link that requires plugin support in JDownloader?
LOTS of links = 200k-500k, 1 mio or more
Did the increased heap solve the issue?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #26  
Old 21.03.2017, 14:34
verheiratet1952 verheiratet1952 is offline
Bandwidth Beast
 
Join Date: Jan 2016
Posts: 145
Default

Quote:
Originally Posted by Jiaz View Post
Is it a direct http link or a link that requires plugin support in JDownloader?
LOTS of links = 200k-500k, 1 mio or more
Did the increased heap solve the issue?
sorry for delay, but I was away from keyboard...

it is direct http/https links... no plugin in between...

changing to 8gb is a great performance push... taskmanager is now showing at about 8.800.000 K...

what about adding another or 2nd TMP/TEMP folder (for JD only) ?
I could use a 128gb USB 3.0 drive...




I was a little bit surprised that moving/copying a csv file from D to C drive was linkcrawler to start crawling for all urls... better saying - it was CTRL+C that started jd with linkcrawler...
Reply With Quote
  #27  
Old 21.03.2017, 14:43
verheiratet1952 verheiratet1952 is offline
Bandwidth Beast
 
Join Date: Jan 2016
Posts: 145
Default

would you suggest to close and open JD again before adding 2nd 3rd csv file?

I was using csv splitter to have 10-12'000 links only with one single csv file...
for me it seems to have better performance after close-reopen JD...
Reply With Quote
  #28  
Old 21.03.2017, 15:28
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 62,452
Default

I guess the csv got auto added because of a bug in auto extension learning (will be fixed with next core update). JDownloader learned to autoparse .csv files in your case, that's why it also works when you copy the file to clipboard.

There should be no need to close-reopen JDownloader. 8GB heap should easily handle 1 million links. With enough free heap available there is no need for it. You can check about Dialog. In case it becomes slow again, please provide a new screenshot of the about dialog.

Of course increasing max heap will result in better performance
__________________
JD-Dev & Server-Admin
Reply With Quote
  #29  
Old 21.03.2017, 15:29
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 62,452
Default

Quote:
Originally Posted by verheiratet1952 View Post
what about adding another or 2nd TMP/TEMP folder (for JD only) ?
I could use a 128gb USB 3.0 drive...
I'm sorry but I don't understand. JDownloader does not use any tmp folder.
What do you hope to achieve from this?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #30  
Old 21.03.2017, 17:33
verheiratet1952 verheiratet1952 is offline
Bandwidth Beast
 
Join Date: Jan 2016
Posts: 145
Default

Quote:
Originally Posted by Jiaz View Post
I guess the csv got auto added because of a bug in auto extension learning (will be fixed with next core update). JDownloader learned to autoparse .csv files in your case, that's why it also works when you copy the file to clipboard.

There should be no need to close-reopen JDownloader. 8GB heap should easily handle 1 million links. With enough free heap available there is no need for it. You can check about Dialog. In case it becomes slow again, please provide a new screenshot of the about dialog.

Of course increasing max heap will result in better performance
I have 16gb ram, I can also change to 12gb instead of using 8gb

replacing -Xmx8g with -Xmx12g ?
Reply With Quote
  #31  
Old 21.03.2017, 17:42
verheiratet1952 verheiratet1952 is offline
Bandwidth Beast
 
Join Date: Jan 2016
Posts: 145
Default

Quote:
Originally Posted by Jiaz View Post
I'm sorry but I don't understand. JDownloader does not use any tmp folder.
What do you hope to achieve from this?
I am not into coding, I'm just saying...

it is only about trying to find some way to result in better performance...
Reply With Quote
  #32  
Old 21.03.2017, 17:51
verheiratet1952 verheiratet1952 is offline
Bandwidth Beast
 
Join Date: Jan 2016
Posts: 145
Default

Quote:
Originally Posted by Jiaz View Post
I guess the csv got auto added because of a bug in auto extension learning (will be fixed with next core update). JDownloader learned to autoparse .csv files in your case, that's why it also works when you copy the file to clipboard.

There should be no need to close-reopen JDownloader. 8GB heap should easily handle 1 million links. With enough free heap available there is no need for it. You can check about Dialog. In case it becomes slow again, please provide a new screenshot of the about dialog.

Of course increasing max heap will result in better performance
it is slow again... system is now using all 8gb... (8.889.362 K taskmanager)
Attached Thumbnails
jd-about-8gb.JPG  
Reply With Quote
  #33  
Old 21.03.2017, 17:55
verheiratet1952 verheiratet1952 is offline
Bandwidth Beast
 
Join Date: Jan 2016
Posts: 145
Default

Quote:
Originally Posted by verheiratet1952 View Post
it is slow again... system is now using all 8gb... (8.889.362 K taskmanager)
how to optimize CPU usage?
Attached Thumbnails
jd-cpu-usage.JPG  
Reply With Quote
  #34  
Old 21.03.2017, 17:56
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 62,452
Default

How many Links do you have in List? That is important to know! Please provide any numbers. Screenshot shows that JDownloader is taking all the memory so either you have hundrets of thousands of links in JDownloader or something you do causes memleak.
8GB heap should easily be enough for 1mio and more links
__________________
JD-Dev & Server-Admin
Reply With Quote
  #35  
Old 21.03.2017, 17:57
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 62,452
Default

No need to optimize CPU. Java is working near max memory limit and therefore spends lots of time in memory management.

How many links do you have in list? Can you provide example csv to support@jdownloader.org ?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #36  
Old 21.03.2017, 18:20
verheiratet1952 verheiratet1952 is offline
Bandwidth Beast
 
Join Date: Jan 2016
Posts: 145
Default

Quote:
Originally Posted by Jiaz View Post
How many Links do you have in List? That is important to know! Please provide any numbers. Screenshot shows that JDownloader is taking all the memory so either you have hundrets of thousands of links in JDownloader or something you do causes memleak.
8GB heap should easily be enough for 1mio and more links
you talk about number of linkcrawler section only?
or download section is also important?
Reply With Quote
  #37  
Old 21.03.2017, 18:25
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 62,452
Default

Overall in JDownloader. You can enable overview panel that will show number of links/packages. I just need number of downloadlist/linkgrabber to be able to say that is has to to with number of links or must be some kind of memleak.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #38  
Old 21.03.2017, 18:25
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 62,452
Default

PS: you can also write in german
__________________
JD-Dev & Server-Admin
Reply With Quote
  #39  
Old 21.03.2017, 18:39
verheiratet1952 verheiratet1952 is offline
Bandwidth Beast
 
Join Date: Jan 2016
Posts: 145
Default

Quote:
Originally Posted by Jiaz View Post
Overall in JDownloader. You can enable overview panel that will show number of links/packages. I just need number of downloadlist/linkgrabber to be able to say that is has to to with number of links or must be some kind of memleak.

download has at about 3'500 folders with 1,2 million image links

linkgrabber has 30'000 folders with 1,6 million image links...
Reply With Quote
  #40  
Old 21.03.2017, 18:41
verheiratet1952 verheiratet1952 is offline
Bandwidth Beast
 
Join Date: Jan 2016
Posts: 145
Default

does it makes sense to save as DLC and re-import when ready to download?
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 20:59.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.