JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 08.12.2021, 20:03
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default swap file? high RAM usage

how to solve high RAM usage?
I use -Xmx24576m with 32GB RAM

how to configure swap file usage to decrease high RAM usage?



EDIT
does anyone need more details?

Last edited by verheiratet1952; 09.12.2021 at 16:26.
Reply With Quote
  #2  
Old 15.12.2021, 17:55
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by verheiratet1952 View Post
how to solve high RAM usage?
I use -Xmx24576m with 32GB RAM

how to configure swap file usage to decrease high RAM usage?



EDIT
does anyone need more details?


sorry?!
Reply With Quote
  #3  
Old 15.12.2021, 17:57
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,339
Default

@verheiratet1952: you allow java to use max 24GB of heap memory, why?
of course windows will then start to swap out memory of other stuff.
how many links do you have in list?
you can also tweak java to free(give back to OS) unused memory, please use google/search for the correct JVM parameters. but in first place you should not provide that amount of memory without really have to.
I'm ready to further analyse the memory usage and try to optimize/reduce it
__________________
JD-Dev & Server-Admin
Reply With Quote
  #4  
Old 17.12.2021, 16:56
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by Jiaz View Post
@verheiratet1952: you allow java to use max 24GB of heap memory, why?
of course windows will then start to swap out memory of other stuff.
how many links do you have in list?
you can also tweak java to free(give back to OS) unused memory, please use google/search for the correct JVM parameters. but in first place you should not provide that amount of memory without really have to.
I'm ready to further analyse the memory usage and try to optimize/reduce it
by the way my desktop PC has 32GB... my notebook has 16GB...

we already had the same conversation about RAM
please see https://board.jdownloader.org/showthread.php?t=86151

you told me to assign RAM manually "You can also use -Xmx24g instead of -Xmx24576m"

I also made some static pagefile.sys on a M.2 using NVMe PCIe 3.0 for best speed performance...


to answer your questions:
-how many links do you have in list?
the biggest one is about 4mio image links with nearly 90k folders where I do add/update new images/folders daily/weekly... I need to have all those links to prevent duplicates...


I do always save CFG folder with its downloadlist and linkcollector ZIPs...

just as an example:

if I start JD with those 4mio links and its 90k folders it has at about 10GB usage at the beginning, but it gets higher and higher within few hours while crawling new content... or when just downloading files and dont run linkcrawler...

Last edited by verheiratet1952; 17.12.2021 at 17:22.
Reply With Quote
  #5  
Old 17.12.2021, 18:45
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,339
Default

@verheiratet1952: sorry, yes, I remember
would it be possible to send me a copy of such a large list, so I can do testing/optimization on a *real world* list. eg upload to mega and send me link to support@jdownloader.org
It would really help me a lot getting hands on a real large list and not create dummy ones myself as they do not reflect *real usage*
__________________
JD-Dev & Server-Admin
Reply With Quote
  #6  
Old 17.12.2021, 20:54
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by Jiaz View Post
@verheiratet1952: sorry, yes, I remember
would it be possible to send me a copy of such a large list, so I can do testing/optimization on a *real world* list. eg upload to mega and send me link to support@jdownloader.org
It would really help me a lot getting hands on a real large list and not create dummy ones myself as they do not reflect *real usage*


you should have 2 emails from transfer website for large files...

subject is thread 494979
Reply With Quote
  #7  
Old 17.12.2021, 21:14
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by verheiratet1952 View Post
you should have 2 emails from transfer website for large files...

subject is thread 494979

JD info
Attached Images
File Type: png jd2-info20211217.PNG (32.8 KB, 1 views)
Reply With Quote
  #8  
Old 21.12.2021, 12:03
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,339
Default

@verheiratet1952: thanks, I will try to find simple/easy to implement optimizations to reduce memory usage for such large lists.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #9  
Old 21.12.2021, 15:25
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by Jiaz View Post
@verheiratet1952: thanks, I will try to find simple/easy to implement optimizations to reduce memory usage for such large lists.
you could already re-produce high RAM usage?
it takes some time to reach heap level after starting JD, as said before....
Reply With Quote
  #10  
Old 21.12.2021, 15:37
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,339
Default

@verheiratet1952: did not work on it yet and first goal is to reduce overal memory consumption for the links which helps to reduce memory pressure and usage
__________________
JD-Dev & Server-Admin
Reply With Quote
  #11  
Old 21.12.2021, 16:29
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by Jiaz View Post
@verheiratet1952: did not work on it yet and first goal is to reduce overal memory consumption for the links which helps to reduce memory pressure and usage

but you downloaded downloadlist and linkcollector zips?
the website does NOT store that long...
Reply With Quote
  #12  
Old 21.12.2021, 16:37
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,339
Default

yes, I have them both. Thanks again for your help!
__________________
JD-Dev & Server-Admin
Reply With Quote
  #13  
Old 22.12.2021, 18:02
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,339
Default

@verheiratet1952: thanks again for your help. with this list I was able to massiv reduce memory consumption in runtime. still doing tests and trying to reduce memory usage even more. I'm sure you'll notice the reduction in memory usage instantly!
__________________
JD-Dev & Server-Admin
Reply With Quote
  #14  
Old 22.12.2021, 20:01
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by Jiaz View Post
@verheiratet1952: thanks again for your help. with this list I was able to massiv reduce memory consumption in runtime. still doing tests and trying to reduce memory usage even more. I'm sure you'll notice the reduction in memory usage instantly!
already working or do I have to wait for core update?

I use multiOS Version where I can assign JAVA...
better to have 2 oder 3 JAVA installs that I can assign those if running more than only one JD at the same time?

Last edited by verheiratet1952; 22.12.2021 at 20:07.
Reply With Quote
  #15  
Old 22.12.2021, 20:03
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,339
Default

@verheiratet1952: I will finally release the update by tomorrow and it will contain the optimizations as well.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #16  
Old 22.12.2021, 20:09
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by Jiaz View Post
@verheiratet1952: I will finally release the update by tomorrow and it will contain the optimizations as well.
I use multiOS Version where I can assign JAVA path using shortcut plus -xmx24g ...
better to have 2 oder 3 JAVA installs that I can assign those if running more than only one JD at the same time?
Reply With Quote
  #17  
Old 22.12.2021, 20:10
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,339
Default

@verheiratet1952: every JDownloader is the same. Only different way to start (JDownloader.jar, .exe Launcher...) Doesn't really matter which one you use.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #18  
Old 22.12.2021, 20:14
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by Jiaz View Post
@verheiratet1952: every JDownloader is the same. Only different way to start (JDownloader.jar, .exe Launcher...) Doesn't really matter which one you use.
because of shortcut to assign max RAM and what JAVA if there is more than just one installed...
Reply With Quote
  #19  
Old 23.12.2021, 11:11
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,339
Default

@verheiratet1952:
JDownloader.jar:
-double click will start with default JVM configuration and system installed Java
-you can create shortcut/batch/script to start with different JVM and configuration, eg
Quote:
C:\.....\bin\java.exe -Xmx.... -jar c:\.....\JDownloader.jar
You can use the same JVM for multiple JDownloader installations/instances with different configuration
or if you like also different JVM versions

JDownloader.exe:
-depends on the *age* of the launcher(exe) but in case there are .vmoptions files, you can place custom
configuration in there, see https://support.jdownloader.org/Know...vmoptions-file
-the launcher(exe) will use the JVM from java or jre subfolder relative to its location.
You can do multiple installations of JDownloader and then each launcher will use its own .vmoptions and JVM
__________________
JD-Dev & Server-Admin
Reply With Quote
  #20  
Old 23.12.2021, 18:03
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,339
Default

Update is live!
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 10:23.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.