JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 08.12.2021, 20:03
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default swap file? high RAM usage

how to solve high RAM usage?
I use -Xmx24576m with 32GB RAM

how to configure swap file usage to decrease high RAM usage?



EDIT
does anyone need more details?

Last edited by verheiratet1952; 09.12.2021 at 16:26.
Reply With Quote
  #2  
Old 15.12.2021, 17:55
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by verheiratet1952 View Post
how to solve high RAM usage?
I use -Xmx24576m with 32GB RAM

how to configure swap file usage to decrease high RAM usage?



EDIT
does anyone need more details?


sorry?!
Reply With Quote
  #3  
Old 15.12.2021, 17:57
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

@verheiratet1952: you allow java to use max 24GB of heap memory, why?
of course windows will then start to swap out memory of other stuff.
how many links do you have in list?
you can also tweak java to free(give back to OS) unused memory, please use google/search for the correct JVM parameters. but in first place you should not provide that amount of memory without really have to.
I'm ready to further analyse the memory usage and try to optimize/reduce it
__________________
JD-Dev & Server-Admin
Reply With Quote
  #4  
Old 17.12.2021, 16:56
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by Jiaz View Post
@verheiratet1952: you allow java to use max 24GB of heap memory, why?
of course windows will then start to swap out memory of other stuff.
how many links do you have in list?
you can also tweak java to free(give back to OS) unused memory, please use google/search for the correct JVM parameters. but in first place you should not provide that amount of memory without really have to.
I'm ready to further analyse the memory usage and try to optimize/reduce it
by the way my desktop PC has 32GB... my notebook has 16GB...

we already had the same conversation about RAM
please see https://board.jdownloader.org/showthread.php?t=86151

you told me to assign RAM manually "You can also use -Xmx24g instead of -Xmx24576m"

I also made some static pagefile.sys on a M.2 using NVMe PCIe 3.0 for best speed performance...


to answer your questions:
-how many links do you have in list?
the biggest one is about 4mio image links with nearly 90k folders where I do add/update new images/folders daily/weekly... I need to have all those links to prevent duplicates...


I do always save CFG folder with its downloadlist and linkcollector ZIPs...

just as an example:

if I start JD with those 4mio links and its 90k folders it has at about 10GB usage at the beginning, but it gets higher and higher within few hours while crawling new content... or when just downloading files and dont run linkcrawler...

Last edited by verheiratet1952; 17.12.2021 at 17:22.
Reply With Quote
  #5  
Old 17.12.2021, 18:45
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

@verheiratet1952: sorry, yes, I remember
would it be possible to send me a copy of such a large list, so I can do testing/optimization on a *real world* list. eg upload to mega and send me link to support@jdownloader.org
It would really help me a lot getting hands on a real large list and not create dummy ones myself as they do not reflect *real usage*
__________________
JD-Dev & Server-Admin
Reply With Quote
  #6  
Old 17.12.2021, 20:54
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by Jiaz View Post
@verheiratet1952: sorry, yes, I remember
would it be possible to send me a copy of such a large list, so I can do testing/optimization on a *real world* list. eg upload to mega and send me link to support@jdownloader.org
It would really help me a lot getting hands on a real large list and not create dummy ones myself as they do not reflect *real usage*


you should have 2 emails from transfer website for large files...

subject is thread 494979
Reply With Quote
  #7  
Old 17.12.2021, 21:14
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by verheiratet1952 View Post
you should have 2 emails from transfer website for large files...

subject is thread 494979

JD info
Attached Images
File Type: png jd2-info20211217.PNG (32.8 KB, 1 views)
Reply With Quote
  #8  
Old 21.12.2021, 12:03
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

@verheiratet1952: thanks, I will try to find simple/easy to implement optimizations to reduce memory usage for such large lists.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #9  
Old 21.12.2021, 15:25
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by Jiaz View Post
@verheiratet1952: thanks, I will try to find simple/easy to implement optimizations to reduce memory usage for such large lists.
you could already re-produce high RAM usage?
it takes some time to reach heap level after starting JD, as said before....
Reply With Quote
  #10  
Old 21.12.2021, 15:37
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

@verheiratet1952: did not work on it yet and first goal is to reduce overal memory consumption for the links which helps to reduce memory pressure and usage
__________________
JD-Dev & Server-Admin
Reply With Quote
  #11  
Old 21.12.2021, 16:29
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by Jiaz View Post
@verheiratet1952: did not work on it yet and first goal is to reduce overal memory consumption for the links which helps to reduce memory pressure and usage

but you downloaded downloadlist and linkcollector zips?
the website does NOT store that long...
Reply With Quote
  #12  
Old 21.12.2021, 16:37
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

yes, I have them both. Thanks again for your help!
__________________
JD-Dev & Server-Admin
Reply With Quote
  #13  
Old 22.12.2021, 18:02
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

@verheiratet1952: thanks again for your help. with this list I was able to massiv reduce memory consumption in runtime. still doing tests and trying to reduce memory usage even more. I'm sure you'll notice the reduction in memory usage instantly!
__________________
JD-Dev & Server-Admin
Reply With Quote
  #14  
Old 22.12.2021, 20:01
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by Jiaz View Post
@verheiratet1952: thanks again for your help. with this list I was able to massiv reduce memory consumption in runtime. still doing tests and trying to reduce memory usage even more. I'm sure you'll notice the reduction in memory usage instantly!
already working or do I have to wait for core update?

I use multiOS Version where I can assign JAVA...
better to have 2 oder 3 JAVA installs that I can assign those if running more than only one JD at the same time?

Last edited by verheiratet1952; 22.12.2021 at 20:07.
Reply With Quote
  #15  
Old 22.12.2021, 20:03
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

@verheiratet1952: I will finally release the update by tomorrow and it will contain the optimizations as well.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #16  
Old 22.12.2021, 20:09
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by Jiaz View Post
@verheiratet1952: I will finally release the update by tomorrow and it will contain the optimizations as well.
I use multiOS Version where I can assign JAVA path using shortcut plus -xmx24g ...
better to have 2 oder 3 JAVA installs that I can assign those if running more than only one JD at the same time?
Reply With Quote
  #17  
Old 22.12.2021, 20:10
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

@verheiratet1952: every JDownloader is the same. Only different way to start (JDownloader.jar, .exe Launcher...) Doesn't really matter which one you use.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #18  
Old 22.12.2021, 20:14
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by Jiaz View Post
@verheiratet1952: every JDownloader is the same. Only different way to start (JDownloader.jar, .exe Launcher...) Doesn't really matter which one you use.
because of shortcut to assign max RAM and what JAVA if there is more than just one installed...
Reply With Quote
  #19  
Old 23.12.2021, 11:11
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

@verheiratet1952:
JDownloader.jar:
-double click will start with default JVM configuration and system installed Java
-you can create shortcut/batch/script to start with different JVM and configuration, eg
Quote:
C:\.....\bin\java.exe -Xmx.... -jar c:\.....\JDownloader.jar
You can use the same JVM for multiple JDownloader installations/instances with different configuration
or if you like also different JVM versions

JDownloader.exe:
-depends on the *age* of the launcher(exe) but in case there are .vmoptions files, you can place custom
configuration in there, see https://support.jdownloader.org/Know...vmoptions-file
-the launcher(exe) will use the JVM from java or jre subfolder relative to its location.
You can do multiple installations of JDownloader and then each launcher will use its own .vmoptions and JVM
__________________
JD-Dev & Server-Admin
Reply With Quote
  #20  
Old 23.12.2021, 18:03
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

Update is live!
__________________
JD-Dev & Server-Admin
Reply With Quote
  #21  
Old 23.12.2021, 18:59
mdmdfss mdmdfss is offline
Modem User
 
Join Date: Nov 2019
Posts: 1
Default

And fails to update:

org.jdownloader.update.SelfUpdateException: Selftest has been terminated unexpectedly
at org.jdownloader.update.SelfUpdateClient.finishSelfUpdate(SelfUpdateClient.java:353)
at org.jdownloader.update.UpdateManager.runFinishSelfUpdate(UpdateManager.java:951)
at org.jdownloader.update.UpdateManager$12.run(UpdateManager.java:866)
at org.jdownloader.update.UpdateManager.installPendingUpdates(UpdateManager.java:886)
at org.jdownloader.updatev2.UpdateController.installUpdates(UpdateController.java:536)
at org.jdownloader.updatev2.UpdateController.onResults(UpdateController.java:408)
at org.jdownloader.update.UpdateManager$21.onResults(UpdateManager.java:1507)
at org.jdownloader.update.UpdateManager.runUpdateLoop(UpdateManager.java:1036)
at org.jdownloader.update.PendingUpdate.run(PendingUpdate.java:19)
at org.jdownloader.update.UpdateManager$23.run(UpdateManager.java:1686)
at java.lang.Thread.run(Thread.java:748)

Running jdownloader on an M1 MAC with:

openjdk 17.0.1 2021-10-19
OpenJDK Runtime Environment Temurin-17.0.1+12 (build 17.0.1+12)
OpenJDK 64-Bit Server VM Temurin-17.0.1+12 (build 17.0.1+12, mixed mode)
Reply With Quote
  #22  
Old 23.12.2021, 19:07
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

@mdmdfss: please don't hijack threads
https://board.jdownloader.org/showthread.php?t=89344
__________________
JD-Dev & Server-Admin
Reply With Quote
  #23  
Old 24.12.2021, 15:22
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by Jiaz View Post
Update is live!
it really works great, much better than before...
thx a lot!

85k folders with 1.6mio images
Usage: 4,75 GB - Allocated: 12,80 GB - Max: 21,33 GB

90k folder with 4.2mio images
(you have CFG folder zips to re-produce a 'real world')
Usage: 6,40 GB - Allocated: 12,95 GB - Max: 21,33 GB



will give more info when having tried on all my different projects...

merry x-mas
Reply With Quote
  #24  
Old 27.12.2021, 11:08
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

@verheiratet1952: Thanks for the feedback and you're welcome! Yes, I was able to heavily reduce memory consumption for your list. could you please provide example list of the 85k folders and 1.6mio images as well?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #25  
Old 28.12.2021, 17:00
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by verheiratet1952 View Post
it really works great, much better than before...
thx a lot!

85k folders with 1.6mio images
Usage: 4,75 GB - Allocated: 12,80 GB - Max: 21,33 GB

90k folder with 4.2mio images
(you have CFG folder zips to re-produce a 'real world')
Usage: 6,40 GB - Allocated: 12,95 GB - Max: 21,33 GB



will give more info when having tried on all my different projects...

merry x-mas


performance update from a 'real world'

(new desktop PC with 128gb RAM)
downloadlist with
70k folders with 3.2 mio images

added new 25k folders with 1.1mio images at
linkcollector running nearly 24hours...
Usage: 44,62 GB - Allocated: 46,31 GB - Max: 85,33 GB




crawling one single sub-reddit for 24hours
Usage: 1,15 GB - Allocated: 1,80 GB - Max: 26,67 GB

had 18k folder with 29k images, crawling new 75k folder with 118k images...

now after another 18 hours it has 122k folders with 188k images
Usage: 990,31 MB - Allocated: 3,18 GB - Max: 26,67 GB
Reply With Quote
  #26  
Old 28.12.2021, 17:27
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

@verheiratet1952: if possible you should stay with max heap below 32GB (eg -Xmx30g). Above 32GB java has to switch to different reference/pointer size and thus will require more memory.

Quote:
Originally Posted by verheiratet1952 View Post
Usage: 44,62 GB - Allocated: 46,31 GB - Max: 85,33 GB
That seems too much usage for the use case.
Would it be possible to do Teamviewer Session together, then I could do further checks on memory consumption and try to find possible optimizations. Right now I worked on optimization on loading lists into memory and reducing memory footprint. But very likely the processing within a running session can be optimized even further but that would require getting *hands on* such a JDownloader instance. Teamviewer and then using jvisual vm to create/analyze heap dumps.
Contact me via support@jdownloader.org
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 28.12.2021 at 17:31.
Reply With Quote
  #27  
Old 28.12.2021, 18:59
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by Jiaz View Post
@verheiratet1952: if possible you should stay with max heap below 32GB (eg -Xmx30g). Above 32GB java has to switch to different reference/pointer size and thus will require more memory.


That seems too much usage for the use case.
Would it be possible to do Teamviewer Session together, then I could do further checks on memory consumption and try to find possible optimizations. Right now I worked on optimization on loading lists into memory and reducing memory footprint. But very likely the processing within a running session can be optimized even further but that would require getting *hands on* such a JDownloader instance. Teamviewer and then using jvisual vm to create/analyze heap dumps.
Contact me via support@jdownloader.org

Usage: 44,62 GB - Allocated: 46,31 GB - Max: 85,33 GB

but if usage and allocated is more than 32gb I think JD needs more than just 32gb of RAM?

you think/know that is should work perfectly with max of 32gb?



by the way:
we can do same Teamviewer, sure I will let you know via email...
Reply With Quote
  #28  
Old 28.12.2021, 19:44
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

sometimes it looks like saving downloadlist and linkcollector zips takes very very long :(

I was also trying on max speed drive m2 Samsung PCIe 4.0 x4, but same same...
Reply With Quote
  #29  
Old 29.12.2021, 10:37
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

@verheiratet1952 : No optimization yet on load/save of lists. And the limiting factor is NOT your hardware/system but the amount of links/data and the current way of storage. I will work on this topic once I have finished the optimization on memory consumption.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #30  
Old 29.12.2021, 10:41
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

Quote:
Originally Posted by verheiratet1952 View Post
but if usage and allocated is more than 32gb I think JD needs more than just 32gb of RAM?

you think/know that is should work perfectly with max of 32gb?
the goal of the optimizations is to reduce memory usage so just throwing more memory at java doesn't make it *better*. once you use larger heaps than ca. 32GB, the internal size of references/pointers are larger and thus the overal memory usage will be larger with same amount of links in list. therefore my recommendation to prefer max heap at max 32gb.

Quote:
Originally Posted by verheiratet1952 View Post
Usage: 44,62 GB - Allocated: 46,31 GB - Max: 85,33 GB
but if usage and allocated is more than 32gb I think JD needs more than just 32gb of RAM?
Java will not use more heap than allowed/configured. And it's strange that memory usage has grown that large despite just adding *a few * more links in list, so my question about teamviewer
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 29.12.2021 at 10:44.
Reply With Quote
  #31  
Old 30.12.2021, 03:29
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by Jiaz View Post
@verheiratet1952 : No optimization yet on load/save of lists. And the limiting factor is NOT your hardware/system but the amount of links/data and the current way of storage. I will work on this topic once I have finished the optimization on memory consumption.

downloadlist and linkcollector zips have at about 1-2.5gb

I think it is about my hdd/ssd/m2...
JD is doing Updates on downloadlist and linkcollector zips every few minutes if anything is being added...

writing new downloadlist and linkcollector zips take much longer if HDD not SSD, or if SSD not m.2 PCIe 4.0 x4
Reply With Quote
  #32  
Old 30.12.2021, 11:19
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

@verheiratet1952: you can increase delays to write changes to disk, see
https://board.jdownloader.org/showpo...7&postcount=15
I will optimize the storage of lists too, but first I want to further optimize memory usage
__________________
JD-Dev & Server-Admin
Reply With Quote
  #33  
Old 30.12.2021, 17:24
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

@verheiratet1952: meanwhile would be great if you can try latest Java11, see
adoptium.net/?variant=openjdk11
It contains String storage optimizations which should also work great in your case!
__________________
JD-Dev & Server-Admin
Reply With Quote
  #34  
Old 31.12.2021, 19:38
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by Jiaz View Post
@verheiratet1952: meanwhile would be great if you can try latest Java11, see
adoptium.net/?variant=openjdk11
It contains String storage optimizations which should also work great in your case!
what changes should I see when using openjdk11?
Reply With Quote
  #35  
Old 04.01.2022, 02:00
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by verheiratet1952 View Post
what changes should I see when using openjdk11?
by the way:

wattage/electric power usage is shown as extremely high in task manager...
Reply With Quote
  #36  
Old 04.01.2022, 12:10
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

Quote:
Originally Posted by verheiratet1952 View Post
what changes should I see when using openjdk11?
less memory usage with same workload in comparission with java1.8
Quote:
Originally Posted by verheiratet1952 View Post
by the way:
wattage/electric power usage is shown as extremely high in task manager...
depends on when? in idle or while working? of course this amount of data requires cpu/disk/energy to load/save.
as explained, that part is not yet optimized at all
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 15:47.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.