#1
|
|||
|
|||
swap file? high RAM usage
how to solve high RAM usage?
I use -Xmx24576m with 32GB RAM how to configure swap file usage to decrease high RAM usage? EDIT does anyone need more details? Last edited by verheiratet1952; 09.12.2021 at 16:26. |
#2
|
|||
|
|||
Quote:
sorry?! |
#3
|
||||
|
||||
@verheiratet1952: you allow java to use max 24GB of heap memory, why?
of course windows will then start to swap out memory of other stuff. how many links do you have in list? you can also tweak java to free(give back to OS) unused memory, please use google/search for the correct JVM parameters. but in first place you should not provide that amount of memory without really have to. I'm ready to further analyse the memory usage and try to optimize/reduce it
__________________
JD-Dev & Server-Admin |
#4
|
|||
|
|||
Quote:
we already had the same conversation about RAM please see https://board.jdownloader.org/showthread.php?t=86151 you told me to assign RAM manually "You can also use -Xmx24g instead of -Xmx24576m" I also made some static pagefile.sys on a M.2 using NVMe PCIe 3.0 for best speed performance... to answer your questions: -how many links do you have in list? the biggest one is about 4mio image links with nearly 90k folders where I do add/update new images/folders daily/weekly... I need to have all those links to prevent duplicates... I do always save CFG folder with its downloadlist and linkcollector ZIPs... just as an example: if I start JD with those 4mio links and its 90k folders it has at about 10GB usage at the beginning, but it gets higher and higher within few hours while crawling new content... or when just downloading files and dont run linkcrawler... Last edited by verheiratet1952; 17.12.2021 at 17:22. |
#5
|
||||
|
||||
@verheiratet1952: sorry, yes, I remember
would it be possible to send me a copy of such a large list, so I can do testing/optimization on a *real world* list. eg upload to mega and send me link to support@jdownloader.org It would really help me a lot getting hands on a real large list and not create dummy ones myself as they do not reflect *real usage*
__________________
JD-Dev & Server-Admin |
#6
|
|||
|
|||
Quote:
you should have 2 emails from transfer website for large files... subject is thread 494979 |
#7
|
|||
|
|||
Quote:
JD info |
#8
|
||||
|
||||
@verheiratet1952: thanks, I will try to find simple/easy to implement optimizations to reduce memory usage for such large lists.
__________________
JD-Dev & Server-Admin |
#9
|
|||
|
|||
Quote:
it takes some time to reach heap level after starting JD, as said before.... |
#10
|
||||
|
||||
@verheiratet1952: did not work on it yet and first goal is to reduce overal memory consumption for the links which helps to reduce memory pressure and usage
__________________
JD-Dev & Server-Admin |
#11
|
|||
|
|||
Quote:
but you downloaded downloadlist and linkcollector zips? the website does NOT store that long... |
#12
|
||||
|
||||
yes, I have them both. Thanks again for your help!
__________________
JD-Dev & Server-Admin |
#13
|
||||
|
||||
@verheiratet1952: thanks again for your help. with this list I was able to massiv reduce memory consumption in runtime. still doing tests and trying to reduce memory usage even more. I'm sure you'll notice the reduction in memory usage instantly!
__________________
JD-Dev & Server-Admin |
#14
|
|||
|
|||
Quote:
I use multiOS Version where I can assign JAVA... better to have 2 oder 3 JAVA installs that I can assign those if running more than only one JD at the same time? Last edited by verheiratet1952; 22.12.2021 at 20:07. |
#15
|
||||
|
||||
@verheiratet1952: I will finally release the update by tomorrow and it will contain the optimizations as well.
__________________
JD-Dev & Server-Admin |
#16
|
|||
|
|||
Quote:
better to have 2 oder 3 JAVA installs that I can assign those if running more than only one JD at the same time? |
#17
|
||||
|
||||
@verheiratet1952: every JDownloader is the same. Only different way to start (JDownloader.jar, .exe Launcher...) Doesn't really matter which one you use.
__________________
JD-Dev & Server-Admin |
#18
|
|||
|
|||
because of shortcut to assign max RAM and what JAVA if there is more than just one installed...
|
#19
|
||||
|
||||
@verheiratet1952:
JDownloader.jar: -double click will start with default JVM configuration and system installed Java -you can create shortcut/batch/script to start with different JVM and configuration, eg Quote:
or if you like also different JVM versions JDownloader.exe: -depends on the *age* of the launcher(exe) but in case there are .vmoptions files, you can place custom configuration in there, see https://support.jdownloader.org/Know...vmoptions-file -the launcher(exe) will use the JVM from java or jre subfolder relative to its location. You can do multiple installations of JDownloader and then each launcher will use its own .vmoptions and JVM
__________________
JD-Dev & Server-Admin |
#20
|
||||
|
||||
Update is live!
__________________
JD-Dev & Server-Admin |
#21
|
|||
|
|||
And fails to update:
org.jdownloader.update.SelfUpdateException: Selftest has been terminated unexpectedly at org.jdownloader.update.SelfUpdateClient.finishSelfUpdate(SelfUpdateClient.java:353) at org.jdownloader.update.UpdateManager.runFinishSelfUpdate(UpdateManager.java:951) at org.jdownloader.update.UpdateManager$12.run(UpdateManager.java:866) at org.jdownloader.update.UpdateManager.installPendingUpdates(UpdateManager.java:886) at org.jdownloader.updatev2.UpdateController.installUpdates(UpdateController.java:536) at org.jdownloader.updatev2.UpdateController.onResults(UpdateController.java:408) at org.jdownloader.update.UpdateManager$21.onResults(UpdateManager.java:1507) at org.jdownloader.update.UpdateManager.runUpdateLoop(UpdateManager.java:1036) at org.jdownloader.update.PendingUpdate.run(PendingUpdate.java:19) at org.jdownloader.update.UpdateManager$23.run(UpdateManager.java:1686) at java.lang.Thread.run(Thread.java:748) Running jdownloader on an M1 MAC with: openjdk 17.0.1 2021-10-19 OpenJDK Runtime Environment Temurin-17.0.1+12 (build 17.0.1+12) OpenJDK 64-Bit Server VM Temurin-17.0.1+12 (build 17.0.1+12, mixed mode) |
#22
|
||||
|
||||
@mdmdfss: please don't hijack threads
https://board.jdownloader.org/showthread.php?t=89344
__________________
JD-Dev & Server-Admin |
#23
|
|||
|
|||
it really works great, much better than before...
thx a lot! 85k folders with 1.6mio images Usage: 4,75 GB - Allocated: 12,80 GB - Max: 21,33 GB 90k folder with 4.2mio images (you have CFG folder zips to re-produce a 'real world') Usage: 6,40 GB - Allocated: 12,95 GB - Max: 21,33 GB will give more info when having tried on all my different projects... merry x-mas |
#24
|
||||
|
||||
@verheiratet1952: Thanks for the feedback and you're welcome! Yes, I was able to heavily reduce memory consumption for your list. could you please provide example list of the 85k folders and 1.6mio images as well?
__________________
JD-Dev & Server-Admin |
#25
|
|||
|
|||
Quote:
performance update from a 'real world' (new desktop PC with 128gb RAM) downloadlist with 70k folders with 3.2 mio images added new 25k folders with 1.1mio images at linkcollector running nearly 24hours... Usage: 44,62 GB - Allocated: 46,31 GB - Max: 85,33 GB crawling one single sub-reddit for 24hours Usage: 1,15 GB - Allocated: 1,80 GB - Max: 26,67 GB had 18k folder with 29k images, crawling new 75k folder with 118k images... now after another 18 hours it has 122k folders with 188k images Usage: 990,31 MB - Allocated: 3,18 GB - Max: 26,67 GB |
#26
|
||||
|
||||
@verheiratet1952: if possible you should stay with max heap below 32GB (eg -Xmx30g). Above 32GB java has to switch to different reference/pointer size and thus will require more memory.
That seems too much usage for the use case. Would it be possible to do Teamviewer Session together, then I could do further checks on memory consumption and try to find possible optimizations. Right now I worked on optimization on loading lists into memory and reducing memory footprint. But very likely the processing within a running session can be optimized even further but that would require getting *hands on* such a JDownloader instance. Teamviewer and then using jvisual vm to create/analyze heap dumps. Contact me via support@jdownloader.org
__________________
JD-Dev & Server-Admin Last edited by Jiaz; 28.12.2021 at 17:31. |
#27
|
|||
|
|||
Quote:
Usage: 44,62 GB - Allocated: 46,31 GB - Max: 85,33 GB but if usage and allocated is more than 32gb I think JD needs more than just 32gb of RAM? you think/know that is should work perfectly with max of 32gb? by the way: we can do same Teamviewer, sure I will let you know via email... |
#28
|
|||
|
|||
sometimes it looks like saving downloadlist and linkcollector zips takes very very long :(
I was also trying on max speed drive m2 Samsung PCIe 4.0 x4, but same same... |
#29
|
||||
|
||||
@verheiratet1952 : No optimization yet on load/save of lists. And the limiting factor is NOT your hardware/system but the amount of links/data and the current way of storage. I will work on this topic once I have finished the optimization on memory consumption.
__________________
JD-Dev & Server-Admin |
#30
|
||||
|
||||
Quote:
Java will not use more heap than allowed/configured. And it's strange that memory usage has grown that large despite just adding *a few * more links in list, so my question about teamviewer
__________________
JD-Dev & Server-Admin Last edited by Jiaz; 29.12.2021 at 10:44. |
#31
|
|||
|
|||
Quote:
downloadlist and linkcollector zips have at about 1-2.5gb I think it is about my hdd/ssd/m2... JD is doing Updates on downloadlist and linkcollector zips every few minutes if anything is being added... writing new downloadlist and linkcollector zips take much longer if HDD not SSD, or if SSD not m.2 PCIe 4.0 x4 |
#32
|
||||
|
||||
@verheiratet1952: you can increase delays to write changes to disk, see
https://board.jdownloader.org/showpo...7&postcount=15 I will optimize the storage of lists too, but first I want to further optimize memory usage
__________________
JD-Dev & Server-Admin |
#33
|
||||
|
||||
@verheiratet1952: meanwhile would be great if you can try latest Java11, see
adoptium.net/?variant=openjdk11 It contains String storage optimizations which should also work great in your case!
__________________
JD-Dev & Server-Admin |
#34
|
|||
|
|||
what changes should I see when using openjdk11?
|
#35
|
|||
|
|||
by the way:
wattage/electric power usage is shown as extremely high in task manager... |
#36
|
||||
|
||||
less memory usage with same workload in comparission with java1.8
Quote:
as explained, that part is not yet optimized at all
__________________
JD-Dev & Server-Admin |
Thread Tools | |
Display Modes | |
|
|