#21
|
|||
|
|||
And fails to update:
org.jdownloader.update.SelfUpdateException: Selftest has been terminated unexpectedly at org.jdownloader.update.SelfUpdateClient.finishSelfUpdate(SelfUpdateClient.java:353) at org.jdownloader.update.UpdateManager.runFinishSelfUpdate(UpdateManager.java:951) at org.jdownloader.update.UpdateManager$12.run(UpdateManager.java:866) at org.jdownloader.update.UpdateManager.installPendingUpdates(UpdateManager.java:886) at org.jdownloader.updatev2.UpdateController.installUpdates(UpdateController.java:536) at org.jdownloader.updatev2.UpdateController.onResults(UpdateController.java:408) at org.jdownloader.update.UpdateManager$21.onResults(UpdateManager.java:1507) at org.jdownloader.update.UpdateManager.runUpdateLoop(UpdateManager.java:1036) at org.jdownloader.update.PendingUpdate.run(PendingUpdate.java:19) at org.jdownloader.update.UpdateManager$23.run(UpdateManager.java:1686) at java.lang.Thread.run(Thread.java:748) Running jdownloader on an M1 MAC with: openjdk 17.0.1 2021-10-19 OpenJDK Runtime Environment Temurin-17.0.1+12 (build 17.0.1+12) OpenJDK 64-Bit Server VM Temurin-17.0.1+12 (build 17.0.1+12, mixed mode) |
#22
|
||||
|
||||
@mdmdfss: please don't hijack threads
https://board.jdownloader.org/showthread.php?t=89344
__________________
JD-Dev & Server-Admin |
#23
|
|||
|
|||
it really works great, much better than before...
thx a lot! 85k folders with 1.6mio images Usage: 4,75 GB - Allocated: 12,80 GB - Max: 21,33 GB 90k folder with 4.2mio images (you have CFG folder zips to re-produce a 'real world') Usage: 6,40 GB - Allocated: 12,95 GB - Max: 21,33 GB will give more info when having tried on all my different projects... merry x-mas |
#24
|
||||
|
||||
@verheiratet1952: Thanks for the feedback and you're welcome! Yes, I was able to heavily reduce memory consumption for your list. could you please provide example list of the 85k folders and 1.6mio images as well?
__________________
JD-Dev & Server-Admin |
#25
|
|||
|
|||
Quote:
performance update from a 'real world' (new desktop PC with 128gb RAM) downloadlist with 70k folders with 3.2 mio images added new 25k folders with 1.1mio images at linkcollector running nearly 24hours... Usage: 44,62 GB - Allocated: 46,31 GB - Max: 85,33 GB crawling one single sub-reddit for 24hours Usage: 1,15 GB - Allocated: 1,80 GB - Max: 26,67 GB had 18k folder with 29k images, crawling new 75k folder with 118k images... now after another 18 hours it has 122k folders with 188k images Usage: 990,31 MB - Allocated: 3,18 GB - Max: 26,67 GB |
#26
|
||||
|
||||
@verheiratet1952: if possible you should stay with max heap below 32GB (eg -Xmx30g). Above 32GB java has to switch to different reference/pointer size and thus will require more memory.
That seems too much usage for the use case. Would it be possible to do Teamviewer Session together, then I could do further checks on memory consumption and try to find possible optimizations. Right now I worked on optimization on loading lists into memory and reducing memory footprint. But very likely the processing within a running session can be optimized even further but that would require getting *hands on* such a JDownloader instance. Teamviewer and then using jvisual vm to create/analyze heap dumps. Contact me via support@jdownloader.org
__________________
JD-Dev & Server-Admin Last edited by Jiaz; 28.12.2021 at 17:31. |
#27
|
|||
|
|||
Quote:
Usage: 44,62 GB - Allocated: 46,31 GB - Max: 85,33 GB but if usage and allocated is more than 32gb I think JD needs more than just 32gb of RAM? you think/know that is should work perfectly with max of 32gb? by the way: we can do same Teamviewer, sure I will let you know via email... |
#28
|
|||
|
|||
sometimes it looks like saving downloadlist and linkcollector zips takes very very long :(
I was also trying on max speed drive m2 Samsung PCIe 4.0 x4, but same same... |
#29
|
||||
|
||||
@verheiratet1952 : No optimization yet on load/save of lists. And the limiting factor is NOT your hardware/system but the amount of links/data and the current way of storage. I will work on this topic once I have finished the optimization on memory consumption.
__________________
JD-Dev & Server-Admin |
#30
|
||||
|
||||
Quote:
Java will not use more heap than allowed/configured. And it's strange that memory usage has grown that large despite just adding *a few * more links in list, so my question about teamviewer
__________________
JD-Dev & Server-Admin Last edited by Jiaz; 29.12.2021 at 10:44. |
#31
|
|||
|
|||
Quote:
downloadlist and linkcollector zips have at about 1-2.5gb I think it is about my hdd/ssd/m2... JD is doing Updates on downloadlist and linkcollector zips every few minutes if anything is being added... writing new downloadlist and linkcollector zips take much longer if HDD not SSD, or if SSD not m.2 PCIe 4.0 x4 |
#32
|
||||
|
||||
@verheiratet1952: you can increase delays to write changes to disk, see
https://board.jdownloader.org/showpo...7&postcount=15 I will optimize the storage of lists too, but first I want to further optimize memory usage
__________________
JD-Dev & Server-Admin |
#33
|
||||
|
||||
@verheiratet1952: meanwhile would be great if you can try latest Java11, see
adoptium.net/?variant=openjdk11 It contains String storage optimizations which should also work great in your case!
__________________
JD-Dev & Server-Admin |
#34
|
|||
|
|||
what changes should I see when using openjdk11?
|
#35
|
|||
|
|||
by the way:
wattage/electric power usage is shown as extremely high in task manager... |
#36
|
||||
|
||||
less memory usage with same workload in comparission with java1.8
Quote:
as explained, that part is not yet optimized at all
__________________
JD-Dev & Server-Admin |
Thread Tools | |
Display Modes | |
|
|