JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #21  
Old 23.12.2021, 18:59
mdmdfss mdmdfss is offline
Modem User
 
Join Date: Nov 2019
Posts: 1
Default

And fails to update:

org.jdownloader.update.SelfUpdateException: Selftest has been terminated unexpectedly
at org.jdownloader.update.SelfUpdateClient.finishSelfUpdate(SelfUpdateClient.java:353)
at org.jdownloader.update.UpdateManager.runFinishSelfUpdate(UpdateManager.java:951)
at org.jdownloader.update.UpdateManager$12.run(UpdateManager.java:866)
at org.jdownloader.update.UpdateManager.installPendingUpdates(UpdateManager.java:886)
at org.jdownloader.updatev2.UpdateController.installUpdates(UpdateController.java:536)
at org.jdownloader.updatev2.UpdateController.onResults(UpdateController.java:408)
at org.jdownloader.update.UpdateManager$21.onResults(UpdateManager.java:1507)
at org.jdownloader.update.UpdateManager.runUpdateLoop(UpdateManager.java:1036)
at org.jdownloader.update.PendingUpdate.run(PendingUpdate.java:19)
at org.jdownloader.update.UpdateManager$23.run(UpdateManager.java:1686)
at java.lang.Thread.run(Thread.java:748)

Running jdownloader on an M1 MAC with:

openjdk 17.0.1 2021-10-19
OpenJDK Runtime Environment Temurin-17.0.1+12 (build 17.0.1+12)
OpenJDK 64-Bit Server VM Temurin-17.0.1+12 (build 17.0.1+12, mixed mode)
Reply With Quote
  #22  
Old 23.12.2021, 19:07
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,330
Default

@mdmdfss: please don't hijack threads
https://board.jdownloader.org/showthread.php?t=89344
__________________
JD-Dev & Server-Admin
Reply With Quote
  #23  
Old 24.12.2021, 15:22
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by Jiaz View Post
Update is live!
it really works great, much better than before...
thx a lot!

85k folders with 1.6mio images
Usage: 4,75 GB - Allocated: 12,80 GB - Max: 21,33 GB

90k folder with 4.2mio images
(you have CFG folder zips to re-produce a 'real world')
Usage: 6,40 GB - Allocated: 12,95 GB - Max: 21,33 GB



will give more info when having tried on all my different projects...

merry x-mas
Reply With Quote
  #24  
Old 27.12.2021, 11:08
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,330
Default

@verheiratet1952: Thanks for the feedback and you're welcome! Yes, I was able to heavily reduce memory consumption for your list. could you please provide example list of the 85k folders and 1.6mio images as well?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #25  
Old 28.12.2021, 17:00
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by verheiratet1952 View Post
it really works great, much better than before...
thx a lot!

85k folders with 1.6mio images
Usage: 4,75 GB - Allocated: 12,80 GB - Max: 21,33 GB

90k folder with 4.2mio images
(you have CFG folder zips to re-produce a 'real world')
Usage: 6,40 GB - Allocated: 12,95 GB - Max: 21,33 GB



will give more info when having tried on all my different projects...

merry x-mas


performance update from a 'real world'

(new desktop PC with 128gb RAM)
downloadlist with
70k folders with 3.2 mio images

added new 25k folders with 1.1mio images at
linkcollector running nearly 24hours...
Usage: 44,62 GB - Allocated: 46,31 GB - Max: 85,33 GB




crawling one single sub-reddit for 24hours
Usage: 1,15 GB - Allocated: 1,80 GB - Max: 26,67 GB

had 18k folder with 29k images, crawling new 75k folder with 118k images...

now after another 18 hours it has 122k folders with 188k images
Usage: 990,31 MB - Allocated: 3,18 GB - Max: 26,67 GB
Reply With Quote
  #26  
Old 28.12.2021, 17:27
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,330
Default

@verheiratet1952: if possible you should stay with max heap below 32GB (eg -Xmx30g). Above 32GB java has to switch to different reference/pointer size and thus will require more memory.

Quote:
Originally Posted by verheiratet1952 View Post
Usage: 44,62 GB - Allocated: 46,31 GB - Max: 85,33 GB
That seems too much usage for the use case.
Would it be possible to do Teamviewer Session together, then I could do further checks on memory consumption and try to find possible optimizations. Right now I worked on optimization on loading lists into memory and reducing memory footprint. But very likely the processing within a running session can be optimized even further but that would require getting *hands on* such a JDownloader instance. Teamviewer and then using jvisual vm to create/analyze heap dumps.
Contact me via support@jdownloader.org
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 28.12.2021 at 17:31.
Reply With Quote
  #27  
Old 28.12.2021, 18:59
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by Jiaz View Post
@verheiratet1952: if possible you should stay with max heap below 32GB (eg -Xmx30g). Above 32GB java has to switch to different reference/pointer size and thus will require more memory.


That seems too much usage for the use case.
Would it be possible to do Teamviewer Session together, then I could do further checks on memory consumption and try to find possible optimizations. Right now I worked on optimization on loading lists into memory and reducing memory footprint. But very likely the processing within a running session can be optimized even further but that would require getting *hands on* such a JDownloader instance. Teamviewer and then using jvisual vm to create/analyze heap dumps.
Contact me via support@jdownloader.org

Usage: 44,62 GB - Allocated: 46,31 GB - Max: 85,33 GB

but if usage and allocated is more than 32gb I think JD needs more than just 32gb of RAM?

you think/know that is should work perfectly with max of 32gb?



by the way:
we can do same Teamviewer, sure I will let you know via email...
Reply With Quote
  #28  
Old 28.12.2021, 19:44
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

sometimes it looks like saving downloadlist and linkcollector zips takes very very long :(

I was also trying on max speed drive m2 Samsung PCIe 4.0 x4, but same same...
Reply With Quote
  #29  
Old 29.12.2021, 10:37
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,330
Default

@verheiratet1952 : No optimization yet on load/save of lists. And the limiting factor is NOT your hardware/system but the amount of links/data and the current way of storage. I will work on this topic once I have finished the optimization on memory consumption.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #30  
Old 29.12.2021, 10:41
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,330
Default

Quote:
Originally Posted by verheiratet1952 View Post
but if usage and allocated is more than 32gb I think JD needs more than just 32gb of RAM?

you think/know that is should work perfectly with max of 32gb?
the goal of the optimizations is to reduce memory usage so just throwing more memory at java doesn't make it *better*. once you use larger heaps than ca. 32GB, the internal size of references/pointers are larger and thus the overal memory usage will be larger with same amount of links in list. therefore my recommendation to prefer max heap at max 32gb.

Quote:
Originally Posted by verheiratet1952 View Post
Usage: 44,62 GB - Allocated: 46,31 GB - Max: 85,33 GB
but if usage and allocated is more than 32gb I think JD needs more than just 32gb of RAM?
Java will not use more heap than allowed/configured. And it's strange that memory usage has grown that large despite just adding *a few * more links in list, so my question about teamviewer
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 29.12.2021 at 10:44.
Reply With Quote
  #31  
Old 30.12.2021, 03:29
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by Jiaz View Post
@verheiratet1952 : No optimization yet on load/save of lists. And the limiting factor is NOT your hardware/system but the amount of links/data and the current way of storage. I will work on this topic once I have finished the optimization on memory consumption.

downloadlist and linkcollector zips have at about 1-2.5gb

I think it is about my hdd/ssd/m2...
JD is doing Updates on downloadlist and linkcollector zips every few minutes if anything is being added...

writing new downloadlist and linkcollector zips take much longer if HDD not SSD, or if SSD not m.2 PCIe 4.0 x4
Reply With Quote
  #32  
Old 30.12.2021, 11:19
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,330
Default

@verheiratet1952: you can increase delays to write changes to disk, see
https://board.jdownloader.org/showpo...7&postcount=15
I will optimize the storage of lists too, but first I want to further optimize memory usage
__________________
JD-Dev & Server-Admin
Reply With Quote
  #33  
Old 30.12.2021, 17:24
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,330
Default

@verheiratet1952: meanwhile would be great if you can try latest Java11, see
adoptium.net/?variant=openjdk11
It contains String storage optimizations which should also work great in your case!
__________________
JD-Dev & Server-Admin
Reply With Quote
  #34  
Old 31.12.2021, 19:38
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by Jiaz View Post
@verheiratet1952: meanwhile would be great if you can try latest Java11, see
adoptium.net/?variant=openjdk11
It contains String storage optimizations which should also work great in your case!
what changes should I see when using openjdk11?
Reply With Quote
  #35  
Old 04.01.2022, 02:00
verheiratet1952 verheiratet1952 is offline
JD VIP
 
Join Date: Jan 2016
Posts: 325
Default

Quote:
Originally Posted by verheiratet1952 View Post
what changes should I see when using openjdk11?
by the way:

wattage/electric power usage is shown as extremely high in task manager...
Reply With Quote
  #36  
Old 04.01.2022, 12:10
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,330
Default

Quote:
Originally Posted by verheiratet1952 View Post
what changes should I see when using openjdk11?
less memory usage with same workload in comparission with java1.8
Quote:
Originally Posted by verheiratet1952 View Post
by the way:
wattage/electric power usage is shown as extremely high in task manager...
depends on when? in idle or while working? of course this amount of data requires cpu/disk/energy to load/save.
as explained, that part is not yet optimized at all
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 14:18.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.