JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #801  
Old 17.06.2019, 17:32
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

VMMap is a process virtual and physical memory analysis utility:

Screenshot:
https://i.postimg.cc/k9h2RsXh/Screen...t-05-30-PM.jpg
Reply With Quote
  #802  
Old 19.07.2019, 22:36
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default How to set Heap correctly?

When update, for example, "Event Scripter" or "Core Update" or plug-ins, some error during the restart, but during the second attempt always starts correctly.

Code:
------------------------Thread: 365:org.jdownloader.updatev2.restart.WindowsRestarter.log-----------------------
--ID:365TS:1563567340243-19.07.19 22:15:40 -  [org.appwork.utils.logging2.InputStreamLogger(readInputStreamToString)] -> Error occurred during initialization of VM
--ID:365TS:1563567340244-19.07.19 22:15:40 -  [org.appwork.utils.logging2.InputStreamLogger(readInputStreamToString)] -> Unable to allocate 163840KB bitmaps for parallel garbage collection for the requested 5242880KB heap.
5242880KB = 5GB ???


My Parameters to run:
"C:\Program Files\Java\jre1.8.0_191\bin\javaw.exe" -Xms2G -Xmx5G -jar C:\JD\JDownloader.jar

Physical memory still available, so I do not know what the problem is?
screen: https://i.postimg.cc/HWyGt2Jd/Screen...t-10-16-PM.jpg

Still need to increase -Xmx?

Download List: 70K
LinkGrabber List: 63K

19.07.19 22.17.14 <--> 19.07.19 22.32.09 jdlog://0992186935451/
Reply With Quote
  #803  
Old 24.07.2019, 14:59
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

I had a similar problem with some recent updates.
If the problem persists, I will publish the logs.


https://i.postimg.cc/CM9XzvmT/Screen...t-12-11-AM.jpg
Reply With Quote
  #804  
Old 24.07.2019, 15:23
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,602
Default

That error is not caused by JDownloader. Java fails to start. Most likely because your system is out of memory
__________________
JD-Dev & Server-Admin
Reply With Quote
  #805  
Old 24.07.2019, 16:05
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

This is not an error of out of memory, because the last pid with this type of error is from July 19.

JVM initialization fails with parallel GC

probably should use parameter start???

-XX:-UseParallelGC

https://board.jdownloader.org/showthread.php?t=81097
Reply With Quote
  #806  
Old 24.07.2019, 16:55
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

Expected results: The JVM should not attempt to allocate memory beyond that required to accommodate the specified maximum heap size when using the parallel garbage collector.
Reply With Quote
  #807  
Old 24.07.2019, 18:36
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,602
Default

something made your JVM fail to start and it's not JDownloader. It's up to you what parameters/JVM settings you set.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #808  
Old 24.07.2019, 18:39
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,602
Default

Quote:
Originally Posted by djmakinera View Post
This is not an error of out of memory, because the last pid with this type of error is from July 19.
JVM does NOT create hs_err files when not started Invalid combination of parameters or failing to start does NOT create hs_err files
__________________
JD-Dev & Server-Admin
Reply With Quote
  #809  
Old 24.07.2019, 20:11
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

Closing or running the application works correctly. Only in some specific conditions during the upgrade with the required restart it encounters an error.
It's just that the error does not appear during the second attempt - running with still the same parameters.

The process may also be busy or working in the background, which may prevent a proper restart when trying to update. The first attempt may fail, but the second may not.
Reply With Quote
  #810  
Old 25.07.2019, 09:42
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,602
Default

Quote:
Originally Posted by djmakinera View Post
The process may also be busy or working in the background, which may prevent a proper restart when trying to update. The first attempt may fail, but the second may not.
That's not the case.

Quote:
Originally Posted by djmakinera View Post
The first attempt may fail, but the second may not.
Because your system runs out of memory and JVM fails to start


Quote:
Originally Posted by djmakinera View Post
Closing or running the application works correctly. Only in some specific conditions during the upgrade with the required restart it encounters an error.
It's just that the error does not appear during the second attempt - running with still the same parameters.
Because your system runs out of memory and JVM fails to start


The answer is very simply. You specify the parameter -Xms2g which makes JVM allocate minimum 2Gbyte of Memory.
During update JD starts another instance and then you have up to 3 instances running, 3*2Gbyte Memory -> the new processes
fail to start because out of memory.
Removing this parameter or reduce it should solve it for your setup
__________________
JD-Dev & Server-Admin
Reply With Quote
  #811  
Old 25.07.2019, 12:16
thecoder2012's Avatar
thecoder2012 thecoder2012 is offline
Official 9kw.eu Support
 
Join Date: Feb 2013
Location: Internet
Posts: 1,324
Default

Quote:
Originally Posted by Jiaz View Post
fail to start because out of memory.
Removing this parameter or reduce it should solve it for your setup
Or buy more memory for your computer. I have 64 GB.
__________________
Join 9kw.eu Captcha Service now and let your JD continue downloads while you sleep.
Reply With Quote
  #812  
Old 25.07.2019, 12:53
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

Quote:
Originally Posted by thecoder2012 View Post
Or buy more memory for your computer. I have 64 GB.

Really, do not need a lot of physical memory, my 12GB, it seems to be optimal. Many people have even less physical memory, 32-64GB is for advanced and hardcore people who will not actually use that much physical memory. Who is it for?
Physical memory can be replaced with virtual memory, and this does not slow down the operation of the computer and programs when you have a very fast SOLID STATE DRIVE (Tested!)
Is something changed in the long run? I've been using JD2 for a long time and I have not had any problems for a long time, now it seems that JD2 needs more and more memory to run JD2, but it does not need a lot RAM while downloading.
Reply With Quote
  #813  
Old 25.07.2019, 13:14
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,602
Default

Quote:
Originally Posted by djmakinera View Post
Download List: 70K
LinkGrabber List: 63K
The more links you've got in list, the more memory is required.
It highly depends on link/host type if a link requires less/more memory.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #814  
Old 25.07.2019, 13:15
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,602
Default

Quote:
Originally Posted by djmakinera View Post
Physical memory can be replaced with virtual memory, and this does not slow down the operation of the computer and programs when you have a very fast SOLID STATE DRIVE (Tested!)
I'm sorry but NO it can not! But I already gave up about this
__________________
JD-Dev & Server-Admin
Reply With Quote
  #815  
Old 25.07.2019, 13:29
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,615
Default

my thoughts, under Jiaz explanation.
why does the update module need to start with the same vm parameters as main program? Surely it should not not have the same memory requirements. You would only have 2*3GB and say 64MB for the update requirements.

and if the original instance is still running at end of updating, maybe it should wait until the original pid is cleared? or at least an option to do so. Doing so would only ever mean one instance of the higher memory allocation running at any time.

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #816  
Old 25.07.2019, 13:35
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,602
Default

@raztoki: for sure we could *filter out* some parameters but in that case the parameters are lost for the *after update* process.
and the first process will shutdown once the 2nd proccess has validated the pending update.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #817  
Old 25.07.2019, 13:36
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

Quote:
Originally Posted by Jiaz View Post
I'm sorry but NO it can not! But I already gave up about this
Yes and no.

It depends on the type of program / application you use.
WEB browsers, movie and music players, text editor, simple programs do not require the need of having large physical memory. Maybe for professionals who render the film or nobody knows what else.
There is an editor that can open 248 GB of text data, but still uses a small amount of RAM, and uses disk storage, if a very large amount of memory is required.

I have the most soundcloud.com host links
I do not know if it matters.
It's not much, anyway ... especially when in the distant past I had 10 times more links, and it was very good (And the usage was in a similar range of RAM)

Decrease the value of -Xms0,5G to see the effects.
Reply With Quote
  #818  
Old 25.07.2019, 13:41
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,602
Default

Quote:
Originally Posted by djmakinera View Post
Decrease the value of -Xms0,5G to see the effects.
You can leave out the -Xms parameter
__________________
JD-Dev & Server-Admin
Reply With Quote
  #819  
Old 25.07.2019, 13:42
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,602
Default

Quote:
Originally Posted by djmakinera View Post
There is an editor that can open 248 GB of text data, but still uses a small amount of RAM, and uses disk storage, if a very large amount of memory is required.
I don't know why a filesize should be connected to application? video player also playback gbyte of video without using that much memory.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #820  
Old 25.07.2019, 13:43
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,602
Default

Quote:
Originally Posted by djmakinera View Post
Yes and no.

It depends on the type of program / application you use.
correct, and when you tell Java to start with 2 Gbyte of memory (-Xms2g) then it will allocate 2gbyte of memory and either the os is able to provide that much memory or not. In your case, it doesn't have enough memory left.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #821  
Old 25.07.2019, 13:45
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,602
Default

Quote:
Originally Posted by djmakinera View Post
I have the most soundcloud.com host links
I do not know if it matters.
It's not much, anyway ... especially when in the distant past I had 10 times more links, and it was very good (And the usage was in a similar range of RAM)
Yes, this matters as different links require different amount of memory.
and of course we add more features to plugins and that will increase memory usage. For example plugins with customizable filename pattern do require more memory because the link needs to hold all possible meta information.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #822  
Old 25.07.2019, 13:46
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,602
Default

you may send me a link to your downloadList/linkcollector zip and I can check for possible optimizations in memory requirements
__________________
JD-Dev & Server-Admin
Reply With Quote
  #823  
Old 25.07.2019, 13:52
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

Quote:
Originally Posted by Jiaz View Post
I don't know why a filesize should be connected to application?
Example. How a program designed that does not need a lot of RAM, even with huge multi operations, and performance, still high.

Read FEATURE:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

Screen Feature:
https://i.postimg.cc/xCK63knX/Screen...t-01-46-PM.jpg
Reply With Quote
  #824  
Old 25.07.2019, 14:01
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

Quote:
Originally Posted by Jiaz View Post
you may send me a link to your downloadList/linkcollector zip and I can check for possible optimizations in memory requirements

Respect... I want to keep privacy, just like most users.

I can provide links only in this way:
50000 links:

**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #825  
Old 25.07.2019, 14:03
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,602
Default

Quote:
Originally Posted by djmakinera View Post
Example. How a program designed that does not need a lot of RAM, even with huge multi operations, and performance, still high.
That's what I meant. Large filesize doesn't mean large memory usage.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #826  
Old 25.07.2019, 14:05
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,602
Default

Quote:
Originally Posted by djmakinera View Post
Respect... I want to keep privacy, just like most users.
I'm sorry but I have more important things to do than to recreate a situation like yours and adding/waiting for 130k links to be added/processed.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #827  
Old 25.07.2019, 14:06
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,602
Default

There's no need to continue here. The issue is caused by system memory and your jvm settings.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #828  
Old 23.10.2019, 19:54
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default JD2 and page

JD2 after starting the use of pagefile.sys increases significantly from 1GB to 4GB, but after closing JD2 it still uses as much disk space all the time. Release memory can only take place when the system is restarted, otherwise the maximum size will be constant all the time.
How do you force JD2 to use only physical memory?

"C:\Program Files\Java\jre1.8.0_231\bin\javaw.exe" -Xms512M -Xmx5G -XX:-UseParallelGC -jar C:\JD\JDownloader.jar
Reply With Quote
  #829  
Old 24.10.2019, 09:55
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,602
Default

Quote:
Originally Posted by djmakinera View Post
How do you force JD2 to use only physical memory?
I'm not aware that this is possible for user space applications at all and even kernel space
"Because contiguous physical memory is usually in short supply, it should be used sparingly and only when necessary." so even drivers should only allocate small portions of such memory.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #830  
Old 24.10.2019, 12:24
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

My SSD drive is not large.
That's why I try to manage space as much as I can.
But after running JD2 or performing what works, the space for it just immediately drops to 0 bytes, even though there was, say, 5-7 GB free space.

Another problem is that the logs will be created indefinitely if there are errors, and it is not interrupted automatically, but it is possible manually.
In the advanced JD2 settings there is an option - the number of attempts after which stop the download attempt or the infinite error loop?
I asked about it many years ago, any new information?
Reply With Quote
  #831  
Old 24.10.2019, 14:37
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,602
Default

Normal applications cannot allocate *non pagable* memory because that way you could easily *make your os non functional*
Please provide more information about "infinite error loop" and the logs are limited in size and can even be disabled (see advanced settings)
__________________
JD-Dev & Server-Admin
Reply With Quote
  #832  
Old 24.10.2019, 18:01
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

The repeated error indefinitely generates a log record all the time. And no automatic interruption of the download process.
Please fix this unhandled error

Log:
24.10.19 17.57.43 <--> 24.10.19 18.00.13 jdlog://4895130900751/
Reply With Quote
  #833  
Old 24.10.2019, 19:17
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,602
Default




the loop is caused by connection issue and JDownloader does auto retry because this error can be temp. or IP dependant.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #834  
Old 25.10.2019, 17:54
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

This link will also repeat the unknown error indefinitely and generate a log.

**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #835  
Old 25.10.2019, 18:26
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,602
Default

And? JDownloader retries. You can easily set limits for logs in Advanced Settings. Without a log, my guess is that the error that has happened doesn't lead to a *don't try again*
__________________
JD-Dev & Server-Admin
Reply With Quote
  #836  
Old 25.10.2019, 18:45
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default

Quote:
And?
You can limit the log limit, but it depends on all hosts. How to exclude logs only for specific hosts?
Reply With Quote
  #837  
Old 25.10.2019, 19:31
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,602
Default

You can't and I don't see any need for this. Just reduce log size.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #838  
Old 28.12.2019, 17:14
djmakinera djmakinera is offline
Banned
 
Join Date: May 2010
Location: Poland
Posts: 8,387
Default memory management

There is a conflict with memory management between the Chrome browser and JD2. I don't know how to solve this problem.

Error: Out Of Memory. JD2 Auto-Terminate

RAM: 12 GB


28.12.19 17.08.39 <--> 28.12.19 17.12.40 jdlog://9553330900751/

"C:\Program Files\Java\jre1.8.0_231\bin\javaw.exe" -Xms512M -Xmx5G -XX:-UseParallelGC -jar C:\JD\JDownloader.jar
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 11:09.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.