JDownloader Community - Appwork GmbH
 

Notices

 
 
Thread Tools Display Modes
  #41  
Old 29.09.2009, 04:14
drbits's Avatar
drbits drbits is offline
JD English Support (inactive)
 
Join Date: Sep 2009
Location: Physically in Los Angeles, CA, USA
Posts: 4,434
Default More data

I hope your fix works. However if it does not, here is more data:

Win XP sp3 (up to date), 1GB mem, 2 GHz pentium. JVM 6.0.160.1 with 7834 links, 384 packages, several different hosters (Rapidshare, Megaupload, storage.to, hotfile, depositfiles, netload [had one Megashare, uploading.com I do by hand, I am also trying YouTube]),

Turning off all column in the Downloads page except for Package/Hostname, disabling logging, and minimizing to the tray slows the increase in memory usage to a tolerable level, but it doesn't stop completely. CPU usage is still jumping around between 70% and 25% (and points in between) creating a sawtooth pattern on a CPU usage graph.

The logging should only increase memory usage if the log is kept in memory. However, if Strings are used to create log entries (instead of CharacterArray or StringBuffer), we can expect logging to increase memory usage.

I think the Password Cracking and Capcha decyphering are used only end or beginning of a file. I don't think that matches the pattern I am seeing.

I find it strange than none of the staff are seeing this. You might check your OS versions, JVM versions, Processor speed, and memory size against all of the reports. For example, it is possible that we are just overloading the GC and you are not seeing that with much more memory and much faster systems. Or you might not see it if you are still on JVM 15.x.

By the way, Process Explorer (sysinternals.com) now from Microsoft is probably the best free tool for examining memory and CPU usage. Taskman doesn't do a good job. Make sure to look at Private Bytes, not Virtual Bytes or Workspace (WS Private).
  #42  
Old 29.09.2009, 10:29
remi
Guest
 
Posts: n/a
Cool

I hope they found the main leaking hole. Nevertheless, your detailed explanations are appreciated and will certainly be taken into consideration by the devs.

That you're seeing patterns in the memory usage is certainly a sign of intelligence.
  #43  
Old 29.09.2009, 13:04
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

we see this post and the high cpu/ram issue is not caused by the strings. of course we know about the performance with strings and stringsbuffer and use them as good as we can

pls wait for upcoming update and check again

thanks for the detailed report
__________________
JD-Dev & Server-Admin
  #44  
Old 30.09.2009, 23:13
Seregwethrin
Guest
 
Posts: n/a
Default

1) What was causing the issue?

2) It's clear that new JD is really faster. I can feel it when I click the interface.

3) So far so good, I hope no CPU issue will happen. I'll let you know again.
  #45  
Old 30.09.2009, 23:30
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

see comment here
http://svn.jdownloader.org/repositor...vision/jd/8736
__________________
JD-Dev & Server-Admin
  #46  
Old 01.10.2009, 21:03
Seregwethrin
Guest
 
Posts: n/a
Default

I guess the problem is solved. There's no problem so far.

Really thank you. You really did it this time. jD is perfect now!
  #47  
Old 02.10.2009, 08:55
LordSteiner
Guest
 
Posts: n/a
Default

Works fine now. Thanks a lot
  #48  
Old 02.10.2009, 09:56
nonessentials
Guest
 
Posts: n/a
Default

Back on August 21, 2009, I wrote the following on another thread:

Quote:
Originally Posted by nonessentials View Post
The standard skin (SyntheticaStandard) seems to be the easiest on the CPU, although with several downloads going, there is some spiking. I tried a couple of the Substance skins, and, wow, what a difference! The CPU load with the Download window showing was as bad as the 0.6.xxx generation of jDownloader (on my machine). What the choice of skins has to do with this is a mystery to me. Maybe somebody could explain that?
I should have used the term 'themes' when I posted this, but I still find that there's a connection between CPU usage and theme choice. I updated JD yesterday to -Pinky_2- Version 0.8.821, and the problem of CPU usage above 50% when the Download tab is visible remains. What I notice is that the usage can peg at 100%, depending on which theme I use. The Substance themes are still the worst. The Windows theme is the only one that is able to keep the CPU usage around 50% when the Download tab is visible, but that's not saying a lot.

So what happened to the SyntheticaStandard theme? Could this be brought back? That's the theme that appeared to me to put the lowest load on the CPU back when I wrote the above-quoted passage, although whether that would still be true today is a big question.

Frankly, I'm not sure that this problem is solvable. Those of us who have it might well be stuck with it.
  #49  
Old 02.10.2009, 10:31
drbits's Avatar
drbits drbits is offline
JD English Support (inactive)
 
Join Date: Sep 2009
Location: Physically in Los Angeles, CA, USA
Posts: 4,434
Default

Works much better. It now does not increase in memory use if it is not displaying. However, it will increase in memory use if left open on the display page. Leaving it all day will result in over double the initial memory use.

The speed is much better. I have an old, slow computer, and JD still stays below 15% CPU most of the time.

Congratulations to the programming staff!
  #50  
Old 02.10.2009, 11:11
nonessentials
Guest
 
Posts: n/a
Default

Hey, drbits, are you saying that with several links downloading, with the Download tab visible on the desktop, and using the Windows theme, your CPU usage rarely goes above 15%? I want to compare apples with apples; even my CPU usage drops to below 15% if the Download tab is not visible on the desktop. That's the key, for me.
  #51  
Old 02.10.2009, 15:20
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

visible download tag, 2k links in list, 2 running around 10-12 cpu (because of linux and no directrendeing), under windows 0-1 here
__________________
JD-Dev & Server-Admin
  #52  
Old 03.10.2009, 23:26
drbits's Avatar
drbits drbits is offline
JD English Support (inactive)
 
Join Date: Sep 2009
Location: Physically in Los Angeles, CA, USA
Posts: 4,434
Default

Nonessentials:

With 6 active downloads (over 8500 links total), with the Download page open (3 downloads and a wait visible - the rest lower in the list), javaw is using around 38.5% of my CPU time. If JD is covered by other apps, it drops to about 24%. Minimized, it is between 7% and 12.5%. After a while, minimized it dropped to 3-5% for a while, then went back to averaging around 11%.

This is with a total CPU usage of under 33%, so other applications are not starving JD.

I just restarted my computer, so it is using 155 MB of private bytes. Private Workspace (allocated RAM) is 46MB so there are a lot of page faults, almost all are to the cache (commit charge just below physical memory). Virtual memory about 756MB. Private bytes are not increasing when minimized. Private bytes does seem to increase as I add new links, this increase stays until I restart the program. Private bytes increase slowly with time with the Download page open, and possibly as I work (moving packages, examining properties, restarting failed downloads, etc.). I will have to watch this. Last night, javaw was using 375MB private bytes for a while (eventually, my system locked up for other reasons and required a reboot).

While doing an UnRar, javaw and UnRar each used about 24% of my CPU time. When I quickly scroll through my packages, javaw can use almost 80% of CPU time (obviously with the Download page visible). With the Download page open, just circling the mouse quickly can increase Javaw CPU time to 76%, so that tree-grid class is expensive.

JD version Pinky_2 v0.8.821 - Windows theme (no skin)
java version "1.6.0_16"
Java(TM) SE Runtime Environment (build 1.6.0_16-b01)
Java HotSpot(TM) Client VM (build 14.2-b01, mixed mode, sharing)
Windows XP SP3
1 GB memory
1.98 GHz Pentium 4 (7 years old) with original BIOS.
Saving to external Maxtor 1TB drive via FireWire 400.

Security package:
PCTools Antivirus, Firewall, and Threatfire. Active scanning of .part files is turned off, but not .avi or .rar files. Turning off scanning of .part (and .!ut for uTorrent when I use it) reduces antivirus overhead tremendously and I recommend it for all who have "real-time" antivirus.
DEP is on with no hardware support. The antivirus also has a DEP-like functionality.
Hardware firewall in router (in addition to the software firewall).
Many services designed for LAN use disabled (such as the Server service, Browser service, etc.)

------------------------------------------------------------

My Private Bytes increased to 225MB when I added a package with two links and then did an Open Directory (I am not sure which added the memory usage). The memory usage has not gone down, even after minimizing the program. CPU usage has now also jumped to the 51-86% range with the Download page open or with the program minimized, although eventually the minimized CPU time went back down. My guess is at least one synchronized object (allocated when a package is created and added) is not being freed. Adding additional packages does not further increase the memory (except for a 9MB short term spike).

Having the LinkGrabber page open increases the CPU time by about 2% over having the program minimized.

----------------------------------------

Restarted JD. Four of the links added just before closing were in the clipboard, so they opened in the LinkGrabber (in Orange). Private Bytes 164MB. Clearing the list does not affect memory.

Added 4 new links to LinkGrabber, which increased memory use by about 1/2MB. Again, clearing the list did not restore memory. Re-adding the links only affected memory use for a few seconds. Adding four new links to LinkGrabber again increased memory by about 1/2MB.

Opening JD with LinkGrabber showing and then minimizing it can cost several hundred KB that don't go away. Now up to 168,640KB Private bytes (not affected by CPU spikes that come at regular intervals).

I noticed with LinkGrabber open, occasionally the CPU% will jump to around 35%. I assume this is GC. They do not seem to affect total memory use.

Added four new links to LinkGrabber (each given a package). PB=168,760KB. Merged the links into one package and saved 4KB. PB=168,756KB.

Changed to Download page and then back to LinkGrabber. After settling time, PB=169,040KB

Clicked on Continue to add the new package. Memory eventually stabilized at 172,568KB.

Highlighted 4 packages. Memory slowly increased to 182,964KB. Removing highlight did not reduce memory use. A few minutes later, PB=183,692KB (no reason known for increase). Note that most of the program display is covered so I can type this (only Progress column shows). Eventually, memory reduced to PB=183,664KB.

I went away for 62 minutes with the Download page open and now PB=211,472KB. About 27.5 MB/hour with no interaction.

Last edited by drbits; 04.10.2009 at 02:06. Reason: Mouse moves increase CPU usage to 75%, RAM jump
  #53  
Old 04.10.2009, 01:11
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

The Table EATS alot of CPU when its redrawing, thats kinda normal. Not much more optimizie potential there (only can try different themes).

the rest sounds normal (for 8k links )
__________________
JD-Dev & Server-Admin
  #54  
Old 04.10.2009, 01:20
nonessentials
Guest
 
Posts: n/a
Default

drbits:

Thanks for the very informative reply. It seems that you, too, are experiencing higher CPU loads when the Download tab is visible, although not quite as high as I do. And both of us see a very dramatic drop in CPU usage when, say, the Settings tab is visible. I guess this is something that has to be lived with since going back to the low CPU usage with Download tab visible of the early 0.7.xxx versions of JD doesn't appear possible. I still wonder what happened to that SyntheticaStandard theme, though . . .

Best regards,

--nonessentials

P.S. Even with the (to me) aggravating CPU problem, I'm still quite impressed with JDownloader. The developers have done a helluva job with the program, and I'm pleased to be using it. I see that you are, too, with that huge links list! I thought that I stacked up lots of links!
  #55  
Old 04.10.2009, 01:24
nonessentials
Guest
 
Posts: n/a
Default

Quote:
Originally Posted by Jiaz View Post
The Table EATS alot of CPU when its redrawing, thats kinda normal. Not much more optimizie potential there (only can try different themes).
Jiaz, what happened to that SyntheticaStandard theme? It really did seem to be less CPU-intensive than any of the themes now available, including the Windows theme. Just curious.
  #56  
Old 04.10.2009, 01:41
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

all themes are still there, there is no difference in theme selection
syntheticastandard will be back with next update

btw. just for info. java does not free memory instant, so after increasing heap, it will only free heap when memory usage drops below a specific percent and the additional heap is no longer needed
and it does only help little to tell the memory usage of the java process. better use a tool that shows the jd memory only (jconsole, jprofile and such tools) because we dont know when/what memory java will use itself and when to free it

we tested current jd very long for memory leaks and till now we could not find any. switching first time to some tabs or doing some things for the first time = its normal that memory gets used.
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 04.10.2009 at 03:51.
  #57  
Old 05.10.2009, 08:31
drbits's Avatar
drbits drbits is offline
JD English Support (inactive)
 
Join Date: Sep 2009
Location: Physically in Los Angeles, CA, USA
Posts: 4,434
Default

Thank you Jiaz.

I was not complaining, just performing a couple of experiments. I do know about the memory monitoring issues you raise, and took that into consideration when reporting.

I will be able to provide better information once I have a development environment installed here.

-----------------------------
nonessentials,

I doubt the themes are a serious efficiency issue. Skins have been simple and efficient since Java 1.2.1 (J2SE).

Decisions that make a big difference are things like object encoding, object recycling, background processing, or trade-offs between memory and disk use for large display tables.

I spent over a year at a start-up optimizing Java code (while keeping it portable). It is not an easy task, but one does learn where to look.

Martin
  #58  
Old 05.10.2009, 14:26
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by drbits View Post
I doubt the themes are a serious efficiency issue. Skins have been simple and efficient since Java 1.2.1 (J2SE).
I'm sorry to tell , but the Themes are the real CPU Killer here thats why you have high cpu while watching a download/linkgrabber list, because the table redraw is cpu intensiv.

switch to trayicon and i have 0-1% with 4 downloads and 4k links in list
__________________
JD-Dev & Server-Admin
  #59  
Old 05.10.2009, 14:28
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by drbits View Post
I was not complaining, just performing a couple of experiments.
Edit by Jiaz: i know
I will be able to provide better information once I have a development environment installed here.
Edit by jiaz: help is always welcomed

Decisions that make a big difference are things like object encoding, object recycling, background processing, or trade-offs between memory and disk use for large display tables.
Edit by Jiaz: if you wanna help to improve performance, your welcome to provide patches/join the team
see comments
__________________
JD-Dev & Server-Admin
  #60  
Old 07.10.2009, 16:45
nonessentials
Guest
 
Posts: n/a
Default

Quote:
Originally Posted by Jiaz View Post
I'm sorry to tell , but the Themes are the real CPU Killer here thats why you have high cpu while watching a download/linkgrabber list, because the table redraw is cpu intensiv.

switch to trayicon and i have 0-1% with 4 downloads and 4k links in list
Yes, I noticed some time ago that the themes appeared to be the source of the CPU usage problem. Now that SyntheticaStandard is back on the list, I find that it, too, has succumbed to the problem. I'm still getting between 45% and 75% CPU usage when the Download tab is visible and links are being downloaded. My workaround is to keep the Settings tab visible except when I'm checking something specific on my links list in the Download tab, or, as you say, Jiaz, minimizing the entire JD program to the System Tray. Both workarounds are sort of annoying, but, hey, you can't have everything (if you did, nobody else would have anything!).
 

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 08:39.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.