#1
|
|||
|
|||
1 CPU core at 100%, downloads stuck in "starting"
I think it started sometime mid-December 2016 or shortly after that.
JDownloader started maxing out one core almost continuously, but only while downloading. At the same time, the single files took longer and longer to start downloading and spent multiple seconds to minutes in the "starting" state. But they eventually started downloading. That went on a few weeks, and then it got so bad that the downloads would not start anymore but rather be stuck in "starting". So a few days ago I started from scratch with a new JDownloader setup to rule out any possible configuration nonsense. That seemed to revert to the "1 core at max and slow starts" situation, which kept going on a couple of days, until a few days ago, when JDownloader spent a whole day without downloading anything. So I stopped using it for now. A few notes:
Anyone seen this behavior or have any hints? I have experience in Java development. Would it perhaps be possible for me to let JDownloader run in an Eclipse debug session to investigate what it is doing? A screenshot of the "About" window is attached as "JDownloader.2017-01-20_01-00-20.png" |
#2
|
||||
|
||||
hi
openjava has caused this in the past, checkout oracle java and see if it resolves the issue. are you saving to usb device? I have seen 100% lockup in this case also. raztoki
__________________
raztoki @ jDownloader reporter/developer http://svn.jdownloader.org/users/170 Don't fight the system, use it to your advantage. :] |
#3
|
||||
|
||||
You can connect via JVisualVM and create Thread-Dumps. In case of high cpu load, simply create a few Thread-Dumps fast and send me them to support@jdownloader.org
Or maybe you see something strange for yourself and can inform us. When CPU usage becomes high, How many Links do you have in List? Please provide a screenshot of the About Dialog when CPU is high (maybe java spends lot time for garbage collection, possible memleak). Is CPU high only during download or also when no downloads are running at all?
__________________
JD-Dev & Server-Admin |
#4
|
||||
|
||||
Quote:
__________________
JD-Dev & Server-Admin |
#5
|
|||
|
|||
All right, here goes, I ran JDownloader in debug mode and attached Eclipse, and then I paused each thread and checked the CPU usage.
For the record, I used this command, Code:
java -Xdebug -Xrunjdwp:transport=dt_socket,address=8001,server=y,suspend=y -jar JDownloader.jar **External links are only visible to Support Staff****External links are only visible to Support Staff** (stackoverflow-dot-com/a/22636491) Turns out, the offending thread is the "WatchDog: downloadWatchDog" thread. As for its stack, have a look at the attached screenshot, "JDownloader.Eclipse.debug.high-CPU-thread.2017-01-21_02-49-44.png". Pausing the thread did not cause the links to start downloading, though. @Jiaz: The high CPU usage only happens when the downloads are running. When I press the stop button, the CPU load goes away. |
#6
|
|||
|
|||
New findings: Seems like I just overdid it with the number of links.
I removed everything from the download queue and then added some single items. Works like a charm for now. (insert :facepalm: smiley) Sorry for making such a fuss. I might look into this anyway – I do suspect that it should be possible to coordinate a virtually unlimited number of download links on today's hardwares, especially in Java which nowadays is a really efficient runtime If I manage to get anything done I'll tell. Thank you for your support, much appreciated. |
#7
|
||||
|
||||
out of interest how many links did you have in the queue?
the code you had breakpointed was just evaluating links to start downloading. It shouldn't be that CPU intensive. If you're downloading lots of small files (say images) it could seem to be hogging more CPU as continuously starting new links. raztoki
__________________
raztoki @ jDownloader reporter/developer http://svn.jdownloader.org/users/170 Don't fight the system, use it to your advantage. :] |
#8
|
|||
|
|||
Quote:
That usually wasn't a problem, though. I just re-added a rather large package with about 8k links to 4k files (2 hosters), and this seems to trigger the bug. At least the CPU load goes to 100% on one core as soon as I move it from the Linkgrabber to the Download area, and stays at 100%. The same is the case with a different package containing 5k links to 2.5k files. Granted that may be a lot of links, but still, that's why one uses JDownloader in the first place My current workaround is to have the bulk of the links in "disabled" state and enable them as JDownloader progresses at downloading them. I think there's room for improvement. Judging from the debug screenshot there seems to be some sort of hash-based lookup thing going on, maybe these hashes are always recalculated? They could perhaps be calculated once and then be kept for the rest of the application runtime. I personally wouldn't mind the memory requirements for keeping some 5-digit number of hashes around. |
#9
|
||||
|
||||
hashes are looked for, or calculated, but its used for mirror determination.
large pacakages isn't great, your better off with smaller packages if you can help it. each time a slots opens, it will compare each many/all of the objects within package (for memory). raztoki
__________________
raztoki @ jDownloader reporter/developer http://svn.jdownloader.org/users/170 Don't fight the system, use it to your advantage. :] |
#10
|
||||
|
||||
Only (available heap) memory limits the max numbers of links in JDownloader. We have some special users out there that easily reach 2 mio links in JDownloader
Could we do a Teamviewer session so we check JVisualVM and its thread dumps together. Thread Dumps provide more information as you can easily see if a thread has a tight loop in it. Also there is still room left for optimizations For example the *find next download link* causes lots of garbage. For example *ugly* use of List.removeAll So I suggest we do a teamviewer session (contact me at support@jdownloader.org) or you can provide some thread dumps in short sequence. Also please test the following. Start downloads and set *max downloads* to <= current downloading. Does this reduce cpu usage?
__________________
JD-Dev & Server-Admin |
#11
|
|||
|
|||
Quote:
I now have an Eclipse workspace with JDownloader checked out from SVN, and it starts up all right as far as I can tell, so I could go wild with breakpoints and such. Kudos on the howto by the way, much appreciated. One question here: Where does JDownloader store its user data? Because it sure did not use the data of my "regular" instance when I launched it in Eclipse. (Which, in retrospect, wasn't a smart thing to do on my part, considering it might have done. With the version differences and in-development state it might have screwed up my regular instance.) Anyhow, now that I know the Eclipse instance has its own user data, I'd like to copy the user data of my regular JDownloader instance to the Eclipse workspace one, so I can debug into the real thing. Maybe we can apply the mighty HashMap and harness its insane lookup powers |
#12
|
||||
|
||||
eclipse
workspace is where ever you placed it, for me its c:\workspace\ jd_home will be placed on windows within Users\Username\.jd_home
__________________
raztoki @ jDownloader reporter/developer http://svn.jdownloader.org/users/170 Don't fight the system, use it to your advantage. :] |
#13
|
||||
|
||||
In case you have additional questions, you can contact us via
support@jdownloader.org IRC, freenode, #jdteam personal email (ask via support@jdownloader.org) The loop that selects next download candidate is in DownloadWatchDog.next(DownloadLinkCandidateSelector selector) nextDownloadLinkCandidate(DownloadLinkCandidateSelector selector) -> here the ugly list usage happens -> causes lots of garbage The complete candidate selection is very complex, so instead of trying to figure it out by yourself you should ask/contact us you can reach us in chat I can also provide little tour in source via teamviewer if you like
__________________
JD-Dev & Server-Admin Last edited by Jiaz; 24.01.2017 at 11:01. |
Thread Tools | |
Display Modes | |
|
|