I think it started sometime mid-December 2016 or shortly after that.

JDownloader started maxing out one core almost continuously, but only while downloading.

At the same time, the single files took longer and longer to start downloading and spent multiple seconds to minutes in the "starting" state. But they eventually started downloading.

That went on a few weeks, and then it got so bad that the downloads would not start anymore but rather be stuck in "starting".

So a few days ago I started from scratch with a new JDownloader setup to rule out any possible configuration nonsense.
That seemed to revert to the "1 core at max and slow starts" situation, which kept going on a couple of days, until a few days ago, when JDownloader spent a whole day without downloading anything. So I stopped using it for now.

A few notes:

• It does not seem to matter which hoster is in use
• I did import the download links from the previous instance using a .dlc, and this too took over a day with one core floored, also it seemed that the "is online" detection wrongly detected some online links as offline links
• This is on Arch Linux, using the OpenJDK 8 runtime shipped by the distribution
• Unlike other problem reports this is not dependent on which tab is active, the problem exists even if the application is minimized. Disabling the clipboard observer did not solve the issue. Also, the user interface always stays responsive

Anyone seen this behavior or have any hints?
I have experience in Java development. Would it perhaps be possible for me to let JDownloader run in an Eclipse debug session to investigate what it is doing?

A screenshot of the "About" window is attached as "JDownloader.2017-01-20_01-00-20.png"

[raztoki]

hi
openjava has caused this in the past, checkout oracle java and see if it resolves the issue.
are you saving to usb device? I have seen 100% lockup in this case also.

raztoki

[Jiaz]

You can connect via JVisualVM and create Thread-Dumps. In case of high cpu load, simply create a few Thread-Dumps fast and send me them to support@jdownloader.org
Or maybe you see something strange for yourself and can inform us.
When CPU usage becomes high, How many Links do you have in List? Please provide a screenshot of the About Dialog when CPU is high (maybe java spends lot time for garbage collection, possible memleak).

Is CPU high only during download or also when no downloads are running at all?

[Jiaz]

Originally Posted by eomanis

I have experience in Java development. Would it perhaps be possible for me to let JDownloader run in an Eclipse debug session to investigate what it is doing?

Sure. You can either hook up via JVisualVM and create Thread-Dumps or CPU Usage/Memory Usage. You can also setup Eclipse and remote debug into JDownloader. You can contact us via support@jdownloader.org or chat (irc, freenode, #jdteam) for more help. You can also join developer team

All right, here goes, I ran JDownloader in debug mode and attached Eclipse, and then I paused each thread and checked the CPU usage.

For the record, I used this command,

java -Xdebug -Xrunjdwp:transport=dt_socket,address=8001,server=y,suspend=y -jar JDownloader.jar

Which I found here:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
(stackoverflow-dot-com/a/22636491)

Turns out, the offending thread is the "WatchDog: downloadWatchDog" thread.
As for its stack, have a look at the attached screenshot, "JDownloader.Eclipse.debug.high-CPU-thread.2017-01-21_02-49-44.png".

Pausing the thread did not cause the links to start downloading, though.

@Jiaz: The high CPU usage only happens when the downloads are running. When I press the stop button, the CPU load goes away.

New findings: Seems like I just overdid it with the number of links.

I removed everything from the download queue and then added some single items.
Works like a charm for now.

(insert :facepalm: smiley)

Sorry for making such a fuss.

I might look into this anyway – I do suspect that it should be possible to coordinate a virtually unlimited number of download links on today's hardwares, especially in Java which nowadays is a really efficient runtime

If I manage to get anything done I'll tell.

Thank you for your support, much appreciated.

[raztoki]

out of interest how many links did you have in the queue?

the code you had breakpointed was just evaluating links to start downloading. It shouldn't be that CPU intensive.

If you're downloading lots of small files (say images) it could seem to be hogging more CPU as continuously starting new links.

raztoki

Originally Posted by raztoki

out of interest how many links did you have in the queue?

Upwards of 20k maybe, with file sizes in the lower 3 digits of megabytes.
That usually wasn't a problem, though.

I just re-added a rather large package with about 8k links to 4k files (2 hosters), and this seems to trigger the bug. At least the CPU load goes to 100% on one core as soon as I move it from the Linkgrabber to the Download area, and stays at 100%.
The same is the case with a different package containing 5k links to 2.5k files.

Granted that may be a lot of links, but still, that's why one uses JDownloader in the first place

My current workaround is to have the bulk of the links in "disabled" state and enable them as JDownloader progresses at downloading them.

I think there's room for improvement.
Judging from the debug screenshot there seems to be some sort of hash-based lookup thing going on, maybe these hashes are always recalculated? They could perhaps be calculated once and then be kept for the rest of the application runtime. I personally wouldn't mind the memory requirements for keeping some 5-digit number of hashes around.

[raztoki]

hashes are looked for, or calculated, but its used for mirror determination.

large pacakages isn't great, your better off with smaller packages if you can help it.

each time a slots opens, it will compare each many/all of the objects within package (for memory).

raztoki

[Jiaz]

Only (available heap) memory limits the max numbers of links in JDownloader. We have some special users out there that easily reach 2 mio links in JDownloader

Could we do a Teamviewer session so we check JVisualVM and its thread dumps together. Thread Dumps provide more information as you can easily see if a thread has a tight loop in it.

Also there is still room left for optimizations

For example the *find next download link* causes lots of garbage. For example *ugly* use of List.removeAll

So I suggest we do a teamviewer session (contact me at support@jdownloader.org) or you can provide some thread dumps in short sequence.
Also please test the following. Start downloads and set *max downloads* to <= current downloading. Does this reduce cpu usage?

Originally Posted by Jiaz

Start downloads and set *max downloads* to <= current downloading. Does this reduce cpu usage?

Yes, after a few seconds the CPU load drops to sane levels.

I now have an Eclipse workspace with JDownloader checked out from SVN, and it starts up all right as far as I can tell, so I could go wild with breakpoints and such.
Kudos on the howto by the way, much appreciated.

One question here: Where does JDownloader store its user data?
Because it sure did not use the data of my "regular" instance when I launched it in Eclipse.
(Which, in retrospect, wasn't a smart thing to do on my part, considering it might have done. With the version differences and in-development state it might have screwed up my regular instance.)

Anyhow, now that I know the Eclipse instance has its own user data, I'd like to copy the user data of my regular JDownloader instance to the Eclipse workspace one, so I can debug into the real thing.

Maybe we can apply the mighty HashMap and harness its insane lookup powers

[raztoki]

eclipse
workspace is where ever you placed it, for me its c:\workspace\
jd_home will be placed on windows within Users\Username\.jd_home

[Jiaz]

In case you have additional questions, you can contact us via
support@jdownloader.org
IRC, freenode, #jdteam
personal email (ask via support@jdownloader.org)

The loop that selects next download candidate is in
DownloadWatchDog.next(DownloadLinkCandidateSelector selector)
nextDownloadLinkCandidate(DownloadLinkCandidateSelector selector) -> here the ugly list usage happens -> causes lots of garbage
The complete candidate selection is very complex, so instead of trying to figure it out by yourself you should ask/contact us you can reach us in chat

I can also provide little tour in source via teamviewer if you like