[sgghostrider]

First and foremost i shall state that i have no idea of Java(i can read it perfectly fine but could not write a functionality without checking google even to save my life), now that i clarified that lets begin.

I have been having problems downloading deviantart galleries since few months ago and out of frustration (and a little reminder of the source code availability by a dev) i jumped head first to try and see why it was taking so long(sometimes up to minutes) to process 1 single link.

What i found is that on each start of each new link, jdownloader is going through EVERY SINGLE ACTIVE DOWNLOAD in queue before trying to actually download the link that it already started processing.

All of that happens inside:
File: "JDownloader/src/jd/controlling/downloadcontroller/DownloadWatchDog.java"
Line: Around 1455(did quite some System.out.println and other modifications, line num is not exact)
Function: "nextDownloadLinkCandidates()"

It loops the "while (!selector.isStopMarkReached() && newDLStartAllowed(currentSession))" loop for every active link and it can take up to several minutes to finish, which IMHO (and my ignorance) is very unnecessary.

My solution: As i don't know shiet about how jDownloader works i don't know how to get variables like "Max simultaneous downloads" or the Plugins' specific "simultaneous limit" either, so i just put a counter and a fixed limit of 10 to limit the max loop executions.

int maxLoops = 10;
int currentLoops = 0;
while (!selector.isStopMarkReached() && newDLStartAllowed(currentSession) && currentLoops++ < maxLoops)

My suggestion for who know what are they doing:
First: It should be limited by the plugins' "Max simultaneous downloads"
Second: It should be limited by the Global "Max simultaneous downloads"

Result:
It improved the links' processing time from minutes per link to ONLY few seconds.

Note: If i'm wrong in anything said here feel free to correct me

[Jiaz]

@sgghostrider: Do you think it would be possible to send me copy of your download list, so I can optimize this with *real world* list that has this issue? you can contact me via support@jdownloader.org

[pspzockerscene]

I just sent you a list (7k DA items + 2k other items).

[Jiaz]

@pspzockerscene: would be better to have the *original* your packages are way too small

[sgghostrider]

Originally Posted by Jiaz

@pspzockerscene: would be better to have the *original* your packages are way too small

Actually i was not testing with a much larger library myself either, with around 15k items was enough to make the process slow enough for me to find the problematic part just using the usual "current time - previous time" on the start of functions and before their returns to find the one with an unusual execution time.

With 9k items probably the wait would be around half of a minute per processed link, i did not do a comparison with varying quanties of links to check if it scales linearly, but during my testing it appeared like it was. It usually took around 50 seconds to process 15k items with a variation of 5-10 seconds.

Anyway if you still want a bigger list, i will leave a text file attached with the links to the deviantart profiles i used for testing initially(before i started tinkering with the source code) which include 1 additional artist with a sum of around 25k elements or more, but only because jdownloader stops crawling after 10k, actually the last artist has 25k submissions by himself

[pspzockerscene]

Jiaz didn't want to add the links himself - he wanted you to send him your latest downloadlistXXX.zip file for easier testing.

[sgghostrider]

Originally Posted by pspzockerscene

Jiaz didn't want to add the links himself - he wanted you to send him your latest **External links are only visible to Support Staff**... for easier testing.

The thing is that i had them deleted like a week ago or more, but i found a backup in the cfg folder from one of my initial attempts, luckily jdownloader did not purge them yet and i was able to restore them into the program and it resaved them into a new clean backup after closing the program.

The problem is i can't attach it here, filesize too large, but i have uploaded it to wetransfer, the link should last like 7 days(maximum allowed), but should be plenty of time: **External links are only visible to Support Staff****External links are only visible to Support Staff**