[sgghostrider]

First and foremost i shall state that i have no idea of Java(i can read it perfectly fine but could not write a functionality without checking google even to save my life), now that i clarified that lets begin.

I have been having problems downloading deviantart galleries since few months ago and out of frustration (and a little reminder of the source code availability by a dev) i jumped head first to try and see why it was taking so long(sometimes up to minutes) to process 1 single link.

What i found is that on each start of each new link, jdownloader is going through EVERY SINGLE ACTIVE DOWNLOAD in queue before trying to actually download the link that it already started processing.

All of that happens inside:
File: "JDownloader/src/jd/controlling/downloadcontroller/DownloadWatchDog.java"
Line: Around 1455(did quite some System.out.println and other modifications, line num is not exact)
Function: "nextDownloadLinkCandidates()"

It loops the "while (!selector.isStopMarkReached() && newDLStartAllowed(currentSession))" loop for every active link and it can take up to several minutes to finish, which IMHO (and my ignorance) is very unnecessary.

My solution: As i don't know shiet about how jDownloader works i don't know how to get variables like "Max simultaneous downloads" or the Plugins' specific "simultaneous limit" either, so i just put a counter and a fixed limit of 10 to limit the max loop executions.

int maxLoops = 10;
int currentLoops = 0;
while (!selector.isStopMarkReached() && newDLStartAllowed(currentSession) && currentLoops++ < maxLoops)

My suggestion for who know what are they doing:
First: It should be limited by the plugins' "Max simultaneous downloads"
Second: It should be limited by the Global "Max simultaneous downloads"

Result:
It improved the links' processing time from minutes per link to ONLY few seconds.

Note: If i'm wrong in anything said here feel free to correct me

[Jiaz]

@sgghostrider: Do you think it would be possible to send me copy of your download list, so I can optimize this with *real world* list that has this issue? you can contact me via support@jdownloader.org

[pspzockerscene]

I just sent you a list (7k DA items + 2k other items).

[Jiaz]

@pspzockerscene: would be better to have the *original* your packages are way too small

[sgghostrider]

Originally Posted by Jiaz

@pspzockerscene: would be better to have the *original* your packages are way too small

Actually i was not testing with a much larger library myself either, with around 15k items was enough to make the process slow enough for me to find the problematic part just using the usual "current time - previous time" on the start of functions and before their returns to find the one with an unusual execution time.

With 9k items probably the wait would be around half of a minute per processed link, i did not do a comparison with varying quanties of links to check if it scales linearly, but during my testing it appeared like it was. It usually took around 50 seconds to process 15k items with a variation of 5-10 seconds.

Anyway if you still want a bigger list, i will leave a text file attached with the links to the deviantart profiles i used for testing initially(before i started tinkering with the source code) which include 1 additional artist with a sum of around 25k elements or more, but only because jdownloader stops crawling after 10k, actually the last artist has 25k submissions by himself

[pspzockerscene]

Jiaz didn't want to add the links himself - he wanted you to send him your latest downloadlistXXX.zip file for easier testing.

[sgghostrider]

Originally Posted by pspzockerscene

Jiaz didn't want to add the links himself - he wanted you to send him your latest **External links are only visible to Support Staff**... for easier testing.

The thing is that i had them deleted like a week ago or more, but i found a backup in the cfg folder from one of my initial attempts, luckily jdownloader did not purge them yet and i was able to restore them into the program and it resaved them into a new clean backup after closing the program.

The problem is i can't attach it here, filesize too large, but i have uploaded it to wetransfer, the link should last like 7 days(maximum allowed), but should be plenty of time: **External links are only visible to Support Staff****External links are only visible to Support Staff**

[mwl]

Is anyone still looking at this? I have a 600MB downloadList.zip where this is very noticeable - enabling a couple of large packages causes download processing to slow to a crawl.

**External links are only visible to Support Staff****External links are only visible to Support Staff**

There is one group enabled. If I enable the next group, progress slows to a crawl. If I enable the next several groups, things almost come to a stop. I have a script that tracks remaining downloads. With just the one group enabled, in a 15 minute period around 300-400 downloads a processed. With the next one enabled, that drops to around 20.

[pspzockerscene]

Please wait until another developer finds time to look into this.

[JackOfAllTrades]

hello,

I am having an issue when downloading a lot of files, images and videos mainly.

Is there a setting so that if an attempt to connect is made once, or twice, it will skip the file and comeback to it only once everything else has been gone through? Because I just get a lot of files that get stuck (Screenshot attached) on running and finalizing, never actually downloading.

Context; It was the same before I have done some domainrule changes, I set domainrules to be minimum 51, but it eventually gets stuck on 51 or so files... I have 50 proxies and +2gbps connection.

How do I fix this? I am downloading a massive amount of files and I just want to speed up the whole process. Most of them are small pictures.

[Jiaz]

@JackOfAllTrades: already have a guess what is going on. Please provide a log, see https://support.jdownloader.org/de/k...d-session-logs
when this happens and post logID here
Most likely same case as moved thread here

[Jiaz]

@mwl: I will try to solve this next, sorry for long waiting

[JackOfAllTrades]

Originally Posted by Jiaz

@JackOfAllTrades: already have a guess what is going on. Please provide a log, see **External links are only visible to Support Staff**...
when this happens and post logID here
Most likely same case as moved thread here

Hello, I made logs - 23.11.24 05.09.40 <--> 23.11.24 05.09.07 jdlog://3696411370661/

Not sure which one has the issue, but the same file has been "reconnecting" for multiple days. Here's another log if the previous one is empty; 21.11.24 06.53.29 <--> 23.11.24 05.08.25 jdlog://4696411370661/

[JackOfAllTrades]

Originally Posted by Jiaz

@JackOfAllTrades: already have a guess what is going on. Please provide a log, see **External links are only visible to Support Staff**...
when this happens and post logID here
Most likely same case as moved thread here

Hello, any updates on what the issue is or how can i fix it?

[Jiaz]

Originally Posted by Jiaz

@mwl: I will try to solve this next, sorry for long waiting

I'm on it

[JackOfAllTrades]

Originally Posted by Jiaz

I'm on it

Hello,

Since my last post, everything downloaded one day overnight, it's almost something was "turned on" and just like that, files that were idling for weeks had downloaded.

I though the issue is fixed, ran all updates, restarted machine, loaded up many more files thinking of moving on and immediately same issue.

[Jiaz]

Originally Posted by Jiaz

I'm on it

The issue is still being worked on