[sgghostrider]

First and foremost i shall state that i have no idea of Java(i can read it perfectly fine but could not write a functionality without checking google even to save my life), now that i clarified that lets begin.

I have been having problems downloading deviantart galleries since few months ago and out of frustration (and a little reminder of the source code availability by a dev) i jumped head first to try and see why it was taking so long(sometimes up to minutes) to process 1 single link.

What i found is that on each start of each new link, jdownloader is going through EVERY SINGLE ACTIVE DOWNLOAD in queue before trying to actually download the link that it already started processing.

All of that happens inside:
File: "JDownloader/src/jd/controlling/downloadcontroller/DownloadWatchDog.java"
Line: Around 1455(did quite some System.out.println and other modifications, line num is not exact)
Function: "nextDownloadLinkCandidates()"

It loops the "while (!selector.isStopMarkReached() && newDLStartAllowed(currentSession))" loop for every active link and it can take up to several minutes to finish, which IMHO (and my ignorance) is very unnecessary.

My solution: As i don't know shiet about how jDownloader works i don't know how to get variables like "Max simultaneous downloads" or the Plugins' specific "simultaneous limit" either, so i just put a counter and a fixed limit of 10 to limit the max loop executions.

int maxLoops = 10;
int currentLoops = 0;
while (!selector.isStopMarkReached() && newDLStartAllowed(currentSession) && currentLoops++ < maxLoops)

My suggestion for who know what are they doing:
First: It should be limited by the plugins' "Max simultaneous downloads"
Second: It should be limited by the Global "Max simultaneous downloads"

Result:
It improved the links' processing time from minutes per link to ONLY few seconds.

Note: If i'm wrong in anything said here feel free to correct me

[Jiaz]

@sgghostrider: Do you think it would be possible to send me copy of your download list, so I can optimize this with *real world* list that has this issue? you can contact me via support@jdownloader.org

[pspzockerscene]

I just sent you a list (7k DA items + 2k other items).

[Jiaz]

@pspzockerscene: would be better to have the *original* your packages are way too small

[sgghostrider]

Originally Posted by Jiaz

@pspzockerscene: would be better to have the *original* your packages are way too small

Actually i was not testing with a much larger library myself either, with around 15k items was enough to make the process slow enough for me to find the problematic part just using the usual "current time - previous time" on the start of functions and before their returns to find the one with an unusual execution time.

With 9k items probably the wait would be around half of a minute per processed link, i did not do a comparison with varying quanties of links to check if it scales linearly, but during my testing it appeared like it was. It usually took around 50 seconds to process 15k items with a variation of 5-10 seconds.

Anyway if you still want a bigger list, i will leave a text file attached with the links to the deviantart profiles i used for testing initially(before i started tinkering with the source code) which include 1 additional artist with a sum of around 25k elements or more, but only because jdownloader stops crawling after 10k, actually the last artist has 25k submissions by himself

[pspzockerscene]

Jiaz didn't want to add the links himself - he wanted you to send him your latest downloadlistXXX.zip file for easier testing.

[sgghostrider]

Originally Posted by pspzockerscene

Jiaz didn't want to add the links himself - he wanted you to send him your latest **External links are only visible to Support Staff**... for easier testing.

The thing is that i had them deleted like a week ago or more, but i found a backup in the cfg folder from one of my initial attempts, luckily jdownloader did not purge them yet and i was able to restore them into the program and it resaved them into a new clean backup after closing the program.

The problem is i can't attach it here, filesize too large, but i have uploaded it to wetransfer, the link should last like 7 days(maximum allowed), but should be plenty of time: **External links are only visible to Support Staff****External links are only visible to Support Staff**

[mwl]

Is anyone still looking at this? I have a 600MB downloadList.zip where this is very noticeable - enabling a couple of large packages causes download processing to slow to a crawl.

**External links are only visible to Support Staff****External links are only visible to Support Staff**

There is one group enabled. If I enable the next group, progress slows to a crawl. If I enable the next several groups, things almost come to a stop. I have a script that tracks remaining downloads. With just the one group enabled, in a 15 minute period around 300-400 downloads a processed. With the next one enabled, that drops to around 20.

[pspzockerscene]

Please wait until another developer finds time to look into this.

[JackOfAllTrades]

hello,

I am having an issue when downloading a lot of files, images and videos mainly.

Is there a setting so that if an attempt to connect is made once, or twice, it will skip the file and comeback to it only once everything else has been gone through? Because I just get a lot of files that get stuck (Screenshot attached) on running and finalizing, never actually downloading.

Context; It was the same before I have done some domainrule changes, I set domainrules to be minimum 51, but it eventually gets stuck on 51 or so files... I have 50 proxies and +2gbps connection.

How do I fix this? I am downloading a massive amount of files and I just want to speed up the whole process. Most of them are small pictures.

[Jiaz]

@JackOfAllTrades: already have a guess what is going on. Please provide a log, see https://support.jdownloader.org/de/k...d-session-logs
when this happens and post logID here
Most likely same case as moved thread here

[Jiaz]

@mwl: I will try to solve this next, sorry for long waiting

[JackOfAllTrades]

Originally Posted by Jiaz

@JackOfAllTrades: already have a guess what is going on. Please provide a log, see **External links are only visible to Support Staff**...
when this happens and post logID here
Most likely same case as moved thread here

Hello, I made logs - 23.11.24 05.09.40 <--> 23.11.24 05.09.07 jdlog://3696411370661/

Not sure which one has the issue, but the same file has been "reconnecting" for multiple days. Here's another log if the previous one is empty; 21.11.24 06.53.29 <--> 23.11.24 05.08.25 jdlog://4696411370661/

[JackOfAllTrades]

Originally Posted by Jiaz

@JackOfAllTrades: already have a guess what is going on. Please provide a log, see **External links are only visible to Support Staff**...
when this happens and post logID here
Most likely same case as moved thread here

Hello, any updates on what the issue is or how can i fix it?

[Jiaz]

Originally Posted by Jiaz

@mwl: I will try to solve this next, sorry for long waiting

I'm on it

[JackOfAllTrades]

Originally Posted by Jiaz

I'm on it

Hello,

Since my last post, everything downloaded one day overnight, it's almost something was "turned on" and just like that, files that were idling for weeks had downloaded.

I though the issue is fixed, ran all updates, restarted machine, loaded up many more files thinking of moving on and immediately same issue.

[Jiaz]

Originally Posted by Jiaz

I'm on it

The issue is still being worked on

[JackOfAllTrades]

Originally Posted by Jiaz

The issue is still being worked on

Hello, any updates?

[Jiaz]

Originally Posted by JackOfAllTrades

Hello, any updates?

Not yet, but I'm making good progress

[MasterCATZ]

Thanks , let me know when you want a test run to see if they stop getting hung on Finalizing ,

also why is Finalizing the first step ? I would have thought that was a process after a download was done ? for a while their I was thinking it was a crc check but nope its happening before any information saved to disk


for now I am getting by doing the pause trick ...

currently it stopped hanging when I set 10x downloads at a time instead of 1x 2x , but I also used pause / un-pause once

I am just about to close jdownloader and try again to see which solved it

*edit*
maybe in my case it was the decade old standalone java that was bundled with original install
did not realize it was not using system java
jre-8u441-linux-x64 it behaved since tossing that into the jre folder


I have now knocked it down to 10x with only 1x allowed from each host , just about to knock it all back down to 1x and 2x and see what happens

so far 1 is hanging on finalizing again ,
but their are some incomplete downloads in the que I am waiting to complete before restarting

I also noticed when this finalizing hang starts extraction of downloaded files also seems to be delayed

also wish their was a way to know what proxies are being used ?


*edit*
when going back to only having a few download limits it started hanging again "Finalizing" ,
went back to 10x and it just kept cruising on again , without noticing "Finalizing" at any stage


also only using chunk size of 1x


8gb memory usage 12gb allocated 28gb max , no idea how to increase it further than that 500Gb Free

-Xmx32g -Xmx64g were tried it seems to ignore those settings ?

[Jiaz]

Originally Posted by MasterCATZ

also why is Finalizing the first step ?

thanks for the hint, just wrong text shown, will be fixed with next update.

[raztoki]

Originally Posted by MasterCATZ

also wish their was a way to know what proxies are being used ?

yes mouse over the connection column of an item that downloading

[MasterCATZ]

yes but that takes forever to pop up and I can only find the current section in que when their is a download by sorting by speed them backtracking over the download logs


any quicker way to ID what proxies need IP adress rotated
**External links are only visible to Support Staff****External links are only visible to Support Staff**


the red thumbs down hardly ever triggers when bandwidths exceeded so I can not use that either

[Jiaz]

Originally Posted by MasterCATZ

any quicker way to ID what proxies need IP adress rotated

What do you mean by that? Why not just add more proxies? and let JDownloader use/try the next best connection? or do you manually change the IP of the proxy endpoint?

What exactly do you have in mind? What do you want to achieve? We can help better then.

[mgpai]

Originally Posted by MasterCATZ

... that takes forever to pop up ...

settings > advanced settings > GraphicalUserInterfaceSettings.tooltipdelay

[raztoki]

Originally Posted by mgpai

settings > advanced settings > GraphicalUserInterfaceSettings.tooltipdelay

or double click in said column & row and it comes up instantly