JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 19.09.2024, 23:06
sgghostrider sgghostrider is offline
JD Alpha
 
Join Date: Oct 2013
Posts: 21
Lightbulb Link processing stuck for long time on each link when many links in queue

First and foremost i shall state that i have no idea of Java(i can read it perfectly fine but could not write a functionality without checking google even to save my life), now that i clarified that lets begin.

I have been having problems downloading deviantart galleries since few months ago and out of frustration (and a little reminder of the source code availability by a dev) i jumped head first to try and see why it was taking so long(sometimes up to minutes) to process 1 single link.

What i found is that on each start of each new link, jdownloader is going through EVERY SINGLE ACTIVE DOWNLOAD in queue before trying to actually download the link that it already started processing.

All of that happens inside:
File: "JDownloader/src/jd/controlling/downloadcontroller/DownloadWatchDog.java"
Line: Around 1455(did quite some System.out.println and other modifications, line num is not exact)
Function: "nextDownloadLinkCandidates()"

It loops the "while (!selector.isStopMarkReached() && newDLStartAllowed(currentSession))" loop for every active link and it can take up to several minutes to finish, which IMHO (and my ignorance) is very unnecessary.

My solution: As i don't know shiet about how jDownloader works i don't know how to get variables like "Max simultaneous downloads" or the Plugins' specific "simultaneous limit" either, so i just put a counter and a fixed limit of 10 to limit the max loop executions.

Code:
int maxLoops = 10;
int currentLoops = 0;
while (!selector.isStopMarkReached() && newDLStartAllowed(currentSession) && currentLoops++ < maxLoops)

My suggestion for who know what are they doing:
First: It should be limited by the plugins' "Max simultaneous downloads"
Second: It should be limited by the Global "Max simultaneous downloads"

Result:
It improved the links' processing time from minutes per link to ONLY few seconds.

Note: If i'm wrong in anything said here feel free to correct me
Reply With Quote
  #2  
Old 20.09.2024, 13:43
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,326
Default

@sgghostrider: Do you think it would be possible to send me copy of your download list, so I can optimize this with *real world* list that has this issue? you can contact me via support@jdownloader.org
__________________
JD-Dev & Server-Admin
Reply With Quote
  #3  
Old 20.09.2024, 13:58
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 73,523
Default

I just sent you a list (7k DA items + 2k other items).
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #4  
Old 20.09.2024, 14:06
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,326
Default

@pspzockerscene: would be better to have the *original* your packages are way too small
__________________
JD-Dev & Server-Admin
Reply With Quote
  #5  
Old 20.09.2024, 20:28
sgghostrider sgghostrider is offline
JD Alpha
 
Join Date: Oct 2013
Posts: 21
Default

Quote:
Originally Posted by Jiaz View Post
@pspzockerscene: would be better to have the *original* your packages are way too small
Actually i was not testing with a much larger library myself either, with around 15k items was enough to make the process slow enough for me to find the problematic part just using the usual "current time - previous time" on the start of functions and before their returns to find the one with an unusual execution time.

With 9k items probably the wait would be around half of a minute per processed link, i did not do a comparison with varying quanties of links to check if it scales linearly, but during my testing it appeared like it was. It usually took around 50 seconds to process 15k items with a variation of 5-10 seconds.

Anyway if you still want a bigger list, i will leave a text file attached with the links to the deviantart profiles i used for testing initially(before i started tinkering with the source code) which include 1 additional artist with a sum of around 25k elements or more, but only because jdownloader stops crawling after 10k, actually the last artist has 25k submissions by himself
Attached Files
File Type: txt deviantart - random test galleries.txt (493 Bytes, 0 views)
Reply With Quote
  #6  
Old 23.09.2024, 12:48
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 73,523
Default

Jiaz didn't want to add the links himself - he wanted you to send him your latest downloadlistXXX.zip file for easier testing.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #7  
Old 23.09.2024, 15:09
sgghostrider sgghostrider is offline
JD Alpha
 
Join Date: Oct 2013
Posts: 21
Default

Quote:
Originally Posted by pspzockerscene View Post
Jiaz didn't want to add the links himself - he wanted you to send him your latest **External links are only visible to Support Staff**... for easier testing.
The thing is that i had them deleted like a week ago or more, but i found a backup in the cfg folder from one of my initial attempts, luckily jdownloader did not purge them yet and i was able to restore them into the program and it resaved them into a new clean backup after closing the program.

The problem is i can't attach it here, filesize too large, but i have uploaded it to wetransfer, the link should last like 7 days(maximum allowed), but should be plenty of time: **External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #8  
Old 10.11.2024, 09:37
mwl mwl is offline
Linkgrabbing Monster
 
Join Date: Mar 2013
Posts: 83
Default

Is anyone still looking at this? I have a 600MB downloadList.zip where this is very noticeable - enabling a couple of large packages causes download processing to slow to a crawl.

**External links are only visible to Support Staff****External links are only visible to Support Staff**

There is one group enabled. If I enable the next group, progress slows to a crawl. If I enable the next several groups, things almost come to a stop. I have a script that tracks remaining downloads. With just the one group enabled, in a 15 minute period around 300-400 downloads a processed. With the next one enabled, that drops to around 20.
Reply With Quote
  #9  
Old 10.11.2024, 21:47
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 73,523
Default

Please wait until another developer finds time to look into this.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #10  
Old 17.11.2024, 13:29
JackOfAllTrades JackOfAllTrades is offline
Modem User
 
Join Date: Nov 2024
Posts: 4
Default downloads get stuck

hello,

I am having an issue when downloading a lot of files, images and videos mainly.

Is there a setting so that if an attempt to connect is made once, or twice, it will skip the file and comeback to it only once everything else has been gone through? Because I just get a lot of files that get stuck (Screenshot attached) on running and finalizing, never actually downloading.

Context; It was the same before I have done some domainrule changes, I set domainrules to be minimum 51, but it eventually gets stuck on 51 or so files... I have 50 proxies and +2gbps connection.

How do I fix this? I am downloading a massive amount of files and I just want to speed up the whole process. Most of them are small pictures.
Attached Thumbnails
Screenshot 2024-11-17 132710.jpg  
Reply With Quote
  #11  
Old 17.11.2024, 21:58
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,326
Default

@JackOfAllTrades: already have a guess what is going on. Please provide a log, see https://support.jdownloader.org/de/k...d-session-logs
when this happens and post logID here
Most likely same case as moved thread here
__________________
JD-Dev & Server-Admin
Reply With Quote
  #12  
Old 17.11.2024, 21:58
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,326
Default

@mwl: I will try to solve this next, sorry for long waiting
__________________
JD-Dev & Server-Admin
Reply With Quote
  #13  
Old 24.11.2024, 14:28
JackOfAllTrades JackOfAllTrades is offline
Modem User
 
Join Date: Nov 2024
Posts: 4
Default

Quote:
Originally Posted by Jiaz View Post
@JackOfAllTrades: already have a guess what is going on. Please provide a log, see **External links are only visible to Support Staff**...
when this happens and post logID here
Most likely same case as moved thread here
Hello, I made logs - 23.11.24 05.09.40 <--> 23.11.24 05.09.07 jdlog://3696411370661/

Not sure which one has the issue, but the same file has been "reconnecting" for multiple days. Here's another log if the previous one is empty; 21.11.24 06.53.29 <--> 23.11.24 05.08.25 jdlog://4696411370661/
Reply With Quote
  #14  
Old 29.11.2024, 10:55
JackOfAllTrades JackOfAllTrades is offline
Modem User
 
Join Date: Nov 2024
Posts: 4
Default

Quote:
Originally Posted by Jiaz View Post
@JackOfAllTrades: already have a guess what is going on. Please provide a log, see **External links are only visible to Support Staff**...
when this happens and post logID here
Most likely same case as moved thread here
Hello, any updates on what the issue is or how can i fix it?
Reply With Quote
  #15  
Old 29.11.2024, 10:58
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,326
Default

Quote:
Originally Posted by Jiaz View Post
@mwl: I will try to solve this next, sorry for long waiting
I'm on it
__________________
JD-Dev & Server-Admin
Reply With Quote
  #16  
Old 08.12.2024, 22:32
JackOfAllTrades JackOfAllTrades is offline
Modem User
 
Join Date: Nov 2024
Posts: 4
Default

Quote:
Originally Posted by Jiaz View Post
I'm on it
Hello,

Since my last post, everything downloaded one day overnight, it's almost something was "turned on" and just like that, files that were idling for weeks had downloaded.

I though the issue is fixed, ran all updates, restarted machine, loaded up many more files thinking of moving on and immediately same issue.
Reply With Quote
  #17  
Old 09.12.2024, 12:30
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,326
Default

Quote:
Originally Posted by Jiaz View Post
I'm on it
The issue is still being worked on
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 18:42.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.