JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 19.09.2024, 23:06
sgghostrider sgghostrider is offline
JD Alpha
 
Join Date: Oct 2013
Posts: 21
Lightbulb Link processing stuck for long time on each link when many links in queue

First and foremost i shall state that i have no idea of Java(i can read it perfectly fine but could not write a functionality without checking google even to save my life), now that i clarified that lets begin.

I have been having problems downloading deviantart galleries since few months ago and out of frustration (and a little reminder of the source code availability by a dev) i jumped head first to try and see why it was taking so long(sometimes up to minutes) to process 1 single link.

What i found is that on each start of each new link, jdownloader is going through EVERY SINGLE ACTIVE DOWNLOAD in queue before trying to actually download the link that it already started processing.

All of that happens inside:
File: "JDownloader/src/jd/controlling/downloadcontroller/DownloadWatchDog.java"
Line: Around 1455(did quite some System.out.println and other modifications, line num is not exact)
Function: "nextDownloadLinkCandidates()"

It loops the "while (!selector.isStopMarkReached() && newDLStartAllowed(currentSession))" loop for every active link and it can take up to several minutes to finish, which IMHO (and my ignorance) is very unnecessary.

My solution: As i don't know shiet about how jDownloader works i don't know how to get variables like "Max simultaneous downloads" or the Plugins' specific "simultaneous limit" either, so i just put a counter and a fixed limit of 10 to limit the max loop executions.

Code:
int maxLoops = 10;
int currentLoops = 0;
while (!selector.isStopMarkReached() && newDLStartAllowed(currentSession) && currentLoops++ < maxLoops)

My suggestion for who know what are they doing:
First: It should be limited by the plugins' "Max simultaneous downloads"
Second: It should be limited by the Global "Max simultaneous downloads"

Result:
It improved the links' processing time from minutes per link to ONLY few seconds.

Note: If i'm wrong in anything said here feel free to correct me
Reply With Quote
  #2  
Old 20.09.2024, 13:43
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,049
Default

@sgghostrider: Do you think it would be possible to send me copy of your download list, so I can optimize this with *real world* list that has this issue? you can contact me via support@jdownloader.org
__________________
JD-Dev & Server-Admin
Reply With Quote
  #3  
Old 20.09.2024, 13:58
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 73,118
Default

I just sent you a list (7k DA items + 2k other items).
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #4  
Old 20.09.2024, 14:06
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,049
Default

@pspzockerscene: would be better to have the *original* your packages are way too small
__________________
JD-Dev & Server-Admin
Reply With Quote
  #5  
Old 20.09.2024, 20:28
sgghostrider sgghostrider is offline
JD Alpha
 
Join Date: Oct 2013
Posts: 21
Default

Quote:
Originally Posted by Jiaz View Post
@pspzockerscene: would be better to have the *original* your packages are way too small
Actually i was not testing with a much larger library myself either, with around 15k items was enough to make the process slow enough for me to find the problematic part just using the usual "current time - previous time" on the start of functions and before their returns to find the one with an unusual execution time.

With 9k items probably the wait would be around half of a minute per processed link, i did not do a comparison with varying quanties of links to check if it scales linearly, but during my testing it appeared like it was. It usually took around 50 seconds to process 15k items with a variation of 5-10 seconds.

Anyway if you still want a bigger list, i will leave a text file attached with the links to the deviantart profiles i used for testing initially(before i started tinkering with the source code) which include 1 additional artist with a sum of around 25k elements or more, but only because jdownloader stops crawling after 10k, actually the last artist has 25k submissions by himself
Attached Files
File Type: txt deviantart - random test galleries.txt (493 Bytes, 0 views)
Reply With Quote
  #6  
Old 23.09.2024, 12:48
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 73,118
Default

Jiaz didn't want to add the links himself - he wanted you to send him your latest downloadlistXXX.zip file for easier testing.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #7  
Old 23.09.2024, 15:09
sgghostrider sgghostrider is offline
JD Alpha
 
Join Date: Oct 2013
Posts: 21
Default

Quote:
Originally Posted by pspzockerscene View Post
Jiaz didn't want to add the links himself - he wanted you to send him your latest **External links are only visible to Support Staff**... for easier testing.
The thing is that i had them deleted like a week ago or more, but i found a backup in the cfg folder from one of my initial attempts, luckily jdownloader did not purge them yet and i was able to restore them into the program and it resaved them into a new clean backup after closing the program.

The problem is i can't attach it here, filesize too large, but i have uploaded it to wetransfer, the link should last like 7 days(maximum allowed), but should be plenty of time: **External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 08:24.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.