JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 30.09.2016, 12:56
madcowpig's Avatar
madcowpig madcowpig is offline
Storm
 
Join Date: Aug 2016
Posts: 210
Default Checksum simultaneous download issue/request

When I set for example max simultaneous downloads to ten and start twenty downloads everything fine. When it finishes the download SHA1 checksum start. Let's say three downloads finish and SHA1 start and two are waiting for checksum, only seven of the other downloads are working. I want to be able to have the files wait in queue for checksum while ten other simultaneous downloads work. I can only get this to work with force download which is annoying. I want this to be automatic. I searched the forums and advanced settings and found nothing. If there is a way, please tell me how. If there isn't, please move this thread to request section. If this is bug, then please move to Problems & Bug Reports section. Thank you.
__________________
I don't fight the system, but I do play fighting games on it. PS4 to be exact. On second thought, sometimes it won't work right and I have to hit it a couple of times before it's fixed. So technically, I do fight the system.

Last edited by madcowpig; 15.01.2017 at 20:30.
Reply With Quote
  #2  
Old 30.09.2016, 13:14
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 68,214
Default

Hashcheck is part of download process because the result is required for the final state of the download. That's why the download slot is counted as active. By forcing downloads you simply increase max concurrent download slots.

How fast is your download speed vs your disk speed? Do you download on local or remote drive?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #3  
Old 30.09.2016, 13:19
madcowpig's Avatar
madcowpig madcowpig is offline
Storm
 
Join Date: Aug 2016
Posts: 210
Default

Also if this gets moved to request, there are two other issues with checksum I'd like to be fixed. When downloads finish checksum, you see this,



Then extraction starts and you see this,



When extraction finishes, you see this,



The issue is, I wanna see both Extraction OK and [SHA1] CRC OK. This,



disappears after extraction, so I can't tell if checksum was correct or not.

The other issue is, if I start CRC via right click menu, "Other", "Start CRC task!", it's significantly slower than if I have it set to auto.
__________________
I don't fight the system, but I do play fighting games on it. PS4 to be exact. On second thought, sometimes it won't work right and I have to hit it a couple of times before it's fixed. So technically, I do fight the system.

Last edited by madcowpig; 15.01.2017 at 20:31. Reason: Added a comma.
Reply With Quote
  #4  
Old 30.09.2016, 13:28
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 68,214
Default

I will update column to show both results at once, would that be okay for you?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #5  
Old 30.09.2016, 13:31
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 68,214
Default

Hashcheck after download and manually via *Start CRC Task* are the same. Only difference is that check after download is blocking other checks and manual one runs in parallel. so with download/crc check activity there are of course speed differences.
Also depending on your system, more data might be left in buffer and not needed to read from disk after download than when doing it manually
__________________
JD-Dev & Server-Admin
Reply With Quote
  #6  
Old 30.09.2016, 13:41
madcowpig's Avatar
madcowpig madcowpig is offline
Storm
 
Join Date: Aug 2016
Posts: 210
Default

Quote:
Originally Posted by Jiaz View Post
Hashcheck is part of download process because the result is required for the final state of the download. That's why the download slot is counted as active. By forcing downloads you simply increase max concurrent download slots.
I understand one download slot being used for a hash check, but the issue is when there's multiple in queue. For example,



You see the top finished download is running a hash check, which is fine, but then there are ten other finished downloads, waiting for a hash check and taking up simultaneous download slots. I want to be able to use those download slots for downloading and reserve only one for a hash check, so when it's time to run a hash check on the finished downloads in queue, only one moves into that reserved spot at a time.

This greatly reduces my download speed, because I can't simultaneously download.

Quote:
Originally Posted by Jiaz View Post
How fast is your download speed vs your disk speed? Do you download on local or remote drive?
My download speed is about 170 Mb/s, so about 21.25 MB/s. I usually cap my download speed at 12 MB/s. I don't know about my disc speed. How do I find out my disk speed? I download on a local drive.
__________________
I don't fight the system, but I do play fighting games on it. PS4 to be exact. On second thought, sometimes it won't work right and I have to hit it a couple of times before it's fixed. So technically, I do fight the system.

Last edited by madcowpig; 15.01.2017 at 20:32.
Reply With Quote
  #7  
Old 30.09.2016, 13:43
madcowpig's Avatar
madcowpig madcowpig is offline
Storm
 
Join Date: Aug 2016
Posts: 210
Default

Quote:
Originally Posted by Jiaz View Post
I will update column to show both results at once, would that be okay for you?
This would be greatly appreciated, thank you very much.
__________________
I don't fight the system, but I do play fighting games on it. PS4 to be exact. On second thought, sometimes it won't work right and I have to hit it a couple of times before it's fixed. So technically, I do fight the system.

Last edited by madcowpig; 15.01.2017 at 20:32.
Reply With Quote
  #8  
Old 30.09.2016, 13:46
madcowpig's Avatar
madcowpig madcowpig is offline
Storm
 
Join Date: Aug 2016
Posts: 210
Default

Quote:
Originally Posted by Jiaz View Post
Hashcheck after download and manually via *Start CRC Task* are the same. Only difference is that check after download is blocking other checks and manual one runs in parallel. so with download/crc check activity there are of course speed differences.
I was speaking about only one hash check at a time, but I might be mistaken, since I tested a long time ago. I'll have to retest, to make sure.

Quote:
Originally Posted by Jiaz View Post
Also depending on your system, more data might be left in buffer and not needed to read from disk after download than when doing it manually
I don't understand what this means at all.
__________________
I don't fight the system, but I do play fighting games on it. PS4 to be exact. On second thought, sometimes it won't work right and I have to hit it a couple of times before it's fixed. So technically, I do fight the system.

Last edited by madcowpig; 15.01.2017 at 20:32.
Reply With Quote
  #9  
Old 30.09.2016, 14:50
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 68,214
Default

It is not ONE download slot. It is THE download slot.
Each download has its own slot and hashcheck is part of the download process within that slot.

Why does your system not handle that *not much* IO load that bad? You write at 12 or 21 Mbyte/s but your system is unable to provide higher read speeds? Your screenshot shows that it very long time for 900 MB File? Screenshot shows over 2 mins remaining?
Even at 20 Mbyte/s read speed it would only take 45 secs. System should be able to reach 50-75Mbyte/s easily!?

You can try to increase buffer size in Advanced Settings (please note that this buffer is for each chunk).
__________________
JD-Dev & Server-Admin
Reply With Quote
  #10  
Old 30.09.2016, 16:04
madcowpig's Avatar
madcowpig madcowpig is offline
Storm
 
Join Date: Aug 2016
Posts: 210
Default

Quote:
Originally Posted by Jiaz View Post
It is not ONE download slot. It is THE download slot.
Each download has its own slot and hashcheck is part of the download process within that slot.
I completely understand this. I just didn't explain well enough before. What I meant was, say I have one finished download running a hash check and ten finished downloads waiting in queue, like this,



I want JD to disable the other ten until the first hash check finishes, then enable them one at a time, to finish the hash checks singularly, so I can use all downloading slots for downloading, except the one reserved for the hash checks. Does that make sense?

Quote:
Originally Posted by Jiaz View Post
Why does your system not handle that *not much* IO load that bad? You write at 12 or 21 Mbyte/s but your system is unable to provide higher read speeds? Your screenshot shows that it very long time for 900 MB File? Screenshot shows over 2 mins remaining?
Even at 20 Mbyte/s read speed it would only take 45 secs.
HAHAHAHAHAHAHAHA Your confusion is 100% my fault. HAHAHAHAHAHAHAHA I'm so sorry, let me explain. The reason the screen cap shows this, is because I only took a partial screen cap, so you couldn't see the whole picture. If you could've seen the whole picture, you would've also seen that I had nine more forced downloads running, with ten connections each, as well as a four file, 3.4 GB extraction process running, while watching a 1080p video on YouTube, at the same time as the checksum. Of course, a checksum by itself, is very fast.

Quote:
Originally Posted by Jiaz View Post
System should be able to reach 50-75Mbyte/s easily!?
You're right on the money. I checked the task manager and JD was using 75 MB/s, which is 100% Disk for my system.
__________________
I don't fight the system, but I do play fighting games on it. PS4 to be exact. On second thought, sometimes it won't work right and I have to hit it a couple of times before it's fixed. So technically, I do fight the system.

Last edited by madcowpig; 15.01.2017 at 20:32.
Reply With Quote
  #11  
Old 30.09.2016, 16:56
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 68,214
Default

I will think about a solution (will be an advanced setting for sure) to exclude pending hashchecks from active slot calculations
__________________
JD-Dev & Server-Admin
Reply With Quote
  #12  
Old 30.09.2016, 16:57
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 68,214
Default

__________________
JD-Dev & Server-Admin
Reply With Quote
  #13  
Old 30.09.2016, 17:45
madcowpig's Avatar
madcowpig madcowpig is offline
Storm
 
Join Date: Aug 2016
Posts: 210
Default

Quote:
Originally Posted by Jiaz View Post
I will think about a solution (will be an advanced setting for sure) to exclude pending hashchecks from active slot calculations
Wow, thanks a lot, man. Although, I'm trying to come up with some type of workaround in the mean time. See, I was downloading 92 files, set to, 10 simultaneous downloads, with 10 connections each. Broken down to, 23 packages with 4 files each, set to automatically hash check after each file finished downloading and automatically extract, after each package finished all hash checks. Is there a way to set automatic extract to start after all packages finish, instead of a singular package? I believe this might help speed up the checksums, so the simultaneous downloads will start quicker, atleast until you add this fix to the advanced settings.
__________________
I don't fight the system, but I do play fighting games on it. PS4 to be exact. On second thought, sometimes it won't work right and I have to hit it a couple of times before it's fixed. So technically, I do fight the system.

Last edited by madcowpig; 15.01.2017 at 20:33. Reason: Added a comma.
Reply With Quote
  #14  
Old 30.09.2016, 17:51
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 68,214
Default

Why not simple disable hashcheck and then write/use eventscripter to wait for downloads to be finished and then extract all of them. Mgpai has done a lot of work in this field!
__________________
JD-Dev & Server-Admin
Reply With Quote
  #15  
Old 30.09.2016, 18:12
madcowpig's Avatar
madcowpig madcowpig is offline
Storm
 
Join Date: Aug 2016
Posts: 210
Default

Again, thanks for the advice. I'll look in to it. I prefer the advanced option though, because it's all within JD and automatic. Also, you just put a lot of pressure on this "Mgpai" person.
__________________
I don't fight the system, but I do play fighting games on it. PS4 to be exact. On second thought, sometimes it won't work right and I have to hit it a couple of times before it's fixed. So technically, I do fight the system.

Last edited by madcowpig; 15.01.2017 at 20:34. Reason: Added a comma.
Reply With Quote
  #16  
Old 30.09.2016, 18:16
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 68,214
Default

I'm sure mgpai is able to say when he has no time I'm just saying that he creates really cool stuff with help of the eventscripter
__________________
JD-Dev & Server-Admin
Reply With Quote
  #17  
Old 30.09.2016, 18:20
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 991
Default

@madcowpig

You could use "GeneralSettings: Auto Max Downloads Speed Limit" as a workaround till the advanced setting is implemented.
Reply With Quote
  #18  
Old 30.09.2016, 18:21
madcowpig's Avatar
madcowpig madcowpig is offline
Storm
 
Join Date: Aug 2016
Posts: 210
Default

So, I'm assuming this Event Scripter thing is a tool within jdownloader.org, right?
__________________
I don't fight the system, but I do play fighting games on it. PS4 to be exact. On second thought, sometimes it won't work right and I have to hit it a couple of times before it's fixed. So technically, I do fight the system.

Last edited by madcowpig; 15.01.2017 at 20:40. Reason: Removed a comma
Reply With Quote
  #19  
Old 30.09.2016, 18:26
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 68,214
Default

Its an extension to JDownloader, see Settings, scroll down.
It listens to specific events and then executes customized scripts (in javascript)
__________________
JD-Dev & Server-Admin
Reply With Quote
  #20  
Old 30.09.2016, 18:29
madcowpig's Avatar
madcowpig madcowpig is offline
Storm
 
Join Date: Aug 2016
Posts: 210
Default

Quote:
Originally Posted by mgpai View Post
@madcowpig

You could use "GeneralSettings: Auto Max Downloads Speed Limit" as a workaround till the advanced setting is implemented.
What does it do specifically?
__________________
I don't fight the system, but I do play fighting games on it. PS4 to be exact. On second thought, sometimes it won't work right and I have to hit it a couple of times before it's fixed. So technically, I do fight the system.

Last edited by madcowpig; 15.01.2017 at 20:42. Reason: Added a question mark.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 02:30.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.