JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 28.05.2015, 00:30
MrGonzo's Avatar
MrGonzo MrGonzo is offline
JD VIP
 
Join Date: Dec 2010
Posts: 473
Lightbulb Anyone ever noticed this issue?

Hi again guys,

I thought id finally bring up an issue thats been around for quite awhile. Maybe uve all noticed it maybe not. BUT.....here we go....lol

What happens is with almost ANY big/BIG file (1GB-100GB+) DL will start and the DL speed will spike high for a moment then start to slow to a crawl dead! The DL will EVENTUALLY stop dead and give "connection problem" error message.

And the only way ive found to fix it is stop and restart. Sounds simple? Not quite.

With the BIG DLs there is a stop cooldown which can take as much as 2 to 5-8mins depending on the file size (I think). So i have to wait for the cooldown to restart. Once i do get it restarted, the DL speed goes just fine

So my q guys is, do any of u know about this? U seen it? Can u explain what this might be?

Anyway to fix this so I dont have to do this
start--stop---cooldown---start = thing anymore? haha lol

Thxs guys
Reply With Quote
  #2  
Old 28.05.2015, 06:23
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 16,143
Default

I've seen it before, and its most likely a result of preallocation of chunk ranges.

if you use more than 1 chunk additional chunks need to be pre allocated in the following fashion.

example 100gb file
2 chunks = preallocates the first 50gb and expands the last chunk as it downloads
3 chunks = preallocates 66.6gb and expands the last chunk as it downloads
4 chunks = preallocates 75gb and expands the last chunk as it downloads.

algo, x = filesize / number of chunks; preallocation data = (number of chunks - 1) * x;

preallocating on devices in Java takes time, as it has to write to said values. Apparently there is no quick way to do this (I'm not sure just what Jiaz told me in the past). Preallocation takes place once the download starts, because we don't know the final filesize from UNTRUSTED linkchecking routines and very few providers give trusted information.

Typically when this happens on slow write devices it can take several minutes to preallocate, in which the download stops after ~8-10MB (on the systems I've used). On Virtual Disks say from VM or USB or Network drives these tasks can even be slower than a standard physical drive.

This is why I hate chunk resetting code in plugins, it triggers that event because of the several timeout events that happen waiting for preallocation of chunks to finish. Once finished it resets back to 1 chunk and all that time spent waiting is lost!!

Jiaz solution in the new download core is to not preallocate in this fashion, but instead use ranges closer to each other and the beginning of the file, thus less data has to be instantly pre allocated.


raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]

Last edited by raztoki; 28.05.2015 at 07:59.
Reply With Quote
  #3  
Old 28.05.2015, 07:25
MrGonzo's Avatar
MrGonzo MrGonzo is offline
JD VIP
 
Join Date: Dec 2010
Posts: 473
Default

Wow sound a bit complicated. But i kinda get ya. If its a big file, JD2 needs time to "allocate" the file chunks before it can really go. (And yes i do use chucks, 4 max)

But i thought id bring it up and see if anyone has noticed this. Good to know someone else finally said something with me haha. its a known issue....good. But i can tell it will be a bit before a solution is found.

Oh well, ive been doing this workaround for this for awhile now. lol. Hope this cooldown is fixed one day

Thxs Raz

Last edited by MrGonzo; 28.05.2015 at 07:27. Reason: add
Reply With Quote
  #4  
Old 28.05.2015, 10:25
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 63,945
Default

Raztoki explained it already.
At the moment you can either set max connections to 1 or wait for update. I will try to add support for sparse files.
Also the next version of downloadsystem will handle those things much better.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #5  
Old 28.05.2015, 23:27
MrGonzo's Avatar
MrGonzo MrGonzo is offline
JD VIP
 
Join Date: Dec 2010
Posts: 473
Default

Haha no worries Jiaz I got most of what he was saying. And i think ive already got it set to 1.

Good to know sounds like this issue could be fixed sooner then i thought. Great!

Will "wait for core update"

+Was wondering i pretty much only use multihosters. Does that have anything to do with this or is it a JD2 thing?

Last edited by MrGonzo; 28.05.2015 at 23:31. Reason: add
Reply With Quote
  #6  
Old 29.05.2015, 04:41
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 16,143
Default

It's a download core issue, so it effects each plugin that allows more than 1 chunk. When you use multihoster providers they offer you access to premium services which will typically mean access to chunking ability. Assuming you have max chunks above 1 it will preallocate 50% + as described above. The larger the file, the longer it takes.

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #7  
Old 30.05.2015, 06:27
MrGonzo's Avatar
MrGonzo MrGonzo is offline
JD VIP
 
Join Date: Dec 2010
Posts: 473
Default

Thxs Raz & Jiaz. I get ya. Hopefully this new core update will at least help with the issue. Good to know this is an issue that can be fixed from JD2s end.

Haha...dealing with multihoster tech support is a nice guy! lol

Last edited by MrGonzo; 30.05.2015 at 06:33. Reason: add
Reply With Quote
  #8  
Old 30.05.2015, 10:54
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 63,945
Default

__________________
JD-Dev & Server-Admin
Reply With Quote
  #9  
Old 30.05.2015, 22:04
MrGonzo's Avatar
MrGonzo MrGonzo is offline
JD VIP
 
Join Date: Dec 2010
Posts: 473
Default

Thxs again Jiaz

Last edited by MrGonzo; 30.05.2015 at 22:08. Reason: add
Reply With Quote
  #10  
Old 07.06.2015, 00:38
MrGonzo's Avatar
MrGonzo MrGonzo is offline
JD VIP
 
Join Date: Dec 2010
Posts: 473
Default

Hi

Sorry for the dbl post, I just wanted to say thxs u guys. It seems u may have fixed this problem. Since the last 2-3 updates ive noticed that the "cooldown" has been automated. Now JD2 recognizes when a connection fails and it now just auto-resets itself and the DL starts again....full speed.

So thxs again guys for the fix. Its saving me a lot of grief and time.
Reply With Quote
  #11  
Old 07.06.2015, 01:55
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 16,143
Default

sparse support doesn't create timeout situations, so you shouldn't get restart or situations where data reset & chunk reset happens due to silly work arounds within plugins to fix core issue.
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #12  
Old 07.06.2015, 03:37
MrGonzo's Avatar
MrGonzo MrGonzo is offline
JD VIP
 
Join Date: Dec 2010
Posts: 473
Default

Hi

Dont quite get everything ur saying. But just letting u know the issue seems to be fixed. I dont have to wait for it to cooldown myself, it does it for me now and everything is good!
Reply With Quote
  #13  
Old 15.06.2015, 08:45
nickles nickles is offline
Baby Loader
 
Join Date: Nov 2013
Posts: 9
Default

How would I control this option (e.g. turn it off)?

I noticed that even for compressed files (7z, zip, rar, cb?, etc.) the flag is set in Windows. This is not desirable!

Last edited by tony2long; 17.06.2015 at 07:37. Reason: Merge posts, please use edit.
Reply With Quote
  #14  
Old 15.06.2015, 12:02
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 63,945
Default

At the moment it is hardcoded(enabled) for all downloads. I can only make it global on/off. Do you wish an option to disable it? Don't see any reason why not to use it. As it affects multi connections downloads only and the only disadvantage might be higher fragmentation for multichonnection downloads
__________________
JD-Dev & Server-Admin
Reply With Quote
  #15  
Old 16.06.2015, 12:54
nickles nickles is offline
Baby Loader
 
Join Date: Nov 2013
Posts: 9
Default

An option to disable it would be fine because

1) my CLI tool colorizes files with the P flag set differently
2) setting the P flag for compressed files makes no sense.

I'd greatly appreciate such an option!
Reply With Quote
  #16  
Old 16.06.2015, 12:58
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 63,945
Default

__________________
JD-Dev & Server-Admin
Reply With Quote
  #17  
Old 16.06.2015, 17:06
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 16,143
Default

@nickles
re: 1) I don't really see how colour would be an issue?
re: 2) can you please explain what you mean ? are you inferring that content is going to be extracted anyway so it doesn't matter? or compression files serves no benefit to sparse flag. Do you realise that sparse option allows you to start downloading without preallocation issues? see https://board.jdownloader.org/showpo...97&postcount=2
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #18  
Old 16.06.2015, 22:32
nickles nickles is offline
Baby Loader
 
Join Date: Nov 2013
Posts: 9
Default

@raztoki
1) For me, color is an issue.
2) No problem with that, but please reset that flag when the download is complete.
Reply With Quote
  #19  
Old 17.06.2015, 02:41
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 16,143
Default

@nickles
as I understand you can't reset the flag as I understand its controlled by the operating system/filesystem, it indicates that the file was created with sparse. To me the benefit of sparse out weighs any issues introduced by a 3rd party program that displays sparse created files in colour.
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #20  
Old 17.06.2015, 05:32
MrGonzo's Avatar
MrGonzo MrGonzo is offline
JD VIP
 
Join Date: Dec 2010
Posts: 473
Default

Thxs guys for workin hard this issue. Razi please make sure this is "always on" with the option (adv sett) for user preference yes.

Good job guys
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 19:47.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.