JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 28.05.2015, 00:30
MrGonzo's Avatar
MrGonzo MrGonzo is offline
JD Legend
 
Join Date: Dec 2010
Posts: 613
Lightbulb Anyone ever noticed this issue?

Hi again guys,

I thought id finally bring up an issue thats been around for quite awhile. Maybe uve all noticed it maybe not. BUT.....here we go....lol

What happens is with almost ANY big/BIG file (1GB-100GB+) DL will start and the DL speed will spike high for a moment then start to slow to a crawl dead! The DL will EVENTUALLY stop dead and give "connection problem" error message.

And the only way ive found to fix it is stop and restart. Sounds simple? Not quite.

With the BIG DLs there is a stop cooldown which can take as much as 2 to 5-8mins depending on the file size (I think). So i have to wait for the cooldown to restart. Once i do get it restarted, the DL speed goes just fine

So my q guys is, do any of u know about this? U seen it? Can u explain what this might be?

Anyway to fix this so I dont have to do this
start--stop---cooldown---start = thing anymore? haha lol

Thxs guys
Reply With Quote
  #2  
Old 28.05.2015, 06:23
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,668
Default

I've seen it before, and its most likely a result of preallocation of chunk ranges.

if you use more than 1 chunk additional chunks need to be pre allocated in the following fashion.

example 100gb file
2 chunks = preallocates the first 50gb and expands the last chunk as it downloads
3 chunks = preallocates 66.6gb and expands the last chunk as it downloads
4 chunks = preallocates 75gb and expands the last chunk as it downloads.

algo, x = filesize / number of chunks; preallocation data = (number of chunks - 1) * x;

preallocating on devices in Java takes time, as it has to write to said values. Apparently there is no quick way to do this (I'm not sure just what Jiaz told me in the past). Preallocation takes place once the download starts, because we don't know the final filesize from UNTRUSTED linkchecking routines and very few providers give trusted information.

Typically when this happens on slow write devices it can take several minutes to preallocate, in which the download stops after ~8-10MB (on the systems I've used). On Virtual Disks say from VM or USB or Network drives these tasks can even be slower than a standard physical drive.

This is why I hate chunk resetting code in plugins, it triggers that event because of the several timeout events that happen waiting for preallocation of chunks to finish. Once finished it resets back to 1 chunk and all that time spent waiting is lost!!

Jiaz solution in the new download core is to not preallocate in this fashion, but instead use ranges closer to each other and the beginning of the file, thus less data has to be instantly pre allocated.


raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]

Last edited by raztoki; 28.05.2015 at 07:59.
Reply With Quote
  #3  
Old 28.05.2015, 07:25
MrGonzo's Avatar
MrGonzo MrGonzo is offline
JD Legend
 
Join Date: Dec 2010
Posts: 613
Default

Wow sound a bit complicated. But i kinda get ya. If its a big file, JD2 needs time to "allocate" the file chunks before it can really go. (And yes i do use chucks, 4 max)

But i thought id bring it up and see if anyone has noticed this. Good to know someone else finally said something with me haha. its a known issue....good. But i can tell it will be a bit before a solution is found.

Oh well, ive been doing this workaround for this for awhile now. lol. Hope this cooldown is fixed one day

Thxs Raz

Last edited by MrGonzo; 28.05.2015 at 07:27. Reason: add
Reply With Quote
  #4  
Old 28.05.2015, 10:25
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,013
Default

Raztoki explained it already.
At the moment you can either set max connections to 1 or wait for update. I will try to add support for sparse files.
Also the next version of downloadsystem will handle those things much better.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #5  
Old 28.05.2015, 23:27
MrGonzo's Avatar
MrGonzo MrGonzo is offline
JD Legend
 
Join Date: Dec 2010
Posts: 613
Default

Haha no worries Jiaz I got most of what he was saying. And i think ive already got it set to 1.

Good to know sounds like this issue could be fixed sooner then i thought. Great!

Will "wait for core update"

+Was wondering i pretty much only use multihosters. Does that have anything to do with this or is it a JD2 thing?

Last edited by MrGonzo; 28.05.2015 at 23:31. Reason: add
Reply With Quote
  #6  
Old 29.05.2015, 04:41
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,668
Default

It's a download core issue, so it effects each plugin that allows more than 1 chunk. When you use multihoster providers they offer you access to premium services which will typically mean access to chunking ability. Assuming you have max chunks above 1 it will preallocate 50% + as described above. The larger the file, the longer it takes.

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #7  
Old 30.05.2015, 06:27
MrGonzo's Avatar
MrGonzo MrGonzo is offline
JD Legend
 
Join Date: Dec 2010
Posts: 613
Default

Thxs Raz & Jiaz. I get ya. Hopefully this new core update will at least help with the issue. Good to know this is an issue that can be fixed from JD2s end.

Haha...dealing with multihoster tech support is a nice guy! lol

Last edited by MrGonzo; 30.05.2015 at 06:33. Reason: add
Reply With Quote
  #8  
Old 30.05.2015, 10:54
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,013
Default

__________________
JD-Dev & Server-Admin
Reply With Quote
  #9  
Old 30.05.2015, 22:04
MrGonzo's Avatar
MrGonzo MrGonzo is offline
JD Legend
 
Join Date: Dec 2010
Posts: 613
Default

Thxs again Jiaz

Last edited by MrGonzo; 30.05.2015 at 22:08. Reason: add
Reply With Quote
  #10  
Old 07.06.2015, 00:38
MrGonzo's Avatar
MrGonzo MrGonzo is offline
JD Legend
 
Join Date: Dec 2010
Posts: 613
Default

Hi

Sorry for the dbl post, I just wanted to say thxs u guys. It seems u may have fixed this problem. Since the last 2-3 updates ive noticed that the "cooldown" has been automated. Now JD2 recognizes when a connection fails and it now just auto-resets itself and the DL starts again....full speed.

So thxs again guys for the fix. Its saving me a lot of grief and time.
Reply With Quote
  #11  
Old 07.06.2015, 01:55
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,668
Default

sparse support doesn't create timeout situations, so you shouldn't get restart or situations where data reset & chunk reset happens due to silly work arounds within plugins to fix core issue.
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #12  
Old 07.06.2015, 03:37
MrGonzo's Avatar
MrGonzo MrGonzo is offline
JD Legend
 
Join Date: Dec 2010
Posts: 613
Default

Hi

Dont quite get everything ur saying. But just letting u know the issue seems to be fixed. I dont have to wait for it to cooldown myself, it does it for me now and everything is good!
Reply With Quote
  #13  
Old 15.06.2015, 08:45
nickles
Guest
 
Posts: n/a
Default

How would I control this option (e.g. turn it off)?

I noticed that even for compressed files (7z, zip, rar, cb?, etc.) the flag is set in Windows. This is not desirable!

Last edited by tony2long; 17.06.2015 at 07:37. Reason: Merge posts, please use edit.
Reply With Quote
  #14  
Old 15.06.2015, 12:02
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,013
Default

At the moment it is hardcoded(enabled) for all downloads. I can only make it global on/off. Do you wish an option to disable it? Don't see any reason why not to use it. As it affects multi connections downloads only and the only disadvantage might be higher fragmentation for multichonnection downloads
__________________
JD-Dev & Server-Admin
Reply With Quote
  #15  
Old 16.06.2015, 12:54
nickles
Guest
 
Posts: n/a
Default

An option to disable it would be fine because

1) my CLI tool colorizes files with the P flag set differently
2) setting the P flag for compressed files makes no sense.

I'd greatly appreciate such an option!
Reply With Quote
  #16  
Old 16.06.2015, 12:58
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,013
Default

__________________
JD-Dev & Server-Admin
Reply With Quote
  #17  
Old 16.06.2015, 17:06
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,668
Default

@nickles
re: 1) I don't really see how colour would be an issue?
re: 2) can you please explain what you mean ? are you inferring that content is going to be extracted anyway so it doesn't matter? or compression files serves no benefit to sparse flag. Do you realise that sparse option allows you to start downloading without preallocation issues? see https://board.jdownloader.org/showpo...97&postcount=2
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #18  
Old 16.06.2015, 22:32
nickles
Guest
 
Posts: n/a
Default

@raztoki
1) For me, color is an issue.
2) No problem with that, but please reset that flag when the download is complete.
Reply With Quote
  #19  
Old 17.06.2015, 02:41
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,668
Default

@nickles
as I understand you can't reset the flag as I understand its controlled by the operating system/filesystem, it indicates that the file was created with sparse. To me the benefit of sparse out weighs any issues introduced by a 3rd party program that displays sparse created files in colour.
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #20  
Old 17.06.2015, 05:32
MrGonzo's Avatar
MrGonzo MrGonzo is offline
JD Legend
 
Join Date: Dec 2010
Posts: 613
Default

Thxs guys for workin hard this issue. Razi please make sure this is "always on" with the option (adv sett) for user preference yes.

Good job guys
Reply With Quote
  #21  
Old 17.06.2015, 09:41
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,013
Default

What CLI tool do you use? Maybe you can setup to ignore/not highlight sparse files.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #22  
Old 17.06.2015, 09:42
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,013
Default

@MrGonzo: Thanks for feedback and no worry, it will be default enabled
__________________
JD-Dev & Server-Admin
Reply With Quote
  #23  
Old 17.06.2015, 18:09
nickles
Guest
 
Posts: n/a
Default

The tool is Take Command (though I could configure this). However, I see no sense in haven e.g. PDF file carrying the P flag after they were downloaded successfully (which leaves them in a "non-sparse" state and creates some overhead for the OS in handling them). I wouldn't mind this change, if it could only be turned off with a advanced setting...
Reply With Quote
  #24  
Old 17.06.2015, 18:23
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,013
Default

I don't know what P Flag means. But in JDownloader the file is either default os or default+sparse. For example default on NTFS is compression on. I still don't understand the issue though
__________________
JD-Dev & Server-Admin
Reply With Quote
  #25  
Old 19.06.2015, 15:58
nickles
Guest
 
Posts: n/a
Default

P stands for sparse.
Please also have a look at this thread **External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #26  
Old 19.06.2015, 16:03
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,013
Default

__________________
JD-Dev & Server-Admin
Reply With Quote
  #27  
Old 17.06.2016, 23:14
MrGonzo's Avatar
MrGonzo MrGonzo is offline
JD Legend
 
Join Date: Dec 2010
Posts: 613
Question

Hi guys

Sorry to reopen an oldy lol but Ive been noticing lately that ive had to stop and start a big DLs sometimes now to get it to finish DLing at max speed.

So my question is....it looks like u added an of/off switch to the sparseing of chunks to fix this cooldown problem.....or whatever lol

I just wanna know what should i search for in the advanced setting so I can make sure this is set "on" for me?

Last edited by MrGonzo; 17.06.2016 at 23:15. Reason: add
Reply With Quote
  #28  
Old 25.06.2016, 16:37
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,013
Default

i'm sorry but there is no option yet to disable sparse. will work on this next week.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #29  
Old 26.06.2016, 02:35
MrGonzo's Avatar
MrGonzo MrGonzo is offline
JD Legend
 
Join Date: Dec 2010
Posts: 613
Default

Thanks Jiaz....please let me know here or PM when uve added the on/off switch for it lol
Reply With Quote
  #30  
Old 26.06.2016, 12:11
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,013
Default

I will post here once the tickets are done
__________________
JD-Dev & Server-Admin
Reply With Quote
  #31  
Old 04.11.2018, 16:12
lj8787
Guest
 
Posts: n/a
Default

Quote:
Originally Posted by raztoki View Post

Typically when this happens on slow write devices it can take several minutes to preallocate, in which the download stops after ~8-10MB (on the systems I've used). On Virtual Disks say from VM or USB or Network drives these tasks can even be slower than a standard physical drive.


raztoki
Sorry to bump a very old thread, but it's relevant to this post. I am noticing terribly slow transfer speeds in a VirtualBox Windows 7 Guest when saving files to a shared drive from the Host. If I download something in JDownloader to the same drive but outside of the VM environment, the pre-allocation is almost instant and the file download doesn't pause at all. In the VM though, saving to the same drive on the Host, it can pause for a very long time, even on smaller files. I know running 1 chunk fixes the issue, but it also means terrible download speed from Real-Debrid (about 600Kbps I'm getting, per chunk). Is there an explanation as to why pre-allocating is so slow over networked drive in the VM? I'd love to be able to fix it. It's certainly not a drive speed bottleneck...

I should also note I've tried disable TCP Offloading in Device Manager and in the Registry, but that had no effect.

Appreciate any help.
Reply With Quote
  #32  
Old 28.04.2020, 09:24
MrGonzo's Avatar
MrGonzo MrGonzo is offline
JD Legend
 
Join Date: Dec 2010
Posts: 613
Default

Sorry to bring this issue up again...

Im not having to "start-stop-start" as much these days...but there seems to be an issue mostly near the end of a DL from about-85-95+%, the chunks start falling off quickly, killing DL speed. A lot of the time these days I have to stop the DL in the 85-95% completion range, cause when the DL drops to "1" chunk, the finishing DL speed can be quite bad (like 50kbs/s bad) and the DL "crawls" to the finish line. But if I stop the DL when it does this, and restart it...the DL usually finishes strong from that point to the finish.

Any ideas on how to fix this constant "1" chunk speed killer issue? It happens on many DLs these days.

Thanks
Reply With Quote
  #33  
Old 28.04.2020, 12:24
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,668
Default

Quote:
Originally Posted by lj8787 View Post
Sorry to bump a very old thread, but it's relevant to this post. I am noticing terribly slow transfer speeds in a VirtualBox Windows 7 Guest when saving files to a shared drive from the Host. If I download something in JDownloader to the same drive but outside of the VM environment, the pre-allocation is almost instant and the file download doesn't pause at all. In the VM though, saving to the same drive on the Host, it can pause for a very long time, even on smaller files. I know running 1 chunk fixes the issue, but it also means terrible download speed from Real-Debrid (about 600Kbps I'm getting, per chunk). Is there an explanation as to why pre-allocating is so slow over networked drive in the VM? I'd love to be able to fix it. It's certainly not a drive speed bottleneck...

I should also note I've tried disable TCP Offloading in Device Manager and in the Registry, but that had no effect.

Appreciate any help.
yes I ran JD on a friends VM and experienced the same thing, I believe that sparse isn't supported. Hung to the point where the tcp/ip connection would timeout waiting to pre-allocate the component (2 chunks = 50%) filesize required to rewrite to location.


Quote:
Originally Posted by MrGonzo View Post
Sorry to bring this issue up again...

Im not having to "start-stop-start" as much these days...but there seems to be an issue mostly near the end of a DL from about-85-95+%, the chunks start falling off quickly, killing DL speed. A lot of the time these days I have to stop the DL in the 85-95% completion range, cause when the DL drops to "1" chunk, the finishing DL speed can be quite bad (like 50kbs/s bad) and the DL "crawls" to the finish line. But if I stop the DL when it does this, and restart it...the DL usually finishes strong from that point to the finish.

Any ideas on how to fix this constant "1" chunk speed killer issue? It happens on many DLs these days.

Thanks
slow downloads at the end of the file is due to chunks finishing and existing chunks are slowwwww. Jiaz has written a new dl core years ago (but its not public) which addresses this, but until new download core becomes standard only work around like killing download if speed is under x and resuming it (event scripter)
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]

Last edited by raztoki; 28.04.2020 at 18:08. Reason: hang to hung
Reply With Quote
  #34  
Old 28.04.2020, 16:58
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,013
Default

@MrGonzo: In case you're talking about Uploaded, we got several similiar reports. Seems they have server issues
__________________
JD-Dev & Server-Admin
Reply With Quote
  #35  
Old 28.04.2020, 18:34
MrGonzo's Avatar
MrGonzo MrGonzo is offline
JD Legend
 
Join Date: Dec 2010
Posts: 613
Default

Quote:
Originally Posted by raztoki View Post
slow downloads at the end of the file is due to chunks finishing and existing chunks are slowwwww. Jiaz has written a new dl core years ago (but its not public) which addresses this, but until new download core becomes standard only work around like killing download if speed is under x and resuming it (event scripter)
Just what I was wondering....its good to know this issue is known and "fixed" (with new core) ....I just wonder now how long before this new core is implemented....would be nice to not have to deal with this issue anymore because its been an issue (with the current core) for a long time now. And happens quite regularly. Quite annoying actually, but at least now I know Im not the only one with this problem, and I am doing all that can be done for now (stop & restart) ....I await the new core!
Reply With Quote
  #36  
Old 29.04.2020, 14:02
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 72,951
Default

Sorry, we cannot give you any ETA.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #37  
Old 28.08.2024, 12:56
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 72,951
Default

Removing the sparse flag when download is complete has been added just recently:
https://board.jdownloader.org/showpo...48&postcount=9

Thus the setting to enable/disable sparse support is not needed anymore (RE ticket #66190).
Closed ticket 66190.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 03:23.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.