[MrGonzo]

Hi again guys,

I thought id finally bring up an issue thats been around for quite awhile. Maybe uve all noticed it maybe not. BUT.....here we go....lol

What happens is with almost ANY big/BIG file (1GB-100GB+) DL will start and the DL speed will spike high for a moment then start to slow to a crawl dead! The DL will EVENTUALLY stop dead and give "connection problem" error message.

And the only way ive found to fix it is stop and restart. Sounds simple? Not quite.

With the BIG DLs there is a stop cooldown which can take as much as 2 to 5-8mins depending on the file size (I think). So i have to wait for the cooldown to restart. Once i do get it restarted, the DL speed goes just fine

So my q guys is, do any of u know about this? U seen it? Can u explain what this might be?

Anyway to fix this so I dont have to do this
start--stop---cooldown---start = thing anymore? haha lol

Thxs guys

[raztoki]

I've seen it before, and its most likely a result of preallocation of chunk ranges.

if you use more than 1 chunk additional chunks need to be pre allocated in the following fashion.

example 100gb file
2 chunks = preallocates the first 50gb and expands the last chunk as it downloads
3 chunks = preallocates 66.6gb and expands the last chunk as it downloads
4 chunks = preallocates 75gb and expands the last chunk as it downloads.

algo, x = filesize / number of chunks; preallocation data = (number of chunks - 1) * x;

preallocating on devices in Java takes time, as it has to write to said values. Apparently there is no quick way to do this (I'm not sure just what Jiaz told me in the past). Preallocation takes place once the download starts, because we don't know the final filesize from UNTRUSTED linkchecking routines and very few providers give trusted information.

Typically when this happens on slow write devices it can take several minutes to preallocate, in which the download stops after ~8-10MB (on the systems I've used). On Virtual Disks say from VM or USB or Network drives these tasks can even be slower than a standard physical drive.

This is why I hate chunk resetting code in plugins, it triggers that event because of the several timeout events that happen waiting for preallocation of chunks to finish. Once finished it resets back to 1 chunk and all that time spent waiting is lost!!

Jiaz solution in the new download core is to not preallocate in this fashion, but instead use ranges closer to each other and the beginning of the file, thus less data has to be instantly pre allocated.


raztoki

[MrGonzo]

Wow sound a bit complicated. But i kinda get ya. If its a big file, JD2 needs time to "allocate" the file chunks before it can really go. (And yes i do use chucks, 4 max)

But i thought id bring it up and see if anyone has noticed this. Good to know someone else finally said something with me haha. its a known issue....good. But i can tell it will be a bit before a solution is found.

Oh well, ive been doing this workaround for this for awhile now. lol. Hope this cooldown is fixed one day

Thxs Raz

[Jiaz]

Raztoki explained it already.
At the moment you can either set max connections to 1 or wait for update. I will try to add support for sparse files.
Also the next version of downloadsystem will handle those things much better.

[MrGonzo]

Haha no worries Jiaz I got most of what he was saying. And i think ive already got it set to 1.

Good to know sounds like this issue could be fixed sooner then i thought. Great!

Will "wait for core update"

+Was wondering i pretty much only use multihosters. Does that have anything to do with this or is it a JD2 thing?

[raztoki]

It's a download core issue, so it effects each plugin that allows more than 1 chunk. When you use multihoster providers they offer you access to premium services which will typically mean access to chunking ability. Assuming you have max chunks above 1 it will preallocate 50% + as described above. The larger the file, the longer it takes.

raztoki

[MrGonzo]

Thxs Raz & Jiaz. I get ya. Hopefully this new core update will at least help with the issue. Good to know this is an issue that can be fixed from JD2s end.

Haha...dealing with multihoster tech support is a nice guy! lol

[Jiaz]

[MrGonzo]

Thxs again Jiaz

[MrGonzo]

Hi

Sorry for the dbl post, I just wanted to say thxs u guys. It seems u may have fixed this problem. Since the last 2-3 updates ive noticed that the "cooldown" has been automated. Now JD2 recognizes when a connection fails and it now just auto-resets itself and the DL starts again....full speed.

So thxs again guys for the fix. Its saving me a lot of grief and time.

[raztoki]

sparse support doesn't create timeout situations, so you shouldn't get restart or situations where data reset & chunk reset happens due to silly work arounds within plugins to fix core issue.

[MrGonzo]

Hi

Dont quite get everything ur saying. But just letting u know the issue seems to be fixed. I dont have to wait for it to cooldown myself, it does it for me now and everything is good!

How would I control this option (e.g. turn it off)?

I noticed that even for compressed files (7z, zip, rar, cb?, etc.) the flag is set in Windows. This is not desirable!

[Jiaz]

At the moment it is hardcoded(enabled) for all downloads. I can only make it global on/off. Do you wish an option to disable it? Don't see any reason why not to use it. As it affects multi connections downloads only and the only disadvantage might be higher fragmentation for multichonnection downloads

An option to disable it would be fine because

1) my CLI tool colorizes files with the P flag set differently
2) setting the P flag for compressed files makes no sense.

I'd greatly appreciate such an option!

[Jiaz]

[raztoki]

@nickles
re: 1) I don't really see how colour would be an issue?
re: 2) can you please explain what you mean ? are you inferring that content is going to be extracted anyway so it doesn't matter? or compression files serves no benefit to sparse flag. Do you realise that sparse option allows you to start downloading without preallocation issues? see https://board.jdownloader.org/showpo...97&postcount=2

@raztoki
1) For me, color is an issue.
2) No problem with that, but please reset that flag when the download is complete.

[raztoki]

@nickles
as I understand you can't reset the flag as I understand its controlled by the operating system/filesystem, it indicates that the file was created with sparse. To me the benefit of sparse out weighs any issues introduced by a 3rd party program that displays sparse created files in colour.

[MrGonzo]

Thxs guys for workin hard this issue. Razi please make sure this is "always on" with the option (adv sett) for user preference yes.

Good job guys

[Jiaz]

What CLI tool do you use? Maybe you can setup to ignore/not highlight sparse files.

[Jiaz]

@MrGonzo: Thanks for feedback and no worry, it will be default enabled

The tool is Take Command (though I could configure this). However, I see no sense in haven e.g. PDF file carrying the P flag after they were downloaded successfully (which leaves them in a "non-sparse" state and creates some overhead for the OS in handling them). I wouldn't mind this change, if it could only be turned off with a advanced setting...

[Jiaz]

I don't know what P Flag means. But in JDownloader the file is either default os or default+sparse. For example default on NTFS is compression on. I still don't understand the issue though

P stands for sparse.
Please also have a look at this thread **External links are only visible to Support Staff****External links are only visible to Support Staff**

[Jiaz]

[MrGonzo]

Hi guys

Sorry to reopen an oldy lol but Ive been noticing lately that ive had to stop and start a big DLs sometimes now to get it to finish DLing at max speed.

So my question is....it looks like u added an of/off switch to the sparseing of chunks to fix this cooldown problem.....or whatever lol

I just wanna know what should i search for in the advanced setting so I can make sure this is set "on" for me?

[Jiaz]

i'm sorry but there is no option yet to disable sparse. will work on this next week.

[MrGonzo]

Thanks Jiaz....please let me know here or PM when uve added the on/off switch for it lol

[Jiaz]

I will post here once the tickets are done

Originally Posted by raztoki

Typically when this happens on slow write devices it can take several minutes to preallocate, in which the download stops after ~8-10MB (on the systems I've used). On Virtual Disks say from VM or USB or Network drives these tasks can even be slower than a standard physical drive.


raztoki

Sorry to bump a very old thread, but it's relevant to this post. I am noticing terribly slow transfer speeds in a VirtualBox Windows 7 Guest when saving files to a shared drive from the Host. If I download something in JDownloader to the same drive but outside of the VM environment, the pre-allocation is almost instant and the file download doesn't pause at all. In the VM though, saving to the same drive on the Host, it can pause for a very long time, even on smaller files. I know running 1 chunk fixes the issue, but it also means terrible download speed from Real-Debrid (about 600Kbps I'm getting, per chunk). Is there an explanation as to why pre-allocating is so slow over networked drive in the VM? I'd love to be able to fix it. It's certainly not a drive speed bottleneck...

I should also note I've tried disable TCP Offloading in Device Manager and in the Registry, but that had no effect.

Appreciate any help.

[MrGonzo]

Sorry to bring this issue up again...

Im not having to "start-stop-start" as much these days...but there seems to be an issue mostly near the end of a DL from about-85-95+%, the chunks start falling off quickly, killing DL speed. A lot of the time these days I have to stop the DL in the 85-95% completion range, cause when the DL drops to "1" chunk, the finishing DL speed can be quite bad (like 50kbs/s bad) and the DL "crawls" to the finish line. But if I stop the DL when it does this, and restart it...the DL usually finishes strong from that point to the finish.

Any ideas on how to fix this constant "1" chunk speed killer issue? It happens on many DLs these days.

Thanks

[raztoki]

Originally Posted by lj8787

Sorry to bump a very old thread, but it's relevant to this post. I am noticing terribly slow transfer speeds in a VirtualBox Windows 7 Guest when saving files to a shared drive from the Host. If I download something in JDownloader to the same drive but outside of the VM environment, the pre-allocation is almost instant and the file download doesn't pause at all. In the VM though, saving to the same drive on the Host, it can pause for a very long time, even on smaller files. I know running 1 chunk fixes the issue, but it also means terrible download speed from Real-Debrid (about 600Kbps I'm getting, per chunk). Is there an explanation as to why pre-allocating is so slow over networked drive in the VM? I'd love to be able to fix it. It's certainly not a drive speed bottleneck...

I should also note I've tried disable TCP Offloading in Device Manager and in the Registry, but that had no effect.

Appreciate any help.

yes I ran JD on a friends VM and experienced the same thing, I believe that sparse isn't supported. Hung to the point where the tcp/ip connection would timeout waiting to pre-allocate the component (2 chunks = 50%) filesize required to rewrite to location.

Originally Posted by MrGonzo

Sorry to bring this issue up again...

Im not having to "start-stop-start" as much these days...but there seems to be an issue mostly near the end of a DL from about-85-95+%, the chunks start falling off quickly, killing DL speed. A lot of the time these days I have to stop the DL in the 85-95% completion range, cause when the DL drops to "1" chunk, the finishing DL speed can be quite bad (like 50kbs/s bad) and the DL "crawls" to the finish line. But if I stop the DL when it does this, and restart it...the DL usually finishes strong from that point to the finish.

Any ideas on how to fix this constant "1" chunk speed killer issue? It happens on many DLs these days.

Thanks

slow downloads at the end of the file is due to chunks finishing and existing chunks are slowwwww. Jiaz has written a new dl core years ago (but its not public) which addresses this, but until new download core becomes standard only work around like killing download if speed is under x and resuming it (event scripter)

[Jiaz]

@MrGonzo: In case you're talking about Uploaded, we got several similiar reports. Seems they have server issues

[MrGonzo]

Originally Posted by raztoki

slow downloads at the end of the file is due to chunks finishing and existing chunks are slowwwww. Jiaz has written a new dl core years ago (but its not public) which addresses this, but until new download core becomes standard only work around like killing download if speed is under x and resuming it (event scripter)

Just what I was wondering....its good to know this issue is known and "fixed" (with new core) ....I just wonder now how long before this new core is implemented....would be nice to not have to deal with this issue anymore because its been an issue (with the current core) for a long time now. And happens quite regularly. Quite annoying actually, but at least now I know Im not the only one with this problem, and I am doing all that can be done for now (stop & restart) ....I await the new core!

[pspzockerscene]

Sorry, we cannot give you any ETA.

-psp-

[pspzockerscene]

Removing the sparse flag when download is complete has been added just recently:
https://board.jdownloader.org/showpo...48&postcount=9

Thus the setting to enable/disable sparse support is not needed anymore (RE ticket #66190).
Closed ticket 66190.