[bugnotme]

Have something changed in the file creation algorithm in the last month? I'm having problems with the files downloaded by JD2. If I make a copy of the file everything works fine, but the original created by JD2 doesn't work with all programs like the files downloaded 3-4 weeks ago (moving or renaming the file doesn't help).

Win7 x64, Java 7u79.

P.S. The problem is present only when accessing the downloaded files with some programs. Here is a download link to a program that can be used to reproduce the problem: **External links are only visible to Support Staff****External links are only visible to Support Staff**

[Jiaz]

Files are now created with sparse flag(to avoid preallocation issue with large files and slow disk/network drives).
Sparse Files exists since many years and if they really cause problems then it looks like issue in the tool you use.

What exactly is the problem? What error shows up?

[bugnotme]

The program can't access the file even after JD2 is shutdown and the pc is restarted. Is there an option to disable the new function?

BTW the problem is not in the software because it works fine with sparse files created by other software.

[Jiaz]

Can we do teamviewer? Sparse is the only change. What error? Can't access...error message/screenshot?

[bugnotme]

The program log is not helpful:

Information:
Statistics:

Erased area = 0 bytes

Cluster tips = 0 bytes



Data written = 0 bytes

Write time = 0.17 s

Write speed = 0 kB/s

Failures:
Failed: X:\...\File.part1.rar

BTW Shouldn't the sparse flag be removed once the file is downloaded?

[Jiaz]

What version are you using?
2255: Fixed a few minor documentation errors on task list compatibility and on NTFS compressed/encrypted/sparse files

[bugnotme]

Build date: Fri Jun 19 11:26:56 CEST 2015
Auto update is on.

[Jiaz]

The sparse flag cannot be removed. It's a flag from beginning, cannot be changed once the file exists.

My guess is that the tool has issues with sparse files. Cause sparse file works different and if the tool is not prepared for it, you wont need to. Also I guess you use Eraser to delete files.
Might be right, trac.heidi.ie/changeset/1149

[bugnotme]

Wrong on both counts:
The problem is only when using FL2KB mode (delete only first and wast 2KB of data) which I'm not using. And was fixed 2009 as can be seen by the link you provided.

"fsutil sparse setflag file_name.ext 0" removes the sparse flag, also torrent clients that use sparse files remove the flag once the files is downloaded.

[Jiaz]

Didn't know that ntfs allows "desparse" a file at the end.
Will try to add support for this.

[Jiaz]

I will try to use same method as libtorrent, just some experimenting with jna

[bugnotme]

Great, thanks.

[Jiaz]

Sorry for the circumstances!

Originally Posted by Jiaz

I will try to use same method as libtorrent, just some experimenting with jna

Will a fix come as an update, or new build?

[raztoki]

when ticket closes at 100% compete && core update.

[Jiaz]

Will do it this week

Originally Posted by Jiaz

Will do it this week

Thank you!

Is there any progress with this?

[Jiaz]

I'm sorry, not yet. Always something that stole my time. Will try to do this finally this week. Sorry for the big delay

Thanks again.

Still no resolution? I wonder how many people have stopped using JDownloader because of this?

[raztoki]

honestly? bugmenot and you are the only ones that has made any noise about this. Most wouldn't even know that its taking place.

We had a few queries about sparse been added, and function not working when Jiaz first implemented. But since then, its been very quiet.

raztoki

That is fair comment, but still doesn't say how many security conscious users may have stopped using JDownloader because secure erase of downloaded files is borked.

[raztoki]

How could you account that? to me that's pie in the sky figure that you would never know definitively. We can only go from those that communicate and its been like just you guys. You could extrapolate a factor maybe for those that don't/wont communicate over the user base? Will only ever been approximation at best.

In my opinion very few users even know about secure wiping data or have the options natively supported by their OS (I see MacOSX does). As windows is the most popular Operating system and doesn't offer this natively, users would then have to install something like BCWipe themselves. So the odds are very low if you look at availability.

To me secure delete isn't be all to end all, for example: it's more than likely what's still on your on your computer that's likely to ever get you into trouble, compared to what "might be deleted" (traditional delete = index is wiped).

PS. secure wipe will reduce the read/write life cycle of your storage device since its writing to those sectors multiple times in order to ensure secure wipe outcome. This is on the assumption that there is a limited read/write cycle of each sector, which is case for traditional spinning disks, and now even more so with SSD.

I just noticed this bug and found this thread. Is this still being worked on?

[Jiaz]

This is no bug, more a sign that the tool you use does not proper handle sparse files. No, the ticket for this feature is still open, still no option to disable sparse at the end. I will work on this soon

Originally Posted by Jiaz

This is no bug, more a sign that the tool you use does not proper handle sparse files. No, the ticket for this feature is still open, still no option to disable sparse at the end. I will work on this soon

Thanks man for working on this. Yep it's not a jdownloader bug, it's eraser acting weird

[Jiaz]

I will work on this asap

Looks like the feature is added, thank you man!

[raztoki]

Don't think any changes have been made, though I could be wrong.

raztoki

Never mind, it worked for one file but not the other, weird!

Is it possible to disable sparse completely in JD2 so files get preallocated completely at start? Its better to keep fragmentation down.

[Jiaz]

Not possible yet. fragmentatin should not be real problem though, only when your disk is near full. And in case of downloading archive parts the fragmentation is no problem at all since JDownloader will extract them after download.

[netgearjd]

While I wouldn't mind an advanced option to enable/disable sparse file support, an option (or default) to remove the sparse flag when the download's finished is most definitely required. I can see that fsutil sparse setflag <FileName> 0 does the job, but I'd rather have JD do that itself and frankly it doesn't sound like it would be hard to implement.

So here's another vote for:


Note #1: If you really need a reason for why I came searching for this specific issue, Windows 8/8.1/10 RTM and certain Server versions such as 2012 have problems mounting ISOs and VHDs with the sparse attribute set (see links below). I know, how dumb, right? Still, JD's current default behavior does cause an issue for those of us downloading such files on these Windows versions, and thus having this more than a year old issue be finally fixed would be appreciated.

**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

Note #2: After checking further, it seems fsutil sparse setflag <FileName> 0 doesn't work in Windows Vista, 7, 8 and many Server versions according to the TechNet link below since setflag only accepts <FileName> as a parameter.

**External links are only visible to Support Staff****External links are only visible to Support Staff**

I don't know about how it works in Windows 8.1 or Windows 10 RTM 10240, but in Windows 10 1511 and above (i.e. 1607 for now) it works properly since setflag accepts both <FileName> and 1/0 as parameters. So if you call fsutil as part of your fix, please keep in mind that it may or may not work depending on the version of Windows.

[Jiaz]

Sorry for late response and thanks for the feedback. I will try to work on this this week. I can only work within a VM because I'm on ubuntu/linux. I will add both (option to enable/disable sparse) and also remove the sparse flag when file is finished.

Originally Posted by Jiaz

I will add both (option to enable/disable sparse) and also remove the sparse flag when file is finished.

It would be appreciated if you could get to do this.

Originally Posted by Jiaz

fragmentatin should not be real problem though

I don't know why but on my system (Win7) only JD2 produces files that are that massively fragmented even on a freshly defragmented drive (like 58 fragments on a 7.41MB file).

[Jiaz]

@LazyJones. Do you download multiple files in parallel? I will work on this (option to disable sparse) as soon as I can

Yes, i usually download 5 files at the same time.

[Jiaz]

Can you check if it is the same when you download 1 by 1 ?