JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 19.06.2015, 11:52
bugnotme bugnotme is offline
BugMeNot Account
 
Join Date: Apr 2013
Posts: 375
Default Strange problem with downloaded files

Have something changed in the file creation algorithm in the last month? I'm having problems with the files downloaded by JD2. If I make a copy of the file everything works fine, but the original created by JD2 doesn't work with all programs like the files downloaded 3-4 weeks ago (moving or renaming the file doesn't help).

Win7 x64, Java 7u79.

P.S. The problem is present only when accessing the downloaded files with some programs. Here is a download link to a program that can be used to reproduce the problem: **External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #2  
Old 19.06.2015, 12:01
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Files are now created with sparse flag(to avoid preallocation issue with large files and slow disk/network drives).
Sparse Files exists since many years and if they really cause problems then it looks like issue in the tool you use.

What exactly is the problem? What error shows up?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #3  
Old 19.06.2015, 14:23
bugnotme bugnotme is offline
BugMeNot Account
 
Join Date: Apr 2013
Posts: 375
Default

The program can't access the file even after JD2 is shutdown and the pc is restarted. Is there an option to disable the new function?

BTW the problem is not in the software because it works fine with sparse files created by other software.
Reply With Quote
  #4  
Old 19.06.2015, 14:31
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Can we do teamviewer? Sparse is the only change. What error? Can't access...error message/screenshot?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #5  
Old 19.06.2015, 14:45
bugnotme bugnotme is offline
BugMeNot Account
 
Join Date: Apr 2013
Posts: 375
Default

The program log is not helpful:
Quote:
Information:
Statistics:

Erased area = 0 bytes

Cluster tips = 0 bytes



Data written = 0 bytes

Write time = 0.17 s

Write speed = 0 kB/s

Failures:
Failed: X:\...\File.part1.rar
BTW Shouldn't the sparse flag be removed once the file is downloaded?

Last edited by bugnotme; 19.06.2015 at 14:48.
Reply With Quote
  #6  
Old 19.06.2015, 14:48
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

What version are you using?
2255: Fixed a few minor documentation errors on task list compatibility and on NTFS compressed/encrypted/sparse files
__________________
JD-Dev & Server-Admin
Reply With Quote
  #7  
Old 19.06.2015, 14:56
bugnotme bugnotme is offline
BugMeNot Account
 
Join Date: Apr 2013
Posts: 375
Default

Build date: Fri Jun 19 11:26:56 CEST 2015
Auto update is on.
Reply With Quote
  #8  
Old 19.06.2015, 15:16
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

The sparse flag cannot be removed. It's a flag from beginning, cannot be changed once the file exists.

My guess is that the tool has issues with sparse files. Cause sparse file works different and if the tool is not prepared for it, you wont need to. Also I guess you use Eraser to delete files.
Might be right, trac.heidi.ie/changeset/1149
__________________
JD-Dev & Server-Admin
Reply With Quote
  #9  
Old 19.06.2015, 15:35
bugnotme bugnotme is offline
BugMeNot Account
 
Join Date: Apr 2013
Posts: 375
Default

Wrong on both counts:
The problem is only when using FL2KB mode (delete only first and wast 2KB of data) which I'm not using. And was fixed 2009 as can be seen by the link you provided.

"fsutil sparse setflag file_name.ext 0" removes the sparse flag, also torrent clients that use sparse files remove the flag once the files is downloaded.
Reply With Quote
  #10  
Old 19.06.2015, 15:49
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Didn't know that ntfs allows "desparse" a file at the end.
Will try to add support for this.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #11  
Old 19.06.2015, 16:06
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

I will try to use same method as libtorrent, just some experimenting with jna
__________________
JD-Dev & Server-Admin
Reply With Quote
  #12  
Old 09.07.2015, 01:43
raymondjpg
Guest
 
Posts: n/a
Default

Quote:
Originally Posted by Jiaz View Post
I will try to use same method as libtorrent, just some experimenting with jna
Will a fix come as an update, or new build?
Reply With Quote
  #13  
Old 19.06.2015, 16:18
bugnotme bugnotme is offline
BugMeNot Account
 
Join Date: Apr 2013
Posts: 375
Default

Great, thanks.
Reply With Quote
  #14  
Old 19.06.2015, 16:24
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Sorry for the circumstances!
__________________
JD-Dev & Server-Admin
Reply With Quote
  #15  
Old 09.07.2015, 06:05
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

when ticket closes at 100% compete && core update.
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #16  
Old 09.07.2015, 10:23
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Will do it this week
__________________
JD-Dev & Server-Admin
Reply With Quote
  #17  
Old 09.07.2015, 10:46
raymondjpg
Guest
 
Posts: n/a
Default

Quote:
Originally Posted by Jiaz View Post
Will do it this week
Thank you!
Reply With Quote
  #18  
Old 22.07.2015, 03:30
raymondjpg
Guest
 
Posts: n/a
Default

Is there any progress with this?
Reply With Quote
  #19  
Old 22.07.2015, 11:10
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

I'm sorry, not yet. Always something that stole my time. Will try to do this finally this week. Sorry for the big delay
__________________
JD-Dev & Server-Admin
Reply With Quote
  #20  
Old 23.07.2015, 02:17
raymondjpg
Guest
 
Posts: n/a
Default

Thanks again.
Reply With Quote
  #21  
Old 22.08.2015, 03:25
raymondjpg
Guest
 
Posts: n/a
Default

Still no resolution? I wonder how many people have stopped using JDownloader because of this?
Reply With Quote
  #22  
Old 22.08.2015, 04:22
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

honestly? bugmenot and you are the only ones that has made any noise about this. Most wouldn't even know that its taking place.

We had a few queries about sparse been added, and function not working when Jiaz first implemented. But since then, its been very quiet.

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]

Last edited by raztoki; 22.08.2015 at 05:39. Reason: used added too many times!
Reply With Quote
  #23  
Old 22.08.2015, 05:09
raymondjpg
Guest
 
Posts: n/a
Default

That is fair comment, but still doesn't say how many security conscious users may have stopped using JDownloader because secure erase of downloaded files is borked.
Reply With Quote
  #24  
Old 22.08.2015, 05:49
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

How could you account that? to me that's pie in the sky figure that you would never know definitively. We can only go from those that communicate and its been like just you guys. You could extrapolate a factor maybe for those that don't/wont communicate over the user base? Will only ever been approximation at best.

In my opinion very few users even know about secure wiping data or have the options natively supported by their OS (I see MacOSX does). As windows is the most popular Operating system and doesn't offer this natively, users would then have to install something like BCWipe themselves. So the odds are very low if you look at availability.

To me secure delete isn't be all to end all, for example: it's more than likely what's still on your on your computer that's likely to ever get you into trouble, compared to what "might be deleted" (traditional delete = index is wiped).

PS. secure wipe will reduce the read/write life cycle of your storage device since its writing to those sectors multiple times in order to ensure secure wipe outcome. This is on the assumption that there is a limited read/write cycle of each sector, which is case for traditional spinning disks, and now even more so with SSD.
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]

Last edited by raztoki; 22.08.2015 at 05:53.
Reply With Quote
  #25  
Old 17.04.2016, 19:28
user228afse
Guest
 
Posts: n/a
Default

I just noticed this bug and found this thread. Is this still being worked on?
Reply With Quote
  #26  
Old 18.04.2016, 10:35
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

This is no bug, more a sign that the tool you use does not proper handle sparse files. No, the ticket for this feature is still open, still no option to disable sparse at the end. I will work on this soon
__________________
JD-Dev & Server-Admin
Reply With Quote
  #27  
Old 19.04.2016, 22:11
user228afse
Guest
 
Posts: n/a
Default

Quote:
Originally Posted by Jiaz View Post
This is no bug, more a sign that the tool you use does not proper handle sparse files. No, the ticket for this feature is still open, still no option to disable sparse at the end. I will work on this soon
Thanks man for working on this. Yep it's not a jdownloader bug, it's eraser acting weird
Reply With Quote
  #28  
Old 20.04.2016, 11:32
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

I will work on this asap
__________________
JD-Dev & Server-Admin
Reply With Quote
  #29  
Old 08.05.2016, 03:28
user228afse
Guest
 
Posts: n/a
Default

Looks like the feature is added, thank you man!
Reply With Quote
  #30  
Old 08.05.2016, 06:04
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

Don't think any changes have been made, though I could be wrong.

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #31  
Old 08.05.2016, 06:36
user228afse
Guest
 
Posts: n/a
Default

Never mind, it worked for one file but not the other, weird!
Reply With Quote
  #32  
Old 06.06.2016, 05:36
LazyJones
Guest
 
Posts: n/a
Default

Is it possible to disable sparse completely in JD2 so files get preallocated completely at start? Its better to keep fragmentation down.
Reply With Quote
  #33  
Old 06.06.2016, 12:29
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default



Not possible yet. fragmentatin should not be real problem though, only when your disk is near full. And in case of downloading archive parts the fragmentation is no problem at all since JDownloader will extract them after download.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #34  
Old 09.08.2016, 04:20
netgearjd netgearjd is offline
Giga Loader
 
Join Date: Aug 2014
Posts: 92
Exclamation

While I wouldn't mind an advanced option to enable/disable sparse file support, an option (or default) to remove the sparse flag when the download's finished is most definitely required. I can see that fsutil sparse setflag <FileName> 0 does the job, but I'd rather have JD do that itself and frankly it doesn't sound like it would be hard to implement.

So here's another vote for:


Note #1: If you really need a reason for why I came searching for this specific issue, Windows 8/8.1/10 RTM and certain Server versions such as 2012 have problems mounting ISOs and VHDs with the sparse attribute set (see links below). I know, how dumb, right? Still, JD's current default behavior does cause an issue for those of us downloading such files on these Windows versions, and thus having this more than a year old issue be finally fixed would be appreciated.

**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

Note #2: After checking further, it seems fsutil sparse setflag <FileName> 0 doesn't work in Windows Vista, 7, 8 and many Server versions according to the TechNet link below since setflag only accepts <FileName> as a parameter.

**External links are only visible to Support Staff****External links are only visible to Support Staff**

I don't know about how it works in Windows 8.1 or Windows 10 RTM 10240, but in Windows 10 1511 and above (i.e. 1607 for now) it works properly since setflag accepts both <FileName> and 1/0 as parameters. So if you call fsutil as part of your fix, please keep in mind that it may or may not work depending on the version of Windows.

Last edited by netgearjd; 09.08.2016 at 04:58.
Reply With Quote
  #35  
Old 11.12.2016, 01:19
LazyJones
Guest
 
Posts: n/a
Default

Quote:
Originally Posted by Jiaz View Post
fragmentatin should not be real problem though
I don't know why but on my system (Win7) only JD2 produces files that are that massively fragmented even on a freshly defragmented drive (like 58 fragments on a 7.41MB file).
Attached Images
File Type: png sparse.png (104.1 KB, 13 views)
Reply With Quote
  #36  
Old 15.08.2016, 19:24
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Sorry for late response and thanks for the feedback. I will try to work on this this week. I can only work within a VM because I'm on ubuntu/linux. I will add both (option to enable/disable sparse) and also remove the sparse flag when file is finished.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #37  
Old 02.11.2016, 10:34
raymondjpg
Guest
 
Posts: n/a
Default

Quote:
Originally Posted by Jiaz View Post
I will add both (option to enable/disable sparse) and also remove the sparse flag when file is finished.
It would be appreciated if you could get to do this.
Reply With Quote
  #38  
Old 12.12.2016, 16:55
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@LazyJones. Do you download multiple files in parallel? I will work on this (option to disable sparse) as soon as I can
__________________
JD-Dev & Server-Admin
Reply With Quote
  #39  
Old 12.12.2016, 18:43
LazyJones
Guest
 
Posts: n/a
Default

Yes, i usually download 5 files at the same time.
Reply With Quote
  #40  
Old 12.12.2016, 18:45
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Can you check if it is the same when you download 1 by 1 ?
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 01:10.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.