JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 02.09.2016, 09:55
xped123's Avatar
xped123 xped123 is offline
JD Beta
 
Join Date: Nov 2011
Posts: 58
Default Size and Size on Disk discrepancy

I've noticed a lot of folders that contain content downloaded from Instagram lately have been largely different between Size and Size on Disk.

I also am using the new setting of original filenames in instagram plugin. My filesystem is NTFS, Windows 8.1.

Reply With Quote
  #2  
Old 02.09.2016, 11:08
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 66,134
Default

I guess high filesystem fragmentation or not good blocksize chosen for filesystem or some ntfs features enabled that cause this.
JDownloader has no influence on that.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #3  
Old 02.09.2016, 21:36
xped123's Avatar
xped123 xped123 is offline
JD Beta
 
Join Date: Nov 2011
Posts: 58
Default

Interesting. My drive is not fragmented for other files, my blocksize is fine (again, for other normal picture and video files), and I have not tweaked my NTFS to be anything but the default.

Are you sure it's not JD2 having writing issues?

Edit: The reason I posted this is that I have never encountered these issues with JD2 before - only from what I'm seeing now with .jpg files saved by JD2 from instagram. Before, every other filetype has been fine as far as I've seen in their Size/Size on Disk ratio.

Last edited by xped123; 02.09.2016 at 21:38.
Reply With Quote
  #4  
Old 03.09.2016, 00:50
SourTon SourTon is offline
Modem User
 
Join Date: May 2013
Posts: 4
Default

(890306560-748671920)/4347 = ... Cluster size 32KB? Google "file slack".
Reply With Quote
  #5  
Old 05.09.2016, 12:41
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 66,134
Default

@SourTon: yes thanks, I found same google results
It must have to do with:
-cluster size
-fragmentation
-data streams (maybe some tools append additional infos)

@xped123: no changes in JDownloader
__________________
JD-Dev & Server-Admin
Reply With Quote
  #6  
Old 05.09.2016, 13:24
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 16,524
Default

and if you use raid you can compound the issue as its raid block size * cluster size.

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #7  
Old 05.09.2016, 15:28
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 713
Default

The average slack space (difference between physical and logical size) per file is roughly half of the cluster size. For e.g. the total file slack for 4347 files depending on the cluster size would be:

4K = 8.49 MB
8K = 16.98 MB
16K = 33.96 MB
32K = 67.92 MB
64K = 135.84 MB

While the difference between size and size on disk is comparatively small with 4K cluster, the same will be considerably high with a 64K cluster.
Reply With Quote
  #8  
Old 05.09.2016, 16:00
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 66,134
Default

@mgpai: thanks for the numbers
__________________
JD-Dev & Server-Admin
Reply With Quote
  #9  
Old 07.09.2016, 03:04
xped123's Avatar
xped123 xped123 is offline
JD Beta
 
Join Date: Nov 2011
Posts: 58
Default

Thanks for all the support ! I will inspect what's going on with all the resources here and find what's my own problem
Reply With Quote
  #10  
Old 30.09.2016, 03:50
xped123's Avatar
xped123 xped123 is offline
JD Beta
 
Join Date: Nov 2011
Posts: 58
Default

I'd like to revive this thread.
I would like to think it's something relating to BOTH JDownloader and my system.

Here's an example:
Image downloaded by JDOWNLOADER from INSTAGRAM using "Original Filename"




Image COPIED by the system to a different folder, same drive:







What gives? Why is there a descepancy between my Size and Size on Disk?
I checked out my CLUSTER, it's 4,096 bytes (4K) allocation.
There's no reason why some of these images should be almost double in their size when downloaded by JDOWNLOADER.

EDIT: Checking another folder of mine relating to "imagefap.com", I can find in 2015, between MAY and AUGUST 2015, the folders then start having issues of these size discrepancies. This is also on a DIFFERENT drive, also have 4K clusters.

i.e. Folders BEFORE May 2015:



i.e. Folders AFTER August 2015:



Windows 8.1, OS install predates both of the drives in question. Both drives are 2TB in question. Both subfolders are NOT within a Junctioned Directory. Both drives are 4K Cluster size.

Windows has not been updated in 2 years, so no Microsoft wankery.
The only thing that has changed between these variables is literally JD2 and updating it and resuming the same methods of download (Change directory for when JD2 parses the page=> to new directory of downloading than the Default Download Folder).

Last edited by xped123; 30.09.2016 at 04:11.
Reply With Quote
  #11  
Old 30.09.2016, 11:05
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 66,134
Default

A possible cause of this may be the use of sparse files.

I will add advanced option to disable it soon
How many connections do you use to download?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #12  
Old 02.10.2016, 02:43
xped123's Avatar
xped123 xped123 is offline
JD Beta
 
Join Date: Nov 2011
Posts: 58
Default

Quote:
Originally Posted by Jiaz View Post
A possible cause of this may be the use of sparse files.

I will add advanced option to disable it soon
How many connections do you use to download?
2 max chunks per download
4-5 max simultaneous downloads

I hope that helps, let me know whatever else I can provide for my current JD2 settings + my OS settings.

I swear I'm not crazy. Nothing else on my system has yielded this much of a difference between Size and Size on Disk.

Thank you for your hard work!
Reply With Quote
  #13  
Old 04.10.2016, 15:24
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 66,134
Default

I will inform you once I've added support to disable sparse, then you can check. I'm sure it is because of many concurrent downloads in combination with multiple connections. Without sparse flag it should no longer hapen
__________________
JD-Dev & Server-Admin
Reply With Quote
  #14  
Old 05.10.2016, 05:50
xped123's Avatar
xped123 xped123 is offline
JD Beta
 
Join Date: Nov 2011
Posts: 58
Default

Thanks for figuring out and catching that issue with instagram. I appreciate it.

I managed to get around it by a simple copy paste, delete of original, but obviously unless I use xcopy through cmd in Windows, I won't preserve proper file timestamps for created/modified.

Cheers
Reply With Quote
  #15  
Old 11.11.2016, 04:35
xped123's Avatar
xped123 xped123 is offline
JD Beta
 
Join Date: Nov 2011
Posts: 58
Default

Any progress on this sparse setting feature?

Thanks :D
Reply With Quote
  #16  
Old 11.11.2016, 13:44
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 66,134
Default

I#m sorry, not yet but I will work on this next week.
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 21:49.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.