View Single Post
Old 16.06.2019, 12:12
Rayan Rayan is offline
Modem User
Join Date: Jun 2019
Posts: 1
Default A way to decrease/prevent fragmentation

Firstly, I'd like to say thanks to the developers who made this project possible.

I know that people already have discussed this subject several times, but I couldn't find any solutions for it. I'd like to voice a small complaint about something that has been really bugging me out about JD.

As you probably know, when JD is downloading files, it keeps a small amount of the downloaded data in memory, until the amount reaches 500KBs, when it writes the received data to the disk.
This process is called "buffering" and is a must when downloading simultaneous files from the internet.

However, the "500KB" of RAM buffering IMO is too small, and it is causing two problems for me when I use JD on Windows:

1) Files that were downloaded with 2 or more chunks will end up with TOO MUCH fragmentation on disk, and it will be torture for the poor HDD drive to read the files later.

As the developer has told me, the reasoning behind this is that the compressed files are going to be eventually extracted into a single piece -- so it isn't worth it or even needed to allocate space for the compressed files on the disk beforehand.

However, my argument is that the extraction process for a fragmented file (with over 1000 fragmentations) is going to take a rather large amount of time, than a 1-piece file. Furthermore, not all files are .RAR or .ZIP files. For example, we don't extract a .MP4 or .MKV file at all.

My reasoning is that it would be better if JD could create a blank file with the same file size when each task is added, so that the operating system could "allocate" sectors to that file, thus preventing file fragmentation.

If I try to extract a fragmented .RAR file (that has over 4000 fragmentation pieces) from my hard drive, it will only reach a maximum speed of 30MB/s at best, and I can clearly hear the pain that my drive's head is going through. Now, if I de-fragment the same .RAR file beforehand, the extraction process would go as smoothly as it can, easily reaching 80MB/s of read speed on my drive (considering that we are not writing the extracted content to the same drive).

I should mention that in both cases, I extract the content from the HDD drive directly to my primary SSD drive.

2) A buffer size of 500KBs keeps the HDD drive in active load in all time, which in turn keeps the drive always on, causing overheating issues and reducing the drive's lifespan. Windows is designed to turn all HDD drives off after 10 minutes of inactivity, so if the drive is constantly being written into, it's never going to be turned off.
IMO, it's much better to keep more amounts of downloaded data in memory, before occasionally writing the content to the disk.

I know that we can change a parameter in advanced settings to increase the buffer size (Named: GeneralSettings → Max Buffer Size) but JD has a limitation of 100MBs (or 100480 KBs to be exact) at maximum; and even if the developer extends its range, the Java virtual machine limits us to a ridiculous 512MB ram limit.

I use JD almost 24/7 on my Windows PC. JD never stops downloading, and it works all year so I prefer to keep my HDDs turned off as much as I could, and only write data to them when the download is completed or buffer size has reached about 500-1024MB in memory (obviously we need enough ram for this).

JD is already a wonderful download manager, but I think it really lacks two useful features:
  1. Ability to allocate space on a drive before writing on it, in order to prevent fragmentation. (An option that can be enabled by user)
  2. Ability to increase the buffer size on RAM (100mb or even more... up to 2048mb, if you ask me) before writing them. (An option that can be enabled and changed by the user)

JD is one the single most useful software in my toolbox, I love its UX and how it manages everything, but I can't really can't get my head around as to why the developers have decided to use Java!!

I'm no developer, but IMO, Java is too wasteful on resources, uses a lot of ram and CPU for no apparent reason while still managing to be limited to 512MB of memory (as the default startup) and it really lacks the interface benefits that native Windows programs have.

Last edited by Rayan; 16.06.2019 at 16:30.
Reply With Quote