[Rayan]

Firstly, I'd like to say thanks to the developers who made this project possible.

I know that people already have discussed this subject several times, but I couldn't find any solutions for it. I'd like to voice a small complaint about something that has been really bugging me out about JD.

As you probably know, when JD is downloading files, it keeps a small amount of the downloaded data in memory, until the amount reaches 500KBs, when it writes the received data to the disk.
This process is called "buffering" and is a must when downloading simultaneous files from the internet.

However, the "500KB" of RAM buffering IMO is too small, and it is causing two problems for me when I use JD on Windows:

1) Files that were downloaded with 2 or more chunks will end up with TOO MUCH fragmentation on disk, and it will be torture for the poor HDD drive to read the files later.

As the developer has told me, the reasoning behind this is that the compressed files are going to be eventually extracted into a single piece -- so it isn't worth it or even needed to allocate space for the compressed files on the disk beforehand.

However, my argument is that the extraction process for a fragmented file (with over 1000 fragmentations) is going to take a rather large amount of time, than a 1-piece file. Furthermore, not all files are .RAR or .ZIP files. For example, we don't extract a .MP4 or .MKV file at all.

My reasoning is that it would be better if JD could create a blank file with the same file size when each task is added, so that the operating system could "allocate" sectors to that file, thus preventing file fragmentation.

If I try to extract a fragmented .RAR file (that has over 4000 fragmentation pieces) from my hard drive, it will only reach a maximum speed of 30MB/s at best, and I can clearly hear the pain that my drive's head is going through. Now, if I de-fragment the same .RAR file beforehand, the extraction process would go as smoothly as it can, easily reaching 80MB/s of read speed on my drive (considering that we are not writing the extracted content to the same drive).

I should mention that in both cases, I extract the content from the HDD drive directly to my primary SSD drive.

2) A buffer size of 500KBs keeps the HDD drive in active load in all time, which in turn keeps the drive always on, causing overheating issues and reducing the drive's lifespan. Windows is designed to turn all HDD drives off after 10 minutes of inactivity, so if the drive is constantly being written into, it's never going to be turned off.
IMO, it's much better to keep more amounts of downloaded data in memory, before occasionally writing the content to the disk.

I know that we can change a parameter in advanced settings to increase the buffer size (Named: GeneralSettings → Max Buffer Size) but JD has a limitation of 100MBs (or 100480 KBs to be exact) at maximum; and even if the developer extends its range, the Java virtual machine limits us to a ridiculous 512MB ram limit.

I use JD almost 24/7 on my Windows PC. JD never stops downloading, and it works all year so I prefer to keep my HDDs turned off as much as I could, and only write data to them when the download is completed or buffer size has reached about 500-1024MB in memory (obviously we need enough ram for this).

JD is already a wonderful download manager, but I think it really lacks two useful features:

1. Ability to allocate space on a drive before writing on it, in order to prevent fragmentation. (An option that can be enabled by user)
2. Ability to increase the buffer size on RAM (100mb or even more... up to 2048mb, if you ask me) before writing them. (An option that can be enabled and changed by the user)

JD is one the single most useful software in my toolbox, I love its UX and how it manages everything, but I can't really can't get my head around as to why the developers have decided to use Java!!

I'm no developer, but IMO, Java is too wasteful on resources, uses a lot of ram and CPU for no apparent reason while still managing to be limited to 512MB of memory (as the default startup) and it really lacks the interface benefits that native Windows programs have.

[raztoki]

ok lets clear up some misconceptions here
a) jdownloader request files to be written using apis, it has zero control over where the data is saved to disk. ultimately this is a task of your operating system, and somewhat these days the hardware (or drivers) thats attached to your harddrive (magnetic or sold state)!
b) jd2 does utilise sparse (assuming your os supports such thing) (common with p2p software), this means the entire space requirement is preserved. No significant data is actually written to file, its just pre-reserved at the file table (as I understand it). So when you are downloading and the buffer is fulls, it then sends/flushes said data to that file.

so with a combination of a and b your operating system backend should allocate space in some logical order. The only time this wont happen is storeage device self manages write locations/cycles, or your storeage device is nearly full or heavily fragmented to start with or combinations of all of the above.

with newer harddrives (ssd's), they have limited write cycles (as you referred), so they balance the write cycles semi evenly which means files or part of files are stored all over the drive. This extends the life of your drive. ssds tend to not suffer large performance issues when data is spread out like this, actually it can increase the speed as internal reading from multiple chips can happen at the same time (kinda like how raid array works). they also do no rely on moving parts in order to seek a different location in order for the head to read fragmented data, and less moving parts/motors == less heat.

in respects to ram, Java does NOT have a limit of 512MiB (32bit Java max 4GBytes, and 64bit Java max is more than you can install in your system (at this time), 16EBytes). Be aware that also some operating system prevent the use of the ram figures (have a quick google). you can start it with more or less, depending on your starting parameters. I believe the installer will set a limit based on your systems ram availability. Some of the older installers (if you keep using old ones) can have fixed limits. 'vmoptions' i believe is the file you need to edit to change default.

in resects to buffer/cache, The only control like you have mentioned is the cache (held in memory before sent to write task). This limit is set for the average user. keep in mind that every download * chunks requires memory allocation (20*20 == 400 * 500kb (default)), thus if you start lots of downloads can be memory intensive, even more so if you move to larger allocations like your screen shot indicates. Also there are other buffer settings, filter keyword buffer over max, =].

in respects to extraction it should work in a similar way to JD, it uses java apis in order to read and write files. So location on file is once again out of control of the application far as I'm aware. Fragmentation at this point shouldn't happen either because the uncompressed filesize within the archive volume is known, so when creating file on disk (extracting) it could tell the os the exact byte size, its not like it keeps amending/expanding a file size.

in respects to why java was used? I assumed it was because thats what the original authors learnt at university and they wanted to utilise there skill base over learning another language? makes sense no? Also cross operating system support is native, you have very little operating system specific code, so it makes it open to largest possible userbase. In respects to Java is buggy, its 24 years old... I personally don't agree with your statement. Traditionally its seemed as memory intensive and even slow, but my opinion has changed over time.

hope this helps

raztoki

[lamole234]

Sorry to borrow this topic but I guess there is no need for me to open a new one.
I know there has been work on the past to pre-allocate the files (#2497) but it has been deleted from JD at some point?


I'm asking this because I don't see any option about it -I guess the implementation is from before JD2 came out?- so, please, can you confirm if currently that feature is working and available on JD2?

[raztoki]

No it hasn't been deleted. Support for it was only in JD2. That svn ticket was closed prior to Java and JD2 having support.

Notes:
- There is no user option at this time,
- It's available to Java users using 1.7 or greater.
- Java tries to apply, but its not supported by all File Systems.

if (Application.getJavaVersion() >= Application.JAVA17) {
          SparseFile.createSparseFile(file);
}

java.nio.file.StandardOpenOption.SPARSE

Sparse file. When used with the CREATE_NEW option then this option provides a hint that the new file will be sparse. The option is ignored when the file system does not support the creation of sparse files.