JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 11.12.2019, 00:54
cman cman is offline
I will play nice!
 
Join Date: Dec 2019
Posts: 6
Default Timing issue causes "Invalid download directory"

There have been several reports earlier (for a long time) of downloads failing with "Invalid download directory" message. There are several reasons for this including write privileges, available space, etc., but there is one frequent scenario where this happens because of a timing issue in the Java code.

This scenario is never about permissions or directories that cannot be written to. In fact, manually asking such a failed download to resume immediately causes the download to resume with no problems. This can also happen randomly and not always.

I traced the exceptions when this happens to the following code in Jdownloader at

**External links are only visible to Support Staff****External links are only visible to Support Staff**

Code:
private void createOutputChannel() throws SkipReasonException {
        try {
            String fileOutput = downloadable.getFileOutput();
            logger.info("createOutputChannel for " + fileOutput);
            String finalFileOutput = downloadable.getFinalFileOutput();
            outputCompleteFile = new File(fileOutput);
            outputFinalCompleteFile = outputCompleteFile;
            if (!fileOutput.equals(finalFileOutput)) {
                outputFinalCompleteFile = new File(finalFileOutput);
            }
            outputPartFile = new File(downloadable.getFileOutputPart());
            try {
                if (Application.getJavaVersion() >= Application.JAVA17) {
                    SparseFile.createSparseFile(outputPartFile);
                }
            } catch (IOException e) {
            }
            outputPartFileRaf.set(IO.open(outputPartFile, "rw"));
        } catch (Exception e) {
            LogSource.exception(logger, e);
            throw new SkipReasonException(SkipReason.INVALID_DESTINATION, e);
        }
    }
What happens in the above code is that a file with a given name (a .part) is created first and immediately, there is a call to make it a sparsefile. This is where the timing issue is. The file is not guaranteed to be in the file system for the latter call if done almost immediately.

If the drive is busy or on a network or the file system is slow, or the processor is much faster than the i/o or whatever reason, the file created is not reflected in the file system before the request to make it a sparse file is called and that throws an exception with that "invalid download directory" message.

When one resumes the download manually, the .part would have appeared in the file system and so the create sparse file succeeds and the download proceeds.

Not sure of the exact cause of this file system update delay but even a very small delay (<0.5s) before the create sparse file is called fixes this.

There might be several ways to do this (and some may be a better solution than others) but I suggest that since the root cause is known, that it be fixed with one of the following depending on what is feasible (I am not a Java I/O expert)

Either a blocking call for the initial file creation until the file created is visible in the file system or a flush that forces the creation or an explicit wait before calling the sparse file create or trying again with a small delay when the first exception is thrown by create sparse file or whatever is feasible.

Thanks for your attention.
Reply With Quote
  #2  
Old 11.12.2019, 02:15
cman cman is offline
I will play nice!
 
Join Date: Dec 2019
Posts: 6
Default

Sorry, I was incorrect in the above description. The timing error is between the call to create sparsefile

Code:
SparseFile.createSparseFile(outputPartFile);
and opening the file for writing in

Code:
outputPartFileRaf.set(IO.open(outputPartFile, "rw"));
It is the second call above that throws an exception if done immediately after the sparse file creation in many cases.

It can happen even with local drives depending on how busy the IO subsystem is to finish the first call and have the file be reflected in the file system for the second call to open it for writing but unlikely. This is why it works most of the time with local drives. But this problem is much more frequent in network drives or external drives (not because they do not support sparse file creation but because they take a tiny bit longer to create it and be available for writing again).

It could also be that the sparse file is created in cache and not flushed to disk and the second call fails to find that file for writing immediately. I am not a Java expert. But in any case, separating /forcing the file creation with a sub-second delay in opening it for writing will fix this problem.

The exception lists it as permission denied but this is not a permission issue. The file itself has not been fully created and flushed before the second call to open it for writing. As I said before resuming the download manually from the menu command makes it proceed with no issues.

Code:
java.io.FileNotFoundException: <file path redacted>  (Permission denied)
	at java.io.RandomAccessFile.open0(Native Method)
	at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
	at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
	at org.appwork.utils.IO.open(IO.java:360)
	at jd.plugins.download.raf.OldRAFDownload.createOutputChannel(OldRAFDownload.java:1044)
Reply With Quote
  #3  
Old 11.12.2019, 07:10
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 16,515
Default

Not sure why jiaz made it with a try and catch then to skip invalid des
it should actually state another skip reason for each known IO exception type then generic one for the rest.
sparse support initially was quick implementation, don't think it has really been heavily refined/reviewed since implementation. Did also have a quick look at SparseFile class, seems ok

please wait for jiaz to respond.

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #4  
Old 11.12.2019, 08:59
cman cman is offline
I will play nice!
 
Join Date: Dec 2019
Posts: 6
Default

I posted a reply right after the first because I made a mistake in my OP description. It has not been approved by moderator after many hours. This is painful.

The problem is NOT the sparsefile call failing as I mentioned.

It is the the immediate IO.open for "rw" that is throwing the exception if the sparsefile was being created and had not finished yet.

There needs to be a delay between the creation of the sparsefile and the subsequent IO.open.

What I think happens is that if the sparsefile creation is carried out, it takes a little bit of time to complete (the slower the disk/IO/network, slightly longer it takes) and during that time the file has been created but is locked.

When the IO.open is done immediately, if there was no sparsefile creation, it would create a new file with that name and proceed normally without use of a sparsefile.

However, if that file is being created as a sparsefile and so still locked, the subsequent IO.open for "rw" fails with a FileNotFound/"Permission denied" exception for IO.open. With a fast local drive, it works fine most of the time. But you cannot guarantee this if there is other IO going on and the local disk is busy.

We are not talking a lot of time here. A few 100ms to 0.5 sec delay at most should be fine.
Reply With Quote
  #5  
Old 11.12.2019, 09:57
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 16,515
Default

Sorry I wrote my response this morning before heading out for an appointment but only posted after I returned. your second post (didn't know it was there, as this forum software doesn't show you live posts). So my comments was from your first post.

Your second post, yes sparse does have issues. There are known issues. It really should have an advanced option to utilise it or not.

Network shared drives have lot higher latency and IO related issues.
USB attached devices have are software controlled, have performance issues (shared bus), along with higher latency issues and even IO related issues.

I personally do not recommend saving to either of these, unless you really have to. Its better to utilise devices that are ready to go (not spun down), choose a device thats performance isn't hindered by outside demands. Move content after downloading to a slower device after download has completed (extract to the end location).
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #6  
Old 11.12.2019, 10:01
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 16,515
Default

Personally I don't believe a simple delay is the answer as you don't know the device is under load, don't know the timeframe of said delay. In VM it can be tens of seconds (seen and experienced this), just to name one possibility. Personally timer on permission denied and loop of some type within ms increments is better solution if a timer must be used, with fail after x tries so you can determine a true permission denied. Else its at the detriment of performance users. Ultimately keeping track of said performance to device and or path (since you can mount storage devices to directories in unix, linux, and windows) would be a good solution to at least apply wait time based on previous - minus some.
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]

Last edited by raztoki; 11.12.2019 at 10:06.
Reply With Quote
  #7  
Old 11.12.2019, 12:17
cman cman is offline
I will play nice!
 
Join Date: Dec 2019
Posts: 6
Default

Looking further it turns out that IO.open from the AppWorks utils already does this retry because this is a known bug but only done for Windows. It is explicitly mentioned in the comments.

Code:
public static RandomAccessFile open(File file, String mode) throws IOException {
        if (CrossSystem.isWindows()) {
            int retry = 0;
            while (true) {
                try {
                    return new RandomAccessFile(file, mode);
                } catch (final FileNotFoundException e) {
                    /**
                     * too fast file opening/extraction (eg image gallery) can result in "access denied" exception
                     */
                    if (retry < 3) {
                        if (retry == 2 && CrossSystem.isWindows()) {
                            // http://stackoverflow.com/questions/10516472/file-createnewfile-randomly-fails
                            // http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6213298
                            System.gc();
                        }
                        try {
                            Thread.sleep(500 * retry++);
                        } catch (InterruptedException e1) {
                            throw e;
                        }
                    } else {
                        throw e;
                    }
                }
            }
        } else {
            return new RandomAccessFile(file, mode);
        }
    }
So, I suspect people running jdownloader on Windows will not encounter this issue.

Unfortunately, this happens on a SMB share on a Mac as well and is pretty much unusable most of the time requirig a manual resume more than 90% of the time. Hope you guys can fix this so the above retries happen on all architectures. Otherwise, I will have to look for an alternate downloader.

Using internal drives is too 20th century. Most people who use jdownloader for large files will be downloading to NAS drives or external drives (these are not mission-critical real-time downloads, so reliability of download is more important than speed - a few seconds in 10s of minutes of load is not an issue), not internal drives which are likely SSDs with limited storage and limited work cycles relative to hard drives to use a two step-process.

So, the program should be designed to work with saves to an external NAS or USB drive as the norm than keep the legacy internal drives as the standard.

YMMV, of course. Just feedback from one usage profile. So, I am not going to argue about it if others feel different.
Reply With Quote
  #8  
Old 11.12.2019, 13:27
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 16,515
Default

there you go ;>
work arounds already present, along the lines of what we have already talked about.

have you thought about running JD on the NAS itself?
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #9  
Old 11.12.2019, 19:35
cman cman is offline
I will play nice!
 
Join Date: Dec 2019
Posts: 6
Default

I hope the same workaround can be made available for any platform, not just windows.

Assuming that this software is in active development by the community to be platform independent.

Selecting hardware to fit software otherwise is not an option for me, the other way is much easier.
Reply With Quote
  #10  
Old 12.12.2019, 01:31
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 16,515
Default

for sure its easier, I do understand, just another option though.

many people are running in headless mode, on as little as 64mb/128mb memory

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #11  
Old 13.12.2019, 00:12
cman cman is offline
I will play nice!
 
Join Date: Dec 2019
Posts: 6
Default

The solution is quite simple.

I downloaded the community code and removed the if test for Windows in the AppWorks IO.open listed above and ran it. Works fine with just that change.

Doing those backup multiple tries across all platforms, not just Windows as done now has no cost to anybody for existing use (i.e., no regression issues) and one don't have to worry about external drives, NAS, IO load, etc., even on Windows.

I am not sure if the developers are active on this software and/or interested in doing this fix in the official release.
Reply With Quote
  #12  
Old 13.12.2019, 16:07
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 16,515
Default

not disagreeing with you, as I stated above (post #10). I was just providing you or others alternatives solutions in the interim (post #5, #8). only appwork can make the required changes.
I asked them to look at the forum days ago, yet no one has done so. Its been weeks since any Appwork staff have looked/responded to posts on the forum. Just need to wait (post #3)
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 21:00.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.