[MLi]

This thread was moved from the EventScripter thread, as it doesn't belong there.

Hi PSP,

Originally Posted by pspzockerscene

Hi MLi,
I'm not helping with custom scripts but:

JDownloader is not logged in during linkcheck of mountfile.net links so this can't be happening.
Whatever is taking away your traffic it's not JDownloader.

If you want me to investigate this further please report it in a dedicated thread in the hoster plugins subforum.

Sorry, that's not what I'm seeing... I just tested the following: leave the mountfile account off for two days, to make sure I have a full day (20G) of traffic. Turn it on, downloads start. Let it run until traffic is used up.

End result (from Select All -> Cleanup -> finished): 8.12GB downloaded, (Cleanup -> failed): 16.58 GB skipped. So it doesn't quite add up to the 20GB I expected, but it definitely downloaded a lot less than the 20GB.

So, something is fishy...

Any ideas on how I can test/debug this? There is nothing else talking to mountfile, I use JD2 for all my download needs.

Thanks!

MLi

[pspzockerscene]

Just as a sidenote:
Did you repeat the same test via browser?
Does the traffic add up there?

Also keep in mind that by default, JD will display all traffic values in MiB. You might want to use the same size unit as in browser (MB) beofre comparing anything.
For this go to Settings -> Advanced Settings -> GraphicalUserInterfaceSettings.maxsizeunit -> Set this to TiB

Another important thing to consider:
Some filehosts might double-count traffic if you use more than 1 connection per file (chunk).
This would be a possible explanation since browsers usually use 1 while download-managers can use more than 1.
--> Set the max chunks value in JD to 1 and try again.

[MLi]

I already had max_chunks at 1 from other hosters having this issue, so that wasn't it.

But I think I have a hunch of what the problem is: Mountfile does not display the correct name on the download page. The names displayed on the download pages are capitalized, but when the download starts the name is all lower case. So when JD2 checks the existing file from the download page it doesn't exist, only when the download starts it does, but at that point the credit has already been deducted.

I don't know if that's a problem only for the site I'm getting my mountfile links from or if they do that for everything. I've checked, and I've downloaded ~31000 files with JD2 through mountfile from this one site, and all came out as lower case. So while it's not a very diverse sample, it's a strong signal. I tried to verify it by uploading a file myself, but for some reason I can't upload anything. :(

So my proposal would be for the mountfile downloader to not just check the existence of the capitalized file, but also the lower case version, before starting the download. Unless somebody is stupid enough to have two files that only differ in case (which I'm not sure mountfile would support anyway) it should be a fairly conservative test. Not sure how hard it is to specialize this check or if that's done generically though.

Other ideas?

Thanks!

MLi

[pspzockerscene]

I'm sorry but I can't follow.
How should the filenames be connected to the problem you initially reported?

Another attempt:
If you can reproduce the problem, maybe we should do a Teamviewer session and look into it.
The easiest way would probably be if you could just send me your mountfile.net login credentials for testing.

Then again:
At this moment you're the only user who is reporting this issue.
I'm not saying that you're wrong - I'm just saying that the reson for your high traffic usage might be something else and not JDownloader.

[MLi]

Hm, let me try to explain it differently.

Here is what I think is happening: I am trying to download a file from mountfile, for example **External links are only visible to Support Staff****External links are only visible to Support Staff** . Now JD goes to that page, and from there gets the filename, let's say it is AHappyFileToDownload.txt . The file does not exist in the download directory, but there is a file named ahappyfiletodownload.txt, but that's obviously different. So now JD2 decides to (virtually) push the download button to download the file. But when it does that the file that gets downloaded is actually called ahappyfiletodownload.txt . I'm assuming JD2 is smart enough to check again if that file exists before it overwrites it, and now that one actually does exist, so I'm getting the File exists error. But because JD2 had to click the download button, the credit gets deducted, even though nothing is downloaded.

Does this make more sense? I don't have anything accessing mountfile, so I don't see how there could be anything else consuming credit.

Let me try it again tomorrow, and I'll try to verify it with screenshots.

MLi

[pspzockerscene]

Originally Posted by MLi

ow JD goes to that page, and from there gets the filename, let's say it is AHappyFileToDownload.txt . The file does not exist in the download directory, but there is a file named ahappyfiletodownload.txt, but that's obviously different. So now JD2 decides to (virtually) push the download button to download the file. But when it does that the file that gets downloaded is actually called ahappyfiletodownload.txt . I'm assuming JD2 is smart enough to check again if that file exists before it overwrites it, and now that one actually does exist, so I'm getting the File exists error. But because JD2 had to click the download button, the credit gets deducted, even though nothing is downloaded.

Yes this can happen but where is the bug?
The same would happen if you pushed the download button in your browser while "ahappyfiletodownload.txt" already existed in the final download directory.

What would you recommend doing to avoid this from happening?
For some websites, the final filename is simply not known until the download is started plus this problem would only happen for users who have a lot of same filenames/files already in their download directory -> Edge case if you ask me.

[MLi]

Ok, now I understand the confusion.

My problem is that I've been doing this for a long time, and I have a lot of links for files that are already downloaded (often from other sources, sometimes from the same; my linklist is around 90000 right now). So I would like to avoid downloading them again, and more importantly, not have credit deducted for them. That's why it's important to check if the file exists before incurring credit cost for it...

I could also work around this with checking through the linklist for existing files, but I don't know how to do that. I can write this in python easily, but I need away to get the linklist with download locations out and an API to remove links from an external program, both of which I'm not sure are possible.

[pspzockerscene]

Maybe you should additionally contact the mountfile.net support as a short "tapping on the download button" should not deduct the traffic.

Either way what you want might be possible and here is how:
You basically need to tell JD "hey the filename you get after checking that link is a safe/final filename", then JDownloader should use this for the "file already exists" check before the download is attempted.
Try this:
Settings -> Advanced Settings -> GeneralSettings.allowunsafefilenameforfileexistscheck -> Enable this

Then check if it's working as expected now.

[MLi]

I can try to talk to them, but I'm not optimistic. So I would prefer to find a solution that does not depend on them fixing their problem...

Unfortunately I already had that flag set, so that didn't fix it. :(

I can understand that this is an unusual situation that doesn't really warrant messing up JD2's behavior.

So I need to find another way to identify and remove files that already exist. I can get the files and download dirs through the Copy Information option, but I don't know if there is any way to remove downloads from an outside process.

Any hints welcome!

[mgpai]

script:

/*
    trigger : toolbar button pressed
*/

if (name == "myButton") {
    getAllDownloadLinks().forEach(function(link) {
        if (getPath(link.downloadPath).exists()) {
            link.enabled = false;
            //link.remove();
        }
    })
}

[pspzockerscene]

I just read your post #5 again:

Originally Posted by MLi

Now JD goes to that page, and from there gets the filename, let's say it is AHappyFileToDownload.txt . The file does not exist in the download directory, but there is a file named ahappyfiletodownload.txt, but that's obviously different.

If the filename really does change significantly of course there is no way for JD to know that it exists before the download is attempted.
In my tests it does not matter if the name changes from uppercase to lowercase.
For example:
Filename displayed in JD: GM-BlaBla.rar
Filename of already existing file: gm-blabla.rar
-> Works fine with the setting which I mentioned without attempting a download.

Please provide some more real life examples of the filenames you see in JD and the names of the files on your HDD.

@mgpai
That EventScripter script should not be needed.
If the filenames on his HDD differ from the ones displayed in linkgrabber, your script will fail too.

Originally Posted by MLi

So I need to find another way to identify and remove files that already exist. I can get the files and download dirs through the Copy Information option, but I don't know if there is any way to remove downloads from an outside process.

No you do not.
We need to find out why it doesn't work with the setting I mentioned.
Would you be open to do a Teamviewer session with me?

[MLi]

@mgpai

Nice! That is a nice and compact script! Is there any documentation on the API that is available to the scripts? I couldn't find anything helpful for me.

@pspzockerscene

Hm, that is surprising. Let me do some testing to make sure I'm not missing anything. The only reason I can think of is that my JD2 is running on Linux, which is less tolerant about cases than the usual Windows FS, but that's a little far fetched.

[pspzockerscene]

@Mli
If it really was about upper/lowercase, a script which changes all mountfile.net filenames in JD to lowercase should also do the trick.
At least under Windows, it's working just fine.

Regarding documentation:
See:
EventScripter forum thread:
https://board.jdownloader.org/showthread.php?t=70525
EventScripter help article:
https://support.jdownloader.org/Know...event-scripter

[mgpai]

Originally Posted by pspzockerscene

...script will fail too.

filename on linkcheck : mixed case
filename during download : lower case

File exists check might fail if it looks for a mixed case file on disk, becaise it would have been actually changed to lower case when the download starts. File name on download list and disk will both be in lower case so the script should work just fine.

[pspzockerscene]

As said here under Windows (which is case-insensitive), it's working fine even with mixed case.
What happens if you manually rename 1-2 files to lowercase in JDownloader?
Are they then skipped fine without a download-attempt?

[mgpai]

Originally Posted by pspzockerscene

As said here under Windows (which is case-insensitive), it's working fine even with mixed case.
What happens if you manually rename 1-2 files to lowercase in JDownloader?
Are they then skipped fine without a download-attempt?

1. I too am on windows where it is working fine.
2. It is by default changed/saved in lowercase, which is what is causing problem in linux .
3. In linux, file exists check should work fine if user manually changes the file name from lower case to mixed case (the one that mountfile provides on link check).

To recap, problem is caused by moutnfile providing mixed case file name during link chek but then changing it to lower case on download.

Guess the OP can shed more light.

[pspzockerscene]

@mgpai
kk then we're both on the same side.

The user should ultimately be able to work around this problem by running an EventScripter script which changes all mountfile.net filenames in JDownloader to lowercase.

[mgpai]

Originally Posted by pspzockerscene

@mgpai
kk then we're both on the same side.

The user should ultimately be able to work around this problem by running an EventScripter script which changes all mountfile.net filenames in JDownloader to lowercase.

Other way round. Check post #5.

Intial file name : AHappyFileToDownload.txt
Final file name : ahappyfiletodownload.txt

User has to change the the file name from lower case, back to intial file name case. Cannot be achieved by script since it is not a simple mixed case.

[pspzockerscene]

Yes this is what I meant:
Change them from mixed case in JD to all lowercase to make "file already exists" check work.

By the way welcome back mgpai

[MLi]

That was a great idea! And thanks to mgpai for the script, that was a great starting point. it took a little fiddling, but I think I have a script now that does exactly what I need:

if (name == "Cleanup") {
    var links = dlSelection.getDownloadLinks();
    for (i = 0; i < links.length; i++) {
        var link = links[i];
        
        var downloadURL = link.getContentURL();
        if(String(downloadURL).includes("mountfile"))
        {
            var filename = link.getName();
            var ln = filename.toLowerCase();
            link.setName(ln);
        }
        if (getPath(link.downloadPath).exists()) {
                print("Found",link.downloadPath, "removing", link);
                link.enabled = false;
                link.remove();
            }
            
    }
}

The only problem I have is that I need to run it twice to cleanup mountfile files that are already downloaded, somehow changing the name does not update the downloadPath. If there is a way to update that it would be nicer, but it's not really a big deal.

Thanks for your help and all the work on JD2!

[mgpai]

Originally Posted by MLi

... a way to update that ...

while (link.getName() != ln) {
    sleep(10);
}

[MLi]

Ah! I did not expect that one to be async. Thanks for the hint!

[mgpai]

Originally Posted by pspzockerscene

By the way welcome back mgpai

Thanks @psp.
Still on the sidelines though.

[pspzockerscene]

Originally Posted by mgpai

Still on the sidelines though.

I hope you are doing well

[mgpai]

Originally Posted by pspzockerscene

I hope you are doing well

Yes I am. Thank you.
Hope you are too.

[pspzockerscene]

Yap "Alles gut"