JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 01.11.2023, 05:53
MLi MLi is offline
Mega Loader
 
Join Date: Oct 2010
Posts: 68
Default [EventScripter][NOT a JD bug] Mountfile: link check consumes traffic?

This thread was moved from the EventScripter thread, as it doesn't belong there.

Hi PSP,

Quote:
Originally Posted by pspzockerscene View Post
Hi MLi,
I'm not helping with custom scripts but:

JDownloader is not logged in during linkcheck of mountfile.net links so this can't be happening.
Whatever is taking away your traffic it's not JDownloader.

If you want me to investigate this further please report it in a dedicated thread in the hoster plugins subforum.
Sorry, that's not what I'm seeing... I just tested the following: leave the mountfile account off for two days, to make sure I have a full day (20G) of traffic. Turn it on, downloads start. Let it run until traffic is used up.

End result (from Select All -> Cleanup -> finished): 8.12GB downloaded, (Cleanup -> failed): 16.58 GB skipped. So it doesn't quite add up to the 20GB I expected, but it definitely downloaded a lot less than the 20GB.

So, something is fishy...

Any ideas on how I can test/debug this? There is nothing else talking to mountfile, I use JD2 for all my download needs.

Thanks!

MLi

Last edited by MLi; 04.11.2023 at 13:56. Reason: Typo
Reply With Quote
  #2  
Old 02.11.2023, 10:27
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,121
Default

Just as a sidenote:
Did you repeat the same test via browser?
Does the traffic add up there?

Also keep in mind that by default, JD will display all traffic values in MiB. You might want to use the same size unit as in browser (MB) beofre comparing anything.
For this go to Settings -> Advanced Settings -> GraphicalUserInterfaceSettings.maxsizeunit -> Set this to TiB

Another important thing to consider:
Some filehosts might double-count traffic if you use more than 1 connection per file (chunk).
This would be a possible explanation since browsers usually use 1 while download-managers can use more than 1.
--> Set the max chunks value in JD to 1 and try again.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #3  
Old 05.11.2023, 12:24
MLi MLi is offline
Mega Loader
 
Join Date: Oct 2010
Posts: 68
Default

I already had max_chunks at 1 from other hosters having this issue, so that wasn't it.

But I think I have a hunch of what the problem is: Mountfile does not display the correct name on the download page. The names displayed on the download pages are capitalized, but when the download starts the name is all lower case. So when JD2 checks the existing file from the download page it doesn't exist, only when the download starts it does, but at that point the credit has already been deducted.

I don't know if that's a problem only for the site I'm getting my mountfile links from or if they do that for everything. I've checked, and I've downloaded ~31000 files with JD2 through mountfile from this one site, and all came out as lower case. So while it's not a very diverse sample, it's a strong signal. I tried to verify it by uploading a file myself, but for some reason I can't upload anything. :(

So my proposal would be for the mountfile downloader to not just check the existence of the capitalized file, but also the lower case version, before starting the download. Unless somebody is stupid enough to have two files that only differ in case (which I'm not sure mountfile would support anyway) it should be a fairly conservative test. Not sure how hard it is to specialize this check or if that's done generically though.

Other ideas?

Thanks!

MLi

Last edited by MLi; 05.11.2023 at 12:32. Reason: Forgot to count rar files.
Reply With Quote
  #4  
Old 07.11.2023, 11:08
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,121
Default

I'm sorry but I can't follow.
How should the filenames be connected to the problem you initially reported?

Another attempt:
If you can reproduce the problem, maybe we should do a Teamviewer session and look into it.
The easiest way would probably be if you could just send me your mountfile.net login credentials for testing.

Then again:
At this moment you're the only user who is reporting this issue.
I'm not saying that you're wrong - I'm just saying that the reson for your high traffic usage might be something else and not JDownloader.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #5  
Old 08.11.2023, 02:55
MLi MLi is offline
Mega Loader
 
Join Date: Oct 2010
Posts: 68
Default

Hm, let me try to explain it differently.

Here is what I think is happening: I am trying to download a file from mountfile, for example **External links are only visible to Support Staff****External links are only visible to Support Staff** . Now JD goes to that page, and from there gets the filename, let's say it is AHappyFileToDownload.txt . The file does not exist in the download directory, but there is a file named ahappyfiletodownload.txt, but that's obviously different. So now JD2 decides to (virtually) push the download button to download the file. But when it does that the file that gets downloaded is actually called ahappyfiletodownload.txt . I'm assuming JD2 is smart enough to check again if that file exists before it overwrites it, and now that one actually does exist, so I'm getting the File exists error. But because JD2 had to click the download button, the credit gets deducted, even though nothing is downloaded.

Does this make more sense? I don't have anything accessing mountfile, so I don't see how there could be anything else consuming credit.

Let me try it again tomorrow, and I'll try to verify it with screenshots.

MLi
Reply With Quote
  #6  
Old 08.11.2023, 09:36
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,121
Default

Quote:
Originally Posted by MLi View Post
ow JD goes to that page, and from there gets the filename, let's say it is AHappyFileToDownload.txt . The file does not exist in the download directory, but there is a file named ahappyfiletodownload.txt, but that's obviously different. So now JD2 decides to (virtually) push the download button to download the file. But when it does that the file that gets downloaded is actually called ahappyfiletodownload.txt . I'm assuming JD2 is smart enough to check again if that file exists before it overwrites it, and now that one actually does exist, so I'm getting the File exists error. But because JD2 had to click the download button, the credit gets deducted, even though nothing is downloaded.
Yes this can happen but where is the bug?
The same would happen if you pushed the download button in your browser while "ahappyfiletodownload.txt" already existed in the final download directory.

What would you recommend doing to avoid this from happening?
For some websites, the final filename is simply not known until the download is started plus this problem would only happen for users who have a lot of same filenames/files already in their download directory -> Edge case if you ask me.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #7  
Old 08.11.2023, 12:47
MLi MLi is offline
Mega Loader
 
Join Date: Oct 2010
Posts: 68
Default

Ok, now I understand the confusion.

My problem is that I've been doing this for a long time, and I have a lot of links for files that are already downloaded (often from other sources, sometimes from the same; my linklist is around 90000 right now). So I would like to avoid downloading them again, and more importantly, not have credit deducted for them. That's why it's important to check if the file exists before incurring credit cost for it...

I could also work around this with checking through the linklist for existing files, but I don't know how to do that. I can write this in python easily, but I need away to get the linklist with download locations out and an API to remove links from an external program, both of which I'm not sure are possible.
Reply With Quote
  #8  
Old 08.11.2023, 13:35
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,121
Default

Maybe you should additionally contact the mountfile.net support as a short "tapping on the download button" should not deduct the traffic.

Either way what you want might be possible and here is how:
You basically need to tell JD "hey the filename you get after checking that link is a safe/final filename", then JDownloader should use this for the "file already exists" check before the download is attempted.
Try this:
Settings -> Advanced Settings -> GeneralSettings.allowunsafefilenameforfileexistscheck -> Enable this

Then check if it's working as expected now.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #9  
Old 09.11.2023, 05:35
MLi MLi is offline
Mega Loader
 
Join Date: Oct 2010
Posts: 68
Default

I can try to talk to them, but I'm not optimistic. So I would prefer to find a solution that does not depend on them fixing their problem...

Unfortunately I already had that flag set, so that didn't fix it. :(

I can understand that this is an unusual situation that doesn't really warrant messing up JD2's behavior.

So I need to find another way to identify and remove files that already exist. I can get the files and download dirs through the Copy Information option, but I don't know if there is any way to remove downloads from an outside process.

Any hints welcome!
Reply With Quote
  #10  
Old 09.11.2023, 07:16
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 1,553
Default

script:
Code:
/*
    trigger : toolbar button pressed
*/

if (name == "myButton") {
    getAllDownloadLinks().forEach(function(link) {
        if (getPath(link.downloadPath).exists()) {
            link.enabled = false;
            //link.remove();
        }
    })
}
Reply With Quote
  #11  
Old 09.11.2023, 09:51
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,121
Default

I just read your post #5 again:
Quote:
Originally Posted by MLi View Post
Now JD goes to that page, and from there gets the filename, let's say it is AHappyFileToDownload.txt . The file does not exist in the download directory, but there is a file named ahappyfiletodownload.txt, but that's obviously different.
If the filename really does change significantly of course there is no way for JD to know that it exists before the download is attempted.
In my tests it does not matter if the name changes from uppercase to lowercase.
For example:
Filename displayed in JD: GM-BlaBla.rar
Filename of already existing file: gm-blabla.rar
-> Works fine with the setting which I mentioned without attempting a download.

Please provide some more real life examples of the filenames you see in JD and the names of the files on your HDD.

@mgpai
That EventScripter script should not be needed.
If the filenames on his HDD differ from the ones displayed in linkgrabber, your script will fail too.


Quote:
Originally Posted by MLi View Post
So I need to find another way to identify and remove files that already exist. I can get the files and download dirs through the Copy Information option, but I don't know if there is any way to remove downloads from an outside process.
No you do not.
We need to find out why it doesn't work with the setting I mentioned.
Would you be open to do a Teamviewer session with me?
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #12  
Old 09.11.2023, 22:41
MLi MLi is offline
Mega Loader
 
Join Date: Oct 2010
Posts: 68
Default

@mgpai

Nice! That is a nice and compact script! Is there any documentation on the API that is available to the scripts? I couldn't find anything helpful for me.

@pspzockerscene

Hm, that is surprising. Let me do some testing to make sure I'm not missing anything. The only reason I can think of is that my JD2 is running on Linux, which is less tolerant about cases than the usual Windows FS, but that's a little far fetched.
Reply With Quote
  #13  
Old 10.11.2023, 09:52
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,121
Default

@Mli
If it really was about upper/lowercase, a script which changes all mountfile.net filenames in JD to lowercase should also do the trick.
At least under Windows, it's working just fine.

Regarding documentation:
See:
EventScripter forum thread:
https://board.jdownloader.org/showthread.php?t=70525
EventScripter help article:
https://support.jdownloader.org/Know...event-scripter
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 10.11.2023 at 09:52. Reason: Added EventScripter links
Reply With Quote
  #14  
Old 13.11.2023, 15:06
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 1,553
Default

Quote:
Originally Posted by pspzockerscene View Post
...script will fail too.
filename on linkcheck : mixed case
filename during download : lower case

File exists check might fail if it looks for a mixed case file on disk, becaise it would have been actually changed to lower case when the download starts. File name on download list and disk will both be in lower case so the script should work just fine.
Reply With Quote
  #15  
Old 13.11.2023, 15:10
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,121
Default

As said here under Windows (which is case-insensitive), it's working fine even with mixed case.
What happens if you manually rename 1-2 files to lowercase in JDownloader?
Are they then skipped fine without a download-attempt?
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #16  
Old 15.11.2023, 16:50
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 1,553
Default

Quote:
Originally Posted by pspzockerscene View Post
As said here under Windows (which is case-insensitive), it's working fine even with mixed case.
What happens if you manually rename 1-2 files to lowercase in JDownloader?
Are they then skipped fine without a download-attempt?
  1. I too am on windows where it is working fine.
  2. It is by default changed/saved in lowercase, which is what is causing problem in linux .
  3. In linux, file exists check should work fine if user manually changes the file name from lower case to mixed case (the one that mountfile provides on link check).

To recap, problem is caused by moutnfile providing mixed case file name during link chek but then changing it to lower case on download.

Guess the OP can shed more light.
Reply With Quote
  #17  
Old 15.11.2023, 17:58
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,121
Default

@mgpai
kk then we're both on the same side.

The user should ultimately be able to work around this problem by running an EventScripter script which changes all mountfile.net filenames in JDownloader to lowercase.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #18  
Old 16.11.2023, 11:00
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 1,553
Default

Quote:
Originally Posted by pspzockerscene View Post
@mgpai
kk then we're both on the same side.

The user should ultimately be able to work around this problem by running an EventScripter script which changes all mountfile.net filenames in JDownloader to lowercase.
Other way round. Check post #5.

Intial file name : AHappyFileToDownload.txt
Final file name : ahappyfiletodownload.txt

User has to change the the file name from lower case, back to intial file name case. Cannot be achieved by script since it is not a simple mixed case.
Reply With Quote
  #19  
Old 16.11.2023, 11:21
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,121
Default

Yes this is what I meant:
Change them from mixed case in JD to all lowercase to make "file already exists" check work.

By the way welcome back mgpai
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by raztoki; 16.11.2023 at 11:35.
Reply With Quote
  #20  
Old 17.11.2023, 17:00
MLi MLi is offline
Mega Loader
 
Join Date: Oct 2010
Posts: 68
Default

That was a great idea! And thanks to mgpai for the script, that was a great starting point. it took a little fiddling, but I think I have a script now that does exactly what I need:


Code:
if (name == "Cleanup") {
    var links = dlSelection.getDownloadLinks();
    for (i = 0; i < links.length; i++) {
        var link = links[i];
        
        var downloadURL = link.getContentURL();
        if(String(downloadURL).includes("mountfile"))
        {
            var filename = link.getName();
            var ln = filename.toLowerCase();
            link.setName(ln);
        }
        if (getPath(link.downloadPath).exists()) {
                print("Found",link.downloadPath, "removing", link);
                link.enabled = false;
                link.remove();
            }
            
    }
}
The only problem I have is that I need to run it twice to cleanup mountfile files that are already downloaded, somehow changing the name does not update the downloadPath. If there is a way to update that it would be nicer, but it's not really a big deal.

Thanks for your help and all the work on JD2!
Reply With Quote
  #21  
Old 17.11.2023, 21:25
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 1,553
Default

Quote:
Originally Posted by MLi View Post
... a way to update that ...
Code:
while (link.getName() != ln) {
    sleep(10);
}
Reply With Quote
  #22  
Old 18.11.2023, 12:14
MLi MLi is offline
Mega Loader
 
Join Date: Oct 2010
Posts: 68
Default

Ah! I did not expect that one to be async. Thanks for the hint!
Reply With Quote
  #23  
Old 22.11.2023, 12:37
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 1,553
Default

Quote:
Originally Posted by pspzockerscene View Post
By the way welcome back mgpai
Thanks @psp.
Still on the sidelines though.
Reply With Quote
  #24  
Old 22.11.2023, 13:03
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,121
Default

Quote:
Originally Posted by mgpai View Post
Still on the sidelines though.
I hope you are doing well
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #25  
Old 23.11.2023, 14:45
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 1,553
Default

Quote:
Originally Posted by pspzockerscene View Post
I hope you are doing well
Yes I am. Thank you.
Hope you are too.
Reply With Quote
  #26  
Old 23.11.2023, 15:05
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,121
Default

Yap "Alles gut"
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 22:03.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.