[Cyberavater]

JD2 sometimes doesn't recognize mirror servers/files properly.

Example Link; **External links are only visible to Support Staff****External links are only visible to Support Staff**

Debug-Log; 29.01.20 13.53.32 <--> 29.01.20 22.41.09 jdlog://7522430900751/

[pspzockerscene]

What exactly happens?

Please note that this most likely happens when filenames are different which is often either caused by the uploader or because depending on which site they upload to, these sites also manipulate filenames afterwards.
Adding your URLs, they seem to all have the same filenames. Mirror handling thus is working fine here.

Please also keep in mind that filenames can change once the download starts.
If e.g. the first mirror which is tried is one where the filename changes on downloadstart but the download fails --> Another mirror will be tried but later on, the first attempted file might get downloaded as it now has a different name.
From what I can see, the filenames of all of your example providers stay the same. Only letsupload.com will change for a short time during linkcheck. Did your issue happen with letsupload?

Your log contains a lot of login processes and download attempts - I'am unable to help you like this.
Please provide a screenshot and a clean log which only contains your issue.

-psp-

[Cyberavater]

Here is the SS; https://imgur.com/a/HKIXkmS

And, I'm not sure how I can post a separate log. [I'll need help with that].

[pspzockerscene]

Originally Posted by Cyberavater

And, I'm not sure how I can post a separate log. [I'll need help with that].

Delete your old logs (see jd installation folder), start JD and only cause your mentioned issue without doing anything else --> Then upload a new log

Can you reproduce this issue?
If so, how?

It does not look that fatal as JD clearly finished all downloads and did not download anything twice.

-psp-

[Cyberavater]

Delete your old logs (see JD installation folder), start JD and only cause your mentioned issue without doing anything else --> Then upload a new log

A. Then I'll have to download the whole file again. [Is it not possible to separate the log from whole?]

Can you reproduce this issue?
If so, how?

A. Just finish the download using that link.

It does not look that fatal as JD clearly finished all downloads and did not download anything twice.

A. That's because a prompt was shown, in which I ordered it to not download anything twice.

[pspzockerscene]

As said, there are no known mirror bugs.

You'll either have to download it again or provide some small testfiles which can maybe be used to reproduce this issue more easily.

Probably it would be ideal to have a 1MB testfile uploaded to all the filehosts which you have in your example.

-psp-

[Cyberavater]

Seems like I can't reproduce it anymore. Maybe this is a ghost issue. I'll post such log if I can reproduce it again.

[Cyberavater]

BTW, there is another thing.
As you mentioned ''Please note that this most likely happens when filenames are different which is often either caused by the uploader or because depending on which site they upload to, these sites also manipulate filenames afterward.''
Can't you make a smart algo for that? My file was downloaded twice for another link.
Link Eg; **External links are only visible to Support Staff****External links are only visible to Support Staff**

[pspzockerscene]

Originally Posted by Cyberavater

Seems like I can't reproduce it anymore. Maybe this is a ghost issue. I'll post such log if I can reproduce it again.

To be able to easily test, as said, a small file would be ideal.
I've uploaded a "1mb.rar" file to all of the filehosts which were used in your testlinks and I was unable to reproduce the issue:
ul.to/sszvu646
uptobox.com/lyy5vcpf88bc
letsupload.co/3vH57/1mb.rar
mega.nz/#!9QoWiQxa!LBr4k308_RlWmCEhToeksfeHAQF_B2sctAd20DTqf3U
drive.google.com/open?id=19ybiJRlodHoQx84M5ZvzbAJjU5J1po__

URL to pastebin containing all URLs with 'http' to add them to JD more easily:
pastebin.com/TGmYLyhY

Originally Posted by Cyberavater

BTW, there is another thing.
As you mentioned ''Please note that this most likely happens when filenames are different which is often either caused by the uploader or because depending on which site they upload to, these sites also manipulate filenames afterward.''
Can't you make a smart algo for that?

Well nothing is impossible but this is kind of pointless if you ask me.
These are the reasons:
- Anything could be happening e.g. a filehost adding random numbers to the start or end of filenames or adding their own tags e.g. 'realfilenamePart - myawesomefilehost.zip'
- The uploader could have mistakenly named files differently
- Files could be named slightly different on purpose e.g. if it is a re-upload and not a mirror thus not compatible with the old archives
- MD5hash, if given, could be slightly different although file is a mirror
- Filesizes are often not given exactly or not given at all
- Filenames are sometimes not given until the download really starts

Uploaders should know that especially when uploading to multiple filehost, they should use simply names, best only containing [a-z0-9] and the file-extension.
It is not up to our software to correct their mistakes and also such a feature could cause more harm than it helps because in most cases, the mirror situation is easy.

Originally Posted by Cyberavater

My file was downloaded twice for another link.
Link Eg; **External links are only visible to Support Staff****External links are only visible to Support Staff**

Which one was downloaded twice?
From which filehost?
Zippyshare is GEO-blocked here so I did not want to turn on my VPN atm. just to check this.

Please also note that you can configure the mirror handling.
Settings --> Advanced Settings --> Type in "mirror".

The URLs you initially posted did e.g. all have the same filename but filesize was different/not displayed for one of them.
For such cases, the setting "Mirror Detection Decision" --> "FILENAME" would e.g. be ideal.

-psp-

[Cyberavater]

Thanks for the advice! I'll report if the bug occurs again.

[Cyberavater]

Debug Log; 31.01.20 22.46.01 <--> 31.01.20 23.04.24 jdlog://5472430900751/

[Cyberavater]

Also plz update the algo to regard the " " and "_" as same, since it is the main cause of duplicate download and I don't think there is any side effect of this.

https://imgur.com/VUsi5CG

[pspzockerscene]

Originally Posted by Cyberavater

Debug Log; 31.01.20 22.46.01 <--> 31.01.20 23.04.24 jdlog://5472430900751/

Again no example URLs given, no screenshot.
It is easier to bring everything together and having a log than only having a log and no idea on what to look for ...

Originally Posted by Cyberavater

Also plz update the algo to regard the " " and "_" as same, since it is the main cause of duplicate download and I don't think there is any side effect of this.

There is not 'super smart' replace algo.

If you want to manually modify filenames of single filehosts, you can do this by either using the Packagizer (though it is not designed to easily replace single chars) or an Event Scripter script.

We also have a ticket regarding the feature request for a general filename replacer:


-psp-
EDIT

For more info, see raztokis post below.

[raztoki]

We already have function/handling for this for many years. The handling is within plugins and we map chars that are renamed, so effectively solves this issue, but we NEED to know which hoster renames chars, and the chars that they rename from and to.

Only reason I know this is, I came up with the idea =]

raztoki

[Cyberavater]

Originally Posted by pspzockerscene

Again no example URLs given, no screenshot.
It is easier to bring everything together and having a log than only having a log and no idea on what to look for ...

Link; **External links are only visible to Support Staff****External links are only visible to Support Staff**
SS; https://imgur.com/VUsi5CG
Debug Log; 31.01.20 22.46.01 <--> 31.01.20 23.04.24 jdlog://5472430900751/

[pspzockerscene]

Okay after a deeper investigation I ´found out this:
You are using the multihost real-debrid.
It seems like it is modifying the filenames.

When I simply add your testlinks, all filenames are correct and I'am unable to test this 100% but according to your log, real-debrid was used for the URLs whose filenames contain the space at the beginning (your screenshot).

Please re-check this.

-psp-
EDIT

I've created a log anyways as I'am pretty sure that the above explained is what happened.
As a workaround you can use the packagizer to simply set current filenames as final filenames. Filenames will then not be modified on downloadstart anymore.
Ticket:

[raztoki]

Originally Posted by Cyberavater

Link; **External links are only visible to Support Staff****External links are only visible to Support Staff**
SS; https://imgur.com/VUsi5CG
Debug Log; 31.01.20 22.46.01 <--> 31.01.20 23.04.24 jdlog://5472430900751/

May I ask your mirror setting (GeneralSettings.mirrordetectiondecision) when you downoaded/tested this as it has influence how JD treats links in a package?
so we fix it on links adding to linkgrabber within a package, the new filename is at the highest setting which only package customiser/this method/maybe you in the gui/apis can interface. But the mirror setting is a download stage event and depending on the mirror setting it will check all for a byte value to determine if a given link is a valid mirror. Even still this should not change the setfilename value.

[Cyberavater]

Originally Posted by pspzockerscene

Okay after a deeper investigation I ´found out this:
You are using the multihost real-debrid.
It seems like it is modifying the filenames.

When I simply add your testlinks, all filenames are correct and I'am unable to test this 100% but according to your log, real-debrid was used for the URLs whose filenames contain the space at the beginning (your screenshot).

Please re-check this.

Yes, you are correct I was using real-debrid. But the file name change is not at the beginning rather at the middle as one is "ChikanIF v2.00.part2.rar' and the other is "ChikanIF_v2.00.part2.rar" [Check before v2.00.part2]
https://imgur.com/VUsi5CG

Originally Posted by raztoki

May I ask your mirror setting (GeneralSettings.mirrordetectiondecision) when you downloaded/tested this as it has influence how JD treats links in a package?

Default setting. I'ven't changed it.

[raztoki]

Can you do further tests and change the mirror setting away from auto to just filename an see what happens.

the auto correction stuff work, even shows that by your comments and psp and my testing. It only fixes plugins that change at adding of links / linkchecking.

from my quick look at your screen shot and few plugins I could draw the following conclusion. they are all in plugins which only set .setName and not setFinalFilename. This means on download the source in which you are using a MH and it adopts the filename provided by url or HEADER response. This not a bug or error persay just flaw in how real plugins work, and MH implementation is only used for download. Since the original filename for those hosts never get changed or autocorrection (as deemed correct at the time) the setter at the highest level (.setForcedFileName) isn't set, thus it then can be altered (as your experiencing).

Also as a side note RD is known to switch your URL with ones it has in a internal DB so it can either provide you with faster dl speed or if they have internal issues you wont notice it. Doing this can also break our support as at anytime they could switch to another url, in which we do not know its source just the filename as they provide it in url or header.

To me the only quick fix this would be to implement fail safe solution on when we know they have changed we set finalfilename of the original.

[pspzockerscene]

If nothing helps, here is a packagizer rule which can be used as a possible workaround.
Please keep in mind that this can cause all sorts of other issues.
You couzld improve it by e.g. only applying it on archive filenames.


-psp-

[pspzockerscene]

Changes are live, see:
https://board.jdownloader.org/showthread.php?t=95411

[pspzockerscene]

I've closed the related ticket because:

• I consider this to be an edge case
• There needs to be a source of filenames JD can trust. Anything we'd change that would not trust the multi hoster could cause trouble for other downloads.
• JD has features that allow the user to correct file names
• Bad/broken file names of multi hosts should be considered a bug and should be reported to the multi hosts