JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 29.01.2020, 17:44
Cyberavater Cyberavater is offline
JD VIP
 
Join Date: Feb 2019
Posts: 435
Default Mirror recognization issue using multihoster

JD2 sometimes doesn't recognize mirror servers/files properly.

Example Link; **External links are only visible to Support Staff****External links are only visible to Support Staff**

Debug-Log; 29.01.20 13.53.32 <--> 29.01.20 22.41.09 jdlog://7522430900751/

Last edited by Cyberavater; 29.01.2020 at 17:46.
Reply With Quote
  #2  
Old 29.01.2020, 18:01
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 51,115
Default

What exactly happens?

Please note that this most likely happens when filenames are different which is often either caused by the uploader or because depending on which site they upload to, these sites also manipulate filenames afterwards.
Adding your URLs, they seem to all have the same filenames. Mirror handling thus is working fine here.

Please also keep in mind that filenames can change once the download starts.
If e.g. the first mirror which is tried is one where the filename changes on downloadstart but the download fails --> Another mirror will be tried but later on, the first attempted file might get downloaded as it now has a different name.
From what I can see, the filenames of all of your example providers stay the same. Only letsupload.com will change for a short time during linkcheck. Did your issue happen with letsupload?

Your log contains a lot of login processes and download attempts - I'am unable to help you like this.
Please provide a screenshot and a clean log which only contains your issue.

-psp-
__________________

Ad-free installers || Werbefreie Installer
Windows Setup<--JD2 BETA-->Linux Setup x86 || Linux Setup x64 || Mac Setup
-----=>Support Chat<=-----
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #3  
Old 29.01.2020, 18:09
Cyberavater Cyberavater is offline
JD VIP
 
Join Date: Feb 2019
Posts: 435
Default

Here is the SS; https://imgur.com/a/HKIXkmS

And, I'm not sure how I can post a separate log. [I'll need help with that].
Reply With Quote
  #4  
Old 29.01.2020, 18:17
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 51,115
Default

Quote:
Originally Posted by Cyberavater View Post
And, I'm not sure how I can post a separate log. [I'll need help with that].
Delete your old logs (see jd installation folder), start JD and only cause your mentioned issue without doing anything else --> Then upload a new log

Can you reproduce this issue?
If so, how?

It does not look that fatal as JD clearly finished all downloads and did not download anything twice.

-psp-
__________________

Ad-free installers || Werbefreie Installer
Windows Setup<--JD2 BETA-->Linux Setup x86 || Linux Setup x64 || Mac Setup
-----=>Support Chat<=-----
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #5  
Old 29.01.2020, 18:28
Cyberavater Cyberavater is offline
JD VIP
 
Join Date: Feb 2019
Posts: 435
Default

Delete your old logs (see JD installation folder), start JD and only cause your mentioned issue without doing anything else --> Then upload a new log

A. Then I'll have to download the whole file again. [Is it not possible to separate the log from whole?]

Can you reproduce this issue?
If so, how?

A. Just finish the download using that link.

It does not look that fatal as JD clearly finished all downloads and did not download anything twice.

A. That's because a prompt was shown, in which I ordered it to not download anything twice.
Reply With Quote
  #6  
Old 29.01.2020, 18:41
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 51,115
Default

As said, there are no known mirror bugs.

You'll either have to download it again or provide some small testfiles which can maybe be used to reproduce this issue more easily.

Probably it would be ideal to have a 1MB testfile uploaded to all the filehosts which you have in your example.

-psp-
__________________

Ad-free installers || Werbefreie Installer
Windows Setup<--JD2 BETA-->Linux Setup x86 || Linux Setup x64 || Mac Setup
-----=>Support Chat<=-----
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #7  
Old 29.01.2020, 19:01
Cyberavater Cyberavater is offline
JD VIP
 
Join Date: Feb 2019
Posts: 435
Default

Seems like I can't reproduce it anymore. Maybe this is a ghost issue. I'll post such log if I can reproduce it again.
Reply With Quote
  #8  
Old 29.01.2020, 19:06
Cyberavater Cyberavater is offline
JD VIP
 
Join Date: Feb 2019
Posts: 435
Default

BTW, there is another thing.
As you mentioned ''Please note that this most likely happens when filenames are different which is often either caused by the uploader or because depending on which site they upload to, these sites also manipulate filenames afterward.''
Can't you make a smart algo for that? My file was downloaded twice for another link.
Link Eg; **External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #9  
Old 31.01.2020, 00:19
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 51,115
Default

Quote:
Originally Posted by Cyberavater View Post
Seems like I can't reproduce it anymore. Maybe this is a ghost issue. I'll post such log if I can reproduce it again.
To be able to easily test, as said, a small file would be ideal.
I've uploaded a "1mb.rar" file to all of the filehosts which were used in your testlinks and I was unable to reproduce the issue:
ul.to/sszvu646
uptobox.com/lyy5vcpf88bc
letsupload.co/3vH57/1mb.rar
mega.nz/#!9QoWiQxa!LBr4k308_RlWmCEhToeksfeHAQF_B2sctAd20DTqf3U
drive.google.com/open?id=19ybiJRlodHoQx84M5ZvzbAJjU5J1po__

URL to pastebin containing all URLs with 'http' to add them to JD more easily:
pastebin.com/TGmYLyhY

Quote:
Originally Posted by Cyberavater View Post
BTW, there is another thing.
As you mentioned ''Please note that this most likely happens when filenames are different which is often either caused by the uploader or because depending on which site they upload to, these sites also manipulate filenames afterward.''
Can't you make a smart algo for that?
Well nothing is impossible but this is kind of pointless if you ask me.
These are the reasons:
- Anything could be happening e.g. a filehost adding random numbers to the start or end of filenames or adding their own tags e.g. 'realfilenamePart - myawesomefilehost.zip'
- The uploader could have mistakenly named files differently
- Files could be named slightly different on purpose e.g. if it is a re-upload and not a mirror thus not compatible with the old archives
- MD5hash, if given, could be slightly different although file is a mirror
- Filesizes are often not given exactly or not given at all
- Filenames are sometimes not given until the download really starts

Uploaders should know that especially when uploading to multiple filehost, they should use simply names, best only containing [a-z0-9] and the file-extension.
It is not up to our software to correct their mistakes and also such a feature could cause more harm than it helps because in most cases, the mirror situation is easy.
Quote:
Originally Posted by Cyberavater View Post
My file was downloaded twice for another link.
Link Eg; **External links are only visible to Support Staff****External links are only visible to Support Staff**
Which one was downloaded twice?
From which filehost?
Zippyshare is GEO-blocked here so I did not want to turn on my VPN atm. just to check this.

Please also note that you can configure the mirror handling.
Settings --> Advanced Settings --> Type in "mirror".

The URLs you initially posted did e.g. all have the same filename but filesize was different/not displayed for one of them.
For such cases, the setting "Mirror Detection Decision" --> "FILENAME" would e.g. be ideal.

-psp-
__________________

Ad-free installers || Werbefreie Installer
Windows Setup<--JD2 BETA-->Linux Setup x86 || Linux Setup x64 || Mac Setup
-----=>Support Chat<=-----
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #10  
Old 31.01.2020, 11:50
Cyberavater Cyberavater is offline
JD VIP
 
Join Date: Feb 2019
Posts: 435
Default

Thanks for the advice! I'll report if the bug occurs again.
Reply With Quote
  #11  
Old 31.01.2020, 18:05
Cyberavater Cyberavater is offline
JD VIP
 
Join Date: Feb 2019
Posts: 435
Default

Debug Log; 31.01.20 22.46.01 <--> 31.01.20 23.04.24 jdlog://5472430900751/
Reply With Quote
  #12  
Old 31.01.2020, 18:17
Cyberavater Cyberavater is offline
JD VIP
 
Join Date: Feb 2019
Posts: 435
Default

Also plz update the algo to regard the " " and "_" as same, since it is the main cause of duplicate download and I don't think there is any side effect of this.

https://imgur.com/VUsi5CG
Reply With Quote
  #13  
Old 31.01.2020, 22:50
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 51,115
Default

Quote:
Originally Posted by Cyberavater View Post
Debug Log; 31.01.20 22.46.01 <--> 31.01.20 23.04.24 jdlog://5472430900751/
Again no example URLs given, no screenshot.
It is easier to bring everything together and having a log than only having a log and no idea on what to look for ...

Quote:
Originally Posted by Cyberavater View Post
Also plz update the algo to regard the " " and "_" as same, since it is the main cause of duplicate download and I don't think there is any side effect of this.
There is not 'super smart' replace algo.

If you want to manually modify filenames of single filehosts, you can do this by either using the Packagizer (though it is not designed to easily replace single chars) or an Event Scripter script.

We also have a ticket regarding the feature request for a general filename replacer:


-psp-
EDIT

For more info, see raztokis post below.
__________________

Ad-free installers || Werbefreie Installer
Windows Setup<--JD2 BETA-->Linux Setup x86 || Linux Setup x64 || Mac Setup
-----=>Support Chat<=-----
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist

Last edited by pspzockerscene; 31.01.2020 at 23:13.
Reply With Quote
  #14  
Old 31.01.2020, 23:08
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 16,702
Default

We already have function/handling for this for many years. The handling is within plugins and we map chars that are renamed, so effectively solves this issue, but we NEED to know which hoster renames chars, and the chars that they rename from and to.

Only reason I know this is, I came up with the idea =]

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #15  
Old 31.01.2020, 23:12
Cyberavater Cyberavater is offline
JD VIP
 
Join Date: Feb 2019
Posts: 435
Default

Quote:
Originally Posted by pspzockerscene View Post
Again no example URLs given, no screenshot.
It is easier to bring everything together and having a log than only having a log and no idea on what to look for ...
Link; **External links are only visible to Support Staff****External links are only visible to Support Staff**
SS; https://imgur.com/VUsi5CG
Debug Log; 31.01.20 22.46.01 <--> 31.01.20 23.04.24 jdlog://5472430900751/
Reply With Quote
  #16  
Old 31.01.2020, 23:54
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 51,115
Default

Okay after a deeper investigation I ´found out this:
You are using the multihost real-debrid.
It seems like it is modifying the filenames.

When I simply add your testlinks, all filenames are correct and I'am unable to test this 100% but according to your log, real-debrid was used for the URLs whose filenames contain the space at the beginning (your screenshot).

Please re-check this.

-psp-
EDIT

I've created a log anyways as I'am pretty sure that the above explained is what happened.
As a workaround you can use the packagizer to simply set current filenames as final filenames. Filenames will then not be modified on downloadstart anymore.
Ticket:
__________________

Ad-free installers || Werbefreie Installer
Windows Setup<--JD2 BETA-->Linux Setup x86 || Linux Setup x64 || Mac Setup
-----=>Support Chat<=-----
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist

Last edited by pspzockerscene; 01.02.2020 at 00:03.
Reply With Quote
  #17  
Old 01.02.2020, 00:07
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 16,702
Default

Quote:
Originally Posted by Cyberavater View Post
Link; **External links are only visible to Support Staff****External links are only visible to Support Staff**
SS; https://imgur.com/VUsi5CG
Debug Log; 31.01.20 22.46.01 <--> 31.01.20 23.04.24 jdlog://5472430900751/
May I ask your mirror setting (GeneralSettings.mirrordetectiondecision) when you downoaded/tested this as it has influence how JD treats links in a package?
so we fix it on links adding to linkgrabber within a package, the new filename is at the highest setting which only package customiser/this method/maybe you in the gui/apis can interface. But the mirror setting is a download stage event and depending on the mirror setting it will check all for a byte value to determine if a given link is a valid mirror. Even still this should not change the setfilename value.
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]

Last edited by raztoki; 01.02.2020 at 00:08. Reason: err psp has written extensive response since #15
Reply With Quote
  #18  
Old 01.02.2020, 04:30
Cyberavater Cyberavater is offline
JD VIP
 
Join Date: Feb 2019
Posts: 435
Default

Quote:
Originally Posted by pspzockerscene View Post
Okay after a deeper investigation I ´found out this:
You are using the multihost real-debrid.
It seems like it is modifying the filenames.

When I simply add your testlinks, all filenames are correct and I'am unable to test this 100% but according to your log, real-debrid was used for the URLs whose filenames contain the space at the beginning (your screenshot).

Please re-check this.
Yes, you are correct I was using real-debrid. But the file name change is not at the beginning rather at the middle as one is "ChikanIF v2.00.part2.rar' and the other is "ChikanIF_v2.00.part2.rar" [Check before v2.00.part2]
https://imgur.com/VUsi5CG

Quote:
Originally Posted by raztoki View Post
May I ask your mirror setting (GeneralSettings.mirrordetectiondecision) when you downloaded/tested this as it has influence how JD treats links in a package?
Default setting. I'ven't changed it.

Last edited by Cyberavater; 01.02.2020 at 04:33. Reason: I can share my real-debrid account if you need that to fix this issue
Reply With Quote
  #19  
Old 01.02.2020, 06:17
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 16,702
Default

Can you do further tests and change the mirror setting away from auto to just filename an see what happens.

the auto correction stuff work, even shows that by your comments and psp and my testing. It only fixes plugins that change at adding of links / linkchecking.

from my quick look at your screen shot and few plugins I could draw the following conclusion. they are all in plugins which only set .setName and not setFinalFilename. This means on download the source in which you are using a MH and it adopts the filename provided by url or HEADER response. This not a bug or error persay just flaw in how real plugins work, and MH implementation is only used for download. Since the original filename for those hosts never get changed or autocorrection (as deemed correct at the time) the setter at the highest level (.setForcedFileName) isn't set, thus it then can be altered (as your experiencing).

Also as a side note RD is known to switch your URL with ones it has in a internal DB so it can either provide you with faster dl speed or if they have internal issues you wont notice it. Doing this can also break our support as at anytime they could switch to another url, in which we do not know its source just the filename as they provide it in url or header.

To me the only quick fix this would be to implement fail safe solution on when we know they have changed we set finalfilename of the original.
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]

Last edited by raztoki; 01.02.2020 at 07:51.
Reply With Quote
  #20  
Old 03.02.2020, 14:32
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 51,115
Default

If nothing helps, here is a packagizer rule which can be used as a possible workaround.
Please keep in mind that this can cause all sorts of other issues.
You couzld improve it by e.g. only applying it on archive filenames.


-psp-
__________________

Ad-free installers || Werbefreie Installer
Windows Setup<--JD2 BETA-->Linux Setup x86 || Linux Setup x64 || Mac Setup
-----=>Support Chat<=-----
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 13:12.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.