[madmax2]

I got debrid account which allows downloading from certain hosters

In a package there is many filehosters which may not be part of that debrid service..

Now when I start the downloads
I would like to have ability to set it so the file hosters that are part of the debrid get prioritized as the first hosters to try to download from them first before other hosters which are not part of the debrid service.

Is there any way to do this in the settings
or can this be added as a new feature in the settings

[pspzockerscene]

This should work fine out of the box.
Maybe our mirror detection fails for your parts.
Please post example URLs and a log.

Please post your log-ID here | bitte poste deine Log-ID hier.

-psp-

[madmax2]

Originally Posted by pspzockerscene

This should work fine out of the box.
Maybe our mirror detection fails for your parts.
Please post example URLs and a log.

Please post **External links are only visible to Support Staff**... | bitte poste **External links are only visible to Support Staff**....

-psp-

so jdownloader already prioritizes debrid filehosters over other filehosters mirror of the same file?

[Jiaz]

it's not about prioritization but selection.
that's why it's important that mirror handling detects the links as mirrors
and JDownloader tries best canditates first (eg premium, multihoster over free)

you should notice the other mirrors being marked as mirror when this all works fine

[madmax2]

Originally Posted by Jiaz

it's not about prioritization but selection.
that's why it's important that mirror handling detects the links as mirrors
and JDownloader tries best canditates first (eg premium, multihoster over free)

you should notice the other mirrors being marked as mirror when this all works fine

After watching the problem occurring again,
I think I know why now.

example of links

**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

It seems some filehoster (not part of the debrid service) is naming the mirror filename slightly differently
so jdownloader is not seeing it as a mirror and starting the mirror first in the queue.

Is there anything that can be done about this?

The first link is name differently to the rest of the links.
**External links are only visible to Support Staff****External links are only visible to Support Staff**

The second and third links has a "(2)" added to the name.

**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

So when I start the download queue it is trying the first link which is not part of the debrid filehoster service.. which then shows a captcha test.

Is there a way to change jdownloader selection..
so that it sees that the majority of the files names are mirrors and download those and ignore/skip the slightly different name file that is also a mirror?


I also find that sometimes jdownload will download the exact mirror file twice
mainly because the file was named slightly differently on one of the filehoster..ie with extra word "(2).rar"

And in this case, it did download the mirror twice since there is a mirror (part of the debrid service) that has that extra "(2)" word in the filename.

So now I got two exact same files but one file has that extra word "(2)" at the end of the filename.

Maybe jdownloader mirror identifying algorithm could be improve so that...

-it look at all the words that are matching at the beginning of the filename (until the first non matching word/s occur) and then look at the season+episode name (e.g S06E01),
therefore treat it as is the same match (even if the filename is slightly different) and so it should skip/ignore the slightly different name mirror file

-OR just match on the season and episode number within a package
and treat all filenames that match on that criteria as mirror, even if the filename maybe slightly different on the other filehoster..

-It should also treat files with the extra number/word "(2)" as a mirror (since it matches on the Season and episode number) and ignore/skip it first unless it couldn't download the other mirrors first.


I think this might be the best way to avoid these two issues from occurring..

[pspzockerscene]

@madmax2
If you ask me, improving mirror detection by "tolerating" a lot of seemingly random "mistakes" that filehosts make when renaming "duplicated" files is a very very bad idea.

We do already try to correct smaller mistakes e.g. if a host always replaces specified characters with other specified characters, we try to fix this inside our plugins but your case is different.

If you ask me, you want either one of those two things to happen
- Wrongly named files should get detected as mirrors too
or
- Remove/ignore such files

I would choose the first option.
If that " (2)" issue happens only for megaup.net and letsupload.org, you can make a packagizer rule in JDownloader that removes that part of the filename --> Problem solved.

Additionally, you might want to contact the uploader of these files - chances are, the mistake was made by him (although I don't think so).

Also, I would not consider your issue as a JD issue.
The more you download, the more edge cases you will find and although JD does a lot of work for you already, you cannot expect it to handle such edge cases.
Also keep in mind that, in other cases, such names could have been set intentional which would lead to other failures if JD was to detect these ones as mirrors

-psp-

[madmax2]

Originally Posted by pspzockerscene

@madmax2
If you ask me, improving mirror detection by "tolerating" a lot of seemingly random "mistakes" that filehosts make when renaming "duplicated" files is a very very bad idea.

We do already try to correct smaller mistakes e.g. if a host always replaces specified characters with other specified characters, we try to fix this inside our plugins but your case is different.

If you ask me, you want either one of those two things to happen
- Wrongly named files should get detected as mirrors too
or
- Remove/ignore such files

I would choose the first option.
If that " (2)" issue happens only for megaup.net and letsupload.org, you can make a **External links are only visible to Support Staff**... in JDownloader that removes that part of the filename --> Problem solved.

Additionally, you might want to contact the uploader of these files - chances are, the mistake was made by him (although I don't think so).

Also, I would not consider your issue as a JD issue.
The more you download, the more edge cases you will find and although JD does a lot of work for you already, you cannot expect it to handle such edge cases.
Also keep in mind that, in other cases, such names could have been set intentional which would lead to other failures if JD was to detect these ones as mirrors

-psp-

I don't think my issue should be consider an edge case
cos all these filehosters naming incorrectly is part of jdownloader filehoster plugins...

This problem will occur if anyone has mirrors in a package (that has those filehosters that are naming the file differently)
and then they click the start download triangle button..

If you look at my links I posted you can see
that the filename matches the the 3 words and then it change a bit on the differently filehoster
then match again on the S06E01 word..

What would the problem be if JDownloader matching algorithm sees that it matches those 3 words before it stop matching and then matches again on the S06E01 and identifies it as a mirror and skip/ignore the 3 filehoster that didn't quite match in that package?

The worst that can happen is, jdownloader keeps the file in the package not downloaded, and a human can look at later to see why it was not downloaded..
This is much better than JD wasting people's bandwidth downloading 3 different files which are exactly the same file...

Just had a look at my current hdd now, and noticed it has downloaded the same duplicates
2 duplicate for one show, and 3 duplicates for another show



This is not good cos it has wasted bandwidth + time downloading unnecessary duplicates..as well as wearing out the hdd a bit more now by downloading those dupes...which I would now need to delete as well..

This has been an ongoing issue that I have been experiencing here and there
but I just didn't report it yet, till now..


https://i.imgur.com/Xk0h6zL.png

If a human was looking at those links, they can see clearly that all those links are mirrors of the same file, based on their own human matching logic..
Yet JDownloader is seeing 3 different files within that package...

This is not good matching algorithm, and could / should be improved...

Additionally, you might want to contact the uploader of these files - chances are, the mistake was made by him (although I don't think so).

Personally I don't think the uploader had anything to do with how those filehoster has renamed the file slightly differently...
so no point in contacting them, or even if they would care to fix it (if it was due to them renaming it, which is like you said, unlikely due to them)

Would like to see what Jiaz can offer some input into this issue
and whether the matching algorithm can be tweaked somewhat like I said..

Maybe not matching 100% filename is necessary to say it is a mirror but slightly less than 100% same filename
provided if it has match the beginning of the filename is the same, until it didn't match then it match again on the
season number, episode number (e.g. S06E01)..
If it has match on two of those part of the filename, then JD can be somewhat be confident that is is a mirror..
since a different filename is unlike to match on two of those things ..

CASE 1
e.g
JD scans the links and sees the beginning of the filename...

Matching ("Fear.the.Walking.Dead") + non matching + matching (S06E01) so it can ignore rest of the filename
and confidently say this is a mirror.

<match><not match><match S06E01><ignore>

Conclusion = this is a mirror

A differently filename is unlikely to match the first beginning part of the filename
even if it has the same matching season+episode number

e.g.
Fear.the.Walking.Dead....S06E01.......rar
The.Walking.Dead...S06E01.....rar

<not match><match S06E01>

Conclusion = this is not a mirror

As you can see, S06E01 may have match for both filenames, but it does not match the beginning of the filename..
So they are not mirrors.

CASE 2
Fear.the.Walking.Dead....S06E01.......rar
Fear.the.Walking.Dead....S06E01......(2).rar

<match><match><not match (2)>

Conclusion = this is a mirror

----------------

I don't see how improving the algorithm like how I said should be consider tolerating ..
since it has match the beginning of the filename + the S06E01 filename...

JDownloader should be pretty confident that those links clearly are mirrors..
if the algorithm has match on those two parts of the filename..

This should be consider more of a smart matching improvement rather tolerating
since JD is unlikely to be incorrect in this instance...

A human would be 100% confident that it is mirror links, if they did the same thing..

This is like google improving their searching algorithm to get results

And again what is the worst that can happen?

JD would just ignore/skip those files that are in that package so that a human can look at it later to determine whether to download it or delete it..

This is better than what is happening right now which is
JD wasting bandwidth downloading 3 files that are exactly the same file.. just named a bit different

Note : This smarter algorithm could be added as an advance setting for people who want/need this better algorithm.

I can be a tester if needed, to see if this smarter algorithm work or not work

[pspzockerscene]

Originally Posted by madmax2

I don't think my issue should be consider an edge case
cos all these filehosters naming incorrectly is part of jdownloader filehoster plugins...

This problem will occur if anyone has mirrors in a package (that has those filehosters that are naming the file differently)
and then they click the start download triangle button..

Sorry my opinion really differs from yours here.
You do expect our software to tolerate the mistakes of others in a way that I don't agree with.
Also, in all the years I've done JD support this is maybe the 2nd time I see a request like yours so yes I do consider it an edge case.

Originally Posted by madmax2

This is not good cos it has wasted bandwidth + time downloading unnecessary duplicates..as well as wearing out the hdd a bit more now by downloading those dupes...which I would now need to delete as well..

This has been an ongoing issue that I have been experiencing here and there
but I just didn't report it yet, till now..

This is a result of either the uploaders using bad filenames or the multi upload mirror service or also the complete selection of used hosts - I wouldn't treat this whole situation as any kind of JD failure ...

Originally Posted by madmax2

If a human was looking at those links, they can see clearly that all those links are mirrors of the same file, based on their own human matching logic..
Yet JDownloader is seeing 3 different files within that package...

No a human cannot - a human can only assume it's mirrors.
Files with different names could have different hashes too --> Different files --> Extraction may fail
A real (= strict) mirror detection would match either based on filename & exact size and/or file-hash (sha1/md5).
Ours is already very tolerant.

Originally Posted by madmax2

Personally I don't think the uploader had anything to do with how those filehoster has renamed the file slightly differently...
so no point in contacting them, or even if they would care to fix it (if it was due to them renaming it, which is like you said, unlikely due to them)

As explained above, I have a different opinion on this.
Even if tha uploader had no influence on this, he could at least chose different mirrors that do not automatically change his filenames!

Originally Posted by madmax2

Would like to see what Jiaz can offer some input into this issue
and whether the matching algorithm can be tweaked somewhat like I said..

Jiaz will probably look into this.
Again:
You can easily fix this by adding one single packagizer rule ...

Maybe you do have more examples/example URLs with different filename-differences than the ones you've provided.
If the " (2)" is always your problem, as said, you can easily fix that using a Packagizer rule because our packagizer exists for exactly such cases ...

-psp-

[madmax2]

Originally Posted by pspzockerscene

Sorry my opinion really differs from yours here.
You do expect our software to tolerate the mistakes of others in a way that I don't agree with.
Also, in all the years I've done JD support this is maybe the 2nd time I see a request like yours so yes I do consider it an edge case.

Well I don't consider this tolerating but smarter matching..
since there is logic to what I said in regards to improve the algorithm..
as noted in my two case examples above.

Just cause you say this is the 2nd time you see this does not mean this not a bigger issue experience by many people...

Not everyone bothers to report the problem..
Many people just "tolerate" it.. like myself..

Like I said, this has been an ongoing issue I have noticed with JD but I just chose to tolerate it.. rather than report it till now..

This is a result of either the uploaders using bad filenames or the multi upload mirror service or also the complete selection of used hosts - I wouldn't treat this whole situation as any kind of JD failure ...

So you think this is perfectly normal for JD to continue wasting bandwidth + time downloading multiple duplicates?

If this is not a failure of JD then I don't know what else could be consider a failure..

The whole point of JD as a software is to automate downloading of files from many filehosters including avoiding downloading duplicates of the same file.

If JD can't even get this right, then I might as manually download the single link myself instead of automating it in JD OR manually choose which filehost JD downloads from for each file rather than automating the download.

No a human cannot - a human can only assume it's mirrors.
Files with different names could have different hashes too --> Different files --> Extraction may fail
A real (= strict) mirror detection would match either based on filename & exact size and/or file-hash (sha1/md5).
Ours is already very tolerant.

Well actually a human can safely and smartly assume that the links are mirrors because:

1. A human has copied all those links from the same source uploader who has specified that all these links are mirrors

2. There is no reason for the uploader to have uploaded 3 different files with similar/same matching names + same season/episode number and paste all those links in one URL location.

3. It matches the criteria as mention in the CASE1 and CASE 2 above..

If you are saying JD software is smarter or more correct than a human in this instance of identifying theses mirrors, then you are clearly wrong.

Why?
Because JD has downloaded 3 duplicates of the same file
while a human would not have done this..

So tell me which is smarter?
-a human or the software that have said
"Nope these are not the same file" and downloaded all 3 of the same file.

As explained above, I have a different opinion on this.
Even if tha uploader had no influence on this, he could at least chose different mirrors that do not automatically change his filenames!

Yes, you clearly do have a different opinion on this problem
so no point in try to convince you to see otherwise that this is a big problem/failure of JD..

One of the core function of JD is to automate downloading of files from many filehosters including avoiding downloading duplicates..

As you can clearly see, JD has failed on avoiding downloading duplicates from my screenshot as shown..

I don't know if you do much downloading from uploaders or know how it actually works..
but the uploaders are doing this for free, so telling them to use a different filehoster that does not have this naming issue has 0% chance of happening.

The uploaders will just tell you to download from one of the other filehosts..
They don't care that this duplicate links are being downloaded in JD due to the incorrect identifying of mirrors.

This is a JD software issue, not an uploader issue.
Since we can manually download each link ourselves without automating it in JD..

Jiaz will probably look into this.
Again:
You can easily fix this by adding one single packagizer rule ...

Maybe you do have more examples/example URLs with different filename-differences than the ones you've provided.
If the " (2)" is always your problem, as said, you can easily fix that using a Packagizer rule because our packagizer exists for exactly such cases ...

yes, there are heaps more examples links that are having this issue..

The problem is not just for duplicate links with the (2)
It has also download duplicates that didn't contain the (2)

See the highlighted red box of duplicate files below and the slightly non matching filenames.

https://i.imgur.com/j9BNB55.png

Again I have asked previously

Please tell me what is the worst that can happen if this was implemented?

1) As I said above, the worst that would occur is the links are treated as mirrors inside that package so it would be skipped or ignored those links in that package
and a human can look at it later on to decide whether to download it or delete it.

2) JD can confidently say those links are duplicate mirrors, because before JD had those links, a human had to have copied all those links from an uploader source URL (who has provided those mirrors) , so a human already knows those links are mirrors .. which JD then grabbed the links copied and placed all those links inside the same package.

For JD to then say, nope all these links inside that package are not mirrors is just ridiculous and has wasted the human's time in copying those links into JD in the first place.

This is akin to a GPS device saying there is no cliff, just keep driving ahead
just cos it is using outdated map data..
Now should I trust the GPS that says there is no cliff, just keep driving
or use my own eyes and see there is a cliff?

e.g
**External links are only visible to Support Staff****External links are only visible to Support Staff**

AND...
this can be implemented in advance setting so people who need this can turn it on
while people who don't need or care can keep the current settings as it is..

VS

JD keeps downloading multiple duplicates and wasting bandwidth + time + wear and tear of the hdd + wasting time needing to delete these dupes.

Do you think this is perfectly acceptable for JD to continue doing this?

[KillerDAN]

Originally Posted by madmax2

Do you think this is perfectly acceptable for JD to continue doing this?

Not really but cost benefit relation is not clear and I personally fear unexpected results.

[pspzockerscene]

Oh, a new post!
I may as well use this to answer once again:

Originally Posted by madmax2

Well I don't consider this tolerating but smarter matching..
since there is logic to what I said in regards to improve the algorithm..
as noted in my two case examples above.

Just cause you say this is the 2nd time you see this does not mean this not a bigger issue experience by many people...

Not everyone bothers to report the problem..
Many people just "tolerate" it.. like myself..

Like I said, this has been an ongoing issue I have noticed with JD but I just chose to tolerate it.. rather than report it till now..



So you think this is perfectly normal for JD to continue wasting bandwidth + time downloading multiple duplicates?

If this is not a failure of JD then I don't know what else could be consider a failure..

The whole point of JD as a software is to automate downloading of files from many filehosters including avoiding downloading duplicates of the same file.

If JD can't even get this right, then I might as manually download the single link myself instead of automating it in JD OR manually choose which filehost JD downloads from for each file rather than automating the download.

You're still just trying to force us to work on a feature request of yours that so far, no one else has reported.
Yes I agree with you there is probably annoying behavior of JD going on for years that no one has reported so far but:
- Without reports we cannot know about it
- Assuming that one report/request (like yours) represents the one of multiple users is just not the way to go if you ask me

Originally Posted by madmax2

Well actually a human can safely and smartly assume that the links are mirrors because:

1. A human has copied all those links from the same source uploader who has specified that all these links are mirrors

2. There is no reason for the uploader to have uploaded 3 different files with similar/same matching names + same season/episode number and paste all those links in one URL location.

3. It matches the criteria as mention in the CASE1 and CASE 2 above..

If you are saying JD software is smarter or more correct than a human in this instance of identifying theses mirrors, then you are clearly wrong.

Why?
Because JD has downloaded 3 duplicates of the same file
while a human would not have done this..

That point goes to you though I must admin I might have expressed myself unclear:
A lot of JD users nowdays rely on JD and are actually unable to download manually via browser ... because they never had to.
The same accounts for things like duplicated/missing items or extraction:
A lot of users do not even know what a multipart-zip-archive is and what to do with it ... and they don't have to because in most of all cases, JD will just handle it properly for them

Originally Posted by madmax2

The problem is not just for duplicate links with the (2)
It has also download duplicates that didn't contain the (2)

See the highlighted red box of duplicate files below and the slightly non matching filenames.

https://i.imgur.com/j9BNB55.png

Your screenshot shows a case where either our plugin failed to handle encoding properly or the host failed to do so.
This is again not a case for auto-mirror-matching if you ask me!
You can report such cases to us and ask if that's a bug (if so, we'll fix it) and also report it to the uploader.

Originally Posted by madmax2

Again I have asked previously

Please tell me what is the worst that can happen if this was implemented?

1) As I said above, the worst that would occur is the links are treated as mirrors inside that package so it would be skipped or ignored those links in that package
and a human can look at it later on to decide whether to download it or delete it.

2) JD can confidently say those links are duplicate mirrors, because before JD had those links, a human had to have copied all those links from an uploader source URL (who has provided those mirrors) , so a human already knows those links are mirrors .. which JD then grabbed the links copied and placed all those links inside the same package.

For JD to then say, nope all these links inside that package are not mirrors is just ridiculous and has wasted the human's time in copying those links into JD in the first place.

Again that's your opinion.
But yes I agree you're probably right and even the "worst cases" of wrong mirror detection won't cause dramatic failures.

Originally Posted by madmax2

This is akin to a GPS device saying there is no cliff, just keep driving ahead
just cos it is using outdated map data..
Now should I trust the GPS that says there is no cliff, just keep driving
or use my own eyes and see there is a cliff?

e.g
**External links are only visible to Support Staff****External links are only visible to Support Staff**

Bad comparison.
Basically you're asking us to modify JD so you CAN trust it not to steer you down the cliff ...

Originally Posted by madmax2

AND...
this can be implemented in advance setting so people who need this can turn it on
while people who don't need or care can keep the current settings as it is..

Sure that's what settings are there for.

Originally Posted by madmax2

JD keeps downloading multiple duplicates and wasting bandwidth + time + wear and tear of the hdd + wasting time needing to delete these dupes.

Again that may be the case for you.
I haven't encountered this situation a single time and even if, that would not have caused mentionable additional wear to my hardware.

Originally Posted by madmax2

Do you think this is perfectly acceptable for JD to continue doing this?

Yes.
Also I do not see us having the time to implement any of your suggestions any time in the near future.

My suggestion to you:
- Use an EventScripter script to cover your edge-cases
and/or:
- Contribute code - we're open source

... and if you think there are more users who are interested in a solution for your problem, please let them know about this thread so they can leave a post in this thread.

-psp-