#1
|
|||
|
|||
Multihoster: Prioritizing some filehosts over other filehosts
I got debrid account which allows downloading from certain hosters
In a package there is many filehosters which may not be part of that debrid service.. Now when I start the downloads I would like to have ability to set it so the file hosters that are part of the debrid get prioritized as the first hosters to try to download from them first before other hosters which are not part of the debrid service. Is there any way to do this in the settings or can this be added as a new feature in the settings |
#2
|
||||
|
||||
This should work fine out of the box.
Maybe our mirror detection fails for your parts. Please post example URLs and a log. Please post your log-ID here | bitte poste deine Log-ID hier. -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download Last edited by pspzockerscene; 12.10.2020 at 15:51. |
#3
|
|||
|
|||
Quote:
|
#4
|
||||
|
||||
it's not about prioritization but selection.
that's why it's important that mirror handling detects the links as mirrors and JDownloader tries best canditates first (eg premium, multihoster over free) you should notice the other mirrors being marked as mirror when this all works fine
__________________
JD-Dev & Server-Admin |
#5
|
|||
|
|||
Quote:
I think I know why now. example of links **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** It seems some filehoster (not part of the debrid service) is naming the mirror filename slightly differently so jdownloader is not seeing it as a mirror and starting the mirror first in the queue. Is there anything that can be done about this? The first link is name differently to the rest of the links. **External links are only visible to Support Staff****External links are only visible to Support Staff** The second and third links has a "(2)" added to the name. **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** So when I start the download queue it is trying the first link which is not part of the debrid filehoster service.. which then shows a captcha test. Is there a way to change jdownloader selection.. so that it sees that the majority of the files names are mirrors and download those and ignore/skip the slightly different name file that is also a mirror? I also find that sometimes jdownload will download the exact mirror file twice mainly because the file was named slightly differently on one of the filehoster..ie with extra word "(2).rar" And in this case, it did download the mirror twice since there is a mirror (part of the debrid service) that has that extra "(2)" word in the filename. So now I got two exact same files but one file has that extra word "(2)" at the end of the filename. Maybe jdownloader mirror identifying algorithm could be improve so that... -it look at all the words that are matching at the beginning of the filename (until the first non matching word/s occur) and then look at the season+episode name (e.g S06E01), therefore treat it as is the same match (even if the filename is slightly different) and so it should skip/ignore the slightly different name mirror file -OR just match on the season and episode number within a package and treat all filenames that match on that criteria as mirror, even if the filename maybe slightly different on the other filehoster.. -It should also treat files with the extra number/word "(2)" as a mirror (since it matches on the Season and episode number) and ignore/skip it first unless it couldn't download the other mirrors first. I think this might be the best way to avoid these two issues from occurring.. Last edited by madmax2; 15.10.2020 at 16:19. |
#6
|
||||
|
||||
@madmax2
If you ask me, improving mirror detection by "tolerating" a lot of seemingly random "mistakes" that filehosts make when renaming "duplicated" files is a very very bad idea. We do already try to correct smaller mistakes e.g. if a host always replaces specified characters with other specified characters, we try to fix this inside our plugins but your case is different. If you ask me, you want either one of those two things to happen - Wrongly named files should get detected as mirrors too or - Remove/ignore such files I would choose the first option. If that " (2)" issue happens only for megaup.net and letsupload.org, you can make a packagizer rule in JDownloader that removes that part of the filename --> Problem solved. Additionally, you might want to contact the uploader of these files - chances are, the mistake was made by him (although I don't think so). Also, I would not consider your issue as a JD issue. The more you download, the more edge cases you will find and although JD does a lot of work for you already, you cannot expect it to handle such edge cases. Also keep in mind that, in other cases, such names could have been set intentional which would lead to other failures if JD was to detect these ones as mirrors -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#7
|
|||
|
|||
Quote:
cos all these filehosters naming incorrectly is part of jdownloader filehoster plugins... This problem will occur if anyone has mirrors in a package (that has those filehosters that are naming the file differently) and then they click the start download triangle button.. If you look at my links I posted you can see that the filename matches the the 3 words and then it change a bit on the differently filehoster then match again on the S06E01 word.. What would the problem be if JDownloader matching algorithm sees that it matches those 3 words before it stop matching and then matches again on the S06E01 and identifies it as a mirror and skip/ignore the 3 filehoster that didn't quite match in that package? The worst that can happen is, jdownloader keeps the file in the package not downloaded, and a human can look at later to see why it was not downloaded.. This is much better than JD wasting people's bandwidth downloading 3 different files which are exactly the same file... Just had a look at my current hdd now, and noticed it has downloaded the same duplicates 2 duplicate for one show, and 3 duplicates for another show This is not good cos it has wasted bandwidth + time downloading unnecessary duplicates..as well as wearing out the hdd a bit more now by downloading those dupes...which I would now need to delete as well.. This has been an ongoing issue that I have been experiencing here and there but I just didn't report it yet, till now.. https://i.imgur.com/Xk0h6zL.png If a human was looking at those links, they can see clearly that all those links are mirrors of the same file, based on their own human matching logic.. Yet JDownloader is seeing 3 different files within that package... This is not good matching algorithm, and could / should be improved... Additionally, you might want to contact the uploader of these files - chances are, the mistake was made by him (although I don't think so). Personally I don't think the uploader had anything to do with how those filehoster has renamed the file slightly differently... so no point in contacting them, or even if they would care to fix it (if it was due to them renaming it, which is like you said, unlikely due to them) Would like to see what Jiaz can offer some input into this issue and whether the matching algorithm can be tweaked somewhat like I said.. Maybe not matching 100% filename is necessary to say it is a mirror but slightly less than 100% same filename provided if it has match the beginning of the filename is the same, until it didn't match then it match again on the season number, episode number (e.g. S06E01).. If it has match on two of those part of the filename, then JD can be somewhat be confident that is is a mirror.. since a different filename is unlike to match on two of those things .. CASE 1 e.g JD scans the links and sees the beginning of the filename... Matching ("Fear.the.Walking.Dead") + non matching + matching (S06E01) so it can ignore rest of the filename and confidently say this is a mirror. <match><not match><match S06E01><ignore> Conclusion = this is a mirror A differently filename is unlikely to match the first beginning part of the filename even if it has the same matching season+episode number e.g. Fear.the.Walking.Dead....S06E01.......rar The.Walking.Dead...S06E01.....rar <not match><match S06E01> Conclusion = this is not a mirror As you can see, S06E01 may have match for both filenames, but it does not match the beginning of the filename.. So they are not mirrors. CASE 2 Fear.the.Walking.Dead....S06E01.......rar Fear.the.Walking.Dead....S06E01......(2).rar <match><match><not match (2)> Conclusion = this is a mirror ---------------- I don't see how improving the algorithm like how I said should be consider tolerating .. since it has match the beginning of the filename + the S06E01 filename... JDownloader should be pretty confident that those links clearly are mirrors.. if the algorithm has match on those two parts of the filename.. This should be consider more of a smart matching improvement rather tolerating since JD is unlikely to be incorrect in this instance... A human would be 100% confident that it is mirror links, if they did the same thing.. This is like google improving their searching algorithm to get results And again what is the worst that can happen? JD would just ignore/skip those files that are in that package so that a human can look at it later to determine whether to download it or delete it.. This is better than what is happening right now which is JD wasting bandwidth downloading 3 files that are exactly the same file.. just named a bit different Note : This smarter algorithm could be added as an advance setting for people who want/need this better algorithm. I can be a tester if needed, to see if this smarter algorithm work or not work Last edited by Jiaz; 06.11.2020 at 18:30. |
#8
|
|||||
|
|||||
Quote:
You do expect our software to tolerate the mistakes of others in a way that I don't agree with. Also, in all the years I've done JD support this is maybe the 2nd time I see a request like yours so yes I do consider it an edge case. Quote:
Quote:
Files with different names could have different hashes too --> Different files --> Extraction may fail A real (= strict) mirror detection would match either based on filename & exact size and/or file-hash (sha1/md5). Ours is already very tolerant. Quote:
Even if tha uploader had no influence on this, he could at least chose different mirrors that do not automatically change his filenames! Quote:
Again: You can easily fix this by adding one single packagizer rule ... Maybe you do have more examples/example URLs with different filename-differences than the ones you've provided. If the " (2)" is always your problem, as said, you can easily fix that using a Packagizer rule because our packagizer exists for exactly such cases ... -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download Last edited by pspzockerscene; 15.10.2020 at 18:33. |
#9
|
|||||
|
|||||
Quote:
since there is logic to what I said in regards to improve the algorithm.. as noted in my two case examples above. Just cause you say this is the 2nd time you see this does not mean this not a bigger issue experience by many people... Not everyone bothers to report the problem.. Many people just "tolerate" it.. like myself.. Like I said, this has been an ongoing issue I have noticed with JD but I just chose to tolerate it.. rather than report it till now.. Quote:
If this is not a failure of JD then I don't know what else could be consider a failure.. The whole point of JD as a software is to automate downloading of files from many filehosters including avoiding downloading duplicates of the same file. If JD can't even get this right, then I might as manually download the single link myself instead of automating it in JD OR manually choose which filehost JD downloads from for each file rather than automating the download. Quote:
1. A human has copied all those links from the same source uploader who has specified that all these links are mirrors 2. There is no reason for the uploader to have uploaded 3 different files with similar/same matching names + same season/episode number and paste all those links in one URL location. 3. It matches the criteria as mention in the CASE1 and CASE 2 above.. If you are saying JD software is smarter or more correct than a human in this instance of identifying theses mirrors, then you are clearly wrong. Why? Because JD has downloaded 3 duplicates of the same file while a human would not have done this.. So tell me which is smarter? -a human or the software that have said "Nope these are not the same file" and downloaded all 3 of the same file. Quote:
so no point in try to convince you to see otherwise that this is a big problem/failure of JD.. One of the core function of JD is to automate downloading of files from many filehosters including avoiding downloading duplicates.. As you can clearly see, JD has failed on avoiding downloading duplicates from my screenshot as shown.. I don't know if you do much downloading from uploaders or know how it actually works.. but the uploaders are doing this for free, so telling them to use a different filehoster that does not have this naming issue has 0% chance of happening. The uploaders will just tell you to download from one of the other filehosts.. They don't care that this duplicate links are being downloaded in JD due to the incorrect identifying of mirrors. This is a JD software issue, not an uploader issue. Since we can manually download each link ourselves without automating it in JD.. Quote:
The problem is not just for duplicate links with the (2) It has also download duplicates that didn't contain the (2) See the highlighted red box of duplicate files below and the slightly non matching filenames. https://i.imgur.com/j9BNB55.png Again I have asked previously Please tell me what is the worst that can happen if this was implemented? 1) As I said above, the worst that would occur is the links are treated as mirrors inside that package so it would be skipped or ignored those links in that package and a human can look at it later on to decide whether to download it or delete it. 2) JD can confidently say those links are duplicate mirrors, because before JD had those links, a human had to have copied all those links from an uploader source URL (who has provided those mirrors) , so a human already knows those links are mirrors .. which JD then grabbed the links copied and placed all those links inside the same package. For JD to then say, nope all these links inside that package are not mirrors is just ridiculous and has wasted the human's time in copying those links into JD in the first place. This is akin to a GPS device saying there is no cliff, just keep driving ahead just cos it is using outdated map data.. Now should I trust the GPS that says there is no cliff, just keep driving or use my own eyes and see there is a cliff? e.g **External links are only visible to Support Staff****External links are only visible to Support Staff** AND... this can be implemented in advance setting so people who need this can turn it on while people who don't need or care can keep the current settings as it is.. VS JD keeps downloading multiple duplicates and wasting bandwidth + time + wear and tear of the hdd + wasting time needing to delete these dupes. Do you think this is perfectly acceptable for JD to continue doing this? Last edited by Jiaz; 06.11.2020 at 18:31. |
#10
|
|||
|
|||
Not really but cost benefit relation is not clear and I personally fear unexpected results.
|
#11
|
||||||||
|
||||||||
Oh, a new post!
I may as well use this to answer once again: Quote:
Yes I agree with you there is probably annoying behavior of JD going on for years that no one has reported so far but: - Without reports we cannot know about it - Assuming that one report/request (like yours) represents the one of multiple users is just not the way to go if you ask me Quote:
A lot of JD users nowdays rely on JD and are actually unable to download manually via browser ... because they never had to. The same accounts for things like duplicated/missing items or extraction: A lot of users do not even know what a multipart-zip-archive is and what to do with it ... and they don't have to because in most of all cases, JD will just handle it properly for them Quote:
This is again not a case for auto-mirror-matching if you ask me! You can report such cases to us and ask if that's a bug (if so, we'll fix it) and also report it to the uploader. Quote:
But yes I agree you're probably right and even the "worst cases" of wrong mirror detection won't cause dramatic failures. Quote:
Basically you're asking us to modify JD so you CAN trust it not to steer you down the cliff ... Quote:
Quote:
Again that may be the case for you. I haven't encountered this situation a single time and even if, that would not have caused mentionable additional wear to my hardware. Quote:
Also I do not see us having the time to implement any of your suggestions any time in the near future. My suggestion to you: - Use an EventScripter script to cover your edge-cases and/or: - Contribute code - we're open source ... and if you think there are more users who are interested in a solution for your problem, please let them know about this thread so they can leave a post in this thread. -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download Last edited by Jiaz; 06.11.2020 at 18:31. |
Thread Tools | |
Display Modes | |
|
|