#1
|
||||
|
||||
Issue: Dupes from LinkGrabber list will be crawled, dupes from Download list won't
The issue is, that links, which have duplicates in LinkGrabber list are still being verified and online-checked, even though they won't be added by default.
Links, which have duplicates in Downloads list won't be verified or online-checked- unless I 'restore' them. This is correct! For the full report with screenshots, please read the attachment. And, please verify, if what I'm writing here is correct. If not, I would edit it accordingly, so others can use this as (correct) information. Preface JD provides the uses with an option to 'Restore x Filtered Links' - which is enabled by default. Duplicate handling with default settings works as follows: No duplicates at all will be added to LinkGrabber list, when a link to a duplicate already exists in Download list or in LinkGrabber list. However, you can restore filtered duplicates of links which exist in Download list (but only those!). In order to be able to see restored links, you have to enable 'Already in Downloadlist' in 'Views' pane Issue, I found In my eyes, also verifying and online-checking of duplicates in LinkGrabber list should not be performed. There is different duplicate crawling handling depending on which list the duplicates were found in. Please see my test cases for details - in particular the red/bold phrases: Case 1: When I copy a link into the LinkGrabber window (Analyze and Add Links) whose URL is already in LinkGrabber list, the link is not added. It won't be shown, even with this box checked: But I see the following in the BubbleNotifier: The link is obviously being verified and online-checked. But Found Packages stays at zero. Case 2: If the link is not in LinkGrabber list but in Downloads list, it looks like this: Here too - but only since a more recent update - the link is no longer added to LinkGrabber list. But in this case 2 it is not verified or online-checked - recognizable by the zeros in BubbleNotifier. But the second difference to case 1 is that I have the option to restore the link that was filtered as a duplicate. In fact, it's probably not a restoration. Because - if I click on the 'Restore 1 filtered links button' - then, and only then this link will be verified and online-checked: Differences to case 1 are:
Case 3: If I now try to add the link again, the link is not added once more and is not verified or online-checked either: The behavior is identical to case 2, except that no restore is offered here and it is also not indicated why the link was not entered in the LinkGrabber list. Last edited by StefanM; 12.06.2022 at 16:31. Reason: Corrections after Developer Feedback and more precise description |
#2
|
||||
|
||||
I'm sorry but this is not true at all. This button exists since 31.10.2013! It's functionality has not changed since then.
__________________
JD-Dev & Server-Admin Last edited by Jiaz; 12.06.2022 at 15:03. |
#3
|
||||
|
||||
Quote:
When you use "Already in Download list" as filter condition, then of course it will be filtered/not added except you have a view that matches it, explained here https://board.jdownloader.org/showpo...23&postcount=7
__________________
JD-Dev & Server-Admin Last edited by Jiaz; 12.06.2022 at 15:04. |
#4
|
||||
|
||||
Case 1: you cannot add the same link multipe times to Linkgrabber list. You can disable this behaviour, see https://board.jdownloader.org/showpo...79&postcount=2
__________________
JD-Dev & Server-Admin |
#5
|
||||
|
||||
@Stefan: I've already explained it here, https://board.jdownloader.org/showpo...23&postcount=7
__________________
JD-Dev & Server-Admin |
#6
|
||||
|
||||
Quote:
Quote:
Because you've enabled a filter that has condition "Already in Downloads list".
__________________
JD-Dev & Server-Admin Last edited by Jiaz; 12.06.2022 at 15:12. |
#7
|
||||
|
||||
Found Packages stays at zero because no new package got added to list.
Found links is 1 because the link is NOT filtered, but not added to list because it's already part of the Linkgrabber list.
__________________
JD-Dev & Server-Admin |
#8
|
||||
|
||||
Quote:
Please do NOT attach a pdf of your post!
__________________
JD-Dev & Server-Admin Last edited by Jiaz; 12.06.2022 at 15:29. |
#9
|
||||
|
||||
Quote:
But in all of my numerous installations it just showed up for the first time a few weeks ago. Still remember this, because then I started to search for that, as it did not work as expected (which was due to another custom filter). Is it possible, that somehow a setting in 'advanced settings' it was (without me doing that) set or kept as disabled? I have thousands of dupes in my LinkGrabber list which prove that this option was not there or not enabled in my installations for a very loooong time then. |
#10
|
||||
|
||||
Quote:
In both cases you'll have a custom configuration which does not contain the button.
__________________
JD-Dev & Server-Admin |
#11
|
||||
|
||||
Quote:
__________________
JD-Dev & Server-Admin |
#12
|
||||
|
||||
Quote:
The feature to disable dupe checks in Linkgrabber was added on 31.03.2017 (default enabled). The dupe check itself exists since 2011.
__________________
JD-Dev & Server-Admin Last edited by Jiaz; 12.06.2022 at 15:36. |
#13
|
||||
|
||||
@StefanM: Just to make it clear. Dupe check is NOT done on filename/filesize but ONLY on link/internal link!
Dupes and Mirrors are different things.
__________________
JD-Dev & Server-Admin |
#14
|
||||
|
||||
Quote:
Well, I tested this at least 5 or 6 times back and forth: I can see the grabbing process or better: the verification/online check process. And I think it would be better, that - in case a duplicate was found - no verification/online check would be performed at all. This would also save a lot of time. Real Life Scenario Please note: I am talking about adding a list of links, not the contents of a web page, which would have to be crawled for links first. I add a few hundred links (paste a list of links) to LinkGrabber, and all of them are dupes, which already exist in LinkGrabber list, it would only cost seconds for JD to figure that out. But instead, all links are verified and online-checked first, which can take a lot of time. And after this process those dupes are not added to LinkkGrabber table. This my observation! And when I do the same with a list of links that have duplicates in Downloads list, then they won't be verified or online-checked, which is the behavior I asked for to implement it also for dupes in LinkGrabber list. |
#15
|
||||
|
||||
@StefanM: finally we come to the *real* topic or *issue* you want to be optimized!
So you would like to have an option/optimization that the crawling process should check for existing dupe in linkgrabber list to avoid unnecessary processing of the link, just to later *trash* it because it's already part of Linkgrabber list. So your wish is: add the same link again, abort it as soon as possible. in best case before it's been processed/online checked, right?
__________________
JD-Dev & Server-Admin Last edited by Jiaz; 12.06.2022 at 15:46. |
#16
|
||||
|
||||
Yes, I'm aware of that and remember that you use the hash value for that, as any other duplicate finder software would do. (I created the German GUIs for SpaceMan99 and DuplicateCleaner )
|
#17
|
||||
|
||||
Quote:
If you look at the pdf you will see the screenshot with that many dupes. Those are dupes of links in Downloads list. And I would not have gotten them, in case the filter/restore option would have been enabled. That's what I tried to say. |
#18
|
||||
|
||||
:P
Quote:
Exactly as you already do it with links in Downloads list. And for better understanding I created the pdf, where you can see this from the screenshots I was referring to. |
#19
|
||||
|
||||
@StefanM: I'm sorry but I don't understand! What is the problem here?
You add Link X into Linkgrabber and move to Downloads. Then you will be able to add Link X again to Linkgrabber except you've added a filter with condition *already in download list*. that will prevent this link to be added again. BUT in case you also have a matching view rule, then this link will be added to Linkgrabber because a matching view rule overrides a matching filter rule.
__________________
JD-Dev & Server-Admin |
#20
|
||||
|
||||
perfect! and I guess you're talking about your vk links, so I can use them for testing, right? as explained, this dupe check works on link/internal link, so I must check the plugin what information is available before the processing of the link
__________________
JD-Dev & Server-Admin |
#21
|
||||
|
||||
Quote:
This would make things so easy... I create a screenshot ( I use FastStone Capture) and then I can paste/embed it where I want to have it. But here I have to
That is 30 (!) actions instead of just pasting 5 pics. But there would also be another solution: A script, which would automatically upload my screenshot to a picture hoster and fetch the direct link, copying it to the clipboard. Last edited by StefanM; 12.06.2022 at 16:10. |
#22
|
||||
|
||||
Quote:
There are nice to use tools to create screenshots, auto upload to image hoster xy and copy url to clipboard, for example zscreen.
__________________
JD-Dev & Server-Admin Last edited by Jiaz; 12.06.2022 at 16:11. |
#23
|
|||
|
|||
Quote:
I do agree there are some people who may post sensitive information, but if they are too stupid to do that, it is on them. Responsible users should not have to pay for their mistakes. |
#24
|
||||
|
||||
Quote:
|
#25
|
||||
|
||||
@StefanM, @mgpai:
The board software does not support *paste image from clipboard* feature. Yes, I agree that such a feature would make it easier/more user friendly to post images but the images still would be visible for support staff only. There are reasons why we don't make attachments available to public and use url blacklisting.
__________________
JD-Dev & Server-Admin Last edited by Jiaz; 12.06.2022 at 16:44. |
#26
|
||||
|
||||
Quote:
Addendum: And if somebody is that stupid, it would even be worse for them: Because forced by forum policy to do so, they would upload sensitive information to a picture hoster, which they cannot even delete anymore. So, it would be for their own security to allow embedding screenshots directly, which they (or an admin) can delete any time, |
#27
|
||||
|
||||
Quote:
And you're not forced to do so. You may just upload as an attachment. I've explained it in my previous post and there is no need for further discussion on this.
__________________
JD-Dev & Server-Admin |
#28
|
|||
|
|||
Hear you. Loud and clear.
|
#29
|
||||
|
||||
Quote:
Well, it is your forum (but lots of unnecessary time spent by the forum members) There are indeed some hosters which allow you to delete your uploads again. But for that even more steps are needed: The forum member has to archive the link together with each picture they uploaded. At least two more steps, which makes it at least 8 (!) steps for each picture. Many might think "Is that worth it... ?". This was my last attempt to try to convince you. I guess if this still cannot convince you, I have to give up for this one. |
#30
|
||||
|
||||
It has nothing to do with convincing me. We don't want to actively monitor/check/filter uploaded attachments/links, so they are not available to public. If an attachment is useful for others, there are ways for us to make it accessible for others. But for normal bug reports, there is no need for others to have access to them. Same for links.
__________________
JD-Dev & Server-Admin |
#31
|
||||
|
||||
Quote:
In most cases, at least for me, the screenshot rarly helped me to understand what you really want.
__________________
JD-Dev & Server-Admin |
#32
|
|||
|
|||
I think it will also be useful to members who want to help each other out, but like @StefanM said, "Well, it's your forum".
|
#33
|
||||
|
||||
Quote:
It was a general suggestion to use the same process for both, dupe checking against Downloads list and dupe checking against LinkGrabber list. Links you consider to be dupes shouldn't be verified or online-checked unnecessarily. ... as you already do it with links that have dupes in Downloads list. Cannot see any dependence from the hoster, whatsoever. |
#34
|
||||
|
||||
read here
Quote:
Many plugins work with custom internal links like host://fileID but you cannot check a.com/file/supernice/fileID against that.
__________________
JD-Dev & Server-Admin |
#35
|
||||
|
||||
Quote:
... when there are dupes that are already in the Downloads list, this verification/online-check... is not being performed. The dupe gets 'kicked' out immediately. And only when I click restore... ... then - and only then - verfication/online-check will be performed. A dupe link in LinkGrabber list is not different from a dupe link in Downloads list! It just got moved from one location to another. If I would first add that dupe link from LinkGrabber list to Downloads list, then there would be no verification/online-check... What's the difference then??? |
#36
|
||||
|
||||
Quote:
So it heavily depends on what links you are adding, eg links that must be processed by a decrypter plugin first or direct download links that will be processed directly by a hoster plugin.
__________________
JD-Dev & Server-Admin |
#37
|
||||
|
||||
Correct and I agree. I just want to explain why this won't be possible for all links.
__________________
JD-Dev & Server-Admin |
#38
|
||||
|
||||
Quote:
So, same condition for all links. As I said: when I had copied a link to Downloads list and in the dupe check that was found out => no verification/online-checking... When I did exactly the same with such a link as dupe in LinkGrabber list => this resulted in verification/online-checking... And there is/was no processing at all when moving a link from LinkGrabber list to Downloads list. |
#39
|
||||
|
||||
For me it's the opposite, when the link has already existed in the LinkGrabber list, I immediately get "-> Filtered Dupe" (in the log), no access to the web.
__________________
FAQ: How to upload a Log Last edited by tony2long; 14.06.2022 at 10:34. |
#40
|
||||
|
||||
Quote:
What does it show, if and when it has been enabled? |
Thread Tools | |
Display Modes | |
|
|