JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 12.06.2022, 14:00
StefanM's Avatar
StefanM StefanM is offline
JD VIP
 
Join Date: Oct 2020
Posts: 485
Default Issue: Dupes from LinkGrabber list will be crawled, dupes from Download list won't

The issue is, that links, which have duplicates in LinkGrabber list are still being verified and online-checked, even though they won't be added by default.
Links, which have duplicates in Downloads list won't be verified or online-checked- unless I 'restore' them. This is correct!

For the full report with screenshots, please read the attachment.
And, please verify, if what I'm writing here is correct. If not, I would edit it accordingly, so others can use this as (correct) information.


Preface
JD provides the uses with an option to 'Restore x Filtered Links' - which is enabled by default.
Duplicate handling with default settings works as follows:
No duplicates at all will be added to LinkGrabber list, when a link to a duplicate already exists in Download list or in LinkGrabber list.
However, you can restore filtered duplicates of links which exist in Download list (but only those!). In order to be able to see restored links, you have to enable 'Already in Downloadlist' in 'Views' pane

Issue, I found
In my eyes, also verifying and online-checking of duplicates in LinkGrabber list should not be performed.
There is different duplicate crawling handling depending on which list the duplicates were found in.
Please see my test cases for details - in particular the red/bold phrases:

Case 1:
When I copy a link into the LinkGrabber window (Analyze and Add Links) whose URL is already in LinkGrabber list, the link is not added. It won't be shown, even with this box checked:

But I see the following in the BubbleNotifier: The link is obviously being verified and online-checked.

But Found Packages stays at zero.

Case 2:
If the link is not in LinkGrabber list but in Downloads list, it looks like this:

Here too - but only since a more recent update - the link is no longer added to LinkGrabber list. But in this case 2 it is not verified or online-checked - recognizable by the zeros in BubbleNotifier.
But the second difference to case 1 is that I have the option to restore the link that was filtered as a duplicate.
In fact, it's probably not a restoration. Because - if I click on the 'Restore 1 filtered links button' - then, and only then this link will be verified and online-checked:

Differences to case 1 are:
  1. The file is now added to LinkGrabber list and displayed - provided I have ticked the 'Already in Downloadlist' box. It is highlighted in red as a duplicate.
  2. Found Packages is now displayed as 1.

Case 3:
If I now try to add the link again, the link is not added once more and is not verified or online-checked either:

The behavior is identical to case 2, except that no restore is offered here and it is also not indicated why the link was not entered in the LinkGrabber list.
Attached Files
File Type: zip Duplicate Handling.zip (134.7 KB, 1 views)

Last edited by StefanM; 12.06.2022 at 16:31. Reason: Corrections after Developer Feedback and more precise description
Reply With Quote
  #2  
Old 12.06.2022, 14:48
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by StefanM View Post
With one of the more recent updates an option 'Restore x Filtered Links' was introduced - which is enabled by default.
I'm sorry but this is not true at all. This button exists since 31.10.2013! It's functionality has not changed since then.
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 12.06.2022 at 15:03.
Reply With Quote
  #3  
Old 12.06.2022, 15:24
StefanM's Avatar
StefanM StefanM is offline
JD VIP
 
Join Date: Oct 2020
Posts: 485
Default

Quote:
Originally Posted by Jiaz View Post
I'm sorry but this is not true at all. This button exists since 31.10.2013! It's functionality has not changed since then.
Of course I believe what you say.
But in all of my numerous installations it just showed up for the first time a few weeks ago.

Still remember this, because then I started to search for that, as it did not work as expected (which was due to another custom filter).

Is it possible, that somehow a setting in 'advanced settings' it was (without me doing that) set or kept as disabled?

I have thousands of dupes in my LinkGrabber list which prove that this option was not there or not enabled in my installations for a very loooong time then.
Reply With Quote
  #4  
Old 12.06.2022, 15:27
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by StefanM View Post
Of course I believe what you say.
But in all of my numerous installations it just showed up for the first time a few weeks ago.
One explanation would be that you have customized your menu and either removed or customized it before the botton got added.
In both cases you'll have a custom configuration which does not contain the button.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #5  
Old 12.06.2022, 15:29
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by StefanM View Post
Is it possible, that somehow a setting in 'advanced settings' it was (without me doing that) set or kept as disabled?
Already answered here, https://board.jdownloader.org/showpo...07&postcount=7
__________________
JD-Dev & Server-Admin
Reply With Quote
  #6  
Old 12.06.2022, 15:30
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by StefanM View Post
I have thousands of dupes in my LinkGrabber list which prove that this option was not there or not enabled in my installations for a very loooong time then.
The restore feature has NOTHING to do with dupes. You can only add dupes when having the dupe check disabled, previously answered here https://board.jdownloader.org/showpo...28&postcount=6.
The feature to disable dupe checks in Linkgrabber was added on 31.03.2017 (default enabled). The dupe check itself exists since 2011.
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 12.06.2022 at 15:36.
Reply With Quote
  #7  
Old 12.06.2022, 15:55
StefanM's Avatar
StefanM StefanM is offline
JD VIP
 
Join Date: Oct 2020
Posts: 485
Default

Quote:
Originally Posted by Jiaz View Post
The restore feature has NOTHING to do with dupes. You can only add dupes when having the dupe check disabled, previously answered here **External links are only visible to Support Staff**....
The feature to disable dupe checks in Linkgrabber was added on 31.03.2017 (default enabled). The dupe check itself exists since 2011.
Again a misunderstanding!

If you look at the pdf you will see the screenshot with that many dupes. Those are dupes of links in Downloads list.

And I would not have gotten them, in case the filter/restore option would have been enabled. That's what I tried to say.
Reply With Quote
  #8  
Old 12.06.2022, 14:50
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by StefanM View Post
No duplicates at all will be added to LinkGrabber list, when a link to a duplicate already exists in Download list or in LinkGrabber list.
Again that is not true. I've already explained here, https://board.jdownloader.org/showpo...79&postcount=2. You can add a link that is in Download list without any problem again to Linkgrabber list. As explained you cannot add the same link multiple times to Linkgrabber list.

When you use "Already in Download list" as filter condition, then of course it will be filtered/not added except you have a view that matches it, explained here https://board.jdownloader.org/showpo...23&postcount=7
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 12.06.2022 at 15:04.
Reply With Quote
  #9  
Old 12.06.2022, 14:53
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Case 1: you cannot add the same link multipe times to Linkgrabber list. You can disable this behaviour, see https://board.jdownloader.org/showpo...79&postcount=2
__________________
JD-Dev & Server-Admin
Reply With Quote
  #10  
Old 12.06.2022, 14:58
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@Stefan: I've already explained it here, https://board.jdownloader.org/showpo...23&postcount=7
__________________
JD-Dev & Server-Admin
Reply With Quote
  #11  
Old 12.06.2022, 15:00
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by StefanM View Post
In my eyes, also crawling of duplicates in LinkGrabber list should not be performed.
Quote:
Originally Posted by StefanM View Post
The issue is, that links, which have duplicates in LinkGrabber list are still being crawled, even though they won't be added by default.
JDownloader doesn't know the link being a duplicate until this duplicate check happens. So it must analyze/crawl/process in order to *know* the link (name, internal details..) and then can check for existing duplicate in LinkGrabber. The duplicate check happens AFTER analyze/crawl/process because only then internal details will be available. The duplicate check happens on internal details. for example, some hosts do have multiple domains or multipe url formats that lead to the same file and this information is not known in advance, so the link must be processed first. then duplicate check can detect a.com/test.txt and b.com/text.txt are the same file and not add it twice.

Quote:
Originally Posted by StefanM View Post
Links, which have duplicates in Downloads list won't be crawled - unless I 'restore' them. This is correct!
Because you've enabled a filter that has condition "Already in Downloads list".
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 12.06.2022 at 15:12.
Reply With Quote
  #12  
Old 12.06.2022, 15:19
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by StefanM View Post
But Found Packages stays at zero.
Found Packages stays at zero because no new package got added to list.
Found links is 1 because the link is NOT filtered, but not added to list because it's already
part of the Linkgrabber list.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #13  
Old 12.06.2022, 15:40
StefanM's Avatar
StefanM StefanM is offline
JD VIP
 
Join Date: Oct 2020
Posts: 485
Default

Quote:
Originally Posted by Jiaz View Post
Found Packages stays at zero because no new package got added to list.
Found links is 1 because the link is NOT filtered, but not added to list because it's already
part of the Linkgrabber list.
I guess my wording - using the general terms crawling/grabbing - was misleading.

Well, I tested this at least 5 or 6 times back and forth:
I can see the grabbing process or better: the verification/online check process.

And I think it would be better, that - in case a duplicate was found - no verification/online check would be performed at all. This would also save a lot of time.

Real Life Scenario
Please note: I am talking about adding a list of links, not the contents of a web page, which would have to be crawled for links first.

I add a few hundred links (paste a list of links) to LinkGrabber, and all of them are dupes, which already exist in LinkGrabber list, it would only cost seconds for JD to figure that out.

But instead, all links are verified and online-checked first, which can take a lot of time. And after this process those dupes are not added to LinkkGrabber table.

This my observation!

And when I do the same with a list of links that have duplicates in Downloads list, then they won't be verified or online-checked, which is the behavior I asked for to implement it also for dupes in LinkGrabber list.
Reply With Quote
  #14  
Old 12.06.2022, 15:24
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by StefanM View Post
For the full report with screenshots, please read the attachment.
Either please attach the images OR upload to an image hoster and link them.
Please do NOT attach a pdf of your post!
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 12.06.2022 at 15:29.
Reply With Quote
  #15  
Old 12.06.2022, 16:06
StefanM's Avatar
StefanM StefanM is offline
JD VIP
 
Join Date: Oct 2020
Posts: 485
Default

Quote:
Originally Posted by Jiaz View Post
Either please attach the images OR upload to an image hoster and link them.
Please do NOT attach a pdf of your post!
If only it were possible to paste pics embedded in my posts - same is with so many other forums...

This would make things so easy...
I create a screenshot ( I use FastStone Capture) and then I can paste/embed it where I want to have it.

But here I have to
  1. create the screenshot
  2. copy it to my HDD
  3. copy the path of the screenshot *.jpg
  4. upload it to a picture hoster
  5. get the direct link
  6. click here on 'insert images'
And with e.g. 5 screenshots, I have to repeat that procedure 5 times!
That is 30 (!) actions instead of just pasting 5 pics.

But there would also be another solution:
A script, which would automatically upload my screenshot to a picture hoster and fetch the direct link, copying it to the clipboard.

Last edited by StefanM; 12.06.2022 at 16:10.
Reply With Quote
  #16  
Old 12.06.2022, 16:08
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by StefanM View Post
If only it were possible to paste pics embedded in my posts - same is with so many other forums...
This would make things so easy...
I create a screenshot ( I use FastStone Capture) and then I can paste/embed it where I want to have it.
Just attach them or use image hosters like explained from pspzockerscene.
Quote:
Originally Posted by StefanM View Post
But there would also be another solution:
A script, which would automatically upload my screenshot to a picture hoster and fetch the direct link, copying it to the clipboard.
There are nice to use tools to create screenshots, auto upload to image hoster xy and copy url to clipboard, for example zscreen.
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 12.06.2022 at 16:11.
Reply With Quote
  #17  
Old 12.06.2022, 16:28
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 1,533
Default

Quote:
Originally Posted by Jiaz View Post
Just attach them or use image hosters like explained from pspzockerscene.

There are nice to use tools to create screenshots, auto upload to image hoster xy and copy url to clipboard, for example zscreen.
Attaching them requires the images to be saved to the disk first. Also, for such a simple task we should not have to use a 3rd party software. We should be able to use print screen to copy it to the clipboard and directly paste it in the forum post. Too many forum restrictions all in the name of "for your own security".

I do agree there are some people who may post sensitive information, but if they are too stupid to do that, it is on them. Responsible users should not have to pay for their mistakes.
Reply With Quote
  #18  
Old 12.06.2022, 16:33
StefanM's Avatar
StefanM StefanM is offline
JD VIP
 
Join Date: Oct 2020
Posts: 485
Default

Quote:
Originally Posted by mgpai View Post
Attaching them requires the images to be saved to the disk first. Also, for such a simple task we should not have to use a 3rd party software. We should be able to use print screen to copy it to the clipboard and directly paste it in the forum post. Too many forum restrictions all in the name of "for your own security".

I do agree there are some people who may post sensitive information, but if they are too stupid to do that, it is on them. Responsible users should not have to pay for their mistakes.
Thank you for your support!
Reply With Quote
  #19  
Old 12.06.2022, 16:39
StefanM's Avatar
StefanM StefanM is offline
JD VIP
 
Join Date: Oct 2020
Posts: 485
Default

Quote:
Originally Posted by mgpai View Post
... , but if they are too stupid to do that, it is on them. Responsible users should not have to pay for their mistakes.
@Jiaz, @mgpai
Addendum:
And if somebody is that stupid, it would even be worse for them:
Because forced by forum policy to do so, they would upload sensitive information to a picture hoster, which they cannot even delete anymore.

So, it would be for their own security to allow embedding screenshots directly, which they (or an admin) can delete any time,
Reply With Quote
  #20  
Old 12.06.2022, 15:38
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@StefanM: Just to make it clear. Dupe check is NOT done on filename/filesize but ONLY on link/internal link!
Dupes and Mirrors are different things.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #21  
Old 12.06.2022, 15:44
StefanM's Avatar
StefanM StefanM is offline
JD VIP
 
Join Date: Oct 2020
Posts: 485
Default

Quote:
Originally Posted by Jiaz View Post
@StefanM: Just to make it clear. Dupe check is NOT done on filename/filesize but ONLY on link/internal link!
Dupes and Mirrors are different things.
Yes, I'm aware of that and remember that you use the hash value for that, as any other duplicate finder software would do. (I created the German GUIs for SpaceMan99 and DuplicateCleaner )
Reply With Quote
  #22  
Old 12.06.2022, 15:42
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@StefanM: finally we come to the *real* topic or *issue* you want to be optimized!
So you would like to have an option/optimization that the crawling process should check for existing dupe in linkgrabber list
to avoid unnecessary processing of the link, just to later *trash* it because it's already part of Linkgrabber list.

So your wish is: add the same link again, abort it as soon as possible. in best case before it's been processed/online checked, right?
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 12.06.2022 at 15:46.
Reply With Quote
  #23  
Old 12.06.2022, 15:57
StefanM's Avatar
StefanM StefanM is offline
JD VIP
 
Join Date: Oct 2020
Posts: 485
Default

:P
Quote:
Originally Posted by jiaz View Post
@stefanm: Finally we come to the *real* topic or *issue* you want to be optimized!
so you would like to have an option/optimization that the crawling process should check for existing dupe in linkgrabber list
to avoid unnecessary processing of the link, just to later *trash* it because it's already part of linkgrabber list.

So your wish is: Add the same link again, abort it as soon as possible. In best case before it's been processed/online checked, right?
right!

Exactly as you already do it with links in Downloads list.
And for better understanding I created the pdf, where you can see this from the screenshots I was referring to.
Reply With Quote
  #24  
Old 12.06.2022, 16:05
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by StefanM View Post
:P

right!

Exactly as you already do it with links in Downloads list.
And for better understanding I created the pdf, where you can see this from the screenshots I was referring to.
perfect! and I guess you're talking about your vk links, so I can use them for testing, right? as explained, this dupe check works on link/internal link, so I must check the plugin what information is available before the processing of the link
__________________
JD-Dev & Server-Admin
Reply With Quote
  #25  
Old 13.06.2022, 18:23
StefanM's Avatar
StefanM StefanM is offline
JD VIP
 
Join Date: Oct 2020
Posts: 485
Default

Quote:
Originally Posted by Jiaz View Post
perfect! and I guess you're talking about your vk links, so I can use them for testing, right? as explained, this dupe check works on link/internal link, so I must check the plugin what information is available before the processing of the link
No, this has nothing to do with VK.
It was a general suggestion to use the same process for both, dupe checking against Downloads list and dupe checking against LinkGrabber list.

Links you consider to be dupes shouldn't be verified or online-checked unnecessarily.

... as you already do it with links that have dupes in Downloads list.

Cannot see any dependence from the hoster, whatsoever.
Reply With Quote
  #26  
Old 13.06.2022, 20:44
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by StefanM View Post
Cannot see any dependence from the hoster, whatsoever.
read here
Quote:
Originally Posted by Jiaz View Post
perfect! and I guess you're talking about your vk links, so I can use them for testing, right? as explained, this dupe check works on link/internal link, so I must check the plugin what information is available before the processing of the link
That's why the link must at least reach a state where this information is known.
Many plugins work with custom internal links like host://fileID but you cannot check a.com/file/supernice/fileID against that.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #27  
Old 13.06.2022, 21:05
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by StefanM View Post
Links you consider to be dupes shouldn't be verified or online-checked unnecessarily.
Correct and I agree. I just want to explain why this won't be possible for all links.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #28  
Old 12.06.2022, 16:02
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@StefanM: I'm sorry but I don't understand! What is the problem here?
You add Link X into Linkgrabber and move to Downloads. Then you will be able to add Link X again to Linkgrabber except you've added a filter with condition *already in download list*. that will prevent this link to be added again. BUT in case you also have a matching view rule, then this link will be added to Linkgrabber because a matching view rule overrides a matching filter rule.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #29  
Old 12.06.2022, 16:36
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@StefanM, @mgpai:
The board software does not support *paste image from clipboard* feature.
Yes, I agree that such a feature would make it easier/more user friendly to post images but the images still would be visible for support staff only. There are reasons why we don't make attachments available to public and use url blacklisting.
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 12.06.2022 at 16:44.
Reply With Quote
  #30  
Old 14.06.2022, 08:09
tony2long's Avatar
tony2long tony2long is offline
English Supporter
 
Join Date: Jun 2009
Posts: 6,507
Default

For me it's the opposite, when the link has already existed in the LinkGrabber list, I immediately get "-> Filtered Dupe" (in the log), no access to the web.
__________________
FAQ: How to upload a Log

Last edited by tony2long; 14.06.2022 at 10:34.
Reply With Quote
  #31  
Old 14.06.2022, 12:36
StefanM's Avatar
StefanM StefanM is offline
JD VIP
 
Join Date: Oct 2020
Posts: 485
Default

Quote:
Originally Posted by tony2long View Post
For me it's the opposite, when the link has already existed in the LinkGrabber list, I immediately get "-> Filtered Dupe" (in the log), no access to the web.
Do you have Bubble Notifier for that enabled?
What does it show, if and when it has been enabled?
Reply With Quote
  #32  
Old 14.06.2022, 12:44
tony2long's Avatar
tony2long tony2long is offline
English Supporter
 
Join Date: Jun 2009
Posts: 6,507
Default

Quote:
Originally Posted by StefanM View Post
Do you have Bubble Notifier for that enabled?
What does it show, if and when it has been enabled?
Yes.
Found link(s) 1
-and-
Done
__________________
FAQ: How to upload a Log
Reply With Quote
  #33  
Old 14.06.2022, 13:01
StefanM's Avatar
StefanM StefanM is offline
JD VIP
 
Join Date: Oct 2020
Posts: 485
Default

Quote:
Originally Posted by tony2long View Post
Yes.
Found link(s) 1
-and-
Done
Hmm, I tested that on June 12 several times, and for LinkGrabber dupes Bubble notifier showed
Found link(s) 1
Found package(s) 0
Online: 1
...


But I could check the log, too.

Which file would that be?
Reply With Quote
  #34  
Old 14.06.2022, 13:09
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by tony2long View Post
For me it's the opposite, when the link has already existed in the LinkGrabber list, I immediately get "-> Filtered Dupe" (in the log), no access to the web.
most likely because enabled fast linkcheck support in Settings->Plugins->vk.com
__________________
JD-Dev & Server-Admin
Reply With Quote
  #35  
Old 14.06.2022, 13:18
StefanM's Avatar
StefanM StefanM is offline
JD VIP
 
Join Date: Oct 2020
Posts: 485
Default

Quote:
Originally Posted by Jiaz View Post
most likely because enabled fast linkcheck support in Settings->Plugins->vk.com:)
Just checked jd.controlling.linkcollector.LinkCollector.log.0
It's the only log file that contains that string Filtered Dupe.

And I also cannot find the online check (as indicated by Bubble Notifier there).

But I can see also from the timing that online check is being performed, but - again - only when the dupe exists in LinkGrabber.

And I also have fast linkcheck enabled.

When Bubble Notifier displays an online check, shouldn't one see this also in the log? Or would that be a different log file?
Reply With Quote
  #36  
Old 14.06.2022, 12:59
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@StefanM: and that's exactly why JDownloader uses internal links to dupe checking as the URL may vary a lot. I just wanted to explain that it depends on plugin/implementation if the internal link is available before linkcheck or after linkcheck. and thus can't always be avoided
__________________
JD-Dev & Server-Admin
Reply With Quote
  #37  
Old 14.06.2022, 13:19
tony2long's Avatar
tony2long tony2long is offline
English Supporter
 
Join Date: Jun 2009
Posts: 6,507
Default

Sorry, because there is no vk.com in the title, I tried with simple link like:
tiktok.com/@elinadevia/video/7106693553442655514?lang=en
__________________
FAQ: How to upload a Log
Reply With Quote
  #38  
Old 14.06.2022, 13:27
StefanM's Avatar
StefanM StefanM is offline
JD VIP
 
Join Date: Oct 2020
Posts: 485
Default

Quote:
Originally Posted by tony2long View Post
Sorry, because there is no vk.com in the title, I tried with simple link like:
tiktok.com/@elinadevia/video/7106693553442655514?lang=en
No problem!
I learned that behavior can be different, depending on the hoster.

What I still don't know: Whether or not there is (or can be) unnecessary online-checking for dupes in LinkGrabber.
Reply With Quote
  #39  
Old 14.06.2022, 13:32
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Quote:
Originally Posted by StefanM View Post
What I still don't know: Whether or not there is (or can be) unnecessary online-checking for dupes in LinkGrabber.
If whatever you're adding first needs to be processed by a crawler, additional "linkchecking" (well not linkchecking exactly but "crawling") can be needed.
Examples for that:
- Re-adding a twitter profile
- Adding links to all sorts of video websites for which JD can return multiple results e.g. vimeo.com, pornhub.com
- Adding cloud folders such as google drive
- Adding user profiles e.g. twitter.com, vimeo.com profile, youporn.com profile

Baiscally everything where a crawl process is involved because "before JD cannot know the results".
E.g. user profiles can have more/different results over time.
Cloud folders can get updated and so on.
EDIT

And as always, exception can exist e.g. some plugins are highly optimized and/or got extra settings for that behavior e.g. youtube, vk.com.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #40  
Old 14.06.2022, 13:41
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by StefanM View Post
What I still don't know: Whether or not there is (or can be) unnecessary online-checking for dupes in LinkGrabber.
That heavily depends on the hoster/plugin and I can't give a general statement
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 00:21.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.