JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #41  
Old 14.06.2022, 12:44
tony2long's Avatar
tony2long tony2long is offline
English Supporter
 
Join Date: Jun 2009
Posts: 6,507
Default

Quote:
Originally Posted by StefanM View Post
Do you have Bubble Notifier for that enabled?
What does it show, if and when it has been enabled?
Yes.
Found link(s) 1
-and-
Done
__________________
FAQ: How to upload a Log
Reply With Quote
  #42  
Old 14.06.2022, 12:56
StefanM's Avatar
StefanM StefanM is offline
JD VIP
 
Join Date: Oct 2020
Posts: 485
Default

Quote:
Originally Posted by Jiaz View Post
Links in Downloads list are already processed and have this internal link ready. Links that are getting crawled/processed don't have this at start and the link must reach a certain processing state for this dupe check to work. That's what I'm trying to say.
So it heavily depends on what links you are adding, eg links that must be processed by a decrypter plugin first or direct download links that will be processed directly by a hoster plugin.
An internal (direct) VK.com-link cannot be used for dupe checking, because it will be different every time:

It contains the IP address, the expiration date (which will be different every time...), ...

With vk.com for video(!) dupe checking you need a URL in this or similar format:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
And - as far as I can see it - JD is exactly checking this when it checks for dupes.

Or JD is checking this in the extended 'version':
**External links are only visible to Support Staff****External links are only visible to Support Staff**

However, when JD would use the above (extended) URL, then another resolution of the same video would not be considered as dupe.

Haven't tested that.
Reply With Quote
  #43  
Old 14.06.2022, 12:59
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@StefanM: and that's exactly why JDownloader uses internal links to dupe checking as the URL may vary a lot. I just wanted to explain that it depends on plugin/implementation if the internal link is available before linkcheck or after linkcheck. and thus can't always be avoided
__________________
JD-Dev & Server-Admin
Reply With Quote
  #44  
Old 14.06.2022, 13:01
StefanM's Avatar
StefanM StefanM is offline
JD VIP
 
Join Date: Oct 2020
Posts: 485
Default

Quote:
Originally Posted by tony2long View Post
Yes.
Found link(s) 1
-and-
Done
Hmm, I tested that on June 12 several times, and for LinkGrabber dupes Bubble notifier showed
Found link(s) 1
Found package(s) 0
Online: 1
...


But I could check the log, too.

Which file would that be?
Reply With Quote
  #45  
Old 14.06.2022, 13:09
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by tony2long View Post
For me it's the opposite, when the link has already existed in the LinkGrabber list, I immediately get "-> Filtered Dupe" (in the log), no access to the web.
most likely because enabled fast linkcheck support in Settings->Plugins->vk.com
__________________
JD-Dev & Server-Admin
Reply With Quote
  #46  
Old 14.06.2022, 13:18
StefanM's Avatar
StefanM StefanM is offline
JD VIP
 
Join Date: Oct 2020
Posts: 485
Default

Quote:
Originally Posted by Jiaz View Post
most likely because enabled fast linkcheck support in Settings->Plugins->vk.com:)
Just checked jd.controlling.linkcollector.LinkCollector.log.0
It's the only log file that contains that string Filtered Dupe.

And I also cannot find the online check (as indicated by Bubble Notifier there).

But I can see also from the timing that online check is being performed, but - again - only when the dupe exists in LinkGrabber.

And I also have fast linkcheck enabled.

When Bubble Notifier displays an online check, shouldn't one see this also in the log? Or would that be a different log file?
Reply With Quote
  #47  
Old 14.06.2022, 13:19
tony2long's Avatar
tony2long tony2long is offline
English Supporter
 
Join Date: Jun 2009
Posts: 6,507
Default

Sorry, because there is no vk.com in the title, I tried with simple link like:
tiktok.com/@elinadevia/video/7106693553442655514?lang=en
__________________
FAQ: How to upload a Log
Reply With Quote
  #48  
Old 14.06.2022, 13:27
StefanM's Avatar
StefanM StefanM is offline
JD VIP
 
Join Date: Oct 2020
Posts: 485
Default

Quote:
Originally Posted by tony2long View Post
Sorry, because there is no vk.com in the title, I tried with simple link like:
tiktok.com/@elinadevia/video/7106693553442655514?lang=en
No problem!
I learned that behavior can be different, depending on the hoster.

What I still don't know: Whether or not there is (or can be) unnecessary online-checking for dupes in LinkGrabber.
Reply With Quote
  #49  
Old 14.06.2022, 13:32
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Quote:
Originally Posted by StefanM View Post
What I still don't know: Whether or not there is (or can be) unnecessary online-checking for dupes in LinkGrabber.
If whatever you're adding first needs to be processed by a crawler, additional "linkchecking" (well not linkchecking exactly but "crawling") can be needed.
Examples for that:
- Re-adding a twitter profile
- Adding links to all sorts of video websites for which JD can return multiple results e.g. vimeo.com, pornhub.com
- Adding cloud folders such as google drive
- Adding user profiles e.g. twitter.com, vimeo.com profile, youporn.com profile

Baiscally everything where a crawl process is involved because "before JD cannot know the results".
E.g. user profiles can have more/different results over time.
Cloud folders can get updated and so on.
EDIT

And as always, exception can exist e.g. some plugins are highly optimized and/or got extra settings for that behavior e.g. youtube, vk.com.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #50  
Old 14.06.2022, 13:41
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by StefanM View Post
What I still don't know: Whether or not there is (or can be) unnecessary online-checking for dupes in LinkGrabber.
That heavily depends on the hoster/plugin and I can't give a general statement
__________________
JD-Dev & Server-Admin
Reply With Quote
  #51  
Old 14.06.2022, 16:27
StefanM's Avatar
StefanM StefanM is offline
JD VIP
 
Join Date: Oct 2020
Posts: 485
Default

Quote:
Originally Posted by Jiaz View Post
That heavily depends on the hoster/plugin and I can't give a general statement
In an earlier post, you said that I was right and that I do have a point with my request.

Now, I just want to know whether or not you will (or already have) implement this:
Avoiding unnecessary requests and additional time involved for LinkGrabber run time that is.
Reply With Quote
  #52  
Old 14.06.2022, 16:41
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Which one do you mean?
I don't see any inconsistent answer so my guess is that you simply did not understand how the current system works.

Where do you currently see a problem with this or potential to optimize?
And example case / example URLs?
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #53  
Old 14.06.2022, 16:57
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by StefanM View Post
In an earlier post, you said that I was right and that I do have a point with my request.
Now, I just want to know whether or not you will (or already have) implement this:
Avoiding unnecessary requests and additional time involved for LinkGrabber run time that is.
Yes, I'm working on this.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #54  
Old 14.06.2022, 17:39
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Check again with next core update.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #55  
Old 15.06.2022, 13:48
StefanM's Avatar
StefanM StefanM is offline
JD VIP
 
Join Date: Oct 2020
Posts: 485
Default

Quote:
Originally Posted by pspzockerscene View Post
Where do you currently see a problem with this or potential to optimize?
And example case / example URLs?
vk.com as I already mentioned. Please have a look at the pdf I sent you. There you can see what's happening.
Reply With Quote
  #56  
Old 15.06.2022, 14:14
StefanM's Avatar
StefanM StefanM is offline
JD VIP
 
Join Date: Oct 2020
Posts: 485
Default

Quote:
Originally Posted by Jiaz View Post
Check again with next core update.
Sounds good!

Just to make sure the way I check whether or not a 'next update' already has been released is correct...

When it reads on
https://support.jdownloader.org/Know...pdate-behavior

Quote:
Latest Revision: 46219 Date: Tue Jun 14 13:48:16 CEST 2022
And your post is from 14.06.2022, 16:39

I can conclude it has not been released yet.
And when I find a release dated later than your post, than I can conclude, the next update is available, right?
Reply With Quote
  #57  
Old 15.06.2022, 15:06
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by StefanM View Post
I can conclude it has not been released yet.
That is correct.
Quote:
Originally Posted by StefanM View Post
And when I find a release dated later than your post, than I can conclude, the next update is available, right?
yes for most of the times.
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 01:18.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.