JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 02.06.2022, 17:17
StefanM's Avatar
StefanM StefanM is offline
JD VIP
 
Join Date: Oct 2020
Posts: 485
Default Issue with filtered links in LinkGrabber

After grabbing links you see, that x links have been filtered. You can choose to have those links being displayed, but nothing happens.

Before I provide logs, whatsoever…
Could you please explain what the expected behavior should be and which criteria lead to filtering (such as duplicate links).

Maybe my interpretration/expectation is not correct.
Reply With Quote
  #2  
Old 02.06.2022, 18:23
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@StefanM:
The number disabled in front of a quick filter means how many links are *matching* this condition/filter in the current setup of filters

When you check(to make them visible) then please know that they still may not show up due to other filters still filtering them out.
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 02.06.2022 at 18:27.
Reply With Quote
  #3  
Old 02.06.2022, 18:24
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by StefanM View Post
such as duplicate links
Do you mean *Already in Downloadlist*? If so, this view/filter means that a link is already in Downloads
__________________
JD-Dev & Server-Admin
Reply With Quote
  #4  
Old 02.06.2022, 18:26
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Settings->Linkgrabber Filter->
Filter: do not process/add links to Linkgrabber that match any of the rules (except there is a matching View filter)
Views: provide quick on/off filtering of links in Linkgrabebr that match any of the rules
__________________
JD-Dev & Server-Admin
Reply With Quote
  #5  
Old 02.06.2022, 18:28
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

In case you've got more questions about this,please just ask
__________________
JD-Dev & Server-Admin
Reply With Quote
  #6  
Old 02.06.2022, 19:19
StefanM's Avatar
StefanM StefanM is offline
JD VIP
 
Join Date: Oct 2020
Posts: 485
Default

Quote:
Originally Posted by Jiaz View Post
In case you've got more questions about this,please just ask
Not sure, if I explained it good enough:

At the end or during LinkGrabber is grabbing, you get a popup window showing you the grabbing process (number of files grabbed so far) and next to that figure there is a funnel, and next to that you see the number of filtered links.

Now, I would like to see what has been filtered. And no matter if there are other filters active or not:
When I click to make them visible, there is no change.
I don't see any additional links. Only the number of filtered links is reset to zero.

Last edited by StefanM; 02.06.2022 at 19:57.
Reply With Quote
  #7  
Old 02.06.2022, 19:28
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@StefanM: That is not supported.
Filtered links are *filtered* and either will go into the lower bottom "Restore" button (default on, Settings->Advanced Settings->LinkgrabberSettings.restorebuttonenabled)(allows you to restore/add them later on) or are gone for good.

An idea would be to also add the *filtered* links as normal links into the Linkgrabber and just mark them as filtered,
similiar to those *dummy* entries when a plugin error occurs

__________________
JD-Dev & Server-Admin
Reply With Quote
  #8  
Old 02.06.2022, 19:29
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by StefanM View Post
When I click to make them visible, there is no change.
I don't see any additional links. Only the number of filtered links is reset to zero.
can you provide a screenshot of what you mean? the right quickfilter settings in Linkgrabber, see https://board.jdownloader.org/showpo...84&postcount=2
__________________
JD-Dev & Server-Admin
Reply With Quote
  #9  
Old 02.06.2022, 20:01
StefanM's Avatar
StefanM StefanM is offline
JD VIP
 
Join Date: Oct 2020
Posts: 485
Default

Quote:
Originally Posted by Jiaz View Post
can you provide a screenshot of what you mean? the right quickfilter settings in Linkgrabber, see **External links are only visible to Support Staff**...
Yes I will do that tomorrow.

And I made a mistake:
Actually, I was referring to the "Restore" button.
And when I click on that button, the number of 'filtered links' is reset to zero.
But I still don't see any files that were restored.

That was the problem, I actually wanted to report.
Sorry about the confusion...
Reply With Quote
  #10  
Old 03.06.2022, 13:14
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Quote:
Originally Posted by StefanM View Post
And when I click on that button, the number of 'filtered links' is reset to zero.
But I still don't see any files that were restored.
@Jiaz
We've just recently talked about this.
In the other case this happened because the filtered entries were folderwatch jobs(?) so upon "restore" nothing appeared in the linkgrabber which can be distracting.

Also, wouldn't it be more consistent to simply add another view for filtered items instead of the "restore" button?
Though I guess there is a reason why it has been done like that...
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 03.06.2022 at 13:27.
Reply With Quote
  #11  
Old 03.06.2022, 13:19
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

I will read/answer, no time yet
__________________
JD-Dev & Server-Admin
Reply With Quote
  #12  
Old 03.06.2022, 16:34
StefanM's Avatar
StefanM StefanM is offline
JD VIP
 
Join Date: Oct 2020
Posts: 485
Default

Quote:
Originally Posted by pspzockerscene View Post
@Jiaz
Also, wouldn't it be more consistent to simply add another view for filtered items instead of the "restore" button?
That would be much better :-)
Reply With Quote
  #13  
Old 03.06.2022, 17:01
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

That said there must be a reason why it hasn't yet been done like that.
I've added this information/idea to our internal ticket.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #14  
Old 03.06.2022, 20:16
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by Jiaz View Post
I will read/answer, no time yet
I will read/answer asap
__________________
JD-Dev & Server-Admin
Reply With Quote
  #15  
Old 04.06.2022, 11:38
StefanM's Avatar
StefanM StefanM is offline
JD VIP
 
Join Date: Oct 2020
Posts: 485
Default

Quote:
Originally Posted by pspzockerscene View Post
That said there must be a reason why it hasn't yet been done like that.
I've added this information/idea to our internal ticket.
Additional information

1. The restore feature and the automatic filtering that comes along with it - when not used directly after crawling has finished, i.e. before any other action - will lead to automatic deletion of those dupes. If this is the intention, that feature at least shouldn't be active by default.

2. As already reported, dupes among links in the LinkGrabber pane are not being treated the same way as dupes in the Download pane.

3. One reason - as you already assumed - why filtered links do not show up after restoration, can be other active filters. (Had this case myself)
Reply With Quote
  #16  
Old 11.06.2022, 08:40
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by StefanM View Post
Additional information

1. The restore feature and the automatic filtering that comes along with it - when not used directly after crawling has finished, i.e. before any other action - will lead to automatic deletion of those dupes. If this is the intention, that feature at least shouldn't be active by default.
The restore button is enabled by default so ppl can see that their setup filter rules actually have filtered some links. Elsethey just wonder why nothing is showing up in list.
The links *behind* restore button are NOT stored across sessions and that's intended behaviour.
Quote:
Originally Posted by StefanM View Post
before any other action - will lead to automatic deletion of those dupes.
What exactly do you mean? Do you mean 1.) enable filter -> add link -> link is filtered to restore button 2.) disable filter -> add link -> doesn't show up because it's already in restore button? if so, yes this is a bug. If you mean something else, please explain better
__________________
JD-Dev & Server-Admin
Reply With Quote
  #17  
Old 11.06.2022, 08:44
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by StefanM View Post
2. As already reported, dupes among links in the LinkGrabber pane are not being treated the same way as dupes in the Download pane.
What do you mean? Please explain. In Linkgrabber it's not possible to add the same link multiple times by default (can be disabled). Whereas in Download pane there is no dupe handling at all. You can add the same link multiple times, as often as you like.

Quote:
Originally Posted by StefanM View Post
3. One reason - as you already assumed - why filtered links do not show up after restoration, can be other active filters. (Had this case myself)
The number on the restore button just means that that many links were filtered. But when you hit restore button, then those links will be processed and ONLY added if they are not already part of Linkgrabber (dupe check) and if they are actual supported! That means you can setup a filter that will also match on unsupported links that JDownloader as the filter also is applied at stages before JDownloader does even know if it can handle the link. So the added filtered link in restore just means there is a link you've added that got filtered. It does NOT mean that when you restore it, that there will be a link added to list. Another reason can be that you've added a folder/shortener link that contains no links/does not lead to any other link.

Quote:
Originally Posted by StefanM View Post
can be other active filters. (Had this case myself)
No it can't. When restoring links, they are no longer checked against your filter rules because you actual want to *restore* them. The whole process of filtering does not happen again else you would have to restore it again and again and again, depending on how many filters do match on the link.
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 11.06.2022 at 08:49.
Reply With Quote
  #18  
Old 11.06.2022, 12:59
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

Would it be an idea to have an additional window to see whats behind the restored button? and which filter it matched against (first or all matches), kinda like the test button within rule creation? So you don't actually have to restore ALL behind it, to see whats actively there. Then a function to retrieve desired (selected) links from that window or all button pull them out?

Why I propose this, is specially from (heavy) users who crawl multiple links(jobs) concurrently or basically scrape large components of websites which then return subsequent links(jobs), curiosity to see whats behind filtered links vs restoring in actual linkgrabber then having manually remove them post is annoying! Or the easier method deleting all links, and re-adding all jobs.

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #19  
Old 11.06.2022, 13:31
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@raztoki: instead of making the restore button more complex,eg showing the links *behind it*, I would prefer to include the *filtered* links as dummy within the list, so you can see/work with them directly (eg delete, or retry without filter or keep for later (now part of the list)). similar to the *linkcrawlerretry* links. that way the dummy link could show which rule matched (eg in comment) and easier to see what got filtered

your opinion on this? of course an additional window would be possible but just putting them into the list with all others, you can better work with them, eg quickfilter on right side or move/retry/delete/keep.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #20  
Old 11.06.2022, 17:40
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

Guess there are pros and cons to everything in life and this program isn't going to escape that!

One of the only issues is with power users, they can have hundreds of thousands of links already within linkgrabber, "finding them" would/could be needle in hay stack situation (like it is now) if you hit that restore function. If you design it with that in mind, making it easy would be nice.

I assumed clicking a button & new window appears as its fast and current design doesn't have to change, peoples custom UI layouts still work. You can quickly and efficiently see the results either 'select, restore, clear, close' with ease. Should be quick to implement in the scheme of things. The tables/gui already exists within test functions in package customer && linkfilter, just don't have to run jobs, just display exisiting data held in memory. One would need to store which rule it matched against and also display that data, based on my previous post.

With a customs views filter; would this require you to deselect all view checkboxes && select 'restore' views rule? As in just show the filtered results.
If you just wanted to view filtered links within linkgrabber and/or apply additional view rules it would be nice, and very powerful overlay. The main downside to this would be additional rules hiding results, thus requiring users to disable multiple rules (if they exist), and an additional minor issue could be the support requirement this would cause (say very minor). Personally I see with long lists one would need to scroll and toggle the checkbox to identify if you didn't have like a colour identifier behind the filtered view, else how would you know, without toggling checkboxes while scrolling.

or do we have a different state, and include all links again! and like offline or dupes, they just don't add to the downloadlist unless user interaction (ask window) or user default actions (remembered state). Different colour shading in table view do easily can be identified.

or offline package equivalent, then the DownloadLinks wouldn't be within there proper packages etc. (early day JD design)

those are the current frame works we are using currently to previously from current JD2 back to older versions
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #21  
Old 18.06.2022, 23:48
reddevil reddevil is offline
JD VIP
 
Join Date: Apr 2010
Posts: 369
Default

Hi JD team,

Filtered links are not showing up as 'clear filtered links' in Linkgrabber anymore (since a few updates ago). I have set rule in linkgrabber filter as: downloadURL contains .*\.(jpg|jpeg|png|gif) and it used to collect into filtered links before a few days ago.
Log: 19.06.22 02.08.35 <--> 19.06.22 02.13.32 jdlog://5001035302851/

Please have a look.

Regards.
Reply With Quote
  #22  
Old 19.06.2022, 10:24
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@reddevil: Thank you so much for the report. Issue was caused by a recent change of mine and only happens in filtered situation. will be fixed with next core update later today
__________________
JD-Dev & Server-Admin
Reply With Quote
  #23  
Old 19.06.2022, 15:27
reddevil reddevil is offline
JD VIP
 
Join Date: Apr 2010
Posts: 369
Default

@Jiaz, it's fixed now, Thanks.
Reply With Quote
  #24  
Old 19.06.2022, 17:18
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@reddevil: Thanks for the feedback and thank you for the report
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 23:04.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.