JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 06.08.2017, 18:24
Tyler Tyler is offline
JD Legend
 
Join Date: Jul 2010
Posts: 581
Default Delete duplicate not working

I can't delete duplicate links in linkgrabber, the option is greyed out.

Last edited by Tyler; 06.08.2017 at 18:28.
Reply With Quote
  #2  
Old 07.08.2017, 11:00
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

Are there duplicates available? They have different colors.
Please know that *delete duplicate* refers to duplicate aka link is already in download list.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #3  
Old 07.08.2017, 22:39
Tyler Tyler is offline
JD Legend
 
Join Date: Jul 2010
Posts: 581
Default

no these are duplicate links in the linkgrabber.
Reply With Quote
  #4  
Old 08.08.2017, 04:21
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

you should not get dupes in linkgrabber.
linkgrabber by default prevents duplicates from been added to linkgrabber, you will see 'filtered dupe' within the linkcollector log.

dupe is of the same linkid OR same full url when linkid isn't set by plugins.

raztok
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #5  
Old 08.08.2017, 06:16
Tyler Tyler is offline
JD Legend
 
Join Date: Jul 2010
Posts: 581
Default

Well linkgrabber was going slow so i turned off automatic checking and then loaded up all the links at once again. (2k+ links at 7 seconds per one)

so i ended up with a bunch of duplicates.

though it does happen all the time when links have the url encoding instead of special characters like %5D or %20. So technically they aren't the exact same link and linkgrabber will grab both.

i've brought it up a few times over the years.
Reply With Quote
  #6  
Old 08.08.2017, 06:28
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

Jiaz will need to look at this, provide examples of url encoded dupes.

I've also told jiaz about not url encoding components unless it's required by RFC.
in respects to dedicated plugins, we typically only scan for the least amount of URL for example protocol://domain/fuid and not protocol://domain/fuid/craphere. as its just not needed, and can lead to dupes if the dedicated plugin doesn't set specific linkid.

so its effectively not just a core issue, but could also be solved within plugins.

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #7  
Old 08.08.2017, 07:26
Tyler Tyler is offline
JD Legend
 
Join Date: Jul 2010
Posts: 581
Default

well that is another problem altogether and not what i made this topic for.

this time the duplicates were the exact same link, they were just added manually by me bypassing the linkgrabber.

basically i'd just like it if remove duplicates actually removed duplicate links in the linkgrabber.
Reply With Quote
  #8  
Old 08.08.2017, 10:49
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

its still related, but it could be solved within the core. For example if we urldecoded everything so that the failover setLinkID had a baseline which can not be variable. Effectively would make this issue null and void.
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #9  
Old 08.08.2017, 23:44
Tyler Tyler is offline
JD Legend
 
Join Date: Jul 2010
Posts: 581
Default

how about an option to delete duplicates based on filesize+name?

like right now i have duplicate files in my linkgrabber but they are different links.

some filehosts upload a file twice for some reason. (glitch during upload i guess, not sure)

they're the exact same name and size but i have to manually click on them to delete. Which is quite cumbersome if you have thousands of files over multiple hosts to go through.
Reply With Quote
  #10  
Old 09.08.2017, 04:04
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

just remember that mirrors != duplicates
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #11  
Old 11.08.2017, 22:45
Tyler Tyler is offline
JD Legend
 
Join Date: Jul 2010
Posts: 581
Default

Quote:
Originally Posted by raztoki View Post
just remember that mirrors != duplicates
why not? It's the same file

maybe you guys should call it something else than delete duplicates.
Reply With Quote
  #12  
Old 12.08.2017, 05:54
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

I did say this earlier, but...
deleting duplicates, are of the same linkid or exact same url when you have object within download tab, and now you have added it again within linkgrabber. They will be shaded a different colour.

duplicate is when they are of the same linkid (set by plugin or when not set by plugins core will set it to the url). In respects to Linkgrabber its impossible to add a duplicate when already in the linkgrabber. Only possible when it's in the download tab.

This is what the delete duplicate function is meant to be used for!, It has nothing todo with mirrors ;> So I guess a little misunderstanding by you in respects to this.

Hope this clears up the situation
raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #13  
Old 13.08.2017, 20:19
Tyler Tyler is offline
JD Legend
 
Join Date: Jul 2010
Posts: 581
Default

so then can you guys add the ability to delete duplicate links in the linkgrabber?

Quote:
Originally Posted by raztoki View Post
In respects to Linkgrabber its impossible to add a duplicate when already in the linkgrabber. Only possible when it's in the download tab.
It's possible to add duplicate links if you disable automatic checking first.

Last edited by Tyler; 13.08.2017 at 20:23.
Reply With Quote
  #14  
Old 14.08.2017, 06:42
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

Quote:
Originally Posted by Tyler View Post
so then can you guys add the ability to delete duplicate links in the linkgrabber?
its impossible to add duplicates within linkgrabber when they already exist in linkgrabber.

the only time that isn't the case at this point in time is when components of the url are urlencoded, and I did mention this above.

Working example

https://board.jdownloader.org/images/logo.png
Code:
1618|http links_jd.plugins.hoster.raz_DirectHTTP.log 14/08/17 11:27:39 AM - INFO [ jd.controlling.linkchecker.LinkChecker(updateAvailableStatus) ] -> Link https://board.jdownloader.org/images/logo.png is TRUE
then try adding with parameter to bypass the dupe checking
https://board.jdownloader.org/images/logo.png?test
Code:
THREAD: 1625
1625|http links_jd.plugins.hoster.raz_DirectHTTP.log 14/08/17 11:27:54 AM - INFO [ jd.controlling.linkcrawler.LinkCrawler(processHostPlugin) ] -> Processing: httpsviajd://board.jdownloader.org/images/logo.png?test

THREAD: 1617
1617|jd.controlling.linkcollector.LinkCollector.log 14/08/17 11:27:54 AM - INFO [ jd.controlling.linkcollector.LinkCollector$11(run) ] -> Filtered Dupe: https://board.jdownloader.org/images/logo.png
as you can see above it prevents it been from added

currently the only time when this doesn't happen (as stated above) is when urlencoding is used within url structure

https://board.jdownloader.org/%69%6D...C%6F%67%6F.png
Code:
1651|http links_jd.plugins.hoster.raz_DirectHTTP.log 14/08/17 11:29:19 AM - INFO [ jd.controlling.linkchecker.LinkChecker(updateAvailableStatus) ] -> Link https://board.jdownloader.org/%69%6D%61%67%65%73%2F%6C%6F%67%6F.png is TRUE
not only doesn't it url decode it before sending the request off, it gets the filename wrong also (fall out).

the fix that Jiaz needs to change, is to full URLDecode the linkid when uses the url as failover (also mentioned above). Then and only then will it be impossible for dupes not to add to linkgrabber when already within linkgrabber, since the backend prevents them adding to Linkgrabber 'filtered dupe' event. It would also prevent dupes within linkgrabber when original is within downloadtab!

Quote:
Originally Posted by Tyler View Post
It's possible to add duplicate links if you disable automatic checking first.
typically disabling the automatic linkchecking (for hoster plugins) doesn't cause this. When LinkID is set within a hoster, its done within the correctDownloadLink method. For memory this is called even though linkchecking isn't.

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]

Last edited by raztoki; 14.08.2017 at 06:47.
Reply With Quote
  #15  
Old 20.08.2017, 01:59
Tyler Tyler is offline
JD Legend
 
Join Date: Jul 2010
Posts: 581
Default

Quote:
Originally Posted by raztoki View Post
its impossible to add duplicates within linkgrabber when they already exist in linkgrabber.
I mean duplicates of the same file not the exact same link. Could be done through a combination of filesize+name

so even if its a different link or even a different hoster it would be deleted when you tell jdownloader to delete duplicates.
Reply With Quote
  #16  
Old 20.08.2017, 05:34
tony2long's Avatar
tony2long tony2long is offline
English Supporter
 
Join Date: Jun 2009
Posts: 6,507
Default

Are you forcing your term about duplicates to be accepted as JD's term and we should change all related things about it?
__________________
FAQ: How to upload a Log
Reply With Quote
  #17  
Old 20.08.2017, 06:04
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

@tony2long
*shrugs* everyone has there own agenda at heart. Never the less it's not like we have to comply.

Quote:
Originally Posted by Tyler View Post
I mean duplicates of the same file not the exact same link.
mirrors != duplicates.
at this time this is intended behaviour

Under your idea, the first link added would then be the one you download, vs not been able to select which one you want. Very few if any users would like that or would use said feature. There are features to add links with higher priority (package customiser) or based on selection and adding links which can remove others. Maybe this will suit your needs better?
also note: typically JD doesn't always final filesize unless it comes from a verified source (apis). Download is required in order to determine that.

Think we will call this topic/thread closed, as current behaviour is intended behaviour.

We will look at fixing the url encoded issue for failover linkids


raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #18  
Old 20.08.2017, 19:27
Tyler Tyler is offline
JD Legend
 
Join Date: Jul 2010
Posts: 581
Default

Quote:
Originally Posted by tony2long View Post
Are you forcing your term about duplicates to be accepted as JD's term and we should change all related things about it?
My term?

Definition of duplicate:

Quote:
exactly like something else, especially through having been copied.
i'm talking about files that are exactly the same, the only difference is the link.

JD even puts them in the same package after checking them.
Reply With Quote
  #19  
Old 20.08.2017, 19:39
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

its a mirror as stated like many times in this thread already
mirrors are not duplicates, thus the OP is greyed out, this is intended behaviour
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #20  
Old 20.08.2017, 19:47
tony2long's Avatar
tony2long tony2long is offline
English Supporter
 
Join Date: Jun 2009
Posts: 6,507
Default

Quote:
Originally Posted by Tyler View Post
i'm talking about files that are exactly the same, the only difference is the link.
So now you are talking about duplicate files?
Quote:
Originally Posted by Tyler View Post
I can't delete duplicate links in linkgrabber, the option is greyed out.
JD has "Delete Links: Selected &*Duplicates", there is no "Delete files: Selected & Duplicates".
__________________
FAQ: How to upload a Log

Last edited by tony2long; 20.08.2017 at 19:52.
Reply With Quote
  #21  
Old 22.08.2017, 14:54
Tyler Tyler is offline
JD Legend
 
Join Date: Jul 2010
Posts: 581
Default

Quote:
Originally Posted by raztoki View Post
its a mirror as stated like many times in this thread already
mirrors are not duplicates, thus the OP is greyed out, this is intended behaviour
I'm aware of what you are defining a duplicate as.

i'm asking for the separate option to take care of a separate problem.

Quote:
Originally Posted by tony2long View Post
So now you are talking about duplicate files?

JD has "Delete Links: Selected &*Duplicates", there is no "Delete files: Selected & Duplicates".
No i'm talking about links, i was merely referring to them as duplicate files in the context of them being the same files across different links.
Reply With Quote
  #22  
Old 22.08.2017, 15:09
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

@Tyler
I think your best solution if you want to remove all links other say first returned or via some other means, is to create event scripter script, and bind it to a button and or keyboard shortcut.

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #23  
Old 22.08.2017, 15:12
Tyler Tyler is offline
JD Legend
 
Join Date: Jul 2010
Posts: 581
Default

I don't want to remove all links just ones that are the same as other ones that i already have in the lingrabber.

if you guys want to call this delete mirrors then i'm fine with that too.
Reply With Quote
  #24  
Old 22.08.2017, 15:13
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

I know what you meant

I just don't see us making this function.
a) we do not know what mirrors to remove, where you do. so script solution can work for you!
b) our user base is used to current behaviour, as its always been that way.

as you can understand serves little to no benefit to our userbase. But this doesn't stop you from doing it yourself.

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #25  
Old 22.08.2017, 16:16
Tyler Tyler is offline
JD Legend
 
Join Date: Jul 2010
Posts: 581
Default

Quote:
Originally Posted by raztoki View Post
a) we do not know what mirrors to remove, where you do.
The ones other than the selected?
Reply With Quote
  #26  
Old 22.08.2017, 17:16
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

current implementation
right click > delete/cleanup > keep only selected && add links
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 17:15.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.