#1
|
|||
|
|||
Delete duplicate not working
I can't delete duplicate links in linkgrabber, the option is greyed out.
Last edited by Tyler; 06.08.2017 at 17:28. |
#2
|
||||
|
||||
Are there duplicates available? They have different colors.
Please know that *delete duplicate* refers to duplicate aka link is already in download list.
__________________
JD-Dev & Server-Admin |
#3
|
|||
|
|||
no these are duplicate links in the linkgrabber.
|
#4
|
||||
|
||||
you should not get dupes in linkgrabber.
linkgrabber by default prevents duplicates from been added to linkgrabber, you will see 'filtered dupe' within the linkcollector log. dupe is of the same linkid OR same full url when linkid isn't set by plugins. raztok
__________________
raztoki @ jDownloader reporter/developer http://svn.jdownloader.org/users/170 Don't fight the system, use it to your advantage. :] |
#5
|
|||
|
|||
Well linkgrabber was going slow so i turned off automatic checking and then loaded up all the links at once again. (2k+ links at 7 seconds per one)
so i ended up with a bunch of duplicates. though it does happen all the time when links have the url encoding instead of special characters like %5D or %20. So technically they aren't the exact same link and linkgrabber will grab both. i've brought it up a few times over the years. |
#6
|
||||
|
||||
Jiaz will need to look at this, provide examples of url encoded dupes.
I've also told jiaz about not url encoding components unless it's required by RFC. in respects to dedicated plugins, we typically only scan for the least amount of URL for example protocol://domain/fuid and not protocol://domain/fuid/craphere. as its just not needed, and can lead to dupes if the dedicated plugin doesn't set specific linkid. so its effectively not just a core issue, but could also be solved within plugins. raztoki
__________________
raztoki @ jDownloader reporter/developer http://svn.jdownloader.org/users/170 Don't fight the system, use it to your advantage. :] |
#7
|
|||
|
|||
well that is another problem altogether and not what i made this topic for.
this time the duplicates were the exact same link, they were just added manually by me bypassing the linkgrabber. basically i'd just like it if remove duplicates actually removed duplicate links in the linkgrabber. |
#8
|
||||
|
||||
its still related, but it could be solved within the core. For example if we urldecoded everything so that the failover setLinkID had a baseline which can not be variable. Effectively would make this issue null and void.
__________________
raztoki @ jDownloader reporter/developer http://svn.jdownloader.org/users/170 Don't fight the system, use it to your advantage. :] |
#9
|
|||
|
|||
how about an option to delete duplicates based on filesize+name?
like right now i have duplicate files in my linkgrabber but they are different links. some filehosts upload a file twice for some reason. (glitch during upload i guess, not sure) they're the exact same name and size but i have to manually click on them to delete. Which is quite cumbersome if you have thousands of files over multiple hosts to go through. |
#10
|
||||
|
||||
just remember that mirrors != duplicates
__________________
raztoki @ jDownloader reporter/developer http://svn.jdownloader.org/users/170 Don't fight the system, use it to your advantage. :] |
#11
|
|||
|
|||
why not? It's the same file
maybe you guys should call it something else than delete duplicates. |
#12
|
||||
|
||||
I did say this earlier, but...
deleting duplicates, are of the same linkid or exact same url when you have object within download tab, and now you have added it again within linkgrabber. They will be shaded a different colour. duplicate is when they are of the same linkid (set by plugin or when not set by plugins core will set it to the url). In respects to Linkgrabber its impossible to add a duplicate when already in the linkgrabber. Only possible when it's in the download tab. This is what the delete duplicate function is meant to be used for!, It has nothing todo with mirrors ;> So I guess a little misunderstanding by you in respects to this. Hope this clears up the situation raztoki
__________________
raztoki @ jDownloader reporter/developer http://svn.jdownloader.org/users/170 Don't fight the system, use it to your advantage. :] |
#13
|
|||
|
|||
so then can you guys add the ability to delete duplicate links in the linkgrabber?
It's possible to add duplicate links if you disable automatic checking first. Last edited by Tyler; 13.08.2017 at 19:23. |
#14
|
||||
|
||||
Quote:
the only time that isn't the case at this point in time is when components of the url are urlencoded, and I did mention this above. Working example https://board.jdownloader.org/images/logo.png Code:
1618|http links_jd.plugins.hoster.raz_DirectHTTP.log 14/08/17 11:27:39 AM - INFO [ jd.controlling.linkchecker.LinkChecker(updateAvailableStatus) ] -> Link https://board.jdownloader.org/images/logo.png is TRUE https://board.jdownloader.org/images/logo.png?test Code:
THREAD: 1625 1625|http links_jd.plugins.hoster.raz_DirectHTTP.log 14/08/17 11:27:54 AM - INFO [ jd.controlling.linkcrawler.LinkCrawler(processHostPlugin) ] -> Processing: httpsviajd://board.jdownloader.org/images/logo.png?test THREAD: 1617 1617|jd.controlling.linkcollector.LinkCollector.log 14/08/17 11:27:54 AM - INFO [ jd.controlling.linkcollector.LinkCollector$11(run) ] -> Filtered Dupe: https://board.jdownloader.org/images/logo.png currently the only time when this doesn't happen (as stated above) is when urlencoding is used within url structure https://board.jdownloader.org/%69%6D...C%6F%67%6F.png Code:
1651|http links_jd.plugins.hoster.raz_DirectHTTP.log 14/08/17 11:29:19 AM - INFO [ jd.controlling.linkchecker.LinkChecker(updateAvailableStatus) ] -> Link https://board.jdownloader.org/%69%6D%61%67%65%73%2F%6C%6F%67%6F.png is TRUE the fix that Jiaz needs to change, is to full URLDecode the linkid when uses the url as failover (also mentioned above). Then and only then will it be impossible for dupes not to add to linkgrabber when already within linkgrabber, since the backend prevents them adding to Linkgrabber 'filtered dupe' event. It would also prevent dupes within linkgrabber when original is within downloadtab! Quote:
raztoki
__________________
raztoki @ jDownloader reporter/developer http://svn.jdownloader.org/users/170 Don't fight the system, use it to your advantage. :] Last edited by raztoki; 14.08.2017 at 05:47. |
#15
|
|||
|
|||
Quote:
so even if its a different link or even a different hoster it would be deleted when you tell jdownloader to delete duplicates. |
#16
|
||||
|
||||
Are you forcing your term about duplicates to be accepted as JD's term and we should change all related things about it?
__________________
FAQ: How to upload a Log |
#17
|
||||
|
||||
@tony2long
*shrugs* everyone has there own agenda at heart. Never the less it's not like we have to comply. mirrors != duplicates. at this time this is intended behaviour Under your idea, the first link added would then be the one you download, vs not been able to select which one you want. Very few if any users would like that or would use said feature. There are features to add links with higher priority (package customiser) or based on selection and adding links which can remove others. Maybe this will suit your needs better? also note: typically JD doesn't always final filesize unless it comes from a verified source (apis). Download is required in order to determine that. Think we will call this topic/thread closed, as current behaviour is intended behaviour. We will look at fixing the url encoded issue for failover linkids raztoki
__________________
raztoki @ jDownloader reporter/developer http://svn.jdownloader.org/users/170 Don't fight the system, use it to your advantage. :] |
#18
|
|||
|
|||
Quote:
Definition of duplicate: Quote:
JD even puts them in the same package after checking them. |
#19
|
||||
|
||||
its a mirror as stated like many times in this thread already
mirrors are not duplicates, thus the OP is greyed out, this is intended behaviour
__________________
raztoki @ jDownloader reporter/developer http://svn.jdownloader.org/users/170 Don't fight the system, use it to your advantage. :] |
#20
|
||||
|
||||
Quote:
JD has "Delete Links: Selected &*Duplicates", there is no "Delete files: Selected & Duplicates".
__________________
FAQ: How to upload a Log Last edited by tony2long; 20.08.2017 at 18:52. |
#21
|
|||
|
|||
Quote:
i'm asking for the separate option to take care of a separate problem. No i'm talking about links, i was merely referring to them as duplicate files in the context of them being the same files across different links. |
#22
|
||||
|
||||
@Tyler
I think your best solution if you want to remove all links other say first returned or via some other means, is to create event scripter script, and bind it to a button and or keyboard shortcut. raztoki
__________________
raztoki @ jDownloader reporter/developer http://svn.jdownloader.org/users/170 Don't fight the system, use it to your advantage. :] |
#23
|
|||
|
|||
I don't want to remove all links just ones that are the same as other ones that i already have in the lingrabber.
if you guys want to call this delete mirrors then i'm fine with that too. |
#24
|
||||
|
||||
I know what you meant
I just don't see us making this function. a) we do not know what mirrors to remove, where you do. so script solution can work for you! b) our user base is used to current behaviour, as its always been that way. as you can understand serves little to no benefit to our userbase. But this doesn't stop you from doing it yourself. raztoki
__________________
raztoki @ jDownloader reporter/developer http://svn.jdownloader.org/users/170 Don't fight the system, use it to your advantage. :] |
#25
|
|||
|
|||
The ones other than the selected?
|
#26
|
||||
|
||||
current implementation
right click > delete/cleanup > keep only selected && add links
__________________
raztoki @ jDownloader reporter/developer http://svn.jdownloader.org/users/170 Don't fight the system, use it to your advantage. :] |
|
|