JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 09.08.2022, 15:11
Rocket Rocket is offline
I will play nice!
 
Join Date: Dec 2020
Posts: 5
Default Smarter Duplicate Check based on file size and file name contents.

Hi,

I'd like to have a smarter duplicate check.

Right now JD only recognizes a duplicate, if you are trying to download the exact same file from the exact same link. Which is nice, but not enough.

If the download link has changed, but the file is the same, JD will not recognize it as a dupe and download it without any hint or warning.

Please add a check for same file size and matching parts of the filename.

So that you also can detect duplicates based on file size (which has a good chance of detecting dupes already) and in combination with matching parts of the filename. This will allow to filter such possible duplicates further and avoid to download them or detect same files with lower or higher file size so that you may chose to download or skip them.

Right now I have to clean duplicates via third party apps after the download, which is not efficient.
Reply With Quote
  #2  
Old 09.08.2022, 17:58
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 64,016
Default

Are we talking about same name check or mirror-recognization?

You can already customize the handling for mirror detection but filenames need to be the same until a certain point or a hash needs to be available to match same files 100%.
See Settings -> Advanced Settings -> Search for mirrordetection
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #3  
Old 11.08.2022, 18:53
Rocket Rocket is offline
I will play nice!
 
Join Date: Dec 2020
Posts: 5
Default

Not mirror.

I'm talking about downloading file A from Mirror A.

A few days later the same file is uploaded in a smaller resolution and named a low resolution etc... and available from mirror b.

I wanna have a check that detects that a file named A was already downloaded some days ago with different resolution and size, but it has a comparable name.

I don't wanna download the "same" file again with worse resolution.
Reply With Quote
  #4  
Old 18.08.2022, 17:52
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 78,420
Default

@Rocket: Duplicate handling is about same file/link. eg adding the same link via different ways (eg alternative domains) or different url format (eg long vs short) and JDownloader being able to recognize those as *the same* link. Here the default duplicate handling works on the download link/url. But plugins may be implemented in mode advanced way, depending if it is meaningful in context of plugin/hoster/link.

Mirror handling does NOT work on download link/url at all.But only on name/size/hashes. You can check Settings-> Advanced Settings-> search for mirrordetection.
also see https://board.jdownloader.org/showth...irrordetection
__________________
JD-Dev & Server-Admin
Reply With Quote
  #5  
Old 18.08.2022, 17:57
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 78,420
Default

Quote:
Originally Posted by Rocket View Post
I'm talking about downloading file A from Mirror A.
A few days later the same file is uploaded in a smaller resolution and named a low resolution etc... and available from mirror b.
I wanna have a check that detects that a file named A was already downloaded some days ago with different resolution and size, but it has a comparable name.
I'm sorry but how should JDownloader know ? Just same name doesn't mean it's same *content*. also there is no knowledge about resolution/content at all. A bad recompression can have larger file size than the original, so filesize also doesn't mean anything in this context. JDownloader doesn't know anything about resolution. Once downloaded you can use tools like ffmpeg to query more detailed information but in advanced that's not possible without high false positive rate.
Mirror A might provide name and resolution. Mirror B just name, no details about resolution. Both being different mirrors/sites/plugins, I don't see any meaningful way to proper add support for this. I can easily give lots of examples where filename/resolution will easily fail/result in wrong/unexpected results.

Of course you are free to share your ideas about this!
__________________
JD-Dev & Server-Admin
Reply With Quote
  #6  
Old 18.08.2022, 18:00
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 78,420
Default

Quote:
Originally Posted by Rocket View Post
Please add a check for same file size and matching parts of the filename.
With help of custom scripts for Eventscripter, see https://board.jdownloader.org/showthread.php?t=70525, you can implement the logic you want by yourself.
Better ask in that thread for help/hint. Eg when adding a link it can be compared against all other links with your own compare logic in mind and then
decide what to do, eg rename/disable/remove...
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 08:26.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2022, Jelsoft Enterprises Ltd.