JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 24.06.2018, 17:28
DJwa163's Avatar
DJwa163 DJwa163 is offline
Tera Loader
 
Join Date: Feb 2010
Posts: 176
Exclamation File Name Masks

Dear JD Developers Team,
Thanks for your continued support for the program which has made it into a phenomenal success.

Lately, I found an issue, solving which can add to the efficiency of this wonderful program.

Issue:
Many mirror links have identical file hosted, but with a different syntax in their names. This is resulting in duplicate downloads. (files are identical, but the
names have subtle variation depending on the file host's server rules). Since the names are not 100% identical, JD is treating them as different files, and downloading both of them.

For example:
Spoiler:

Quote:
Set 1:
Link: **External links are only visible to Support Staff****External links are only visible to Support Staff**
File: *****+HDR+2018+v1.2.0.2114

Link:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
File: *****_HDR_2018_v1.2.0.2114

Set 2:
Link:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
File: *****TrueImage2018 12510

Link: **External links are only visible to Support Staff****External links are only visible to Support Staff**
File: *****TrueImage2018_12510

Set 3:
Link: **External links are only visible to Support Staff****External links are only visible to Support Staff**
File: **ezloader 3.1.2.rar

Link: **External links are only visible to Support Staff****External links are only visible to Support Staff**
File: **ezloader 3.1.2.rar

Set 4:
Link: **External links are only visible to Support Staff****External links are only visible to Support Staff**
File: *****_Technical_Suite_2018_v20.1.0.707_x64-P2P.rar

Link: **External links are only visible to Support Staff****External links are only visible to Support Staff**
File: ****_Technical Suite 2018 v20.1.0.707 x64-P2P.rar


Essentially, with this example, I want to make the point that:
a file name on different mirror may have the following name patterns:
MIRRORED_FILE_NAME.extension
MIRRORED FILE NAME.extension
MIRRORED-FILE-NAME.extension
MIRRORED+FILE+NAME.extension

Suggestion:
To implement a "File Name Masking" Mode (which can be disabled/toggled/ticked) Under the Settings Tab, which would allow users to set masks (e.g. _, <space>, +, - or any other characters )

When enabled, this feature should be able to treat all such name variants as identical, so that unnecessary duplicate downloading can be prevented.

Image Illustration:

Attached Images
File Type: png Untitled.png (90.5 KB, 1 views)

Last edited by DJwa163; 26.06.2018 at 22:47.
Reply With Quote
  #2  
Old 25.06.2018, 17:37
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Plugins already can handle this but we have to know that the sites do these replacements so we can update the plugin. I've updated the plugins and your examples. After next update, when you add those links, the filenames will be fixed/corrected.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #3  
Old 25.06.2018, 17:38
DJwa163's Avatar
DJwa163 DJwa163 is offline
Tera Loader
 
Join Date: Feb 2010
Posts: 176
Default

Sir,
Some examples have been quoted above inside the spoiler.
I will try to collect more examples.

Last edited by DJwa163; 27.06.2018 at 07:39.
Reply With Quote
  #4  
Old 25.06.2018, 17:39
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default


Once this feature is added, you will be able to customize replacements.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #5  
Old 25.06.2018, 17:43
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Please know that the plugin handling and feature will fix/replace when settings the filename.

That means that the filename will be updated/changed and NOT kept as original and treated different.
A_B.txt
and
A B.txt
will either become 'A B.txt' or 'A_B.txt' depending on your customized rules.

Keeping the original 'A B.txt' and 'A_B.txt' would result in major changes and many many duplicated code/checks and very slow as this would have to be done on many many parts like FileExists,ExtractionParts, Mirror handling, Download Core, Rename.....
__________________
JD-Dev & Server-Admin
Reply With Quote
  #6  
Old 25.06.2018, 17:55
DJwa163's Avatar
DJwa163 DJwa163 is offline
Tera Loader
 
Join Date: Feb 2010
Posts: 176
Default

Thank you very much, sir.

EDIT:
I seem to have stolen someone else's idea. This proposition originally came way back in 2017! I must credit the original proponent, though I don't know who he/she is. But sadly enough, this doesn't seem to be a very popular concept after all (hence, stayed just as an idea, and never quite materialized for a long time).

I hope, this time it sees some light of success.


Status: New
Created: 2017-06-16T07:58:49Z
Priority: Normal
Progress: 0%
Last Update: 2018-05-14T14:55:18Z

Last edited by DJwa163; 27.06.2018 at 07:50.
Reply With Quote
  #7  
Old 03.07.2018, 17:54
DJwa163's Avatar
DJwa163 DJwa163 is offline
Tera Loader
 
Join Date: Feb 2010
Posts: 176
Default

Thanks a lot, sir.
I can see that this thread has been marked as SOLVED.
But, I didn't find this filter under 'Advanced Settings'.
My JD is updated. But, don't see this feature!!!

Last edited by DJwa163; 03.07.2018 at 18:00.
Reply With Quote
  #8  
Old 03.07.2018, 18:55
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

The feature is not yet available. The wrong status was set by accident
__________________
JD-Dev & Server-Admin
Reply With Quote
  #9  
Old 13.07.2018, 09:58
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 1,533
Default

@DJwa163: Till the feature is implemented, you can also check the eventscripter extension, where you can automatically replace characters in file names using javascript.
Reply With Quote
  #10  
Old 15.07.2018, 10:54
DJwa163's Avatar
DJwa163 DJwa163 is offline
Tera Loader
 
Join Date: Feb 2010
Posts: 176
Default

@mgpai,
Can you please put up an example script?
Thanks

BTW, besides this script, there should be an easier way to bring in the function. That would go better with most users.
So, please don't abandon the project.
Reply With Quote
  #11  
Old 16.07.2018, 13:02
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by DJwa163 View Post
That would go better with most users.
So, please don't abandon the project.
The ticket is still open, I just need free time to work on it
__________________
JD-Dev & Server-Admin
Reply With Quote
  #12  
Old 17.07.2018, 02:42
audio20's Avatar
audio20 audio20 is offline
Fibre Channel User
 
Join Date: Jun 2009
Posts: 125
Smile Love it

I hadn't seen this thread before... Love this idea as well.

Still... I really do understand why it is not quite in the list of priorities.

I'll keep an eye open for when this feature get implemented in the program.
Reply With Quote
  #13  
Old 17.07.2018, 11:20
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

I have lot's of suff on my todo list and around 500 tickets assigned to me
__________________
JD-Dev & Server-Admin
Reply With Quote
  #14  
Old 19.07.2018, 11:23
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 1,533
Default

Quote:
Originally Posted by DJwa163 View Post
Can you please put up an example script?
Code:
github.com/mgpai/resources/blob/master/jdownloader/eventscripter/scripts/DJwa163.js
Reply With Quote
  #15  
Old 08.09.2018, 21:51
audio20's Avatar
audio20 audio20 is offline
Fibre Channel User
 
Join Date: Jun 2009
Posts: 125
Default

Quote:
Originally Posted by mgpai View Post
Code:
github.com/mgpai/resources/blob/master/jdownloader/eventscripter/scripts/DJwa163.js
Hello, mgpai!

The last time I did program anything, it was probably more than 20 years ago and I was still using Pascal...

I took the sample script that you provided and tried to modify it for my own personal use; I started reading some JavaScript reference material around the web but soon realized it wasn't "reeeeeeeally" necessary for this kind of task (as long as I already had your code in hand).

I would like to thank you and leave another reference here just in case anyone else would like to adapt their own script tailored to their use.

Code:
// Script to Replace a few characters in the filename inside LinkGrabber
// Trigger: Packagizer Hook
// Strip a sequence of characters like: "&.#40;"
// Strip Parenthesis and Brackets: "(", ")", "[", "]"
// Strip Apostrophe and Comma: "'", ","
// Replace multiple spaces with a single one

var fn = link.getName();
var re = /(\&#.\d\d?;|[\(\)\[\]',])/g;

if (re.test(fn)) link.setName(fn.replace(re, "").replace(/\s+/g, " "));
Reply With Quote
  #16  
Old 21.07.2018, 13:26
DJwa163's Avatar
DJwa163 DJwa163 is offline
Tera Loader
 
Join Date: Feb 2010
Posts: 176
Exclamation

An error message is popping up.
Do I need to modify the code?

Attached Images
File Type: png Screenshot_1.png (85.1 KB, 1 views)

Last edited by DJwa163; 21.07.2018 at 13:31.
Reply With Quote
  #17  
Old 21.07.2018, 13:35
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 1,533
Default

Quote:
Originally Posted by DJwa163 View Post
An error message is popping up.
Do I need to modify the code?
Have you selected the trigger (Packagizer Hook) specified in the script?
Reply With Quote
  #18  
Old 21.07.2018, 16:38
DJwa163's Avatar
DJwa163 DJwa163 is offline
Tera Loader
 
Join Date: Feb 2010
Posts: 176
Default

Quote:
Originally Posted by mgpai View Post
Have you selected the trigger (Packagizer Hook) specified in the script?
Thanks mgpai,
This time it worked flawlessly.
Reply With Quote
  #19  
Old 21.07.2018, 16:55
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 1,533
Default

@DJwa163: You're Welcome.

While this is useful to replace characters in file names, it may not completely solve the 'duplicate downloading' issue though, as some hosters not only substitute characters, they sometimes also truncate it. If you are familiar with Javascript, you can explore other/better options, possibly identify and rename the mirrors at the download stage, after the first file in the package starts downloading.
Reply With Quote
  #20  
Old 09.09.2018, 01:55
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

use of htmlencoding/character references (&[^;]+;) and subsequent decoding should be addressed in each plugin, if you know of a plugin which needs fixing please let us know.
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 03:39.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.