JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 09.03.2024, 13:07
mmdbuta mmdbuta is offline
Modem User
 
Join Date: Mar 2024
Posts: 4
Default Renaming illegal characters in filename

Hi everyone.

Before I asked, I just used JD, and I'm sorry that I'm using a translator, not an English speaker.

I recently used JD for the first time and touched the settings a little bit, and it was very satisfying.

But soon I had a problem that was hard to solve with my ability, so please let me know about this.

I've searched as much as I can on this forum and Google with my amazing poor English skills, but I still couldn't solve it.

I usually use JD to crawl iwara videos and I want to know how to rename some illegal characters as I wrote in the title.

I'm using the windows system.

for example
/ to /
? to?
like this

I've seen some resolved threads about it, but I'm not sure what "Ticket" is, and the event scripter... let me just say first that I don't know how to code.

Actually, I tried some codes on copilot or other threads, but it didn't work out also.

I found "General Settings: Filename and Path Character Regex Replace Map" in Advanced setting, so I thought this is it! This didn't work either.

I'm sorry if I didn't follow the rules of this forum or the set questioning methods. To excuse myself, there are very few thread-style communities in my nation, so I'm not used to it.

All the links that seemed to have answers were dead... I tried to solve them myself without asking questions, but it wasn't easy.

I don't know if my words were conveyed well. I'm sorry for the rambling.

Anyway, I'm looking for a way to solve this problem. It might be better to learn how to write scripts soon.

Thank you for reading this long article.

I hope you all have a good day.
Reply With Quote
  #2  
Old 10.03.2024, 21:03
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,387
Default

@mmdbuta: Can you please provide example links? JDownloader already should replace forbidden chars (depending on the plugin) and finally remove all forbidden chars.
Best is to provide example links, then we can check it
__________________
JD-Dev & Server-Admin
Reply With Quote
  #3  
Old 11.03.2024, 11:26
mmdbuta mmdbuta is offline
Modem User
 
Join Date: Mar 2024
Posts: 4
Default

@Jiaz: I forgot to suggest the links
Here are some examples
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Forbidden chars just replaced to "_" only
That makes me sad ;_;
Reply With Quote
  #4  
Old 11.03.2024, 16:01
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,985
Default

This has already been done by me.
Next CORE-update (ETA unknown) will:
- Globally replace those chars with better ones
- Allow the user to define a custom regular expression based 'replace map' and also to change our default one
- And yes, same goes for package names
Default replaces (as of the date of writing this post):
Code:
{
 ":"    : ";",
 "\\|"  : "¦",
 "<"    : "[",
 ">"    : "]",
 "/"    : "\u2044",
 "\\\\" : "∖",
 "\\*"  : "#",
 "\\?"  : "¿",
 "\\!"  : "¡",
 "\""   : "'"
}
See the following advanced settings:
Code:
GeneralSettings.filenamecharacterregexreplacemap
and:
Code:
GeneralSettings.packagenamecharacterregexreplacemap
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 14.06.2024 at 16:40. Reason: Added more information
Reply With Quote
  #5  
Old 11.03.2024, 17:58
mmdbuta mmdbuta is offline
Modem User
 
Join Date: Mar 2024
Posts: 4
Default

Oh, does that mean I have to wait a little bit to use that feature in the official version?
Reply With Quote
  #6  
Old 11.03.2024, 18:02
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,985
Default

I cannot provide any ETA.
The feature has been implemented roughly since august 2023 and is still waiting for release.
If you can't wait, another option is to use/test the dev version.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #7  
Old 11.03.2024, 19:06
mmdbuta mmdbuta is offline
Modem User
 
Join Date: Mar 2024
Posts: 4
Default

Oh... august 2023...
Then i'll try dev version.
Thanks for your kindness.
Reply With Quote
  #8  
Old 13.03.2024, 10:55
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,387
Default

@mmdbuta: We're working on releasing finally the changes soon.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #9  
Old 14.06.2024, 16:11
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,985
Default

Changes are live.
EDIT

See the following advanced settings:
Code:
GeneralSettings.filenamecharacterregexreplacemap
and:
Code:
GeneralSettings.packagenamecharacterregexreplacemap
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 14.06.2024 at 16:30. Reason: EDIT
Reply With Quote
  #10  
Old 15.06.2024, 20:40
TGU TGU is offline
Mega Loader
 
Join Date: Jun 2024
Location: International Waters where DRM/DMCA protections are ignored
Posts: 65
Default

Nice to see those regex's settable & fetchable via the settings, I was wondering about the "." and "_" characters being replaced with spaces also (thankfully the new settings description said this).

When are these regexes and cleanup actions applied?, I've noticed things like the subfolderbyplugin still containing "." so I'm assuming these 2 settings are only applied to the link name (filename) and package name exclusively?
So is there another setting for the cleanup of the SaveTo path?

Last edited by TGU; 15.06.2024 at 20:49.
Reply With Quote
  #11  
Old 15.06.2024, 20:43
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,387
Default

Due to a bug those are currently not handled. Please wait for next update. see
https://board.jdownloader.org/showpo...8&postcount=19
__________________
JD-Dev & Server-Admin
Reply With Quote
  #12  
Old 15.06.2024, 21:11
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,387
Default

@TGU: the update is live
__________________
JD-Dev & Server-Admin
Reply With Quote
  #13  
Old 16.06.2024, 18:59
TGU TGU is offline
Mega Loader
 
Join Date: Jun 2024
Location: International Waters where DRM/DMCA protections are ignored
Posts: 65
Default

By default I believe that these Unicode spaces should be replaced with a standard space (if you aren't already handling it), these can cause quite the headache (and even potentially be malicious?). There's also things like changing the direction of the text for Arabic and similar which I have seen being used in malicious attacks with user run executables, which can make a file.txt.exe look like exefile.txt

unicode spaces: "\u2009", "\u200A", "\u200B", "\u200C","\u200D", "\uFEFF", "\u0000-\u001f"
unicode quotes: "\u201C" = “, "\u201D" = ” (Replace with normal quotes?)

:thumbup: that could cover most of the issues I've had with these in the past

Last edited by TGU; 16.06.2024 at 20:34.
Reply With Quote
  #14  
Old 16.06.2024, 20:52
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,387
Default

@TGU:
You're talking about, for example
medium.com/@sam.rothlisberger/embed-a-malicious-executable-in-a-normal-pdf-or-exe-81ee5339707e

I have not looked deeper into this if windows meanwhile better handles or protects for this situation.

I'm sorry but I don't think that we should add those replacements by default.
a.) those replacements do only affect names from plugins to replace forbidden characters (all OS). so no extraction/archives will be auto replaced
b.) it's job of the OS/scanner/defender to detect this and warn user about or block it.
c.) where do we start? where do we end? exe files, bat files? shell scripts? powershell scripts and many more...

I'm okay with creating a wiki article about this and some rules user can copy/paste into their setup, but I don't think it's something
a download tool should do. Would be interesting to know how browsers like Chrome/Firefox handle this? do they mangle/neutralize the filename or use as it is?
Same with other characters. This feature was introduced so ppl can customize the default replacements, not for us adding more default replacements and do stuff that didn't happen before/noone was asking for before.

You are free to create a collection of useful replacement rules and we're happy to include those in an article but for them to be included in default or not, not decided today.
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 16.06.2024 at 21:48.
Reply With Quote
  #15  
Old 16.06.2024, 23:41
TGU TGU is offline
Mega Loader
 
Join Date: Jun 2024
Location: International Waters where DRM/DMCA protections are ignored
Posts: 65
Default

@Jiaz Yup that's basically it, the "Right-To-Left Override (RTO or RTLO)" could be used maliciously but also has its own actual use with some foreign languages. The main issue with it, is that one could think it's not an exe (or any other extension .bat, .ps, .scr doesn't matter) and simply run it themselves. A little warning article about it for the user could be nice.

Browser's just typically download the files as-is within the HTTP Header Content-Disposition, normally filename*encoding_here="filename_here" if not the filename="filename_here"; They don't include any warnings about RTO/RTLO, and neither does the OS.

As for the spaces and quotes I think those should be replaced in general I've seen them used such few times, that when I do see them it can only just cause issues.

:thumbup: But this is some of the stuff I do in my download tools after +10.. 15 years of doing this, feel free to use it or not. I'm sure someone will read these posts and use the information in it. As I'm extracting these texts from websites and then using using them within the OS for paths you can basically refer them as a "Cross-Site Scripting" equivalent.

Last edited by TGU; 16.06.2024 at 23:49.
Reply With Quote
  #16  
Old 17.06.2024, 00:20
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,387
Default

@TGU: Can you provide examples for the unicode spaces?
And maybe an example for RTO/RTLO? then maybe we can simple rule for exe/bat/ps1... files. Need to do some testing. If it can be done easily with pattern/regex, then on 2nd thought, why not have it in default. Won't hurt
__________________
JD-Dev & Server-Admin
Reply With Quote
  #17  
Old 17.06.2024, 02:23
TGU TGU is offline
Mega Loader
 
Join Date: Jun 2024
Location: International Waters where DRM/DMCA protections are ignored
Posts: 65
Default

@Jiaz this should help you **External links are only visible to Support Staff****External links are only visible to Support Staff** I had a hard time with the forum software changing the values.

All of these shouldn't be in a filename as-is except RTO/RTLO for some few languages, many of these are normally used within a text/document context.

Unicode Spaces (for use the \u within double quotes within code): "\u2009"=" ", "\u200A"=" ", "\u200B"="​", "\u200C"="*","\u200D"="*", "\uFEFF"="" (replace with space)

unicode quotes: "\u201C" ="“", "\u201D"="”" (replace with standard quote)

"\u0000-\u001f" is 31 of the control characters which should never be in the filenames, just remove them

The RTO/LTO is filtered by the forum software, but it's a truly evil character especially when they mix them together. This one should be replaced with an _ (some languages are read backwards which "need" it)
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**


(RTO = * = "\u202E")
filename: haha_got_you.*gmi.exe
what the user sees: haha_got_you.*exe.img

Last edited by TGU; 17.06.2024 at 02:31.
Reply With Quote
  #18  
Old 17.06.2024, 11:01
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,985
Default

I'm against doing this by default.
If you start with this, you will find way more security related things such as "should executable files also be launched when a user chooses to 'open file' in JDownloaders GUI" or also "if a multihost supports hosts like 'drive.google.com', it may obtain users private links so maybe that should be turned off by default".

Also if the user is downloading some kind of known/popular malicous file, the users' basic firewall should detect it way before he is even able to open it (that said, if it is a targeted attack that won't happen).

In the end, the user needs to trust his source of data.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #19  
Old 17.06.2024, 23:17
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 80,387
Default

On windows you can block execution of files if they contain such chars, see
superuser.com/questions/408792/what-are-ways-to-prevent-files-with-the-right-to-left-override-rlo-unicode-cha
__________________
JD-Dev & Server-Admin
Reply With Quote
  #20  
Old 19.06.2024, 12:07
goratnik goratnik is offline
Baby Loader
 
Join Date: Dec 2023
Posts: 9
Default

JDownloader has started doing seemingly nonsensical character replacements and reading recent threads has brought me here. I like the idea of the map, I'm going to update mine for files in a moment, there is one logical problem with the packagename-side feature though: it depends on the flag "Clean Up Packagenames", which is still doing its own with modification of "." and "_", which is on the other hand unwanted by me and I had it disabled before. So in order to use a feature that keeps packagenames consistent with filenames, I have to turn on another feature that BREAKS names.

Please do either of these:
* make the "Packagename Replace Map" independent from "Clean Up Packagenames"; disabling the feature would be done by setting the map to empty, just like the map for filenames
* make "Clean Up Packagenames" feature wholly dependent on "Packagename Replace Map", meaning that only matches defined in the map are modified, including explicit definitions for "_" or "." as well; this would include updating the description to indicate that no actions other than defined in that other setting will be taken

Cheers!

Last edited by goratnik; 19.06.2024 at 12:18.
Reply With Quote
  #21  
Old 19.06.2024, 12:26
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,985
Default

Quote:
Originally Posted by goratnik View Post
there is one logical problem with the packagename-side feature though: it depends on the flag "Clean Up Packagenames"
That is not correct.
The replace map is always executed while the "old cleanup" is happening after the map replace and that part depends on the said setting.
We didn't remove it because we first wanted to find out the demand of it.
TL;DR If you turn off "GeneralSettings.cleanuppackagenames", only the map will be used.

If you do not like some of the map replacements, modify the map accordingly. The replaces are vital as some of those characters are not allowed in file paths under some OS and in a lot of cases, package names end up being a part of the file path [if the names were only used in JD GUI, the replacements wouldn't be needed].

Quote:
Originally Posted by goratnik View Post
JDownloader has started doing seemingly nonsensical character replacements and reading recent threads has brought me here.
Can you narrow this down for me?

My point of view:
Replacing a bunch of characters just by "_" was never a good idea but this is what you guys were used to.
The new method is better. If you do not like if, feel free to just change the map to replace all of those characters to "_".
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #22  
Old 19.06.2024, 12:35
goratnik goratnik is offline
Baby Loader
 
Join Date: Dec 2023
Posts: 9
Default

Quote:
Originally Posted by pspzockerscene View Post
That is not correct.
The replace map is always executed while the "old cleanup" is happening after the map replace and that part depends on the said setting.
We didn't remove it because we first wanted to find out the demand of it.
TL;DR If you turn off "GeneralSettings.cleanuppackagenames", only the map will be used.
Thank you, I just confirmed that map is still used with that other option off; I was misguided by description of the map saying "You can change this map to your needs and you can disable this by turning off the setting 'Clean Up Package Names'"; please fix the description and that would be it.

Quote:
Originally Posted by pspzockerscene View Post
If you do not like some of the map replacements, modify the map accordingly. (...) Can you narrow this down for me?
This applied mostly to replacement of "!", which is common in names and apart from corner cases of bash history expansion - rather unlikely to cause problems, I think; I have simply updated the map removing that one mapping and it's really OK with how it's customizable.

Last edited by goratnik; 19.06.2024 at 12:37.
Reply With Quote
  #23  
Old 19.06.2024, 12:40
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,985
Default

Quote:
Originally Posted by goratnik View Post
I was misguided by description of the map saying "You can change this map to your needs and you can disable this by turning off the setting 'Clean Up Package Names'"; please fix the description and that would be it.
It will be fixed with the next set of CORE-updates.
Thanks for reporting!

When the description was added, the goal was to have it this way, in the end plans were changed the wrong description was just forgotten.

Quote:
Originally Posted by goratnik View Post
This applied mostly to replacement of "!", which is common in names and apart from corner cases of bash history expansion - rather unlikely to cause problems, I think; I just updated the map removing that one mapping and it's really OK with how it's customizable.
Well the items which need to be replaced depend on the OS of the user but we didn't want to over-engineer this feature so we decided to try to go the "one for all" way.
Especially under Linux systems afaik you should be able to use anything as filename, even pipe symbols (but not using JDownloader).
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #24  
Old 19.06.2024, 12:49
goratnik goratnik is offline
Baby Loader
 
Join Date: Dec 2023
Posts: 9
Default

A correction: even with an explicit line
Code:
 "!"    : "!",
"!" is still replaced by "¡", which is unwanted by me and is not forced on by the filesystem anyway, so the map can apparently be ignored and the replacements still applied; can you please accept "noop" and further replace with underscores only if the character would be actually illegal, as in, filesystem would return an error on file creation attempt?

While I have nothing against the default, I would prefer to be able to keep some of these characters.
Reply With Quote
  #25  
Old 19.06.2024, 12:53
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,985
Default

You need to replace it with something else - see [future] description of the setting:
Quote:
Returns mapping of common invalid characters as regular expression to be replaced inside filenames. This is used to try to avoid invalid download paths due to forbidden characters in filenames. Any characters which are invalid for file paths and remain after execution of these replacements will be removed by a generic replacement handling with underscores.
Looking at the history, I think we didn't use to replace "!" before said update.
Maybe we should remove it from the default map since it is allowed under both windows and Linux.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #26  
Old 19.06.2024, 13:01
goratnik goratnik is offline
Baby Loader
 
Join Date: Dec 2023
Posts: 9
Default

Voting for removal of "!", it has a low risk factor and fixing it manually is a hassle with how frequent it is.
Reply With Quote
  #27  
Old 19.06.2024, 13:05
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,985
Default

Your got my vote.

Just for reference - the following is the old slash fallback-statement which is removing illegal characters after all of the customizable replacements:
Code:
    public static String alleviatePathParts(String pathPart, boolean removeLeadingHidingDot) {
        if (StringUtils.isEmpty(pathPart)) {
            if (pathPart != null) {
                return pathPart;
            }
            return null;
        }
        pathPart = pathPart.trim();
        /* remove invalid chars */
        /**
         * Integer value zero, sometimes referred to as the ASCII NUL character.
         *
         * Characters whose integer representations are in the range from 1 through 31->\\x00-\\x1f
         *
         * < (less than), * > (greater than), : (colon), " (double quote), / (forward slash), \ (backslash), | (vertical bar or pipe), ?
         * (question* mark) (asterisk)
         *
         * Volume designators (drive letters) are similarly case-insensitive. For example, "D:\" and "d:\" refer to the same volume.
         */
        pathPart = pathPart.replaceAll("([\\\\|<|>|\\||\r|\n|\t|\"|:|\\*|\\?|/|\\x00-\\x1f])+", "_");
        if (CrossSystem.isWindows() || CrossSystem.isOS2()) {
            /**
             * **External links are only visible to Support Staff**
             */
            if (CrossSystem.isForbiddenFilename(pathPart)) {
                pathPart = "_" + pathPart;
            }
        }
        /*
         * replace starting dots by single dot (prevents directory traversal)
         */
        if (removeLeadingHidingDot) {
            pathPart = pathPart.replaceFirst("^\\.+", "");
        } else {
            pathPart = pathPart.replaceFirst("^\\.+", ".");
        }
        /*
         * remove ending dots, not allowed under windows and others os maybe too
         *
         * Do not end a file or directory name with a space or a period.
         */
        pathPart = pathPart.replaceFirst("\\.+$", "");
        pathPart = pathPart.trim();
        if (StringUtils.isEmpty(pathPart)) {
            return "_";
        } else {
            return pathPart;
        }
    }
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #28  
Old 19.06.2024, 13:26
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,985
Default

Next CORE-update will reset the map once and the new default one will not contain "!" anymore.

Bitte auf das nächste CORE-Update warten!

Please wait for the next CORE-Update!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #29  
Old 19.06.2024, 16:36
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,985
Default

Update has been released.
Changelog:
https://board.jdownloader.org/showth...798#post531798
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #30  
Old 21.06.2024, 00:49
TGU TGU is offline
Mega Loader
 
Join Date: Jun 2024
Location: International Waters where DRM/DMCA protections are ignored
Posts: 65
Default

Quote:
Originally Posted by Jiaz View Post
On windows you can block execution of files if they contain such chars, see
superuser.com/questions/408792/what-are-ways-to-prevent-files-with-the-right-to-left-override-rlo-unicode-cha
Ohh
I have to turn that on right away, you never know when it's going to get you. I've only seen it a few times, but anything with an "exe" in it's name is always a :red_flag: to me (but sometime they use the other executable extensions...)

:thumbup: removal of ! -> ¡

Last edited by TGU; 21.06.2024 at 00:55.
Reply With Quote
  #31  
Old 23.06.2024, 02:29
Chicken++ Chicken++ is offline
Baby Loader
 
Join Date: Aug 2023
Posts: 9
Default

Quote:
Originally Posted by pspzockerscene View Post
My point of view:
Replacing a bunch of characters just by "_" was never a good idea but this is what you guys were used to.
The new method is better. If you do not like if, feel free to just change the map to replace all of those characters to "_".

Understandable, but I'd suggest that next time you change the default settings for how files are saved, you should make a popup message appear after the update clarifying this (and how to disable it should the user desire such).

Because while I agree that it does make more logical sense on its own, you need to consider that some people want to update their existing libraries of channel-downloads and are reliant on "Skip if file already exists" --- I only just updated recently and suddenly jDL was downloading 1000+ duplicate files because their file names were all different to the ones I already have, and only realized why that was happening after 3 days of downloading 50 GB I didn't want.


So, I need to restore the old file naming scheme. Oddly enough, simply setting "Filename Replace Map" and "Packagename Replace Map" to "{}" has no effect, it still does the same replacements as before.

I guess I have to replace all those characters with "_" explicitly ? Weird.

I mean, that's what I have to do to "get the old naming scheme back", right?

Also, I'm guessing "Clean up package" should be "off", then?

Last edited by Chicken++; 23.06.2024 at 04:12.
Reply With Quote
  #32  
Old 23.06.2024, 07:11
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,647
Default

@chicken++

I believe there was no setting before (or at least none that was used), file names changes was hard coded. And with package names were (and most likely still) stripped select chars to have cleaner names allowing for easy assignment to a single package '.' '_' '-' for instance replaced with " ". I assume the hardcoding has been removed for settings which could be changed by the end user, so reverting would be not possible without roll back (don't think is possible with the update network) or a copy of an old installation.

Some plugins used to perform internal cleanups also, but that was for other reasons (not illegal chars).

Only way to prevent dupes is to leave them within your download list (done based on url or uid stored). But this does have negative side effects, as memory requirement grows the larger the list becomes. The project really needs to work on an external dupe db in SQL or similar (as addon/extension). This would prevent these sort of issues moving forward. You mentioned channel name I assume youtube? For youtube the uid is most all you would need to store and would have prevented this outcome.

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #33  
Old 23.06.2024, 20:07
Chicken++ Chicken++ is offline
Baby Loader
 
Join Date: Aug 2023
Posts: 9
Default

Quote:
Originally Posted by raztoki View Post
@chicken++
so reverting would be not possible without roll back
I'm not asking for a change in the code, all I want to do is to change my settings such that they emulate the prior behavior. (I don't want to use an outdated version, obviously!) And I was asking for someone to "spell check" my settings.


Quote:
Originally Posted by raztoki View Post
Only way to prevent dupes is to leave them within your download list
Ooooof, sorry, but that really really would not work for me for... a number of reasons.

In short: I already have an organized system, this would throw a massive wrench into all of that. E.g., some duplicates are allowed because otherwise it'd be more chaotic. (Someone's playlist can contain videos from channels you also want to download.)




Quote:
Originally Posted by raztoki View Post
@chicken++
I believe there was no setting before (or at least none that was used), file names changes was hard coded.
Yeah, but I guess until a few months ago I guess it still replaced all of them with "_" instead of unique variations. So changing the replacement maps for files and packages to this:
Code:
{
 ":"    : "_",
 "\\|"  : "_",
 "<"    : "_",
 ">"    : "_",
 "/"    : "_",
 "\\\\" : "_",
 "\\*"  : "_",
 "\\?"  : "_",
 "\""   : "_"
}
...should restore the old naming scheme, as long as I also disable "Clean up package" which would turn "_" into a blankspace otherwise. [OBSOLETE, SEE FURTHER DOWN BELOW]


Quote:
Originally Posted by raztoki View Post
And with package names were (and most likely still) stripped select chars to have cleaner names allowing for easy assignment to a single package '.' '_' '-' for instance replaced with " ".
This one I might need some advice on how to restore the previous behavior.

First of, unlike you said, "-" isn't replaced regardless of "Clean up package" being ON or OFF, so I don't think I need to do anything about that. (why would it be removed, it's allowed by Windows and should cause no problems)

"." is where I'm unsure what the previous behavior was. I've got 2 old folders with a "." in them (I think they might be subfolders in a Google Drive / mega.nz download?) and I have no idea how that was handled before. Now, if I turn "Clean up package" ON, "." disappears. If I turn it off, it stays*. But what was the previous text replacement? An underscore? A blankspace? An empty space?

* For completeness' sake: "It stays", unless the dot is at the end of of a folder name, which Windows won't allow as this would cause issues. This removal happens independent of "Clean up package" ON/OFF. Perfectly sensible.


Are there any other characters affected by "Clean up package" I should be aware of?



...I suppose my attempt at "restoring the previous behavior" doesn't need to be 100% perfect, as long as it prevents the majority of duplicates at least. But some tips would be nice.

Last edited by Chicken++; 05.07.2024 at 22:54. Reason: Mention that first draft of JSON map is obsolete now
Reply With Quote
  #34  
Old 24.06.2024, 09:53
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,985
Default

Quote:
Originally Posted by Chicken++ View Post
Understandable, but I'd suggest that next time you change the default settings for how files are saved, you should make a popup message appear after the update clarifying this (and how to disable it should the user desire such).
No, not planned.

Quote:
Originally Posted by Chicken++ View Post
Because while I agree that it does make more logical sense on its own, you need to consider that some people want to update their existing libraries of channel-downloads and are reliant on "Skip if file already exists" --- I only just updated recently and suddenly jDL was downloading 1000+ duplicate files because their file names were all different to the ones I already have, and only realized why that was happening after 3 days of downloading 50 GB I didn't want.
I understand your troubles but you can easily modify the replace-map to restore the previous behavior.


Quote:
Originally Posted by Chicken++ View Post
I guess I have to replace all those characters with "_" explicitly ? Weird.
Yes and not weird:
1. Yes, change the map so that all characters get replaced to "_".
2. Not weird because this is exactly how it was before, see:
https://board.jdownloader.org/showpo...8&postcount=27

@raztoki
User can easily restore the previous behavior by changing the map.

Quote:
Originally Posted by Chicken++ View Post
...should restore the old naming scheme
Yes.

Quote:
Originally Posted by Chicken++ View Post
Are there any other characters affected by "Clean up package" I should be aware of?
JDownloader is open source.
The following code gets executed whenever "cleanuppackagenames" is enabled:
Code:
    public static String cleanPackagename(final String packageName, boolean allowCleanup) {
        String ret = replaceCharactersByMap(packageName, PACKAGENAME_REPLACEMAP);
        /* if enabled, replace dots and _ with spaces and do further clean ups */
        if (allowCleanup && org.jdownloader.settings.staticreferences.CFG_GENERAL.CLEAN_UP_PACKAGENAMES.isEnabled()) {
            // still wanted by users, so please do not simply remove this
            // TODO: maybe add own cleanup replace map that does this via pattern/replace or add those to package name replace map
            final StringBuilder sb = new StringBuilder();
            char[] cs = ret.toCharArray();
            char lastChar = 'a';
            for (int i = 0; i < cs.length; i++) {
                switch (cs[i]) {
                case '_':
                case '.':
                    if (lastChar != ' ') {
                        sb.append(' ');
                    }
                    lastChar = ' ';
                    break;
                default:
                    lastChar = cs[i];
                    sb.append(cs[i]);
                }
            }
            ret = sb.toString();
        }
        return ret.trim();
    }
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #35  
Old 27.06.2024, 11:58
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,647
Default

@psp
yah, My answer was more in respects to a quick change within the app or reverting version. I went looking for the diff but ran out of time on the weekend, bit hard to give exact advice to the changes without knowing what they were.
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #36  
Old 27.06.2024, 12:05
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,985
Default

No worries.
We've made some mistakes here but I'm confident that this change was a good one and overall helps to get nicer filenames / preserve characters which were previously just dumped.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #37  
Old 28.06.2024, 01:45
Chicken++ Chicken++ is offline
Baby Loader
 
Join Date: Aug 2023
Posts: 9
Default

Quote:
Originally Posted by pspzockerscene View Post
No worries.
We've made some mistakes here but I'm confident that this change was a good one and overall helps to get nicer filenames / preserve characters which were previously just dumped.
Not arguing with that at all - just be more aware in the future that changing filenaming conventions can affect users more than you seemed to realize. ;)


I guess I could make a test-playlist with a few test videos on my end, with titles comprised of unusual characters and run a test-download every time I update the software, to see if anything changed. *shrug*


I just don't wanna spend a whole day (or more) setting up my download que (unfortunately there's some manual oversight/adjustments required for my use case, it's not as simple as "paste-and-forget"), hit "start downloads", and only then realize it's downloading files it should not. ;)




_______________________________________________________



For anybody passing by who needs to restore the old filename behavior for themselves, this is my final setup.
Tested and seems to work as far as I could tell, or at least got rid of most dupe-downloads.

------------------------
Filename Replace Map
Code:
{
 ":"    : "_",
 "\\|"  : "_",
 "<"    : "_",
 ">"    : "_",
 "/"    : "_",
 "\\\\" : "_",
 "\\*"  : "_",
 "\\?"  : "_",
 "\""   : "_"
}
------------------------
Packagename Replace Map
Code:
{
 "\\."        : " ",
 "[ \\.]{3,}" : "  ",
 ":"          : "_",
 "\\|"        : "_",
 "<"          : "_",
 ">"          : "_",
 "/"          : "_",
 "\\\\"       : "_",
 "\\*"        : "_",
 "\\?"        : "_",
 "\""         : "_"
}
------------------------
Clean up package = OFF
------------------------



Quick note on my "Packagename Replace Map" as it is slightly different from "Filename Replace Map":

"\\." : " ", // This does the job replacing "." with a blankspace that "Clean up package" normally would do. "Clean up package" however also replaces "_" with blankspaces, which would result in filenames different from the old behavior, so it needs to be turned off here.

"[ \\.]{3,}" : " ", // I've noticed a minor bug, "Clean up package" will sometimes leave double-blankspaces in package names, e.g. when the package name contains ". ", however it's already been like this for years, and this line replicates the behavior. None of my old downloads have a triple-blankspace in their package name, so I'd guess that did not occur.

Last edited by Chicken++; 05.07.2024 at 22:52. Reason: Correcting a mistake in the JSON map
Reply With Quote
  #38  
Old 28.06.2024, 09:59
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,985
Default

Quote:
Originally Posted by Chicken++ View Post
Not arguing with that at all
No worries this is not how I understood it either

Quote:
Originally Posted by Chicken++ View Post
just be more aware in the future that changing filenaming conventions can affect users more than you seemed to realize.
Agree. That learning is there.

Quote:
Originally Posted by Chicken++ View Post
I guess I could make a test-playlist with a few test videos on my end, with titles comprised of unusual characters and run a test-download every time I update the software, to see if anything changed. *shrug*
Tbh in this case, I'am still unsure what would have been the best solution.
Now afterwards we did discuss some possible solutions like:
- Only use new behavior in new JD installations as default, do not touch current installations
- Ask user via dialog with timeout, maybe provide an example of new/old behavior and if user does not respond to the dialog in time, use old behavior
- Only use new behavior for newly added items and then after some time, let's say 6 months, force new behavior for all items

...but that comes with the downside for us of needing to put more time into the GUI part and also this means, that e.g. dozens of older JD installations will behave differently than the current default one which we always need to consider when providing support for filename related user-inquiries.
Ultimately, JDownloader is not a super critical software like a banking or tax software (if it is for you, I'am taking this as a compliment :D) so we can't always leave things the way they are.
As you can see in this thread, the backlash we got is not too big although yes, we did also get some private support tickets about the filename changes.

Quote:
Originally Posted by Chicken++ View Post
I just don't wanna spend a whole day (or more) setting up my download que (unfortunately there's some manual oversight/adjustments required for my use case, it's not as simple as "paste-and-forget"), hit "start downloads", and only then realize it's downloading files it should not.
I understand.
I guess next time I will at least put instructions into the changelog on how to go back to the previous behavior whenever changes of default behavior will be done in the future.

Quote:
Originally Posted by Chicken++ View Post
"[ \\.]{3,}" : " ", // I've noticed a minor bug, "Clean up package" will sometimes leave double-blankspaces in package names, e.g. when the package name contains ". ", however it's already been like this for years
No one has reported this so far.
I've added a comment about this to the current cleanup code since it might be changed/refactored soon anyways.
We also had the idea of mimicing the current "old cleanup packagenames behavior" into the "Packagename Replace Map" but afaik we've given up on that and also this would mean we'd need to reset the default value of "Packagename Replace Map" for all users one more time which is not a good idea.

One last thing about the maps you've posted here:
They are missing some escaping so just copy & pasting your maps will not work as the syntax is broken.
I highly recommend putting them into our forums "CODE" tag e.g.:
Code:
{
  ":": "_",
  "\\|": "_",
  "<": "_",
  ">": "_",
  "/": "_",
  "": "_",
  "\\*": "_",
  "\\?": "_",
  "\"": "_"
}
--> Use a json editor like jsoneditoronline.org to verify your json before/after posting it anywhere.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #39  
Old 05.07.2024, 22:47
Chicken++ Chicken++ is offline
Baby Loader
 
Join Date: Aug 2023
Posts: 9
Thumbs up

Quote:
Originally Posted by pspzockerscene View Post
Ultimately, JDownloader is not a super critical software like a banking or tax software (if it is for you, I'am taking this as a compliment :D) so we can't always leave things the way they are.
Well, I do use it quite a lot! You may indeed consider yourself flattered! ;)



Quote:
Originally Posted by pspzockerscene View Post
Tbh in this case, I'am still unsure what would have been the best solution.
Now afterwards we did discuss some possible solutions like:
1) Only use new behavior in new JD installations as default, do not touch current installations
2) Ask user via dialog with timeout, maybe provide an example of new/old behavior and if user does not respond to the dialog in time, use old behavior
3) Only use new behavior for newly added items and then after some time, let's say 6 months, force new behavior for all items
Okay, here's my 2 cents on that:


Quote:
Originally Posted by pspzockerscene View Post
1) Only use new behavior in new JD installations as default, do not touch current installations

...but that comes with the downside for us [...] that e.g. dozens of older JD installations will behave differently than the current default one which we always need to consider when providing support for filename related user-inquiries.
Yeah, understandable. I suppose "synchronicity" as default makes sense.




Quote:
Originally Posted by pspzockerscene View Post
3) Only use new behavior for newly added items and then after some time, let's say 6 months, force new behavior for all items
Ah. Err. Hmm. If I understood that correctly (I dunno maybe I have not ?), that'd seem like it would be the most chaotic solution and create a lot of confusion for users, with inconsistent behavior between downloads that also then suddenly changed at some later point. ^_^;;


Quote:
Originally Posted by pspzockerscene View Post
2) Ask user via dialog with timeout, maybe provide an example of new/old behavior and if user does not respond to the dialog in time, use old behavior

[...]

...but that comes with the downside for us of needing to put more time into the GUI part
Well, I'd suggest that — for "big enough changes"* — to make a popup briefly explaining "A change was made to [insert here], this could affect [insert here], here's a link to the relevant changelog post", with two buttons "Read, don't show again" and "skip for now". Similar to what you did here:



No need to make the system switch between new and old behavior. If it turns out people are struggling with the update or need to know how to "revert" the changes for their installation, the forum post for that changelog can be edited and they can do so themselves as they see fit.

* I suppose it'll be up to you to decide what counts as a "big enough change" to qualify showing up as a popup after an update (i.e., is it something that only affects a certain — popular? — plugin, or all of them), it also would not do to get flooded with notifications every time an update was made.

I suppose if the user doesn't update for a while (*cough, cough* like me, I probably hadn't updated in half a year and thus was unaware of changes leading up to this one), and there are multiple such "big changes", the notifications should get appended in the popup? Like, show them with a scrollbar, maybe?





Quote:
Originally Posted by pspzockerscene View Post
One last thing about the maps you've posted here:
They are missing some escaping so just copy & pasting your maps will not work as the syntax is broken.
Huh. How odd. Those mistakes are not in my Jdownloader settings, I don't know how that happened when posting them here.

I'll correct the forum post.

Thanks for the tip! Wouldn't want to provide misleading information for people seeking assistance!

Last edited by Chicken++; 06.07.2024 at 18:39.
Reply With Quote
  #40  
Old 08.07.2024, 11:53
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 71,985
Default

Quote:
Originally Posted by Chicken++ View Post
Ah. Err. Hmm. If I understood that correctly (I dunno maybe I have not ?), that'd seem like it would be the most chaotic solution and create a lot of confusion for users, with inconsistent behavior between downloads that also then suddenly changed at some later point. ^_^;;
Can be. Again it's nothing that I would do anyways - it was just a collection of ideas.

Quote:
Originally Posted by Chicken++ View Post
Similar to what you did here
Bad example. This was just an additional hint because the login works differently for Google Drive.
While users benefit from this, we do also (= the hope of getting less tickets with questions related to the Google login).

Quote:
Originally Posted by Chicken++ View Post
I suppose if the user doesn't update for a while (*cough, cough* like me, I probably hadn't updated in half a year and thus was unaware of changes leading up to this one), and there are multiple such "big changes", the notifications should get appended in the popup? Like, show them with a scrollbar, maybe?
Well as you can already see, such popups or notifications about changes which then maybe provide the user with more settings will also make things more complicated.

Also a short notice about your *not updaing JD* behavior:
The more users do this, the more confusing the reports we get can be as we'd then never know which version the user has used.
Also, a software like JDownloader needs to be up2date because of the nature of it: Plugins need to be updated otherwise many of them will fail whenever a supported website is doing siginficant changes.

Quote:
Originally Posted by Chicken++ View Post
Thanks for the tip! Wouldn't want to provide misleading information for people seeking assistance!
No worries - your initial code you posted would have triggered a syntax error anyways so users would notice this immediately.


So, sticking to this topic:
My next planned filename related change is the auto shortening of too long filenames [under Windows systems]. It is already working in my dev build.
What is your personal opinion about this?
- Default enable
- Default disable
- You don't care because you are not a Windows user
- Notify user and ask


My personal opinion:
Implement, enable by default and don't notify the user.
Reasons for this:
So far problems with too long filenames have caused more trouble/confusion than what will happen when there is an auto shorten feature.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 00:59.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.