[mmdbuta]

Hi everyone.

Before I asked, I just used JD, and I'm sorry that I'm using a translator, not an English speaker.

I recently used JD for the first time and touched the settings a little bit, and it was very satisfying.

But soon I had a problem that was hard to solve with my ability, so please let me know about this.

I've searched as much as I can on this forum and Google with my amazing poor English skills, but I still couldn't solve it.

I usually use JD to crawl iwara videos and I want to know how to rename some illegal characters as I wrote in the title.

I'm using the windows system.

for example
/ to ／
? to？
like this

I've seen some resolved threads about it, but I'm not sure what "Ticket" is, and the event scripter... let me just say first that I don't know how to code.

Actually, I tried some codes on copilot or other threads, but it didn't work out also.

I found "General Settings: Filename and Path Character Regex Replace Map" in Advanced setting, so I thought this is it! This didn't work either.

I'm sorry if I didn't follow the rules of this forum or the set questioning methods. To excuse myself, there are very few thread-style communities in my nation, so I'm not used to it.

All the links that seemed to have answers were dead... I tried to solve them myself without asking questions, but it wasn't easy.

I don't know if my words were conveyed well. I'm sorry for the rambling.

Anyway, I'm looking for a way to solve this problem. It might be better to learn how to write scripts soon.

Thank you for reading this long article.

I hope you all have a good day.

[Jiaz]

@mmdbuta: Can you please provide example links? JDownloader already should replace forbidden chars (depending on the plugin) and finally remove all forbidden chars.
Best is to provide example links, then we can check it

[mmdbuta]

@Jiaz: I forgot to suggest the links
Here are some examples
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Forbidden chars just replaced to "_" only
That makes me sad ;_;

[pspzockerscene]

This has already been done by me.
Next CORE-update (ETA unknown) will:
- Globally replace those chars with better ones
- Allow the user to define a custom regular expression based 'replace map' and also to change our default one
- And yes, same goes for package names
Default replaces (as of the date of writing this post):

{
 ":"    : ";",
 "\\|"  : "¦",
 "<"    : "[",
 ">"    : "]",
 "/"    : "\u2044",
 "\\\\" : "∖",
 "\\*"  : "#",
 "\\?"  : "¿",
 "\\!"  : "¡",
 "\""   : "'"
}

See the following advanced settings:

GeneralSettings.filenamereplacemap

and:

GeneralSettings.packagenamereplacemap

[mmdbuta]

Oh, does that mean I have to wait a little bit to use that feature in the official version?

[pspzockerscene]

I cannot provide any ETA.
The feature has been implemented roughly since august 2023 and is still waiting for release.
If you can't wait, another option is to use/test the dev version.

[mmdbuta]

Oh... august 2023...
Then i'll try dev version.
Thanks for your kindness.

[Jiaz]

@mmdbuta: We're working on releasing finally the changes soon.

[pspzockerscene]

Changes are live. Changelog:
https://board.jdownloader.org/showth...798#post531798
EDIT

See the following advanced settings:

GeneralSettings.filenamecharacterregexreplacemap

and:

GeneralSettings.packagenamecharacterregexreplacemap

[TGU]

Nice to see those regex's settable & fetchable via the settings, I was wondering about the "." and "_" characters being replaced with spaces also (thankfully the new settings description said this).

When are these regexes and cleanup actions applied?, I've noticed things like the subfolderbyplugin still containing "." so I'm assuming these 2 settings are only applied to the link name (filename) and package name exclusively?
So is there another setting for the cleanup of the SaveTo path?

[Jiaz]

Due to a bug those are currently not handled. Please wait for next update. see
https://board.jdownloader.org/showpo...8&postcount=19

[Jiaz]

@TGU: the update is live

[TGU]

By default I believe that these Unicode spaces should be replaced with a standard space (if you aren't already handling it), these can cause quite the headache (and even potentially be malicious?). There's also things like changing the direction of the text for Arabic and similar which I have seen being used in malicious attacks with user run executables, which can make a file.txt.exe look like exefile.txt

unicode spaces: "\u2009", "\u200A", "\u200B", "\u200C","\u200D", "\uFEFF", "\u0000-\u001f"
unicode quotes: "\u201C" = “, "\u201D" = ” (Replace with normal quotes?)

:thumbup: that could cover most of the issues I've had with these in the past

[Jiaz]

@TGU:
You're talking about, for example
medium.com/@sam.rothlisberger/embed-a-malicious-executable-in-a-normal-pdf-or-exe-81ee5339707e

I have not looked deeper into this if windows meanwhile better handles or protects for this situation.

I'm sorry but I don't think that we should add those replacements by default.
a.) those replacements do only affect names from plugins to replace forbidden characters (all OS). so no extraction/archives will be auto replaced
b.) it's job of the OS/scanner/defender to detect this and warn user about or block it.
c.) where do we start? where do we end? exe files, bat files? shell scripts? powershell scripts and many more...

I'm okay with creating a wiki article about this and some rules user can copy/paste into their setup, but I don't think it's something
a download tool should do. Would be interesting to know how browsers like Chrome/Firefox handle this? do they mangle/neutralize the filename or use as it is?
Same with other characters. This feature was introduced so ppl can customize the default replacements, not for us adding more default replacements and do stuff that didn't happen before/noone was asking for before.

You are free to create a collection of useful replacement rules and we're happy to include those in an article but for them to be included in default or not, not decided today.

[TGU]

@Jiaz Yup that's basically it, the "Right-To-Left Override (RTO or RTLO)" could be used maliciously but also has its own actual use with some foreign languages. The main issue with it, is that one could think it's not an exe (or any other extension .bat, .ps, .scr doesn't matter) and simply run it themselves. A little warning article about it for the user could be nice.

Browser's just typically download the files as-is within the HTTP Header Content-Disposition, normally filename*encoding_here="filename_here" if not the filename="filename_here"; They don't include any warnings about RTO/RTLO, and neither does the OS.

As for the spaces and quotes I think those should be replaced in general I've seen them used such few times, that when I do see them it can only just cause issues.

:thumbup: But this is some of the stuff I do in my download tools after +10.. 15 years of doing this, feel free to use it or not. I'm sure someone will read these posts and use the information in it. As I'm extracting these texts from websites and then using using them within the OS for paths you can basically refer them as a "Cross-Site Scripting" equivalent.

[Jiaz]

@TGU: Can you provide examples for the unicode spaces?
And maybe an example for RTO/RTLO? then maybe we can simple rule for exe/bat/ps1... files. Need to do some testing. If it can be done easily with pattern/regex, then on 2nd thought, why not have it in default. Won't hurt

[TGU]

@Jiaz this should help you **External links are only visible to Support Staff****External links are only visible to Support Staff** I had a hard time with the forum software changing the values.

All of these shouldn't be in a filename as-is except RTO/RTLO for some few languages, many of these are normally used within a text/document context.

Unicode Spaces (for use the \u within double quotes within code): "\u2009"=" ", "\u200A"=" ", "\u200B"="​", "\u200C"="*","\u200D"="*", "\uFEFF"="" (replace with space)

unicode quotes: "\u201C" ="“", "\u201D"="”" (replace with standard quote)

"\u0000-\u001f" is 31 of the control characters which should never be in the filenames, just remove them

The RTO/LTO is filtered by the forum software, but it's a truly evil character especially when they mix them together. This one should be replaced with an _ (some languages are read backwards which "need" it)
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**


(RTO = * = "\u202E")
filename: haha_got_you.*gmi.exe
what the user sees: haha_got_you.*exe.img

[pspzockerscene]

I'm against doing this by default.
If you start with this, you will find way more security related things such as "should executable files also be launched when a user chooses to 'open file' in JDownloaders GUI" or also "if a multihost supports hosts like 'drive.google.com', it may obtain users private links so maybe that should be turned off by default".

Also if the user is downloading some kind of known/popular malicous file, the users' basic firewall should detect it way before he is even able to open it (that said, if it is a targeted attack that won't happen).

In the end, the user needs to trust his source of data.

[Jiaz]

On windows you can block execution of files if they contain such chars, see
superuser.com/questions/408792/what-are-ways-to-prevent-files-with-the-right-to-left-override-rlo-unicode-cha

[goratnik]

JDownloader has started doing seemingly nonsensical character replacements and reading recent threads has brought me here. I like the idea of the map, I'm going to update mine for files in a moment, there is one logical problem with the packagename-side feature though: it depends on the flag "Clean Up Packagenames", which is still doing its own with modification of "." and "_", which is on the other hand unwanted by me and I had it disabled before. So in order to use a feature that keeps packagenames consistent with filenames, I have to turn on another feature that BREAKS names.

Please do either of these:
* make the "Packagename Replace Map" independent from "Clean Up Packagenames"; disabling the feature would be done by setting the map to empty, just like the map for filenames
* make "Clean Up Packagenames" feature wholly dependent on "Packagename Replace Map", meaning that only matches defined in the map are modified, including explicit definitions for "_" or "." as well; this would include updating the description to indicate that no actions other than defined in that other setting will be taken

Cheers!

[pspzockerscene]

Originally Posted by goratnik

there is one logical problem with the packagename-side feature though: it depends on the flag "Clean Up Packagenames"

That is not correct.
The replace map is always executed while the "old cleanup" is happening after the map replace and that part depends on the said setting.
We didn't remove it because we first wanted to find out the demand of it.
TL;DR If you turn off "GeneralSettings.cleanuppackagenames", only the map will be used.

If you do not like some of the map replacements, modify the map accordingly. The replaces are vital as some of those characters are not allowed in file paths under some OS and in a lot of cases, package names end up being a part of the file path [if the names were only used in JD GUI, the replacements wouldn't be needed].

Originally Posted by goratnik

JDownloader has started doing seemingly nonsensical character replacements and reading recent threads has brought me here.

Can you narrow this down for me?

My point of view:
Replacing a bunch of characters just by "_" was never a good idea but this is what you guys were used to.
The new method is better. If you do not like if, feel free to just change the map to replace all of those characters to "_".

[goratnik]

Originally Posted by pspzockerscene

That is not correct.
The replace map is always executed while the "old cleanup" is happening after the map replace and that part depends on the said setting.
We didn't remove it because we first wanted to find out the demand of it.
TL;DR If you turn off "GeneralSettings.cleanuppackagenames", only the map will be used.

Thank you, I just confirmed that map is still used with that other option off; I was misguided by description of the map saying "You can change this map to your needs and you can disable this by turning off the setting 'Clean Up Package Names'"; please fix the description and that would be it.

Originally Posted by pspzockerscene

If you do not like some of the map replacements, modify the map accordingly. (...) Can you narrow this down for me?

This applied mostly to replacement of "!", which is common in names and apart from corner cases of bash history expansion - rather unlikely to cause problems, I think; I have simply updated the map removing that one mapping and it's really OK with how it's customizable.

[pspzockerscene]

Originally Posted by goratnik

I was misguided by description of the map saying "You can change this map to your needs and you can disable this by turning off the setting 'Clean Up Package Names'"; please fix the description and that would be it.

It will be fixed with the next set of CORE-updates.
Thanks for reporting!

When the description was added, the goal was to have it this way, in the end plans were changed the wrong description was just forgotten.

Originally Posted by goratnik

This applied mostly to replacement of "!", which is common in names and apart from corner cases of bash history expansion - rather unlikely to cause problems, I think; I just updated the map removing that one mapping and it's really OK with how it's customizable.

Well the items which need to be replaced depend on the OS of the user but we didn't want to over-engineer this feature so we decided to try to go the "one for all" way.
Especially under Linux systems afaik you should be able to use anything as filename, even pipe symbols (but not using JDownloader).

[goratnik]

A correction: even with an explicit line

 "!"    : "!",

"!" is still replaced by "¡", which is unwanted by me and is not forced on by the filesystem anyway, so the map can apparently be ignored and the replacements still applied; can you please accept "noop" and further replace with underscores only if the character would be actually illegal, as in, filesystem would return an error on file creation attempt?

While I have nothing against the default, I would prefer to be able to keep some of these characters.

[pspzockerscene]

You need to replace it with something else - see [future] description of the setting:

Returns mapping of common invalid characters as regular expression to be replaced inside filenames. This is used to try to avoid invalid download paths due to forbidden characters in filenames. Any characters which are invalid for file paths and remain after execution of these replacements will be removed by a generic replacement handling with underscores.

Looking at the history, I think we didn't use to replace "!" before said update.
Maybe we should remove it from the default map since it is allowed under both windows and Linux.

[goratnik]

Voting for removal of "!", it has a low risk factor and fixing it manually is a hassle with how frequent it is.

[pspzockerscene]

Your got my vote.

Just for reference - the following is the old slash fallback-statement which is removing illegal characters after all of the customizable replacements:

    public static String alleviatePathParts(String pathPart, boolean removeLeadingHidingDot) {
        if (StringUtils.isEmpty(pathPart)) {
            if (pathPart != null) {
                return pathPart;
            }
            return null;
        }
        pathPart = pathPart.trim();
        /* remove invalid chars */
        /**
         * Integer value zero, sometimes referred to as the ASCII NUL character.
         *
         * Characters whose integer representations are in the range from 1 through 31->\\x00-\\x1f
         *
         * < (less than), * > (greater than), : (colon), " (double quote), / (forward slash), \ (backslash), | (vertical bar or pipe), ?
         * (question* mark) (asterisk)
         *
         * Volume designators (drive letters) are similarly case-insensitive. For example, "D:\" and "d:\" refer to the same volume.
         */
        pathPart = pathPart.replaceAll("([\\\\|<|>|\\||\r|\n|\t|\"|:|\\*|\\?|/|\\x00-\\x1f])+", "_");
        if (CrossSystem.isWindows() || CrossSystem.isOS2()) {
            /**
             * **External links are only visible to Support Staff**
             */
            if (CrossSystem.isForbiddenFilename(pathPart)) {
                pathPart = "_" + pathPart;
            }
        }
        /*
         * replace starting dots by single dot (prevents directory traversal)
         */
        if (removeLeadingHidingDot) {
            pathPart = pathPart.replaceFirst("^\\.+", "");
        } else {
            pathPart = pathPart.replaceFirst("^\\.+", ".");
        }
        /*
         * remove ending dots, not allowed under windows and others os maybe too
         *
         * Do not end a file or directory name with a space or a period.
         */
        pathPart = pathPart.replaceFirst("\\.+$", "");
        pathPart = pathPart.trim();
        if (StringUtils.isEmpty(pathPart)) {
            return "_";
        } else {
            return pathPart;
        }
    }

[pspzockerscene]

Next CORE-update will reset the map once and the new default one will not contain "!" anymore.

Bitte auf das nächste CORE-Update warten!

Please wait for the next CORE-Update!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[pspzockerscene]

Update has been released.
Changelog:
https://board.jdownloader.org/showth...798#post531798

[TGU]

Originally Posted by Jiaz

On windows you can block execution of files if they contain such chars, see
superuser.com/questions/408792/what-are-ways-to-prevent-files-with-the-right-to-left-override-rlo-unicode-cha

Ohh
I have to turn that on right away, you never know when it's going to get you. I've only seen it a few times, but anything with an "exe" in it's name is always a :red_flag: to me (but sometime they use the other executable extensions...)

:thumbup: removal of ! -> ¡

[Chicken++]

Originally Posted by pspzockerscene

My point of view:
Replacing a bunch of characters just by "_" was never a good idea but this is what you guys were used to.
The new method is better. If you do not like if, feel free to just change the map to replace all of those characters to "_".

Understandable, but I'd suggest that next time you change the default settings for how files are saved, you should make a popup message appear after the update clarifying this (and how to disable it should the user desire such).

Because while I agree that it does make more logical sense on its own, you need to consider that some people want to update their existing libraries of channel-downloads and are reliant on "Skip if file already exists" --- I only just updated recently and suddenly jDL was downloading 1000+ duplicate files because their file names were all different to the ones I already have, and only realized why that was happening after 3 days of downloading 50 GB I didn't want.


So, I need to restore the old file naming scheme. Oddly enough, simply setting "Filename Replace Map" and "Packagename Replace Map" to "{}" has no effect, it still does the same replacements as before.

I guess I have to replace all those characters with "_" explicitly ? Weird.

I mean, that's what I have to do to "get the old naming scheme back", right?

Also, I'm guessing "Clean up package" should be "off", then?

[raztoki]

@chicken++

I believe there was no setting before (or at least none that was used), file names changes was hard coded. And with package names were (and most likely still) stripped select chars to have cleaner names allowing for easy assignment to a single package '.' '_' '-' for instance replaced with " ". I assume the hardcoding has been removed for settings which could be changed by the end user, so reverting would be not possible without roll back (don't think is possible with the update network) or a copy of an old installation.

Some plugins used to perform internal cleanups also, but that was for other reasons (not illegal chars).

Only way to prevent dupes is to leave them within your download list (done based on url or uid stored). But this does have negative side effects, as memory requirement grows the larger the list becomes. The project really needs to work on an external dupe db in SQL or similar (as addon/extension). This would prevent these sort of issues moving forward. You mentioned channel name I assume youtube? For youtube the uid is most all you would need to store and would have prevented this outcome.

raztoki

[Chicken++]

Originally Posted by raztoki

@chicken++
so reverting would be not possible without roll back

I'm not asking for a change in the code, all I want to do is to change my settings such that they emulate the prior behavior. (I don't want to use an outdated version, obviously!) And I was asking for someone to "spell check" my settings.

Originally Posted by raztoki

Only way to prevent dupes is to leave them within your download list

Ooooof, sorry, but that really really would not work for me for... a number of reasons.

In short: I already have an organized system, this would throw a massive wrench into all of that. E.g., some duplicates are allowed because otherwise it'd be more chaotic. (Someone's playlist can contain videos from channels you also want to download.)

Originally Posted by raztoki

@chicken++
I believe there was no setting before (or at least none that was used), file names changes was hard coded.

Yeah, but I guess until a few months ago I guess it still replaced all of them with "_" instead of unique variations. So changing the replacement maps for files and packages to this:

{
 ":"    : "_",
 "\\|"  : "_",
 "<"    : "_",
 ">"    : "_",
 "/"    : "_",
 "\\\\" : "_",
 "\\*"  : "_",
 "\\?"  : "_",
 "\""   : "_"
}

...should restore the old naming scheme, as long as I also disable "Clean up package" which would turn "_" into a blankspace otherwise. [OBSOLETE, SEE FURTHER DOWN BELOW]

Originally Posted by raztoki

And with package names were (and most likely still) stripped select chars to have cleaner names allowing for easy assignment to a single package '.' '_' '-' for instance replaced with " ".

This one I might need some advice on how to restore the previous behavior.

First of, unlike you said, "-" isn't replaced regardless of "Clean up package" being ON or OFF, so I don't think I need to do anything about that. (why would it be removed, it's allowed by Windows and should cause no problems)

"." is where I'm unsure what the previous behavior was. I've got 2 old folders with a "." in them (I think they might be subfolders in a Google Drive / mega.nz download?) and I have no idea how that was handled before. Now, if I turn "Clean up package" ON, "." disappears. If I turn it off, it stays*. But what was the previous text replacement? An underscore? A blankspace? An empty space?

* For completeness' sake: "It stays", unless the dot is at the end of of a folder name, which Windows won't allow as this would cause issues. This removal happens independent of "Clean up package" ON/OFF. Perfectly sensible.


Are there any other characters affected by "Clean up package" I should be aware of?



...I suppose my attempt at "restoring the previous behavior" doesn't need to be 100% perfect, as long as it prevents the majority of duplicates at least. But some tips would be nice.

[pspzockerscene]

Originally Posted by Chicken++

Understandable, but I'd suggest that next time you change the default settings for how files are saved, you should make a popup message appear after the update clarifying this (and how to disable it should the user desire such).

No, not planned.

Originally Posted by Chicken++

Because while I agree that it does make more logical sense on its own, you need to consider that some people want to update their existing libraries of channel-downloads and are reliant on "Skip if file already exists" --- I only just updated recently and suddenly jDL was downloading 1000+ duplicate files because their file names were all different to the ones I already have, and only realized why that was happening after 3 days of downloading 50 GB I didn't want.

I understand your troubles but you can easily modify the replace-map to restore the previous behavior.

Originally Posted by Chicken++

I guess I have to replace all those characters with "_" explicitly ? Weird.

Yes and not weird:
1. Yes, change the map so that all characters get replaced to "_".
2. Not weird because this is exactly how it was before, see:
https://board.jdownloader.org/showpo...8&postcount=27

@raztoki
User can easily restore the previous behavior by changing the map.

Originally Posted by Chicken++

...should restore the old naming scheme

Yes.

Originally Posted by Chicken++

Are there any other characters affected by "Clean up package" I should be aware of?

JDownloader is open source.
The following code gets executed whenever "cleanuppackagenames" is enabled:

    public static String cleanPackagename(final String packageName, boolean allowCleanup) {
        String ret = replaceCharactersByMap(packageName, PACKAGENAME_REPLACEMAP);
        /* if enabled, replace dots and _ with spaces and do further clean ups */
        if (allowCleanup && org.jdownloader.settings.staticreferences.CFG_GENERAL.CLEAN_UP_PACKAGENAMES.isEnabled()) {
            // still wanted by users, so please do not simply remove this
            // TODO: maybe add own cleanup replace map that does this via pattern/replace or add those to package name replace map
            final StringBuilder sb = new StringBuilder();
            char[] cs = ret.toCharArray();
            char lastChar = 'a';
            for (int i = 0; i < cs.length; i++) {
                switch (cs[i]) {
                case '_':
                case '.':
                    if (lastChar != ' ') {
                        sb.append(' ');
                    }
                    lastChar = ' ';
                    break;
                default:
                    lastChar = cs[i];
                    sb.append(cs[i]);
                }
            }
            ret = sb.toString();
        }
        return ret.trim();
    }

[raztoki]

@psp
yah, My answer was more in respects to a quick change within the app or reverting version. I went looking for the diff but ran out of time on the weekend, bit hard to give exact advice to the changes without knowing what they were.

[pspzockerscene]

No worries.
We've made some mistakes here but I'm confident that this change was a good one and overall helps to get nicer filenames / preserve characters which were previously just dumped.

[Chicken++]

Originally Posted by pspzockerscene

No worries.
We've made some mistakes here but I'm confident that this change was a good one and overall helps to get nicer filenames / preserve characters which were previously just dumped.

Not arguing with that at all - just be more aware in the future that changing filenaming conventions can affect users more than you seemed to realize. ;)


I guess I could make a test-playlist with a few test videos on my end, with titles comprised of unusual characters and run a test-download every time I update the software, to see if anything changed. *shrug*


I just don't wanna spend a whole day (or more) setting up my download que (unfortunately there's some manual oversight/adjustments required for my use case, it's not as simple as "paste-and-forget"), hit "start downloads", and only then realize it's downloading files it should not. ;)




_______________________________________________________



For anybody passing by who needs to restore the old filename behavior for themselves, this is my final setup.
Tested and seems to work as far as I could tell, or at least got rid of most dupe-downloads.

------------------------
Filename Replace Map

{
 ":"    : "_",
 "\\|"  : "_",
 "<"    : "_",
 ">"    : "_",
 "/"    : "_",
 "\\\\" : "_",
 "\\*"  : "_",
 "\\?"  : "_",
 "\""   : "_"
}

------------------------
Packagename Replace Map

{
 "\\."        : " ",
 "[ \\.]{3,}" : "  ",
 ":"          : "_",
 "\\|"        : "_",
 "<"          : "_",
 ">"          : "_",
 "/"          : "_",
 "\\\\"       : "_",
 "\\*"        : "_",
 "\\?"        : "_",
 "\""         : "_"
}

------------------------
Clean up package = OFF
------------------------



Quick note on my "Packagename Replace Map" as it is slightly different from "Filename Replace Map":

"\\." : " ", // This does the job replacing "." with a blankspace that "Clean up package" normally would do. "Clean up package" however also replaces "_" with blankspaces, which would result in filenames different from the old behavior, so it needs to be turned off here.

"[ \\.]{3,}" : " ", // I've noticed a minor bug, "Clean up package" will sometimes leave double-blankspaces in package names, e.g. when the package name contains ". ", however it's already been like this for years, and this line replicates the behavior. None of my old downloads have a triple-blankspace in their package name, so I'd guess that did not occur.

[pspzockerscene]

Originally Posted by Chicken++

Not arguing with that at all

No worries this is not how I understood it either

Originally Posted by Chicken++

just be more aware in the future that changing filenaming conventions can affect users more than you seemed to realize.

Agree. That learning is there.

Originally Posted by Chicken++

I guess I could make a test-playlist with a few test videos on my end, with titles comprised of unusual characters and run a test-download every time I update the software, to see if anything changed. *shrug*

Tbh in this case, I'am still unsure what would have been the best solution.
Now afterwards we did discuss some possible solutions like:
- Only use new behavior in new JD installations as default, do not touch current installations
- Ask user via dialog with timeout, maybe provide an example of new/old behavior and if user does not respond to the dialog in time, use old behavior
- Only use new behavior for newly added items and then after some time, let's say 6 months, force new behavior for all items

...but that comes with the downside for us of needing to put more time into the GUI part and also this means, that e.g. dozens of older JD installations will behave differently than the current default one which we always need to consider when providing support for filename related user-inquiries.
Ultimately, JDownloader is not a super critical software like a banking or tax software (if it is for you, I'am taking this as a compliment :D) so we can't always leave things the way they are.
As you can see in this thread, the backlash we got is not too big although yes, we did also get some private support tickets about the filename changes.

Originally Posted by Chicken++

I just don't wanna spend a whole day (or more) setting up my download que (unfortunately there's some manual oversight/adjustments required for my use case, it's not as simple as "paste-and-forget"), hit "start downloads", and only then realize it's downloading files it should not.

I understand.
I guess next time I will at least put instructions into the changelog on how to go back to the previous behavior whenever changes of default behavior will be done in the future.

Originally Posted by Chicken++

"[ \\.]{3,}" : " ", // I've noticed a minor bug, "Clean up package" will sometimes leave double-blankspaces in package names, e.g. when the package name contains ". ", however it's already been like this for years

No one has reported this so far.
I've added a comment about this to the current cleanup code since it might be changed/refactored soon anyways.
We also had the idea of mimicing the current "old cleanup packagenames behavior" into the "Packagename Replace Map" but afaik we've given up on that and also this would mean we'd need to reset the default value of "Packagename Replace Map" for all users one more time which is not a good idea.

One last thing about the maps you've posted here:
They are missing some escaping so just copy & pasting your maps will not work as the syntax is broken.
I highly recommend putting them into our forums "CODE" tag e.g.:

{
  ":": "_",
  "\\|": "_",
  "<": "_",
  ">": "_",
  "/": "_",
  "": "_",
  "\\*": "_",
  "\\?": "_",
  "\"": "_"
}

--> Use a json editor like jsoneditoronline.org to verify your json before/after posting it anywhere.

[Chicken++]

Originally Posted by pspzockerscene

Ultimately, JDownloader is not a super critical software like a banking or tax software (if it is for you, I'am taking this as a compliment :D) so we can't always leave things the way they are.

Well, I do use it quite a lot! You may indeed consider yourself flattered! ;)

Originally Posted by pspzockerscene

Tbh in this case, I'am still unsure what would have been the best solution.
Now afterwards we did discuss some possible solutions like:
1) Only use new behavior in new JD installations as default, do not touch current installations
2) Ask user via dialog with timeout, maybe provide an example of new/old behavior and if user does not respond to the dialog in time, use old behavior
3) Only use new behavior for newly added items and then after some time, let's say 6 months, force new behavior for all items

Okay, here's my 2 cents on that:

Originally Posted by pspzockerscene

1) Only use new behavior in new JD installations as default, do not touch current installations

...but that comes with the downside for us [...] that e.g. dozens of older JD installations will behave differently than the current default one which we always need to consider when providing support for filename related user-inquiries.

Yeah, understandable. I suppose "synchronicity" as default makes sense.

Originally Posted by pspzockerscene

3) Only use new behavior for newly added items and then after some time, let's say 6 months, force new behavior for all items

Ah. Err. Hmm. If I understood that correctly (I dunno maybe I have not ?), that'd seem like it would be the most chaotic solution and create a lot of confusion for users, with inconsistent behavior between downloads that also then suddenly changed at some later point. ^_^;;

Originally Posted by pspzockerscene

2) Ask user via dialog with timeout, maybe provide an example of new/old behavior and if user does not respond to the dialog in time, use old behavior

[...]

...but that comes with the downside for us of needing to put more time into the GUI part

Well, I'd suggest that — for "big enough changes"* — to make a popup briefly explaining "A change was made to [insert here], this could affect [insert here], here's a link to the relevant changelog post", with two buttons "Read, don't show again" and "skip for now". Similar to what you did here:



No need to make the system switch between new and old behavior. If it turns out people are struggling with the update or need to know how to "revert" the changes for their installation, the forum post for that changelog can be edited and they can do so themselves as they see fit.

* I suppose it'll be up to you to decide what counts as a "big enough change" to qualify showing up as a popup after an update (i.e., is it something that only affects a certain — popular? — plugin, or all of them), it also would not do to get flooded with notifications every time an update was made.

I suppose if the user doesn't update for a while (*cough, cough* like me, I probably hadn't updated in half a year and thus was unaware of changes leading up to this one), and there are multiple such "big changes", the notifications should get appended in the popup? Like, show them with a scrollbar, maybe?

Originally Posted by pspzockerscene

One last thing about the maps you've posted here:
They are missing some escaping so just copy & pasting your maps will not work as the syntax is broken.

Huh. How odd. Those mistakes are not in my Jdownloader settings, I don't know how that happened when posting them here.

I'll correct the forum post.

Thanks for the tip! Wouldn't want to provide misleading information for people seeking assistance!

[pspzockerscene]

Originally Posted by Chicken++

Ah. Err. Hmm. If I understood that correctly (I dunno maybe I have not ?), that'd seem like it would be the most chaotic solution and create a lot of confusion for users, with inconsistent behavior between downloads that also then suddenly changed at some later point. ^_^;;

Can be. Again it's nothing that I would do anyways - it was just a collection of ideas.

Originally Posted by Chicken++

Similar to what you did here

Bad example. This was just an additional hint because the login works differently for Google Drive.
While users benefit from this, we do also (= the hope of getting less tickets with questions related to the Google login).

Originally Posted by Chicken++

I suppose if the user doesn't update for a while (*cough, cough* like me, I probably hadn't updated in half a year and thus was unaware of changes leading up to this one), and there are multiple such "big changes", the notifications should get appended in the popup? Like, show them with a scrollbar, maybe?

Well as you can already see, such popups or notifications about changes which then maybe provide the user with more settings will also make things more complicated.

Also a short notice about your *not updaing JD* behavior:
The more users do this, the more confusing the reports we get can be as we'd then never know which version the user has used.
Also, a software like JDownloader needs to be up2date because of the nature of it: Plugins need to be updated otherwise many of them will fail whenever a supported website is doing siginficant changes.

Originally Posted by Chicken++

Thanks for the tip! Wouldn't want to provide misleading information for people seeking assistance!

No worries - your initial code you posted would have triggered a syntax error anyways so users would notice this immediately.


So, sticking to this topic:
My next planned filename related change is the auto shortening of too long filenames [under Windows systems]. It is already working in my dev build.
What is your personal opinion about this?
- Default enable
- Default disable
- You don't care because you are not a Windows user
- Notify user and ask


My personal opinion:
Implement, enable by default and don't notify the user.
Reasons for this:
So far problems with too long filenames have caused more trouble/confusion than what will happen when there is an auto shorten feature.