JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #41  
Old 17.09.2021, 07:09
EquiNox EquiNox is online now
Vacuum Cleaner
 
Join Date: Sep 2021
Posts: 16
Default

Quote:
Originally Posted by pspzockerscene View Post
I've tested this again and I was unable to resume stopped image files (only videos).
I guess it simply wasn#t set correctly before which is why you did not get the warning.
I'm pretty sure picture downloads would also start from zero in the older flickr plugin revisions.

@Jiaz
If you want go ahead and test again but afaik they do not even return the Content-Length for images.

-psp-
Could be like you said although I don't think that I have ever deactivated the warning in JD. And yes, maybe it's better to reset partial downloaded images for not getting corrupted ones in the end.

Thanks for checking!
Reply With Quote
  #42  
Old 17.09.2021, 08:24
EquiNox EquiNox is online now
Vacuum Cleaner
 
Join Date: Sep 2021
Posts: 16
Default

Quote:
Originally Posted by pspzockerscene View Post
Are you using Windows?
You can **External links are only visible to Support Staff**... (scroll down to "Hint for Windows users") but it may cause other trouble.

Apart from that you can use our **External links are only visible to Support Staff**... to limit filename sizes.
As it has been requested multiple times in the past, I've just added an **External links are only visible to Support Staff**... on how to do that.
Ok, thanks, I thought it was hard-wired in the file system although 260 characters total lenght seems to be ridiculous to me nowadays. I guess I should switch to Linux, that probably does not have such silly limitations. Yes, W10 Pro here, I used the Policy editor to change the option. However it's still not working correctly, JD still returns "Invalid download directory" for the very long file names.

Quote:
Originally Posted by pspzockerscene View Post
Please keep in mind that this example works only on the filename without checking the full path.
If you need it to act more intelligent you may need an **External links are only visible to Support Staff**....

Please understand that a plugin specific solution/setting just for flickr wouldn't be target-oriented as the "too long path" issue can happen with any source/file/path.

We do have tickets regarding this topic but it is more complex that you might think because:
- Which path to use if not only the filename but also the path needs to be shortened
- Shorten paths before- or after Packagizer actions?
- Always preserve file-type when shorting filenames: This implies that this is always correctly found and may fail for files without extension or with "unknown" extension

-psp-

I fully understand your objections and it's not my intention to clutter or slow down the plugin with too many options. And I have tried your packagizer-rule with limiting the filename length and it worked very well. However this is not very intelligent solution either imo as it simply cuts off the end of the filename when the limit is reached. For the the custom filename that I use...

Code:
Flickr-*title*_*username_url*_*content_id**extension*
...it would cut off the username and the content_id probably, so I don't anymore know where the file was from.

For the default custom filename that you now suggest...

Code:
*username_url*_*content_id*_*title**extension*
...it does work, however there is no guarantee that the title is always cut off at the same position, when for example I want to change username_internal to the new pathalias a user now has, then the length of the prefix before the title can change so the title will be cut off differently. This can be a problem when updating streams, you just parse all items again and just let those skip where the filename does already exist. When the lenght of the filename has now changed, I am downloading the same file again althought it's already there with a slightly different name. A constant max title length would fix this problem. Just wanted to elaborate what the problem for me is with this issue.

It's not a huge problem and only occurs occasionally but it's an annoying piece of work always having to fix this manually.

Maybe a packagizer-rule can be defined that does only limit the length of the title, but leaves the rest of the filename untouched? I am not to much a regular expressions expert, but I know you can do some magic with that.
Reply With Quote
  #43  
Old 17.09.2021, 10:27
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 58,827
Default

Quote:
Originally Posted by EquiNox View Post
Ok, thanks, I thought it was hard-wired in the file system although 260 characters total lenght seems to be ridiculous to me nowadays.
My guess is that the limit will be removed by default with the next major Windows version (maybe Win 11) though maybe it is still there for good reasons.
I haven't tried removing it as I was afraid of issues it could cause and I'm usually not running into the "too long path" issue.

Quote:
Originally Posted by EquiNox View Post
I guess I should switch to Linux, that probably does not have such silly limitations.
Exactly.

Quote:
Originally Posted by EquiNox View Post
However it's still not working correctly, JD still returns "Invalid download directory" for the very long file names.
Any examples for that?
Does your filename maybe additionally contain unsupported characters?

Quote:
Originally Posted by EquiNox View Post
I fully understand your objections and it's not my intention to clutter or slow down the plugin with too many options.
It's not for these reasons.
I just think it doesn't make sense to create plugin settings for stuff that should be solved by a global feature otherwise you could e.g. add the "crop filename" feature hundrets of times for each plugin -> Makes no sense.
Same for custom filenames because our Packagizer feature can be used to do pretty much the same you can do inside the flickr.com plugin at this moment but for some reason custom filenames have been added as a plugin setting back then which is why I kept it... users hate it when existing features get removed and I can fully understand this

Quote:
Originally Posted by EquiNox View Post
...it would cut off the username and the content_id probably, so I don't anymore know where the file was from.
As described in our Packagizer article, you can also use plugin specific tags inside filenames.
In this case, all tags given in the flickr.com plugin settings are 1:1 the tags you can use in Packagizer.
... so let's pretend that the title is always the problem meaning that "username_url" and "content_id" will always be missing 100% if a filename gets cropped... so we'll just add this info to the new name:
Code:
<jd:orgfilename:1>_<jd:prop:username_url>_<jd:prop:content_id>.<jd:orgfiletype>
Quote:
Originally Posted by EquiNox View Post
...it does work, however there is no guarantee that the title is always cut off at the same position, when for example I want to change username_internal to the new pathalias a user now has, then the length of the prefix before the title can change so the title will be cut off differently.
Sure the name will be cut off differently because it is a different name.
To be honest I wouldn't know how to solve this other than adding a logic to cut off clean after specified tags and still allow different lengths.

Quote:
Originally Posted by EquiNox View Post
Maybe a packagizer-rule can be defined that does only limit the length of the title, but leaves the rest of the filename untouched? I am not to much a regular expressions expert, but I know you can do some magic with that.
That's not possible using only Packagizer rules because:
- Packagizer cannot do String operations (e.g. modify strings, crop strings or do uppercase/lowercase conversions)
- Apart from the defaults needed to create rules, Packagizer cannot do things like "if <jd:prop:username_url> contains 'xxxYYY' then do ..."

You'd have to use an EventScripter script for that.
You can get help creating such scripts in this thread.
With such a script you'd definitely be able to optimize the naming scheme along with obeying your OS' limits up to "perfection".
Using a script, you'd probably simply crop the title and then build your new filename.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist

Last edited by pspzockerscene; 17.09.2021 at 10:28. Reason: Added hyperlinks
Reply With Quote
  #44  
Old 17.09.2021, 10:49
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 72,126
Default

About the *invalid download directory*, can you please provide example link and a log for this?
Most likely the filename contains characters that cause the issue. individual path segments are *as far as I know* still
limited to 255 chars, same as in other OS filesystem.
It looks like you can allow longer path segments, see https://support.jdownloader.org/Know...load-directory
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 17.09.2021 at 10:59.
Reply With Quote
  #45  
Old 17.09.2021, 14:06
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 58,827
Default

Changes for the next flickr plugin update:
- Implemented linkcheck via flickr API for single items --> Now even single added mature content items can be downloaded without account
- Added fast linkcheck setting for videos and enabled it by default (=filesize won't be visible and 'quality' tag won't be available until download is started)
- Disabled account login for flickr as I haven't found any reasons to keep it -> It's up to you to provide some good reasons to re-enable it

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #46  
Old 17.09.2021, 15:46
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 58,827
Default

Regarding the Windows path length limit:

I've now tesdted the method that is linked in our guide to disable that limit.
Unfortunately that did not work for me either but I'm pretty sure that the limit can be disabled.

I'll do some more research the next week.
If you want you can also do a research on that.

Have a nice weekend!

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #47  
Old 19.09.2021, 09:27
EquiNox EquiNox is online now
Vacuum Cleaner
 
Join Date: Sep 2021
Posts: 16
Default

Quote:
Originally Posted by Jiaz View Post
individual path segments are *as far as I know* still
limited to 255 chars, same as in other OS filesystem.
It looks like you can allow longer path segments, see **External links are only visible to Support Staff**...
Ah, ok, thanks for clarifying. That was probably the issue with the examples that I tried. Individual path segments does not mean the entire path+filename but just the single folder- or filename should not be longer than 255 chars. However the old "260 character maximum path limit" does apply to the entire path+filename as far as I understood.

Quote:
Originally Posted by Jiaz View Post
About the *invalid download directory*, can you please provide example link and a log for this?
The examples that I already used above, not a single special character in there.
From this stream that contains many items with very long titles:
Code:
**External links are only visible to Support Staff**

I have now tested this a little more with this link, using the default custom filename:
Code:
**External links are only visible to Support Staff**

Filename that would be complete and correct (it's too long individual filename):
Code:
268 characters: viv_vivekananda_50039847567_Tourist helicopter flights over pristine waters of Great Barrier Reef - in the distance you can see the water colour change and waves which signifies the continental shelf giving way to very deep ocean, Off Cairns, Queensland, Australia.jpg

Longest filename that JD still does allow before showing "Invalid download directory" error, I have renamed manually:
Code:
250 characters: viv_vivekananda_50039847567_Tourist helicopter flights over pristine waters of Great Barrier Reef - in the distance you can see the water colour change and waves which signifies the continental shelf giving way to very deep ocean, Off Cairns, Que.jpg

Afterwards you still can rename this file in Windows Explorer manually but not longer than to this filename when the file was put into the root folder (\):
Code:
254 characters: viv_vivekananda_50039847567_Tourist helicopter flights over pristine waters of Great Barrier Reef - in the distance you can see the water colour change and waves which signifies the continental shelf giving way to very deep ocean, Off Cairns, Queensl.jpg

JDownloader does allow to download this 250 character long filename into a deep folder structure, it's no problem, even if the path alone is longer than 255 characters:
Code:
D:\this_is_a_very_long_foldername\this_is_a_very_long_foldername\this_is_a_very_long_foldername\this_is_a_very_long_foldername\this_is_a_very_long_foldername\this_is_a_very_long_foldername\this_is_a_very_long_foldername\this_is_a_very_long_foldername\this_is_a_very_long_foldername
However you will be running into issues when you want to do something with this file (moving, renaming, viewing). I guess most programs cannot handle such overlong pathes and just return some kind of "file not found" error.

I have also tested the maximum foldername length and have found out that JD can handle 258 characters at a max for a single foldername.

All this I did with "Enabled NTFS long paths". Now I have deactivated the NTFS long paths with the policy editor again, restarted, checked the option in the registry (LongPathsEnabled=0) and tried to do the same again with JDownloader, downloading to this overlong path with the filename 250 characters long. That also worked totally fine. So I don't yet see any difference with the long paths option enabled or not. As long as the filename isn't too long, JD can handle an overlong total path, so such are obviously not limited by the operating system. The only issue is that many other programs and even Windows Explorer do not handle files over the maximum path limit properly. But I also had programs that had not much problems with such a long path, XnView MP for example. XnView Classic could not handle it and crashed. So guess it depends on the program implementation.

Conclusion, JD works all fine as long as filename lengths <=250 and single foldername lengths <= 258 characters.

In this article you get some useful info regarding this issue:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

Quote:
We’re a long way from the mid-1990s, however, and the whole Long Filename thing is (for the most part) firmly ironed out. If you’re running a version of Windows from the last 10 years, you’ve likely never even come across a filename length conflict like we we used to run into back in the DOS/Windows 95 days. That said, we still run into hiccups, as you discovered with your disk cleanup project. But why? If Windows’ Long Filename system supports folders and file names of up to 255 characters per component, what wall are you running into? We can’t blame NTFS (the filesystem that the vast majority of modern Windows machines use) as NTFS will support a chaining of folders and file names up to a total path length of 32,767 characters. That far exceeds the typical directory structure most users would ever need.

Where it all falls apart is an artificial restriction Windows stacks on top of the LFN/NTFS system: the MAX_PATH variable. The MAX_PATH variable specifies that a complete directory structure in Windows can’t exceed 260 total characters, including the drive letter, colon, backslash, and null backlash at the end. Thus you only have a potential real MAX_PATH of 256 characters, e.g. C:\your-256-character-path\.

So what happened when you were cleaning up your computer is that you had a directory with an already long path (either because the folder names were long, the file names were long, or both), and when you attempted to move one or more of those directories into another directory with a long path, the total length of the path name exceeded the 260 character limit imposed by the MAX_PATH variable.

Now, you may be thinking “Ah-hah! We’ll just change the MAX_PATH variable and solve the problem!” Alas, it’s not that simple. Not only is the MAX_PATH variable essentially hard coded into Windows, but even if you went through the enormous hassle of changing it, you’d end up breaking so much it wouldn’t be worth it. Too many applications expect the path variable to be what Windows has long specified it to be. We can’t just go around changing it without creating an enormous mess.
NTFS itself does not have such a limit, the total path length can be 32,767 characters long. The MAX_PATH variable seems to cause all this mess. Just too many apps would not anymore work correctly when exceeding the limit defined in this variable.

Still not sure what this "Enable NTFS long paths" option should actually do as I don't see any difference between having it activated or not.

Last edited by EquiNox; 19.09.2021 at 09:29.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 10:08.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.