#41
|
|||
|
|||
Quote:
Thanks for checking! |
#42
|
|||
|
|||
Quote:
Quote:
I fully understand your objections and it's not my intention to clutter or slow down the plugin with too many options. And I have tried your packagizer-rule with limiting the filename length and it worked very well. However this is not very intelligent solution either imo as it simply cuts off the end of the filename when the limit is reached. For the the custom filename that I use... Code:
Flickr-*title*_*username_url*_*content_id**extension* For the default custom filename that you now suggest... Code:
*username_url*_*content_id*_*title**extension* It's not a huge problem and only occurs occasionally but it's an annoying piece of work always having to fix this manually. Maybe a packagizer-rule can be defined that does only limit the length of the title, but leaves the rest of the filename untouched? I am not to much a regular expressions expert, but I know you can do some magic with that. |
#43
|
|||||||
|
|||||||
Quote:
I haven't tried removing it as I was afraid of issues it could cause and I'm usually not running into the "too long path" issue. Quote:
Quote:
Does your filename maybe additionally contain unsupported characters? Quote:
I just think it doesn't make sense to create plugin settings for stuff that should be solved by a global feature otherwise you could e.g. add the "crop filename" feature hundrets of times for each plugin -> Makes no sense. Same for custom filenames because our Packagizer feature can be used to do pretty much the same you can do inside the flickr.com plugin at this moment but for some reason custom filenames have been added as a plugin setting back then which is why I kept it... users hate it when existing features get removed and I can fully understand this Quote:
In this case, all tags given in the flickr.com plugin settings are 1:1 the tags you can use in Packagizer. ... so let's pretend that the title is always the problem meaning that "username_url" and "content_id" will always be missing 100% if a filename gets cropped... so we'll just add this info to the new name: Code:
<jd:orgfilename:1>_<jd:prop:username_url>_<jd:prop:content_id>.<jd:orgfiletype> Quote:
To be honest I wouldn't know how to solve this other than adding a logic to cut off clean after specified tags and still allow different lengths. Quote:
- Packagizer cannot do String operations (e.g. modify strings, crop strings or do uppercase/lowercase conversions) - Apart from the defaults needed to create rules, Packagizer cannot do things like "if <jd:prop:username_url> contains 'xxxYYY' then do ..." You'd have to use an EventScripter script for that. You can get help creating such scripts in this thread. With such a script you'd definitely be able to optimize the naming scheme along with obeying your OS' limits up to "perfection". Using a script, you'd probably simply crop the title and then build your new filename. -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download Last edited by pspzockerscene; 17.09.2021 at 11:28. Reason: Added hyperlinks |
#44
|
||||
|
||||
About the *invalid download directory*, can you please provide example link and a log for this?
Most likely the filename contains characters that cause the issue. individual path segments are *as far as I know* still limited to 255 chars, same as in other OS filesystem. It looks like you can allow longer path segments, see https://support.jdownloader.org/Know...load-directory
__________________
JD-Dev & Server-Admin Last edited by Jiaz; 17.09.2021 at 11:59. |
#45
|
||||
|
||||
Changes for the next flickr plugin update:
- Implemented linkcheck via flickr API for single items --> Now even single added mature content items can be downloaded without account - Added fast linkcheck setting for videos and enabled it by default (=filesize won't be visible and 'quality' tag won't be available until download is started) - Disabled account login for flickr as I haven't found any reasons to keep it -> It's up to you to provide some good reasons to re-enable it Wartest du auf einen angekündigten Bugfix oder ein neues Feature? Updates werden nicht immer sofort bereitgestellt! Bitte lies unser Update FAQ! | Please read our Update FAQ! --- Are you waiting for recently announced changes to get released? Updates to not necessarily get released immediately! Bitte lies unser Update FAQ! | Please read our Update FAQ! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#46
|
||||
|
||||
Regarding the Windows path length limit:
I've now tesdted the method that is linked in our guide to disable that limit. Unfortunately that did not work for me either but I'm pretty sure that the limit can be disabled. I'll do some more research the next week. If you want you can also do a research on that. Have a nice weekend! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#47
|
|||
|
|||
Quote:
Quote:
From this stream that contains many items with very long titles: Code:
**External links are only visible to Support Staff** I have now tested this a little more with this link, using the default custom filename: Code:
**External links are only visible to Support Staff** Filename that would be complete and correct (it's too long individual filename): Code:
268 characters: viv_vivekananda_50039847567_Tourist helicopter flights over pristine waters of Great Barrier Reef - in the distance you can see the water colour change and waves which signifies the continental shelf giving way to very deep ocean, Off Cairns, Queensland, Australia.jpg Longest filename that JD still does allow before showing "Invalid download directory" error, I have renamed manually: Code:
250 characters: viv_vivekananda_50039847567_Tourist helicopter flights over pristine waters of Great Barrier Reef - in the distance you can see the water colour change and waves which signifies the continental shelf giving way to very deep ocean, Off Cairns, Que.jpg Afterwards you still can rename this file in Windows Explorer manually but not longer than to this filename when the file was put into the root folder (\): Code:
254 characters: viv_vivekananda_50039847567_Tourist helicopter flights over pristine waters of Great Barrier Reef - in the distance you can see the water colour change and waves which signifies the continental shelf giving way to very deep ocean, Off Cairns, Queensl.jpg JDownloader does allow to download this 250 character long filename into a deep folder structure, it's no problem, even if the path alone is longer than 260 characters: Code:
D:\this_is_a_very_long_foldername\this_is_a_very_long_foldername\this_is_a_very_long_foldername\this_is_a_very_long_foldername\this_is_a_very_long_foldername\this_is_a_very_long_foldername\this_is_a_very_long_foldername\this_is_a_very_long_foldername\this_is_a_very_long_foldername I have also tested the maximum foldername length and have found out that JD can handle 255 characters at a max for a single foldername. All this I did with "Enabled NTFS long paths". Now I have deactivated the NTFS long paths with the policy editor again, restarted, checked the option in the registry (LongPathsEnabled=0) and tried to do the same again with JDownloader, downloading to this overlong path with the filename 250 characters long. That also worked totally fine. So I don't yet see any difference with the long paths option enabled or not. As long as the filename isn't too long, JD can handle an overlong total path, so such are obviously not limited by the operating system. The only issue is that many other programs and even Windows Explorer do not handle files over the maximum path limit properly. But I also had programs that had not much problems with such a long path, XnView MP for example. XnView Classic could not handle it and crashed. So guess it depends on the program implementation. Conclusion, JD works all fine as long as filename lengths <=250 and single foldername lengths <= 255 characters. In this article you get some useful info regarding this issue: **External links are only visible to Support Staff****External links are only visible to Support Staff** Quote:
Still not sure what this "Enable NTFS long paths" option should actually do as I don't see any difference between having it activated or not. Last edited by EquiNox; 19.09.2021 at 15:34. |
#48
|
|||
|
|||
During my tests with these overlong paths I ran into the problem that it's not easy getting rid of these afterwards, almost any regular method to delete such folders failed and hardly any program would even recognize those correctly.
But there is a way and the article that I linked to above says that a workaround in the operating system is implemented dealing with such overlong paths. I did it with the WinRAR file manager, there was no problem to rename and delete these paths. When you want command line tools to work with such long paths you should add the prefix "\\?\" to the path. **External links are only visible to Support Staff****External links are only visible to Support Staff** Quote:
Last edited by EquiNox; 20.09.2021 at 13:49. |
#49
|
||||
|
||||
Quote:
According to your test results I guess it's easier to use Linux than to mess with this. As said if you want a nicer solution to cut filenames (or well ... titles to be more precise), try an EventScripter script. Flickr: As described in my recent posts, I've removed the login functionality for now as I was unable to find any reasons to keep/fix it. As for now, I will do some flickr code cleanup but that's it: For now I'm done with the flickr refactoring and I'm not planning on adding more features. -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#50
|
|||
|
|||
Quote:
Thanks for all your help and being patient with me! Quote:
About login functionality, I'll put the information you requested in here. The ability to download albums that are only accessible when you are logged in is one good reason to have this login functionality. And the biggest reason of course, parsing would be much faster. There are many albums at Flickr you can download for free, but you must be logged in and JD2 is the best tool I ever ran across so far to make such things so easy. It's all good, I am sure you've much other things to do. Thanks again for all your hard work! |
#51
|
|||
|
|||
Quote:
Quote:
And yes, there was a time you could login there with a non-yahoo.com address. But as far as I remember there had always been more restrictions regarding visibility and accessibility of content when you were not using yahoo.com address. It was enough to use yahoo.de and you could not see as much as you could with the com-address. At that time I thought it might even be some kind of geoblocking, that depended on the email address. Quote:
When you upload content at Flickr you can set a "safety level" for the uploaded item:
Besides that you can set the "viewing privacy" that configures who actually should be allowed to see your content:
When you are not logged in at Flickr you will only get to see content that is safe and publicly available. When you are logged in and not follow any user you can also see content that is public and set moderate or restricted. Restricted content is limited to registered users over the age of 18. When you have set the viewing privacy to private it's only you who can see the content. When you have set it to Friends and/or Family only Flickr users who follow you and who you added to your family respectively friends group will be allowed to see your content. Just following a member does not give you access to his or her f+f content. For all this you surely need an account. More info on this you can also get here: **External links are only visible to Support Staff****External links are only visible to Support Staff** And example where content is set to "moderate" and you can only view or download with an account, JD2 doesn't show anything when you parse this stream and it would be nice to be able to download the albums as they are shown on the album page: **External links are only visible to Support Staff****External links are only visible to Support Staff** |
#52
|
|||||||
|
|||||||
Quote:
https://support.jdownloader.org/Know...load-directory Quote:
https://board.jdownloader.org/showthread.php?t=70525 Quote:
Quote:
Quote:
Quote:
You can add mature content without account. Quote:
I will try to fix account support. -psp- EDIT Login support ticket:
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download Last edited by pspzockerscene; 21.09.2021 at 14:39. |
#53
|
|||
|
|||
Quote:
**External links are only visible to Support Staff****External links are only visible to Support Staff** Then you have access to all content that is publicly available. Quote:
Another account that only contains moderate or restricted content: **External links are only visible to Support Staff****External links are only visible to Support Staff** |
#54
|
|||||
|
|||||
Quote:
Quote:
Quote:
Quote:
- Albums ONLY containing mature content - Crawling "all albums of a user" when all of those contain 100% mature content - Crawling all favorites of a user when only some of the images are mature content (this works fine - see the example I've sent for testing the "missing images" issue) Quote:
This is also a quite inconsistent behavior of flickr but maybe their API was designed with this in mind: Why shouldn't external apps be able to access mature content if e.g. their users have to confirm their age at some other point than on the flickr website so an age confirmation via flickr account is not necessary anymore. Whatever it is, I guess all content will be crawlable once I've fixed account support. I'm not in the mood of looking for other tools that might be able to get around these limits via special API parameters plus account support will bring the benefit of e.g. being able to download your own private galleries. Again I'm absolutely not a flickr user/customer myself... -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#55
|
||||
|
||||
Next flickr plugin update:
- fixed login - enabled resume for all items: Indeed resuming photo downloads is possible but not for all items: JD cannot know this until it first checks/download the directURL of a picture so I'll just leave it enabled for all items now: After all, images are usually small so when stopping downloads you will only lose ~100MB at max depending on how many unresumable pictures you were downloading simultaneously: I consider this an edge-case. --> After the next update, JD will not warn you regarding resumability anymore when stopping photo downloads. Wartest du auf einen angekündigten Bugfix oder ein neues Feature? Updates werden nicht immer sofort bereitgestellt! Bitte lies unser Update FAQ! | Please read our Update FAQ! --- Are you waiting for recently announced changes to get released? Updates to not necessarily get released immediately! Bitte lies unser Update FAQ! | Please read our Update FAQ! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download Last edited by pspzockerscene; 22.09.2021 at 17:19. |
#56
|
|||
|
|||
Wow! The way was a bit tricky to install the account support for Flickr in JD (I used the incognito mode) but it finally works and even very well it seems! Thank you very much pspzockerscene! This now even doubles the quality of this plugin and makes so many tasks so much easier! Just fantastic!
I'll still keep an eye on how it works and will report if there are any other issues we yet haven't run into. Many thanks for your work on this, pspzockerscene!! |
#57
|
||||
|
||||
Thanks for your feedback!
Next update includes (this time I was lazy and simply copied my SVN commit message over): - fixed NPE in website mode - fixed missing directurl in website mode - added missing property "set_id" to settings properties description - added new property "gallery_id" - fixed missing set_id property for single added URLs containing that ID - fixed/improved contentURLs in JD so they match the browser/context ones e.g. for items from galleries: flickr.com/photos/<usernameURL>/<content_id>/in/gallery-<galleryUsernameInternal>-<gallery_id>/ -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#58
|
|||
|
|||
Thanks for fixing and the new additions as well!
I already have ideas how to use the new tags in certain situations. Thank you very much psp! Much appreciated! |
#59
|
||||
|
||||
Thanks for your feedback!
I've gone through the code of both plugins one last time and made some small changes. I've removed flickr from my TODO list for now. A few small observations/open points: - Are there videos with other file extensions than .mp4? If so, I need example URLs. - Are there photos with other file extensions than .jpg? If so, I need example URLs. --> For both cases I'd say yes it is possible when original file is downloadable. - If possible, we could set the "resumable" state to be correctly set for all items (once the download has been started) -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#60
|
|||
|
|||
Good you mentioned, I had seen we are not yet done. There are a few more issues that should be checked/worked on.
First of all a minor issue, the *date_taken* tag, very useful but could it also get the date format that is defined in "Define how dates inside filenames should look like"? That would be more flexible and useful in certain situations and more consistent to. Quote:
Code:
asx, avi, avif, bmp, dib, gif, ico, jfif, jpeg, jpg, jxl, mov, mp4, mpeg, mpg, ogm, pjp, pjpeg, png, svg, svgz, tif, tiff, webm,ogv, webp, wmv, xbm **External links are only visible to Support Staff****External links are only visible to Support Staff** This one is a good example as it shows what JD2 currently does, parsing correctly 504 items in this stream but assuming all of them to be JPG-files, although 242 of these are actually PNG files. Examples: **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** **External links are only visible to Support Staff****External links are only visible to Support Staff** JD2 can parse all these links but puts a JPG-extension at the end even when it's a PNG image. Image editing or viewing programs can handle this as they usually scan the header of the files to actually see what filetype it is but it would be better if the extension would be set to PNG, which actually would be correct. Checking these files with an image analysing tool returns something like this "Not a JPEG file: starts with 0x89 0x50". On my search for PNG files I ran across another stream that gave me a big riddle as it was not correctly parsed by JD although all the content should be available without an account. First of all the hq-png image from that stream: **External links are only visible to Support Staff****External links are only visible to Support Staff** Now when you parse this stream, JD returns 756 items although the stream actually has 1191 items. I looked through the pages and all had correctly 100 items and the last page 91, so there were no gaps. **External links are only visible to Support Staff****External links are only visible to Support Staff** When you now parse the albums then you'll get another 206 items, so there are 962 in total, but still far away from 1191. **External links are only visible to Support Staff****External links are only visible to Support Staff** I checked this a long time and thoroughly but I really could not find a reason why JD would not find all the items in this stream, maybe you can help clearing up this mystery, psp. Besides that I still more need to check if downloading all the video links are working, I think I had a few issues with very small res videos. I don't have an example at hand right now but it could be that those examples previously listed in this thread are also affected (Point 2 in this post: https://board.jdownloader.org/showpost.php?p=490685). The overall performance of the plugin is amazing compared to former times. Even the downloading is much faster I'd say. Thank you very much psp! |
Thread Tools | |
Display Modes | |
|
|