JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #41  
Old 17.09.2021, 08:09
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

Quote:
Originally Posted by pspzockerscene View Post
I've tested this again and I was unable to resume stopped image files (only videos).
I guess it simply wasn#t set correctly before which is why you did not get the warning.
I'm pretty sure picture downloads would also start from zero in the older flickr plugin revisions.

@Jiaz
If you want go ahead and test again but afaik they do not even return the Content-Length for images.

-psp-
Could be like you said although I don't think that I have ever deactivated the warning in JD. And yes, maybe it's better to reset partial downloaded images for not getting corrupted ones in the end.

Thanks for checking!
Reply With Quote
  #42  
Old 17.09.2021, 09:24
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

Quote:
Originally Posted by pspzockerscene View Post
Are you using Windows?
You can **External links are only visible to Support Staff**... (scroll down to "Hint for Windows users") but it may cause other trouble.

Apart from that you can use our **External links are only visible to Support Staff**... to limit filename sizes.
As it has been requested multiple times in the past, I've just added an **External links are only visible to Support Staff**... on how to do that.
Ok, thanks, I thought it was hard-wired in the file system although 260 characters total lenght seems to be ridiculous to me nowadays. I guess I should switch to Linux, that probably does not have such silly limitations. Yes, W10 Pro here, I used the Policy editor to change the option. However it's still not working correctly, JD still returns "Invalid download directory" for the very long file names.

Quote:
Originally Posted by pspzockerscene View Post
Please keep in mind that this example works only on the filename without checking the full path.
If you need it to act more intelligent you may need an **External links are only visible to Support Staff**....

Please understand that a plugin specific solution/setting just for flickr wouldn't be target-oriented as the "too long path" issue can happen with any source/file/path.

We do have tickets regarding this topic but it is more complex that you might think because:
- Which path to use if not only the filename but also the path needs to be shortened
- Shorten paths before- or after Packagizer actions?
- Always preserve file-type when shorting filenames: This implies that this is always correctly found and may fail for files without extension or with "unknown" extension

-psp-

I fully understand your objections and it's not my intention to clutter or slow down the plugin with too many options. And I have tried your packagizer-rule with limiting the filename length and it worked very well. However this is not very intelligent solution either imo as it simply cuts off the end of the filename when the limit is reached. For the the custom filename that I use...

Code:
Flickr-*title*_*username_url*_*content_id**extension*
...it would cut off the username and the content_id probably, so I don't anymore know where the file was from.

For the default custom filename that you now suggest...

Code:
*username_url*_*content_id*_*title**extension*
...it does work, however there is no guarantee that the title is always cut off at the same position, when for example I want to change username_internal to the new pathalias a user now has, then the length of the prefix before the title can change so the title will be cut off differently. This can be a problem when updating streams, you just parse all items again and just let those skip where the filename does already exist. When the lenght of the filename has now changed, I am downloading the same file again althought it's already there with a slightly different name. A constant max title length would fix this problem. Just wanted to elaborate what the problem for me is with this issue.

It's not a huge problem and only occurs occasionally but it's an annoying piece of work always having to fix this manually.

Maybe a packagizer-rule can be defined that does only limit the length of the title, but leaves the rest of the filename untouched? I am not to much a regular expressions expert, but I know you can do some magic with that.
Reply With Quote
  #43  
Old 17.09.2021, 11:27
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,920
Default

Quote:
Originally Posted by EquiNox View Post
Ok, thanks, I thought it was hard-wired in the file system although 260 characters total lenght seems to be ridiculous to me nowadays.
My guess is that the limit will be removed by default with the next major Windows version (maybe Win 11) though maybe it is still there for good reasons.
I haven't tried removing it as I was afraid of issues it could cause and I'm usually not running into the "too long path" issue.

Quote:
Originally Posted by EquiNox View Post
I guess I should switch to Linux, that probably does not have such silly limitations.
Exactly.

Quote:
Originally Posted by EquiNox View Post
However it's still not working correctly, JD still returns "Invalid download directory" for the very long file names.
Any examples for that?
Does your filename maybe additionally contain unsupported characters?

Quote:
Originally Posted by EquiNox View Post
I fully understand your objections and it's not my intention to clutter or slow down the plugin with too many options.
It's not for these reasons.
I just think it doesn't make sense to create plugin settings for stuff that should be solved by a global feature otherwise you could e.g. add the "crop filename" feature hundrets of times for each plugin -> Makes no sense.
Same for custom filenames because our Packagizer feature can be used to do pretty much the same you can do inside the flickr.com plugin at this moment but for some reason custom filenames have been added as a plugin setting back then which is why I kept it... users hate it when existing features get removed and I can fully understand this

Quote:
Originally Posted by EquiNox View Post
...it would cut off the username and the content_id probably, so I don't anymore know where the file was from.
As described in our Packagizer article, you can also use plugin specific tags inside filenames.
In this case, all tags given in the flickr.com plugin settings are 1:1 the tags you can use in Packagizer.
... so let's pretend that the title is always the problem meaning that "username_url" and "content_id" will always be missing 100% if a filename gets cropped... so we'll just add this info to the new name:
Code:
<jd:orgfilename:1>_<jd:prop:username_url>_<jd:prop:content_id>.<jd:orgfiletype>
Quote:
Originally Posted by EquiNox View Post
...it does work, however there is no guarantee that the title is always cut off at the same position, when for example I want to change username_internal to the new pathalias a user now has, then the length of the prefix before the title can change so the title will be cut off differently.
Sure the name will be cut off differently because it is a different name.
To be honest I wouldn't know how to solve this other than adding a logic to cut off clean after specified tags and still allow different lengths.

Quote:
Originally Posted by EquiNox View Post
Maybe a packagizer-rule can be defined that does only limit the length of the title, but leaves the rest of the filename untouched? I am not to much a regular expressions expert, but I know you can do some magic with that.
That's not possible using only Packagizer rules because:
- Packagizer cannot do String operations (e.g. modify strings, crop strings or do uppercase/lowercase conversions)
- Apart from the defaults needed to create rules, Packagizer cannot do things like "if <jd:prop:username_url> contains 'xxxYYY' then do ..."

You'd have to use an EventScripter script for that.
You can get help creating such scripts in this thread.
With such a script you'd definitely be able to optimize the naming scheme along with obeying your OS' limits up to "perfection".
Using a script, you'd probably simply crop the title and then build your new filename.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 17.09.2021 at 11:28. Reason: Added hyperlinks
Reply With Quote
  #44  
Old 17.09.2021, 11:49
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

About the *invalid download directory*, can you please provide example link and a log for this?
Most likely the filename contains characters that cause the issue. individual path segments are *as far as I know* still
limited to 255 chars, same as in other OS filesystem.
It looks like you can allow longer path segments, see https://support.jdownloader.org/Know...load-directory
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 17.09.2021 at 11:59.
Reply With Quote
  #45  
Old 17.09.2021, 15:06
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,920
Default

Changes for the next flickr plugin update:
- Implemented linkcheck via flickr API for single items --> Now even single added mature content items can be downloaded without account
- Added fast linkcheck setting for videos and enabled it by default (=filesize won't be visible and 'quality' tag won't be available until download is started)
- Disabled account login for flickr as I haven't found any reasons to keep it -> It's up to you to provide some good reasons to re-enable it

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #46  
Old 17.09.2021, 16:46
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,920
Default

Regarding the Windows path length limit:

I've now tesdted the method that is linked in our guide to disable that limit.
Unfortunately that did not work for me either but I'm pretty sure that the limit can be disabled.

I'll do some more research the next week.
If you want you can also do a research on that.

Have a nice weekend!

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #47  
Old 19.09.2021, 10:27
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

Quote:
Originally Posted by Jiaz View Post
individual path segments are *as far as I know* still
limited to 255 chars, same as in other OS filesystem.
It looks like you can allow longer path segments, see **External links are only visible to Support Staff**...
Ah, ok, thanks for clarifying. That was probably the issue with the examples that I tried. Individual path segments does not mean the entire path+filename but just the single folder- or filename should not be longer than 255 chars. However the old "260 character maximum path limit" does apply to the entire path+filename as far as I understood.

Quote:
Originally Posted by Jiaz View Post
About the *invalid download directory*, can you please provide example link and a log for this?
The examples that I already used above, not a single special character in there.
From this stream that contains many items with very long titles:
Code:
**External links are only visible to Support Staff**

I have now tested this a little more with this link, using the default custom filename:
Code:
**External links are only visible to Support Staff**

Filename that would be complete and correct (it's too long individual filename):
Code:
268 characters: viv_vivekananda_50039847567_Tourist helicopter flights over pristine waters of Great Barrier Reef - in the distance you can see the water colour change and waves which signifies the continental shelf giving way to very deep ocean, Off Cairns, Queensland, Australia.jpg

Longest filename that JD still does allow before showing "Invalid download directory" error, I have renamed manually:
Code:
250 characters: viv_vivekananda_50039847567_Tourist helicopter flights over pristine waters of Great Barrier Reef - in the distance you can see the water colour change and waves which signifies the continental shelf giving way to very deep ocean, Off Cairns, Que.jpg

Afterwards you still can rename this file in Windows Explorer manually but not longer than to this filename when the file was put into the root folder (\):
Code:
254 characters: viv_vivekananda_50039847567_Tourist helicopter flights over pristine waters of Great Barrier Reef - in the distance you can see the water colour change and waves which signifies the continental shelf giving way to very deep ocean, Off Cairns, Queensl.jpg

JDownloader does allow to download this 250 character long filename into a deep folder structure, it's no problem, even if the path alone is longer than 260 characters:
Code:
D:\this_is_a_very_long_foldername\this_is_a_very_long_foldername\this_is_a_very_long_foldername\this_is_a_very_long_foldername\this_is_a_very_long_foldername\this_is_a_very_long_foldername\this_is_a_very_long_foldername\this_is_a_very_long_foldername\this_is_a_very_long_foldername
However you will be running into issues when you want to do something with this file (moving, renaming, viewing). I guess most programs cannot handle such overlong pathes and just return some kind of "file not found" error.

I have also tested the maximum foldername length and have found out that JD can handle 255 characters at a max for a single foldername.

All this I did with "Enabled NTFS long paths". Now I have deactivated the NTFS long paths with the policy editor again, restarted, checked the option in the registry (LongPathsEnabled=0) and tried to do the same again with JDownloader, downloading to this overlong path with the filename 250 characters long. That also worked totally fine. So I don't yet see any difference with the long paths option enabled or not. As long as the filename isn't too long, JD can handle an overlong total path, so such are obviously not limited by the operating system. The only issue is that many other programs and even Windows Explorer do not handle files over the maximum path limit properly. But I also had programs that had not much problems with such a long path, XnView MP for example. XnView Classic could not handle it and crashed. So guess it depends on the program implementation.

Conclusion, JD works all fine as long as filename lengths <=250 and single foldername lengths <= 255 characters.

In this article you get some useful info regarding this issue:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

Quote:
We’re a long way from the mid-1990s, however, and the whole Long Filename thing is (for the most part) firmly ironed out. If you’re running a version of Windows from the last 10 years, you’ve likely never even come across a filename length conflict like we we used to run into back in the DOS/Windows 95 days. That said, we still run into hiccups, as you discovered with your disk cleanup project. But why? If Windows’ Long Filename system supports folders and file names of up to 255 characters per component, what wall are you running into? We can’t blame NTFS (the filesystem that the vast majority of modern Windows machines use) as NTFS will support a chaining of folders and file names up to a total path length of 32,767 characters. That far exceeds the typical directory structure most users would ever need.

Where it all falls apart is an artificial restriction Windows stacks on top of the LFN/NTFS system: the MAX_PATH variable. The MAX_PATH variable specifies that a complete directory structure in Windows can’t exceed 260 total characters, including the drive letter, colon, backslash, and null backlash at the end. Thus you only have a potential real MAX_PATH of 256 characters, e.g. C:\your-256-character-path\.

So what happened when you were cleaning up your computer is that you had a directory with an already long path (either because the folder names were long, the file names were long, or both), and when you attempted to move one or more of those directories into another directory with a long path, the total length of the path name exceeded the 260 character limit imposed by the MAX_PATH variable.

Now, you may be thinking “Ah-hah! We’ll just change the MAX_PATH variable and solve the problem!” Alas, it’s not that simple. Not only is the MAX_PATH variable essentially hard coded into Windows, but even if you went through the enormous hassle of changing it, you’d end up breaking so much it wouldn’t be worth it. Too many applications expect the path variable to be what Windows has long specified it to be. We can’t just go around changing it without creating an enormous mess.
NTFS itself does not have such a limit, the total path length can be 32,767 characters long. The MAX_PATH variable seems to cause all this mess. Just too many apps would not anymore work correctly when exceeding the limit defined in this variable.

Still not sure what this "Enable NTFS long paths" option should actually do as I don't see any difference between having it activated or not.

Last edited by EquiNox; 19.09.2021 at 15:34.
Reply With Quote
  #48  
Old 20.09.2021, 13:44
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

During my tests with these overlong paths I ran into the problem that it's not easy getting rid of these afterwards, almost any regular method to delete such folders failed and hardly any program would even recognize those correctly.

But there is a way and the article that I linked to above says that a workaround in the operating system is implemented dealing with such overlong paths. I did it with the WinRAR file manager, there was no problem to rename and delete these paths. When you want command line tools to work with such long paths you should add the prefix "\\?\" to the path.


**External links are only visible to Support Staff****External links are only visible to Support Staff**
Quote:
If you have a huge number of files with a long path and you don’t want to edit them all (or if you want to delete a ton of old directories that are too long for Windows to deal with when restricted by the MAX_PATH variable), there is a command-line work around. Even though Windows is restricted by the MAX_PATH variable, Windows engineers realized there would be situations wherein users would need to deal with longer path names. As such, the Windows API has a function for dealing with extremely long paths.

In order to take advantage of that API and use command line tools on your unwieldy folders/file names, you simply need to append the directory name with a few extra characters. For example, if you had a huge directory structure that you wanted to delete (but received an error due to the path length when you attempted it), you could change the command from:

Code:
rmdir c:\documents\some-really-super-long-folder-name-scheme\
to:
Code:
rmdir \\?\c:\documents\some-really-super-long-folder-name-scheme\
The key is the addition of the \\?\ portion before the start of the file path; this instructs Windows to disregard the limitations imposed by the MAX_PATH variable and to interact with the path you just supplied as supplied/understood directly by the underlying files system (which can clearly support a longer path). As always, exercise caution at the command prompt to avoid accidentally deleting files or directories you intended to leave intact.

Last edited by EquiNox; 20.09.2021 at 13:49.
Reply With Quote
  #49  
Old 20.09.2021, 21:11
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,920
Default

Quote:
Originally Posted by EquiNox View Post
So I don't yet see any difference with the long paths option enabled or not.
Neither did I.

According to your test results I guess it's easier to use Linux than to mess with this.

As said if you want a nicer solution to cut filenames (or well ... titles to be more precise), try an EventScripter script.

Flickr:
As described in my recent posts, I've removed the login functionality for now as I was unable to find any reasons to keep/fix it.

As for now, I will do some flickr code cleanup but that's it:
For now I'm done with the flickr refactoring and I'm not planning on adding more features.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #50  
Old 20.09.2021, 22:01
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

Quote:
Originally Posted by pspzockerscene View Post
Neither did I.

According to your test results I guess it's easier to use Linux than to mess with this.

As said if you want a nicer solution to cut filenames (or well ... titles to be more precise), try an EventScripter script.
It's all fine. I guess it was just unsatisfying before to run into these errors without exactly knowing what's wrong. Now I have already a solution with the Packagizer. EventScripter I'll maybe have a look later into. I guess my scripting skills in this regard are not yet sufficient.

Thanks for all your help and being patient with me!


Quote:
Originally Posted by pspzockerscene View Post
Flickr:
As described in my recent posts, I've removed the login functionality for now as I was unable to find any reasons to keep/fix it.

As for now, I will do some flickr code cleanup but that's it:
For now I'm done with the flickr refactoring and I'm not planning on adding more features.
I am probably not yet done with all tests but it already works like a charm and it's faster than before too! So thank you very much pspzockerscene for all your work on the plugin and this tremendous improvement!

About login functionality, I'll put the information you requested in here. The ability to download albums that are only accessible when you are logged in is one good reason to have this login functionality. And the biggest reason of course, parsing would be much faster. There are many albums at Flickr you can download for free, but you must be logged in and JD2 is the best tool I ever ran across so far to make such things so easy. It's all good, I am sure you've much other things to do. Thanks again for all your hard work!
Reply With Quote
  #51  
Old 21.09.2021, 09:00
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

Quote:
Originally Posted by pspzockerscene View Post
Hi dear users,
It has been a long time ago since you've reported the flickr.com login issues.
I'm working at the flickr plugin at this moment but I need some more information for you in order to decide how to proceed regarding the flickr login support.

- In which situations is a flickr.com account required (e.g. private content, mature content, content only available when befriended with a user?)
- I need example URLs for content only accessible via account.
I've never used the flickr.com account add-on in JD2 since it never ever worked for me (Plugin-outdated). I was a bit surprised to read here at this forum that there was a time it was ever working.

Quote:
Originally Posted by Jiaz View Post
We need username/password to check/fix it. send them to support@jdownloader.org
The *forced* yahoo.com account isn't yet supported. In the past it was possible to signup *outside* yahoo, as far as I remember.
Yes, I remember that Flickr changed its login procedure a few years ago. It was a bit of a mess somehow because you didn't really know if you were now logging in to your Yahoo or Flickr account. It can be the same login details but different ones too.

And yes, there was a time you could login there with a non-yahoo.com address. But as far as I remember there had always been more restrictions regarding visibility and accessibility of content when you were not using yahoo.com address. It was enough to use yahoo.de and you could not see as much as you could with the com-address. At that time I thought it might even be some kind of geoblocking, that depended on the email address.


Quote:
Originally Posted by pspzockerscene View Post
- In which situations is a flickr.com account required (e.g. private content, mature content, content only available when befriended with a user?)
- I need example URLs for content only accessible via account.
There are different ways to control the visibility and accessibility of content at Flickr.
When you upload content at Flickr you can set a "safety level" for the uploaded item:
  • Safe
  • Moderate
  • Restricted

Besides that you can set the "viewing privacy" that configures who actually should be allowed to see your content:
  • Public
  • Private
  • Friends
  • Family
  • Friends & Family

When you are not logged in at Flickr you will only get to see content that is safe and publicly available. When you are logged in and not follow any user you can also see content that is public and set moderate or restricted. Restricted content is limited to registered users over the age of 18.

When you have set the viewing privacy to private it's only you who can see the content. When you have set it to Friends and/or Family only Flickr users who follow you and who you added to your family respectively friends group will be allowed to see your content. Just following a member does not give you access to his or her f+f content. For all this you surely need an account.

More info on this you can also get here:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

And example where content is set to "moderate" and you can only view or download with an account, JD2 doesn't show anything when you parse this stream and it would be nice to be able to download the albums as they are shown on the album page:

**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #52  
Old 21.09.2021, 13:53
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,920
Default

Quote:
Originally Posted by EquiNox View Post
It's all fine. I guess it was just unsatisfying before to run into these errors without exactly knowing what's wrong.
I've updated our "Invalid download directory" help article and removed the "Windows long path workaround" path as it didn't do what we all thought it would:
https://support.jdownloader.org/Know...load-directory

Quote:
Originally Posted by EquiNox View Post
Now I have already a solution with the Packagizer. EventScripter I'll maybe have a look later into. I guess my scripting skills in this regard are not yet sufficient.
You can ask for help regarding EventScripter scripts in the following thread:
https://board.jdownloader.org/showthread.php?t=70525

Quote:
Originally Posted by EquiNox View Post
I am probably not yet done with all tests but it already works like a charm and it's faster than before too! So thank you very much pspzockerscene for all your work on the plugin and this tremendous improvement!
Thanks for your feedback!

Quote:
Originally Posted by EquiNox View Post
The ability to download albums that are only accessible when you are logged in is one good reason to have this login functionality.
I need examples!

Quote:
Originally Posted by EquiNox View Post
When you are not logged in at Flickr you will only get to see content that is safe and publicly available. When you are logged in and not follow any user you can also see content that is public and set moderate or restricted.
Okay - interesting.

Quote:
Originally Posted by EquiNox View Post
Restricted content is limited to registered users over the age of 18.
As said, this restriction doesn't apply for our current/new plugin:
You can add mature content without account.

Quote:
Originally Posted by EquiNox View Post
And example where content is set to "moderate" and you can only view or download with an account, JD2 doesn't show anything when you parse this stream and it would be nice to be able to download the albums as they are shown on the album page:
That's a great example, thanks!
I will try to fix account support.

-psp-
EDIT

Login support ticket:

__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 21.09.2021 at 14:39.
Reply With Quote
  #53  
Old 21.09.2021, 22:08
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

Quote:
Originally Posted by pspzockerscene View Post
Quote:
Originally Posted by EquiNox View Post
When you are not logged in at Flickr you will only get to see content that is safe and publicly available. When you are logged in and not follow any user you can also see content that is public and set moderate or restricted.
Okay - interesting.
I forgot to say that for viewing the moderate or restricted content you also need to turn off the SafeSearch for your Flickr account. The option you can find on this page:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

Then you have access to all content that is publicly available.

Quote:
Originally Posted by pspzockerscene View Post
As said, this restriction doesn't apply for our current/new plugin:
You can add mature content without account.
Yes, as long as I can remember the plugin always worked like that, single link parsing of mature content. That was already before the big update now. The only big drawback it took a long time to parse many links. A few thousand links could have taken many hours to parse. And first you had to get all the single links which also takes some time. So it wasn't really an option, just now and then. With the new update however single link parsing is much faster, so this is really a great improvement. However when trying to parse albums with mature content LinkGrabber would just not find anything. It is also possible to do all this with the single link method but it would be a lot of manual work album-by-album, not really practical.

Another account that only contains moderate or restricted content:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #54  
Old 22.09.2021, 16:09
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,920
Default

Quote:
Originally Posted by EquiNox View Post
I forgot to say that for viewing the moderate or restricted content you also need to turn off the SafeSearch for your Flickr account. The option you can find on this page:
Thanks - I've already done that.

Quote:
Originally Posted by EquiNox View Post
Yes, as long as I can remember the plugin always worked like that, single link parsing of mature content.
Nope it definitly wasn't possible before (although strangely I haven't seen any reports about this in our forum).

Quote:
Originally Posted by EquiNox View Post
With the new update however single link parsing is much faster, so this is really a great improvement.
Yap it is now about as fast as possible - there isn't a lot of room for improvement left

Quote:
Originally Posted by EquiNox View Post
However when trying to parse albums with mature content LinkGrabber would just not find anything.
There seems to be a difference between:
- Albums ONLY containing mature content
- Crawling "all albums of a user" when all of those contain 100% mature content
- Crawling all favorites of a user when only some of the images are mature content (this works fine - see the example I've sent for testing the "missing images" issue)

Quote:
Originally Posted by EquiNox View Post
It is also possible to do all this with the single link method but it would be a lot of manual work album-by-album, not really practical.
Exactly and this is also what wasn't possible before.
This is also a quite inconsistent behavior of flickr but maybe their API was designed with this in mind:
Why shouldn't external apps be able to access mature content if e.g. their users have to confirm their age at some other point than on the flickr website so an age confirmation via flickr account is not necessary anymore.

Whatever it is, I guess all content will be crawlable once I've fixed account support.
I'm not in the mood of looking for other tools that might be able to get around these limits via special API parameters plus account support will bring the benefit of e.g. being able to download your own private galleries.

Again I'm absolutely not a flickr user/customer myself...

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #55  
Old 22.09.2021, 17:10
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,920
Default

Next flickr plugin update:
- fixed login
- enabled resume for all items: Indeed resuming photo downloads is possible but not for all items: JD cannot know this until it first checks/download the directURL of a picture so I'll just leave it enabled for all items now: After all, images are usually small so when stopping downloads you will only lose ~100MB at max depending on how many unresumable pictures you were downloading simultaneously: I consider this an edge-case.
--> After the next update, JD will not warn you regarding resumability anymore when stopping photo downloads.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 22.09.2021 at 17:19.
Reply With Quote
  #56  
Old 22.09.2021, 20:17
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

Wow! The way was a bit tricky to install the account support for Flickr in JD (I used the incognito mode) but it finally works and even very well it seems! Thank you very much pspzockerscene! This now even doubles the quality of this plugin and makes so many tasks so much easier! Just fantastic!

I'll still keep an eye on how it works and will report if there are any other issues we yet haven't run into.

Many thanks for your work on this, pspzockerscene!!
Reply With Quote
  #57  
Old 23.09.2021, 14:54
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,920
Default

Thanks for your feedback!

Next update includes (this time I was lazy and simply copied my SVN commit message over):
- fixed NPE in website mode
- fixed missing directurl in website mode
- added missing property "set_id" to settings properties description
- added new property "gallery_id"
- fixed missing set_id property for single added URLs containing that ID
- fixed/improved contentURLs in JD so they match the browser/context ones e.g. for items from galleries: flickr.com/photos/<usernameURL>/<content_id>/in/gallery-<galleryUsernameInternal>-<gallery_id>/

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #58  
Old 24.09.2021, 08:57
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

Thanks for fixing and the new additions as well!
I already have ideas how to use the new tags in certain situations.
Thank you very much psp! Much appreciated!
Reply With Quote
  #59  
Old 24.09.2021, 15:39
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,920
Default

Thanks for your feedback!

I've gone through the code of both plugins one last time and made some small changes.
I've removed flickr from my TODO list for now.

A few small observations/open points:

- Are there videos with other file extensions than .mp4? If so, I need example URLs.
- Are there photos with other file extensions than .jpg? If so, I need example URLs.
--> For both cases I'd say yes it is possible when original file is downloadable.
- If possible, we could set the "resumable" state to be correctly set for all items (once the download has been started)

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #60  
Old 27.09.2021, 07:21
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

Quote:
Originally Posted by pspzockerscene View Post
A few small observations/open points:

...

-psp-
Good you mentioned, I had seen we are not yet done. There are a few more issues that should be checked/worked on.

First of all a minor issue, the *date_taken* tag, very useful but could it also get the date format that is defined in "Define how dates inside filenames should look like"? That would be more flexible and useful in certain situations and more consistent to.


Quote:
Originally Posted by pspzockerscene View Post
- Are there videos with other file extensions than .mp4? If so, I need example URLs.
- Are there photos with other file extensions than .jpg? If so, I need example URLs.
--> For both cases I'd say yes it is possible when original file is downloadable.
- If possible, we could set the "resumable" state to be correctly set for all items (once the download has been started)
Yes, there are other file types you can find at Flickr and we should probably take all these into account although the vast majority will probably be jpg and mp4. PNG you can also find here and there. When you try to upload at Flickr there is a long list of filetypes offered to you:
Code:
asx, avi, avif, bmp, dib, gif, ico, jfif, jpeg, jpg, jxl, mov, mp4, mpeg, mpg, ogm, pjp, pjpeg, png, svg, svgz, tif, tiff, webm,ogv, webp, wmv, xbm
Besides that I do know for sure JD previously downloaded .flv video files, but maybe this is just a different type of .mp4 files? I am not sure. It's not easy to find examples since the other filetypes are not used very often, here is a stream with PNG-files:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

This one is a good example as it shows what JD2 currently does, parsing correctly 504 items in this stream but assuming all of them to be JPG-files, although 242 of these are actually PNG files.

Examples:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

JD2 can parse all these links but puts a JPG-extension at the end even when it's a PNG image. Image editing or viewing programs can handle this as they usually scan the header of the files to actually see what filetype it is but it would be better if the extension would be set to PNG, which actually would be correct. Checking these files with an image analysing tool returns something like this "Not a JPEG file: starts with 0x89 0x50".

On my search for PNG files I ran across another stream that gave me a big riddle as it was not correctly parsed by JD although all the content should be available without an account.

First of all the hq-png image from that stream:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

Now when you parse this stream, JD returns 756 items although the stream actually has 1191 items. I looked through the pages and all had correctly 100 items and the last page 91, so there were no gaps.

**External links are only visible to Support Staff****External links are only visible to Support Staff**

When you now parse the albums then you'll get another 206 items, so there are 962 in total, but still far away from 1191.

**External links are only visible to Support Staff****External links are only visible to Support Staff**

I checked this a long time and thoroughly but I really could not find a reason why JD would not find all the items in this stream, maybe you can help clearing up this mystery, psp.

Besides that I still more need to check if downloading all the video links are working, I think I had a few issues with very small res videos. I don't have an example at hand right now but it could be that those examples previously listed in this thread are also affected (Point 2 in this post: https://board.jdownloader.org/showpost.php?p=490685).


The overall performance of the plugin is amazing compared to former times. Even the downloading is much faster I'd say. Thank you very much psp!
Reply With Quote
  #61  
Old 27.09.2021, 16:41
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

@EquiNox:
1.) file extension should be fixed with next update
2.) item parser will be fixed with next update
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 27.09.2021 at 17:10.
Reply With Quote
  #62  
Old 28.09.2021, 07:22
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

Quote:
Originally Posted by Jiaz View Post
@EquiNox:
1.) file extension should be fixed with next update
2.) item parser will be fixed with next update
Hey cool, both seem to work now!

Thank you very much Jiaz!
Reply With Quote
  #63  
Old 28.09.2021, 10:58
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

@EquiNox: Thanks for the feedback and you're welcome
__________________
JD-Dev & Server-Admin
Reply With Quote
  #64  
Old 28.09.2021, 17:08
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,920
Default

Quote:
Originally Posted by EquiNox View Post
First of all a minor issue, the *date_taken* tag, very useful but could it also get the date format that is defined in "Define how dates inside filenames should look like"? That would be more flexible and useful in certain situations and more consistent to.
I've updated our plugin to use the same handling used for "date" for "date_taken" too.
Existing "date_taken" values will be auto-overwritten with the new formatted value.

Quote:
Originally Posted by EquiNox View Post
Yes, there are other file types you can find at Flickr and we should probably take all these into account although the vast majority will probably be jpg and mp4. PNG you can also find here and there. When you try to upload at Flickr there is a long list of filetypes offered to you:
The way it has been implemented by Jiaz it shouldn't matter which filetype an item has - it will be obtained from the direct-URL of that item.

Quote:
Originally Posted by EquiNox View Post
I think I had a few issues with very small res videos.
Please provide some URLs if you find some.
I can imagine how this might happen as we do skip 1-2 frequently broken video qualities so there might be an edge case where only those ones are available...
Please also check if those videos are playable via browser.

Quote:
Originally Posted by EquiNox View Post
The overall performance of the plugin is amazing compared to former times. Even the downloading is much faster I'd say. Thank you very much psp!
Thanks!
Yeah that plugin was really old and when I saw that old stuff I couldn't resist rewriting it.

@Jiaz
Thanks for fixing that last encoding issue!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
EDIT

One more thing about date_taken:


I've noticed that their werbsite/api can contain this:
Code:
"isDateTakenUnknown":true
... though a "datetaken" value will still be given.
My guess is that it is wrong in this case and that whenever "isDateTakenUnknown" is true, I rather should not set the value of "datedaken" as that will simply be the same as the upload date then.

If you want you can do some tests on that e.g. find items where isDateTakenUnknown equals false.
Please let me know your results and I'll update our plugin accordingly.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 28.09.2021 at 17:12.
Reply With Quote
  #65  
Old 29.09.2021, 07:35
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

Ok, many thanks for the last changes, psp!

Quote:
Originally Posted by pspzockerscene View Post
One more thing about date_taken:[/B]

I've noticed that their werbsite/api can contain this:
Code:
"isDateTakenUnknown":true
... though a "datetaken" value will still be given.
My guess is that it is wrong in this case and that whenever "isDateTakenUnknown" is true, I rather should not set the value of "datedaken" as that will simply be the same as the upload date then.

If you want you can do some tests on that e.g. find items where isDateTakenUnknown equals false.
Please let me know your results and I'll update our plugin accordingly.
Yes, this is exactly how I thought it would be working on Flickr. Usually you'll see the term "Taken on... DATE" for single items. Then I think the DateTaken is available. But sometimes you only see the term "Uploaded on.. DATE". Then I think DateTaken is unavailable. But when you open the archives/date-taken/ view in Flickr that I mentioned before, still all items are listed at a specific date, even when DateTaken is not available.

I'll do some futher tests with this too, but later. Busy at work these days.
Reply With Quote
  #66  
Old 29.09.2021, 16:57
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,920
Default

After the next update, the "date_taken" property will only be set when given according to "isDateTakenUnknown" (website) or "takenunknown"/"datetakenunknown" (API).

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #67  
Old 30.09.2021, 07:44
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

That is probably more consistent. Thanks psp! But I think this issue is a little more complicated. When the date-taken is given but the uploader forbids to download the original files (_o) then you can download the 3k, 4k, 5k or whatever files but the exif-header of the file is (always?) empty. So in this case it's really helpful that you have the *date_taken* tag, so you can at least include this information in the filename when it's not stored within the file itself. Just an observation. When it's not that long ago you maybe think it's not so important to have this information, but when you look back 20, 30 years you will be grateful to have the date, when the photo was actually taken.
Reply With Quote
  #68  
Old 30.09.2021, 13:55
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,920
Default

Quote:
Originally Posted by EquiNox View Post
So in this case it's really helpful that you have the *date_taken* tag, so you can at least include this information in the filename when it's not stored within the file itself.
With a little bit of EventScripter magic you could even write a script which checks for the date inside downloaded flickr files EXIF data and puts the "date_taken" in there in case it is missing in the file but given on the DownloadLink object
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #69  
Old 04.10.2021, 08:49
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

Quote:
Originally Posted by pspzockerscene View Post
With a little bit of **External links are only visible to Support Staff**... magic you could even write a script which checks for the date inside downloaded flickr files EXIF data and puts the "date_taken" in there in case it is missing in the file but given on the DownloadLink object
Hey, great idea, I had not yet thought in this direction. And I didn't know this would even possible with EventScripter. I guess my basic guideline prevented my from doing so before - never touch any original files - but when you think of it these Flickr files are touched very often and when you are only allowed to download the k- instead of o-files you do not get any originals either. But I think when the date-taken is already stored within the filename it should be possible to use this info for my favorite tool to write the information in the header of the files, that's **External links are only visible to Support Staff**ExifTool. Very cryptic command line options but extremely powerful for me especially to sort collections chronologically.

Ok, I have tested the plugin little more over the weekend and must say it really works like a charm and so nicely fast and precise!! So thanks again for all the time and efforts you have put into this, psp and Jiaz!

There is however one more issue I ran accross, not a bug but more a feature from Flickr that has not yet been covered adequately I'd say. I think the solution should be very simple and I think that was also the same behaviour of the JD-Flickr plugin before this big update.

The problem is that as a Flickr user you can put the same items into different albums. I think for Flickr this is a very nice feature as you are not limited to put an item only in one album but it does only appear once in the stream. You can make for example theme albums like "sundown" or "nature" where you just combine all pictures from other albums that apply to the theme. Or sometimes users make favorite albums where they put their own favorite pictures together from their other albums. There are many possibilities.

Anyway the current behaviour of JD is that items are only parsed once, and when the same item later appears in another album too it's simply left out, so that album would have less items in JD than stated on the Flickr site. Generally this behaviour is very helpful as you very nicely get exactly the correct number of items in the whole stream (first parse the albums, later the fotostream -> you have all items). But sometimes this behaviour is very unfortunate, when as I described above a user has put same items in more than one album. Then it could be that the original albums have big holes since those items are already part of a theme album and this theme albums was parsed first by JD. When there are only a few albums it's not big issue you just let JD parse the original albums first then the theme albums (copy paste every album singly one-by-one) but when there are hundreds or thousands of albums this is tedious work.

Ok, to cut a long story short, it would be very nice to have a button on the Flickr plugin settings [x] Allow duplicate items, you can switch it on and off. This would solve all these problems that I described above. You then would maybe have a some duplicates in that Flickr collection but you'd know all albums are as complete as they appear on the Flickr site. Then later you can decide if you want to remove any duplicates and from which albums.

The stream where I noticed this issue was this one:
Code:
**External links are only visible to Support Staff**
This user has created albums that combines the top favourited pictures together (20+, 50+, 100+):
Code:
**External links are only visible to Support Staff**
**External links are only visible to Support Staff**
**External links are only visible to Support Staff**
When JD is parsing these albums first then these best pictures are missing in the original albums.

A link of this stream that is even part of three albums, original, fave-album and a camera album.
Code:
**External links are only visible to Support Staff**
Reply With Quote
  #70  
Old 04.10.2021, 20:55
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,920
Default

Quote:
Originally Posted by EquiNox View Post
But I think when the date-taken is already stored within the filename it should be possible to use this info for my favorite tool to write the information in the header of the files, that's ExifTool. Very cryptic command line options but extremely powerful for me especially to sort collections chronologically.
Sure.
It really doesn't matter which way you're doing it.

Quote:
Originally Posted by EquiNox View Post
Ok, I have tested the plugin little more over the weekend and must say it really works like a charm and so nicely fast and precise!! So thanks again for all the time and efforts you have put into this, psp and Jiaz!
Thanks for your feedback!

Quote:
Originally Posted by EquiNox View Post
Anyway the current behaviour of JD is that items are only parsed once, and when the same item later appears in another album too it's simply left out, so that album would have less items in JD than stated on the Flickr site.
Yeah we try to match items by theeir unique IDs by default.
It probably also worked with the old plugin revision but probably only partly as dupes were matched by URL so items as part opf an album had different URLÖs than single items -> No dupe detected.

Quote:
Originally Posted by EquiNox View Post
Ok, to cut a long story short, it would be very nice to have a button on the Flickr plugin settings [x] Allow duplicate items, you can switch it on and off.
We have a general Setting therefore:
Settings -> Advanced Settings ->
Code:
LinkCollector.dupemanagerenabled
Turn that off and restart JD.

An extra setting for flickr which I could imagine would be to set other unique IDs for albums e.g. a picture with ID 123456 gets the unique ID 123456.
If it gets added in context of the album with albumID 55555, we could use the combination of both IDs (55555_123456) as unique ID thus the same item can be added unlimited times as long as it is in a different album then.
Good idea?

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #71  
Old 05.10.2021, 08:50
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

Quote:
Originally Posted by pspzockerscene View Post
Yeah we try to match items by theeir unique IDs by default.
It probably also worked with the old plugin revision but probably only partly as dupes were matched by URL so items as part opf an album had different URLÖs than single items -> No dupe detected.
Could be. I do know for sure - not anymore exactly when it was - maybe 3 or 4 years ago - the plugin was still working exactly the way it works now. You parse the albums -> you get all items in the albums, you parse the fotostream -> you get all the remaining items -> so you have the complete stream. Then there was probably a change on the plugin back then and the behaviour was different. You parse the albums -> you get all items in the albums (I think you are right dupes were left out too), but when you then had parsed the fotostream you'd again get the entire stream, so those items in the albums were included too although they were already in LinkGrabber. This was not a big deal as this behaviour had some advantages too. And about single links, yes they were also always possible even when the items were already parsed by JD in an album or the fotostream, you could always add single items. The only issue with the single items back then, they always contained the *content_id* twice, no idea why but it was like that.

Example:
The filename of an item parsed with an album or the fotostream looked like this:
Code:
Flickr-DSCN1234_flickruser_1234567890.jpg
When you had parsed the same item singly the content_id was always included twice.
Code:
Flickr-1234567890_DSCN1234_flickruser_1234567890.jpg
Back then my custom filename looked like this:
Code:
*owner*-*title*_*username*_*photo_id**extension*
*owner* was always = Flickr -> that way I got my uppercase prefix.

Not a big deal, bulk renaming with regular expressions fixed this within seconds (removing the double id).
Anyway thankfully this issue is now history and was fixed with the great update now.

Quote:
Originally Posted by pspzockerscene View Post
We have a general Setting therefore:
Settings -> Advanced Settings ->
Code:
LinkCollector.dupemanagerenabled
Turn that off and restart JD.
Ok, many thanks! I almost thought already you'd suggest a global option for this as it does make sense.

But I think I need to change two options to get the desired result:
Code:
Advanced Settings\LinkCollector: Dupe Manager -> deactivate
Advanced Settings\LinkgrabberSettings: Default On Added Dupes Links Action -> add them, too

Quote:
Originally Posted by pspzockerscene View Post
An extra setting for flickr which I could imagine would be to set other unique IDs for albums e.g. a picture with ID 123456 gets the unique ID 123456.
If it gets added in context of the album with albumID 55555, we could use the combination of both IDs (55555_123456) as unique ID thus the same item can be added unlimited times as long as it is in a different album then.
Good idea?

-psp-
Ok, would this modify the general behaviour of *content_id*? I think then better not, the way it works now with excluding dupe items is very good in most cases as you do not always want to download hundreds or thousands of duplicates.

When I have deactivated dupe check in Advanced Settings and use this custom filename...
Code:
Flickr-*date_taken*_*title*_*username_url*_*content_id*_*set_id*_*quality**extension*
... the result would be the same as the one you just suggested. I think we can leave it like that.


There is another question I have about the "Menu Manager". There is a very nice function "Copy Information" that offers various tags as pattern for links and packages (right click -> Open Menu Manager -> Copy Information). This function was already so tremendously helpful for me to update collections, finding missing items, sorting into albums. You can easily create scripts with this function. My question, are there more tags that can be used than those that are stated there ({path}, {name}, {url.content}, etc.)? It would be fantastic for example if I could also use the *content_id* tag from the Flickr plugin with this function but I am not sure if that would be possible. Then I could use for example this command:

Code:
rename "*{flickr.content_id}*" "{name}"
...to rename files of previously already downloaded streams but now adding the date-taken for example, that was not there before.
Reply With Quote
  #72  
Old 05.10.2021, 14:48
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,920
Default

Quote:
Originally Posted by EquiNox View Post
But I think I need to change two options to get the desired result:
Code:
Advanced Settings\LinkgrabberSettings: Default On Added Dupes Links Action -> add them, too
Shouldn't be required if the other setting is active (JD restart required after changing this) because dupes won't get flagged as dupes, the "On added dupesl inks action" never gets executed

Quote:
Originally Posted by EquiNox View Post
Ok, would this modify the general behaviour of *content_id*?
Not at all.
content_id != dupe_id
Our internal dupe_ids/"link ids" for flickr.com look like this:
Code:
flickr.com://<content_id>
E.g.:
Code:
flickr.com://123456
This ID is used globally for dupe-checking this it also contains the name of the host.
Nothing regarding filename properties would change even if I introduced the change mentioned in my last post.

Quote:
Originally Posted by EquiNox View Post
My question, are there more tags that can be used than those that are stated there ({path}, {name}, {url.content}, etc.)?
AfaIk no but it looks like it wouldn't be much work to add generic support for plugin specific properties.
Jiaz may also add his opinion on that.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #73  
Old 06.10.2021, 08:48
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

Quote:
Originally Posted by pspzockerscene View Post
Code:
Advanced Settings\LinkgrabberSettings: Default On Added Dupes Links Action -> add them, too
Shouldn't be required if the other setting is active (JD restart required after changing this) because dupes won't get flagged as dupes, the "On added dupesl inks action" never gets executed
Hmm, ok I have tested again the dupe setting and it does not seem to work at all with just one time parsing. When parsing again you get dupes but not always the way I wanted them. I need to test this further.

Quote:
Originally Posted by pspzockerscene View Post
Not at all.
content_id != dupe_id
Our internal dupe_ids/"link ids" for flickr.com look like this:
Code:
flickr.com://<content_id>
E.g.:
Code:
flickr.com://123456
This ID is used globally for dupe-checking this it also contains the name of the host.
Nothing regarding filename properties would change even if I introduced the change mentioned in my last post.
Ok, I understand you mean a new tag so we have a unique id for any item that is included in the albums. Maybe this would better solve my dupe issue from above. When that is easy to implement, I'd say let's try.

Quote:
Originally Posted by pspzockerscene View Post
AfaIk no but it looks like it wouldn't be much work to add generic support for plugin specific properties.
Jiaz may also add his opinion on that.

-psp-
Yes, that would be very helpful. Content_id probably the most useful tag but the other ones too. It would make so many tasks so much easier. And this is not necessarily limited to Flickr. I could imagine the unique YouTube íd would be very useful too, if we could have access to this in this dialog.
Reply With Quote
  #74  
Old 06.10.2021, 10:41
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

@pspzockerscene: contact me
__________________
JD-Dev & Server-Admin
Reply With Quote
  #75  
Old 06.10.2021, 16:13
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,920
Default

Quote:
Originally Posted by EquiNox View Post
Hmm, ok I have tested again the dupe setting and it does not seem to work at all with just one time parsing. When parsing again you get dupes but not always the way I wanted them. I need to test this further.
Works fine here.

Quote:
Originally Posted by EquiNox View Post
Ok, I understand you mean a new tag so we have a unique id for any item that is included in the albums.
No I was just trying to explain how our internal IDs for dupe-checking work/can work.

Quote:
Originally Posted by EquiNox View Post
And this is not necessarily limited to Flickr. I could imagine the unique YouTube íd would be very useful too, if we could have access to this in this dialog.
If implemented, you could access all plugin properties the same way you can in Packagizer:
https://support.jdownloader.org/Know...the-packagizer
--> See:
Code:
<jd:prop:KEY>
-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #76  
Old 06.10.2021, 17:48
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,920
Default

Copying Downloadlink-specific properties via "copy information" will be available after the next CORE-update.
While the pattern may look a bit different, the properties/keys (such as "username_url") themselves are 1:1 the same as in Packagizer and (in this case) in the plugin filename settings.

I haven't done anything regarding the dupe-check behavior.
Just let me know your thoughts on that.
I get that it actually would make sense that, even by default to allow duplicates as long as they were added in a different context...but then again on the other side it makes no sense because e.g. whats the difference between photo galleries and a user e.g. trying to add the same link from two differen other sources such as forum-posts.
...so either a complete new global setting for "fine tuning" would be needed so that we can basically set a "unique ID" and a "unique context ID" on every link...or just stick with the global dupe-check setting we already have.
I still consider this and edge cases/rare issue but I do understand your remarks on this topic.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 06.10.2021 at 17:52.
Reply With Quote
  #77  
Old 07.10.2021, 08:52
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

Quote:
Originally Posted by pspzockerscene View Post
Quote:
Originally Posted by EquiNox View Post
Hmm, ok I have tested again the dupe setting and it does not seem to work at all with just one time parsing. When parsing again you get dupes but not always the way I wanted them. I need to test this further.
Works fine here.
Ok, can you tell me how you get there?

Maybe we should straigthen this out thoroughly because it's still not working for me the way I'd like it to be working.

First of all a question about the "Advanced Settings", I filtered them with "dupe" and get these settings (among a few others):

Code:
1. GeneralSettings: Dupe Manager                                 (Default: true, activated)
2. LinkCollector: Dupe Manager                                   (Default: true, activated)
3. LinkgrabberSettings: Default On Added Dupes Links Action      (Default: ask me every time)
4. LinkgrabberSettings: Handle Dupes On Confirm Latest Selection (Default: add them, too)
The only I changed here is deactivating option No. 2 "LinkCollector: Dupe Manager" (and restart JD), as I think you suggested. What is the difference to option No. 1 "GeneralSettings: Dupe Manager"? I kept the default for that one.


Now I tried to parse again this stream (Download and LinkGrabber list empty):
Code:
**External links are only visible to Support Staff**

My custom filename for this:
Code:
Flickr-*date_taken*_*title*_*username_url*_*content_id*_*set_id*_*quality**extension*

After one time parsing this is what I get in total (Group single files in a 'various package' deactivated):
Code:
6997 links, 280 packages

Now I counted the number of items and albums on the pages manually together and this is what I get,
when all albums have the exact number of items stated on the site:
Code:
8552 items, 288 albums

As said before there are more items in all albums because some items are part of more than one album.


When I parse the albums page a second time this is what I get in total:
Code:
7964 links, 286 packages

When I parse the albums page a third time this is what I get in total:
Code:
8245 links, 286 packages

When I now empty the LinkGrabber and do the same thing again, I get different results but I never get exactly 8552 links in 288 packages as I wanted them. The "Parse Clipboard" bubble always (or in most cases) says "Found link(s): 6997" which is of course the correct number of unique links in the albums but it does not parse the dupes correctly when I have deactivated the option in "Advanced Settings". Dupes show up but often just in the same album again the link was already part of before. Those won't be downloaded because they have exactly the same filename. What am I doing wrong here?

I have added an attachment with a list of all the albums and the number of items in there.

I'd just like to be able to download these albums exactly the way they are listed at Flickr. The main issue for me here is that I want to have the complete model albums and not any gaps in those when those items are already part of another fav-, camera- or whatever-album. So the best would be just to get all albums complete. When you want to remove duplicates you can decide afterwards which album you want to remove them from.
Attached Files
File Type: txt 134430854@N03-288-albums.txt (13.3 KB, 1 views)

Last edited by EquiNox; 07.10.2021 at 13:16.
Reply With Quote
  #78  
Old 07.10.2021, 08:56
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

Quote:
Originally Posted by pspzockerscene View Post
Copying Downloadlink-specific properties via "copy information" will be available after the next CORE-update.
While the pattern may look a bit different, the properties/keys (such as "username_url") themselves are 1:1 the same as in Packagizer and (in this case) in the plugin filename settings.
Looking forward to this new update. That sound great! Many thanks!

Quote:
Originally Posted by pspzockerscene View Post
I haven't done anything regarding the dupe-check behavior.
Just let me know your thoughts on that.
I get that it actually would make sense that, even by default to allow duplicates as long as they were added in a different context...but then again on the other side it makes no sense because e.g. whats the difference between photo galleries and a user e.g. trying to add the same link from two differen other sources such as forum-posts.
...so either a complete new global setting for "fine tuning" would be needed so that we can basically set a "unique ID" and a "unique context ID" on every link...or just stick with the global dupe-check setting we already have.
I still consider this and edge cases/rare issue but I do understand your remarks on this topic.
Ok, when the dupe check or deactivated dupe check would be working as I described in my previous post, then all would be fine for me. It only happens now and then, not too often.
Reply With Quote
  #79  
Old 07.10.2021, 13:20
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

Quote:
Originally Posted by EquiNox View Post
I never get exactly 8552 links in 288 packages as I wanted them.
Ok, this is probably not entirely correct. When albums have the same name (this is allowed at Flickr), they are simply merged in one package in LinkGrabber. That could also have happened here. Still it would be nice if I just get the exact number of links like it is stated on the site.

Of course it is possible to get this done, even when you have activated dup check. When you just parse and download the albums one-by-one. But that's a lot of work.

Last edited by EquiNox; 08.10.2021 at 08:59.
Reply With Quote
  #80  
Old 07.10.2021, 14:13
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,920
Default

First response:
This one is only about adding dupes and the configurable behavior:

Quote:
Originally Posted by EquiNox View Post
Maybe we should straigthen this out thoroughly because it's still not working for me the way I'd like it to be working.
Without replying to all of your further tests, I'll try to add an explanation:
First of all, to simplify tests, I'm using an URL to a single album here:
flickr.com/photos/derodeolifant/albums/72157635508464613
--> Contains 37 items
Reason for this:
Adding thousands of items for testing the dupes behavior simply takes too much time so it doesn't make sense

Keep in mind that the first two dupes settings need a JDownloader restart to become active!

First:
Code:
LinkCollector.dupemanagerenabled
Enabled = Do not allow duplicated items in linkgrabber
--> If you add the above album twice, you will only get all items once.
Disabled = allow duplicated items in linkgrabber
--> If you add the above item twice, resulting in a single package containing 74 URLs. If you add it again and again, that package will get bigger each time.

I've added a setting description for this setting for the next CORE-update.

Now to:
Code:
GeneralSettings.dupemanagerenabled
This does exactly what the settings' description says:
Quote:
If Enabled, the linkgrabber will detect links that are already in the downloadlist
--> You can leave this enabled as it won't stop you from adding duplicates but it will still visually let you know if there is stuff in the linkgrabber that is already in your downloadlist.

Now to the less important stuff:
Code:
LinkgrabberSettings: Default On Added Dupes Links Action
This basically lets you define "what to do when trying to add items from the linkgrabber to your downloadlist that are already in your downloadlist.
If you want JD to shut up and completely don't care about this, set it to "add them, too".

I've added a setting description for this setting for the next CORE-update.

Then to:
Code:
LinkgrabberSettings.handledupesonconfirmlatestselection
This is basically a fine-tuning setting for the last described setting,
Code:
LinkgrabberSettings: Default On Added Dupes Links Action
--> Use this to define the default selection in the "what do you want to do with these duplicates" dialog.
If you disabled the dialog and e.g. always allow JD to add dupes, this last setting is of course absolutely irrelevant for you.

I've added a setting description for this setting for the next CORE-update.

Now I've completely read your post but at this moment there is no point for me to go into details regarding your test results.
When adding many items there is a lot of stuff that could go wrong such hitting one of flickrs rate-limits so better test with one link and whenever you experience "missing items" we can check it and see if we maybe have to slow down our crawler to avoid hitting such rate-limits...though this is just an assumption here.
Until now I've never hit any of their rate-limits during testing.

Conclusion:
All you need to do is to disable:
Code:
LinkCollector.dupemanagerenabled
...and restart JD.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 18:15.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.