JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 06.09.2021, 19:58
poowangdee poowangdee is offline
Baby Loader
 
Join Date: Sep 2021
Posts: 6
Default Flickr.com can't scan with clipboard observer

today i try to download pic,but when i copy link around 20 link it only show 1 file,or when i try copy user link that usually grab all file of that id, it only grab 1 file. how can i fix it?
Reply With Quote
  #2  
Old 06.09.2021, 20:12
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Please provide example links,then we can check/fix it
__________________
JD-Dev & Server-Admin
Reply With Quote
  #3  
Old 06.09.2021, 20:19
poowangdee poowangdee is offline
Baby Loader
 
Join Date: Sep 2021
Posts: 6
Default

Quote:
Originally Posted by Jiaz View Post
Please provide example links,then we can check/fix it
for me it happen all link,
example
**External links are only visible to Support Staff****External links are only visible to Support Staff**
usually if i copy this link to clipboard it should detect around 472 pic(all of her picture)
(i only random user for example)
https://imgur.com/a/zVXFJ2B
Reply With Quote
  #4  
Old 06.09.2021, 20:38
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

I can reproduce the issue,working on it
__________________
JD-Dev & Server-Admin
Reply With Quote
  #5  
Old 06.09.2021, 20:48
poowangdee poowangdee is offline
Baby Loader
 
Join Date: Sep 2021
Posts: 6
Default

Quote:
Originally Posted by Jiaz View Post
I can reproduce the issue,working on it
many many Thanks.
Reply With Quote
  #6  
Old 06.09.2021, 21:09
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

do you prefer the real username (from title) or from url (anan5211) ? within your filenames?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #7  
Old 06.09.2021, 22:00
poowangdee poowangdee is offline
Baby Loader
 
Join Date: Sep 2021
Posts: 6
Default

Quote:
Originally Posted by Jiaz View Post
do you prefer the real username (from title) or from url (anan5211) ? within your filenames?
i use photo title on file name(i always change folder name to "realname - url - date(that i update this folder)")
for this example it will be like this pic
thanks.
https://imgur.com/a/a8ADDAz

**file name will be this rule for me
*date*-*photo_id*-*title**extension*
Reply With Quote
  #8  
Old 07.09.2021, 16:18
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@poowangdee: Thanks for the feedback. pspzocker is currently working on some more refactoring on this plugin
__________________
JD-Dev & Server-Admin
Reply With Quote
  #9  
Old 07.09.2021, 20:36
poowangdee poowangdee is offline
Baby Loader
 
Join Date: Sep 2021
Posts: 6
Default

Quote:
Originally Posted by Jiaz View Post
@poowangdee: Thanks for the feedback. pspzocker is currently working on some more refactoring on this plugin
thanks
i hope it will working soon.
Reply With Quote
  #10  
Old 08.09.2021, 10:42
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@poowangdee: update is available. Please update/restart and test. In case something doesn't work, please report back and provide example links
__________________
JD-Dev & Server-Admin
Reply With Quote
  #11  
Old 08.09.2021, 13:53
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

Hello,

first of all thank you very much for fixing the latest issue with Flickr, a day ago I too noticed that streams or albums were not anymore correctly parsed by JD, just one link showed up in the LinkGrabber. The latest update(s) have fixed this issue successfully.

However now I noticed another issue, that single-id-links from Flickr are not anymore parsed correctly and it's not possible to download those links ("Try restarting this link (plugin outdated?)"). Sometimes I do not want to download the entire stream but just some single links, copy-pasting and first collect them with the LinkGrabber.

Example link:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

Besides that I noticed another specific issue with album names from Flickr. In most cases the albums are correctly parsed and the right names show up in the LinkGrabber. Special characters that some file systems do not support are replaced by an underscore, which is totally fine. However when an album name at Flickr contains double quotes (") then the LinkGrabber does not show the correct name but converts the name into something like this: "flickr com set 123 of user xyz". 123 is the album-id. In this case I always have to fix the album name manually. It would be nice if the double quotes would also automatically be converted, maybe into apostrophes (') like it is done to the Flickr file names.

Example link:
**External links are only visible to Support Staff****External links are only visible to Support Staff**


Currently using JD, Revision: 44953

Last edited by EquiNox; 08.09.2021 at 13:56.
Reply With Quote
  #12  
Old 08.09.2021, 15:07
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@EquiNox: Thanks for the feedback, pspzockerscene is currently working on the plugin and will check/fix those issues as well
__________________
JD-Dev & Server-Admin
Reply With Quote
  #13  
Old 08.09.2021, 16:51
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Hi EquiNox,
both of these issues will be addressed with the next flickr.com update along with many other improvements.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #14  
Old 08.09.2021, 18:37
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Update:
I'm not yet done with my tests so this will have to wait until tomorrow...

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #15  
Old 08.09.2021, 21:49
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

Many thanks!

When the update is ready I'll let you know if everything worked out fine.
Reply With Quote
  #16  
Old 09.09.2021, 14:46
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Changelog for the upcoming flickr plugin upate:
- Faster crawling and faster linkcheck
- Added username, groupname and apikey caches
- Added different fields for username, usernameFull, usernameInternal
- Updated custom filename tags
- Added new custom filename tags: username, username_internal, username_full, quality
- Fixed issues with single photo/video URLs
- Removed duplicated code in hostplugin
- Started work on setting for preferred image quality

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #17  
Old 09.09.2021, 17:28
poowangdee poowangdee is offline
Baby Loader
 
Join Date: Sep 2021
Posts: 6
Default

thanks a lot,it's work great now.
Reply With Quote
  #18  
Old 09.09.2021, 18:36
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Thanks for your feedback.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #19  
Old 10.09.2021, 00:12
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

Ok, that was a big update for the Flickr-plugin. And thank you very much for fixing the single link and double quotes issue!

The custom definition that poowangdee used above works indeed totally fine. However I have now tested several of the possible new definitions with different Flickr sites and noticed some issues.

These are all the definitions:

Code:
*content_id*
*username*
*username_internal*
*username_full*
*date*
*title*
*extension* 
*quality*
*order_id*
This should be the definition that I use for my own preferred custom filename:
Code:
Flickr-*title*_*username*_*content_id**extension*

I think the main issue here is that *username_internal* does not work at all. It never returns the internal username id like 12345678@N04 for example. When using this tag it either returns the string defined under "Char which will be used for empty tags (e.g. missing data)" or when nothing is defined there it simply returns a dash (-).

In case a Flickr site has a real username in the url like...
Code:
**External links are only visible to Support Staff**
...everything is still ok I'd say.

Example link:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

However when a Flickr site does not have such name but only the internal username in the url like...
Code:
**External links are only visible to Support Staff**
...*username* and *username_internal* only return the dash (-).

Example link:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

The preferred scenario would be if there is a "real-username" then *username*=real-username. When there is only the "internal-username" (12345678@N04) then *username_internal*=internal-username and *username*=internal-username also. Hope that was not too confusing. I think that way it worked before with the Flickr plugin.


For the other tags
Code:
*content_id*        -> works fine
*username_full*     -> works fine
*date*              -> works fine
*title*             -> works fine
*extension*         -> works fine
*quality*           -> works fine

Code:
*order_id*
Does return negative values?


I just noticed the single content id parsing still does not work correctly. *date* does always return "1970-01-01" as value, the value for *title* is gone and the *username_full* tag is messed up somehow and does not return the same value as the same id would return within a stream.

Example Link:
**External links are only visible to Support Staff****External links are only visible to Support Staff**


One another issue I'd like to mention. When you add a constant string to the custom file name definition like I did myself...
Code:
Flickr-*title*_*username*_*content_id**extension*
...then all characters are automatically lowercased, so the prefix "Flickr-" becomes "flickr-".
Would it be possible to switch off this conversion?
Reply With Quote
  #20  
Old 10.09.2021, 08:00
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

As an addition to my last post I guess I need some clarification what *username_full* should actually be.

Here is an example link:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

And here is a screenshot from the top of that Flickr page:
https://postimg.cc/XZDHMwwT

Is *username_full* the big headline on the top? Trevor Dobson here.
Or the small subtitle below next to the Pro logo? inefekt69 here.

I was always a bit confused myself about this as Flickr does use different headlines for some views of their pages.

For example when you open the link from above in the archives-date-taken view:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

Then the big headline is gone and just the subtitle from the main url above is displayed.
Reply With Quote
  #21  
Old 10.09.2021, 14:41
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

@EquiNox
Thanks for your detailed feedback!

Changelog for the next flickr plugin update:
- Fixed missing tags "title" and "username_internal" for single added items
- Added new tag "date_taken"
- Fixed "order_id" tag delivering negative values
- Added new superfast linkcheck; server-filenames and "quality" tag is available during crawling right away
- Fixed issues with "date" tag and fixed recognization of missing date to use "-" instead then
- Fixed lowercase filename format string

You will have to either reset- or delete- and re-add all items with corrupt filenames.
For corrupt "order_id" properties, you will need to re-add such URLs in context (e.g. user-profiles/galleries) to get the correct "order_id" properties.

Regarding "username_full" and all of the other "username" tags:
The example I put in our flickr plugin settings should make it clear already.
username_full = "User Name" (not always available)
username = "username1999example"
username_internal = "12345678@N04"

I'm sorry I cannot access the last URL of your last post as it leads to mature content and/or a flickr.com account is required.

Regarding flickr account support:
I'll work on this once all the other stuff is done.

For all mentioned changes in this post, the following information applies:
Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 10.09.2021 at 14:42. Reason: Improved formatting
Reply With Quote
  #22  
Old 10.09.2021, 17:44
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

More changes:
- Fixed 99% of all missing images (mature content as part of sets/galleries/favorites); either flickr itself is boggy or there is still some work left
- Added another tag "real_name": if given, this resembles the full name of the uploader (name and surname)

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
EDIT

I'm actually a bit confused regarding the different "usernames".

Example:
flickr.com/photos/louobedlam/
API:
pathalias: louobedlam --> *username*
realname: Lou Noble --> *username_full*
ownername: Lou O' Bedlam --> *real_name*

Website:
pathAlias: louobedlam --> *username*
realname: Lou Noble --> *username_full*
username: Lou O' Bedlam --> *real_name*

I guess "username" and "username_internal" are correct but for the rest, website and API are using different fields.
Is the current handling fine or do I need to swap the data that goes into "username_full" and "real_name"?
I'am not a flickr user...
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 10.09.2021 at 17:57.
Reply With Quote
  #23  
Old 10.09.2021, 23:06
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

Thanks for the quick new update, pspzockerscene!

Not yet time to test it, but I'll do and let you know how it worked.

Quote:
Originally Posted by pspzockerscene View Post
I'm sorry I cannot access the last URL of your last post as it leads to mature content and/or a flickr.com account is required.
I see, had not paid attention to it before but it looks like you need to be logged in at Flickr to see the /archives/date-taken/ view. It is a chronological view where you can see which photos were taken which year, month and day. There is also another chronological view at Flickr, that works the same way but does not show when the pictures were taken but when they were posted (/archives/date-posted/).

What I wanted to say is that Flickr does not always use the same usernames for their different views. It's a bit confusing I'd say.

The regular Flickr view:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Screenshot:
https://postimg.cc/XZDHMwwT

The /archives/date-taken/ view:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Screenshot:
https://postimg.cc/gxD0z92w
Reply With Quote
  #24  
Old 13.09.2021, 08:16
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

First of all this was indeed a big quality update for the Flickr-plugin with so many new options but also improvements under the surface! Thank you very much pspzockerscene!

The biggest improvement imo, pictures downloaded with JD now should always contain the EXIF-tag when they were uploaded that way at the Flickr page. In the past I wasn't sure why maybe 10% of all pictures downloaded from Flickr were missing the EXIF-tag, although Flickr showed the date taken on their page. Before I thought this was an issue with Flickr but after now all these pictures obviously can be fixed afterwards with a re-download I am almost certain it was an issue with JD. The EXIF-tag is very useful especially for using the original-date-taken stored there by the camera, for chronologically sorting pictures with tools like EXIFtool for example.

A few issues however that could be improved I'd like to mention.

1. First of all the biggest flaw I found, that videos in Flickr-streams will never be downloaded when you parse the stream. You'll just get a jpg-preview of the video instead. However when you copy-paste the single link of the video, then the video file shows up in the LinkGrabber and can be downloaded correctly.

As an example you can use the Flickr page that you used yourself above.
Parse the stream and you don't get any video link in LinkGrabber, just the corresponding jpg-preview for the video:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

However there are 3 videos in this stream:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

Copy paste them singly and the videos show up in the LinkGrabber:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

2. There is another issue with some videos which however more seems to be a Flickr issue and which did not just occur with the new updates. It applies mostly to videos 360p or smaller, but some bigger ones as well.

Some random examples:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

These show up in the LinkGrabber as 'offline'. Trying to download these directly from Flickr mostly returns this error message:

Code:
That 'size' is missing.
404
Unknown:
JSON | PHP | MsgPack
{
    "code": 404,
    "message": "That 'size' is missing."
}
However when you play the video it's possible to save it manually via right click on the video. Would it be possible to get JD also downloading these?


Quote:
Originally Posted by pspzockerscene View Post
- Fixed 99% of all missing images (mature content as part of sets/galleries/favorites); either flickr itself is boggy or there is still some work left
I saw that already works very well. I did notice myself there can always be a certain discrepancy between the amount of pictures stated on the user front page at Flickr and the amount that actually shows up in the stream in the end. And I am not talking about any mature content that is not visible for JD since it does not yet have a working Flickr account support. In most these cases there is however only a difference of a few pictures. In this case I use the /archives/date-taken/ view to crosscheck because in most cases that shows the correct number of items in a stream.


Quote:
Originally Posted by pspzockerscene View Post
-psp-
EDIT

I'm actually a bit confused regarding the different "usernames".

Example:
flickr.com/photos/louobedlam/
API:
pathalias: louobedlam --> *username*
realname: Lou Noble --> *username_full*
ownername: Lou O' Bedlam --> *real_name*

Website:
pathAlias: louobedlam --> *username*
realname: Lou Noble --> *username_full*
username: Lou O' Bedlam --> *real_name*

I guess "username" and "username_internal" are correct but for the rest, website and API are using different fields.
Is the current handling fine or do I need to swap the data that goes into "username_full" and "real_name"?
I'am not a flickr user...
I am confused myself with all the different usernames at Flickr. For the plugin I think *username_full* and *real_name* already works fine. *username_full* the big headline on the top of the front page of the Flickr user and *real_name* the name stated below in the second row. I think we can leave it that way.

3. The username tags already work very well. However there is one change that I'd appreciate since I think the Flickr plugin worked already before that way and it would make updating streams much easier as I would not constantly have to toggle between *username* and *username_internal* for the custom filename. I think we would have a better backward compatibility to the old *username* tag, that was used for many years already.

The problem is that with the new update *username* does not always have a value anymore, only in case there is a pathalias available like you called it above. The change that I'd suggest is that in case there is no pathalias *username* would simply get the same value like *username_internal*. Maybe a simple way to implement this is just using the Flickr url to extract the value for *username*. Then you can decide yourself what username you want to use when you want to download from Flickr with JD. It just depends on what you copy-pasted, either the pathalias url or the internal username url.

To make it more clear taking your example from above:
Lou Noble - Lou O' Bedlam - louobedlam - 49229574@N00, you can open this site in two ways, either by using the pathalias or the internal username url. With the change that I suggest *username* would get different values when you copy paste a link for the LinkGrabber to parse:
Code:
**External links are only visible to Support Staff**
**External links are only visible to Support Staff**
This implementation would also easily fix the issue with *username* being empty in case there is no pathalias. Naming the tag *username_url* would probably make more sense in this case.

Example:
Irina Dimulescu - Irina1010 - no pathalias - 100184150@N02, there is only one way to open this Flickr site:
Code:
**External links are only visible to Support Staff**
I do not believe that this implementation would have any harmful interdependencies. Of course another way to implement this would be in case there is no pathalias to simply set *username* = *username_internal*, however this would not be as flexible as the suggestion above.


4. There was another issue with single link parsing that I have seen, unicode characters in file names are replaced by cryptic codes.

Example:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

Single link parsing returns this cryptic file name (custom filename: Flickr-*title*_*username*_*content_id**extension*):
Code:
Flickr-%u6417%u86CB%u6417%u86CB_anan5211_5067725471.jpg
The correct file name when parsing the stream:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Code:
Flickr-搗蛋搗蛋_anan5211_5067725471.jpg
This unicode-bug seems to apply to any tag of the custom filename when parsing a single Flickr link.

Here is an example where *title* and *username_full* in the filename contain cryptic code:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

When you parse the album the link above is part of it's all fine again:
**External links are only visible to Support Staff****External links are only visible to Support Staff**


Quote:
Originally Posted by pspzockerscene View Post
Regarding flickr account support:
I'll work on this once all the other stuff is done.
@pspzockerscene:
That would of course make more things much easier, although it's already very powerful plugin especially with the latest updates. Thanks again!


I guess that's it for the moment when there are any other issues I will run across, I'll let you know.
Reply With Quote
  #25  
Old 13.09.2021, 18:56
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Quote:
Originally Posted by EquiNox View Post
I see, had not paid attention to it before but it looks like you need to be logged in at Flickr to see the /archives/date-taken/ view.
Doesn't matter.
JD will find that property even without flickr account.

Quote:
Originally Posted by EquiNox View Post
What I wanted to say is that Flickr does not always use the same usernames for their different views. It's a bit confusing I'd say.
Doesn't matter either.
The fact is that there seem to be 4 usernames in total.
As long as they're all consistently/correctly found- and set via plugin it does not matter which one us used by flickr at which place as you can use whichever you want inside JDs filenames.

Quote:
Originally Posted by EquiNox View Post
First of all this was indeed a big quality update for the Flickr-plugin with so many new options but also improvements under the surface! Thank you very much pspzockerscene!
You're welcome.

Quote:
Originally Posted by EquiNox View Post
The biggest improvement imo, pictures downloaded with JD now should always contain the EXIF-tag when they were uploaded that way at the Flickr page. In the past I wasn't sure why maybe 10% of all pictures downloaded from Flickr were missing the EXIF-tag, although Flickr showed the date taken on their page. Before I thought this was an issue with Flickr but after now all these pictures obviously can be fixed afterwards with a re-download I am almost certain it was an issue with JD. The EXIF-tag is very useful especially for using the original-date-taken stored there by the camera, for chronologically sorting pictures with tools like EXIFtool for example.
It was not a JD issue.
JD cannot(yet) write EXIF data.
All data it downloads is downloaded as it is served via the server (in this case via the flickr.com CDN) so there must have been a serverside change...or the place our plugin is getting the downloadlinks from is different and leads to the pictures with EXIF data although I've always only been able to find one unique downloadurl per image-quality.
Mabe EXIF data is only available for images which can be officially downloaded (quality == "ORIGINAL" (/o/))?

Quote:
Originally Posted by EquiNox View Post
1. First of all the biggest flaw I found, that videos in Flickr-streams will never be downloaded when you parse the stream.
Fixed.

Quote:
Originally Posted by EquiNox View Post
2. There is another issue with some videos which however more seems to be a Flickr issue and which did not just occur with the new updates. It applies mostly to videos 360p or smaller, but some bigger ones as well.
Fixed.

Quote:
Originally Posted by EquiNox View Post
However when you play the video it's possible to save it manually via right click on the video. Would it be possible to get JD also downloading these?
Yes.

Quote:
Originally Posted by EquiNox View Post
I saw that already works very well. I did notice myself there can always be a certain discrepancy between the amount of pictures stated on the user front page at Flickr and the amount that actually shows up in the stream in the end. And I am not talking about any mature content that is not visible for JD since it does not yet have a working Flickr account support. In most these cases there is however only a difference of a few pictures. In this case I use the /archives/date-taken/ view to crosscheck because in most cases that shows the correct number of items in a stream.
Do you have example URLs for that?
As said I'll do some more tests but afaik the problem is not the mature content (anymore).
Also, are there URLs which require login but are not mature?
E.g. private photos of other users which are only unlocked for you or other specified users/friends?
If so, I'd need to get some examples to work on/check login support.

Quote:
Originally Posted by EquiNox View Post
3. The username tags already work very well. However there is one change that I'd appreciate since I think the Flickr plugin worked already before that way and it would make updating streams much easier as I would not constantly have to toggle between *username* and *username_internal* for the custom filename. I think we would have a better backward compatibility to the old *username* tag, that was used for many years already.
Please keep in mind that the URLs you add may either contain the value of "username_internal" or "username".
To add the ability to use this inconsistent website behavior in JD too, I've added a new tag "username_url" that will return the username used inside the URL.

Quote:
Originally Posted by EquiNox View Post
4. There was another issue with single link parsing that I have seen, unicode characters in file names are replaced by cryptic codes.
I'll check this soon.

Quote:
Originally Posted by EquiNox View Post
That would of course make more things much easier, although it's already very powerful plugin especially with the latest updates. Thanks again!
As mentioned above, at this moment I've yet to find a reason to use flickr accounts in JD.
If there is one, simply removing account support might also be an option

For all code changes mentioned in this post, the following applies:

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #26  
Old 14.09.2021, 08:51
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

Thanks again for the quick fix and update! Much appreciated!

One issue I noticed already now, parsing albums does not always work anymore.
LinkGrabber only returns "Plugin Defect!flickr.com" or "Plugin Defect!<album-id>".
I am not sure but I think it was not that way before the update.

Example:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

I'll reply to your last post properly later.
Reply With Quote
  #27  
Old 14.09.2021, 12:17
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

@EquiNox: please provide a debug log. Enable Settings->Advanced Settings->Log.debugmodeenabled
restart JDownloader, try to add the link/wait for error and then create log, see https://support.jdownloader.org/Know...d-session-logs
and post shown logID here
__________________
JD-Dev & Server-Admin
Reply With Quote
  #28  
Old 14.09.2021, 14:08
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

@EquiNox
Fixed!
The encoding problem you've reported has been fixed too (thx to Jiaz).
Also please provide some more example URLs for videos, especially higher quality ones.
I want to find out if it makes sense to add a "Preferred quality" option for videos too...

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 15.09.2021 at 13:27. Reason: Fixed typo
Reply With Quote
  #29  
Old 14.09.2021, 19:32
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

@EquiNox
I'm done with my "missing images" tests and so far it looks like a flickr bug.
You seem to be into the flickr system more than I'am so I've send you an URL to a public users favorite gallery.
Number of items I get: 1301
Expected number of items: 1306

I have no idea where those 5 missing images are supposed to be and/or how to get them.
I did all of my tests without a flickr account.

In our plugin I fetch up to 500 items per page while the flickr website only gets 25.
With 500 images per page, 3 of those 5 images are missing on page 2.
I did actually add an extra logger for that:
Code:
249|flickr.com_jd.plugins.decrypter.FlickrCom.log 14.09.21, 18:29:42 - INFO [ jd.plugins.decrypter.FlickrCom(api_handleAPI) ] -> There is probably hidden mature content present? Found only 497 of max 500 items on page 2 although we're not yet on the last page
So via website these images will be missing on another page...but even if we knew the page that wouldn't tell us which ones are missing or why.
During my tests I even used the same "extras" parameter their website is using but the result was the same.
That leads me to the final conclusion that this is a flickr bug.

Maybe we can solve that mystery together.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #30  
Old 14.09.2021, 20:24
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

Thank you very much pspzockerscene!

A quick test showed the album parsing seems to work again.

Quote:
Originally Posted by pspzockerscene View Post
@EquiNox
Fixed!
The encoding problem you've reported has been fixed too (thx to Jiaz).
We talk about the unicode bug for single link parsing? Are you sure it's fixed?
I just tried this example link and it still returns cryptic code:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

The file name I get in the LinkGrabber:
Code:
Flickr-20190421%20-%20%u6E05%u5FE0%u76C3%uFF08%u8A31%u8ED2%u8C6A%u62CD%u651D%uFF09_jj-wng_48336881542.jpg
The file name like it should be (you get when you parse the entire stream or the album):
Code:
Flickr-20190421 - 清忠盃(許軒豪拍攝)_jj-wng_48336881542.jpg
Quote:
Originally Posted by pspzockerscene View Post
Also please provide some more example URLs for videos, especially highe quality ones.
I want to find out if it makes sense to add a "Preferred quality" option for videos too...
Ok, I'll have a look and try to put a list together.

Quote:
Originally Posted by pspzockerscene View Post
@EquiNox
I'm done with my "missing images" tests and so far it looks like a flickr bug.
You seem to be into the flickr system more than I'am so I've send you an URL to a public users favorite gallery.
Number of items I get: 1301
Expected number of items: 1306

I have no idea where those 5 missing images are supposed to be and/or how to get them.
I did all of my tests without a flickr account.
Ok, I get 1301 too, but I'll look for those 5 items you are missing, it should be possible to get them.

Last edited by EquiNox; 14.09.2021 at 20:43.
Reply With Quote
  #31  
Old 15.09.2021, 08:18
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

Quote:
Originally Posted by pspzockerscene View Post
@EquiNox
I'm done with my "missing images" tests and so far it looks like a flickr bug.
You seem to be into the flickr system more than I'am so I've send you an URL to a public users favorite gallery.
Number of items I get: 1301
Expected number of items: 1306

I have no idea where those 5 missing images are supposed to be and/or how to get them.
I did all of my tests without a flickr account.

In our plugin I fetch up to 500 items per page while the flickr website only gets 25.
With 500 images per page, 3 of those 5 images are missing on page 2.
I did actually add an extra logger for that:
Code:
249|flickr.com_jd.plugins.decrypter.FlickrCom.log 14.09.21, 18:29:42 - INFO [ jd.plugins.decrypter.FlickrCom(api_handleAPI) ] -> There is probably hidden mature content present? Found only 497 of max 500 items on page 2 although we're not yet on the last page
So via website these images will be missing on another page...but even if we knew the page that wouldn't tell us which ones are missing or why.
During my tests I even used the same "extras" parameter their website is using but the result was the same.
That leads me to the final conclusion that this is a flickr bug.

Maybe we can solve that mystery together.

-psp-
Ok, let me now respond to this issue.

I just checked this stream myself with an account and getting the same results. 1301 items. There are 100 items per favorite page. Page 10 has only 97 items, page 12 99 items and page 13 also 99 items, so 5 are missing in total assuming that every page should have 100 items.

I did some tests myself with favoring family + friend content that was not publicly available, checking if there was a gap on a page when you call that favorite page without an account. But no, the (non visible) item was missing on the page, the page was however counting exactly 100 items, like it should be. Also when content that you have favored is deleted or made invisible, it should disappear in your favorite stream but the number of items per page (100) should not change.

Then I had a look at my own favorite stream where I have accumulated items over serveral years and noticed it contained the same bug, I hadn't noticed myself before or paid attention to it. At the time I added them all these favorites were publicly available with an account, no hidden f+f content. On page 5 there were only 98 items. So then I added another ~100 items to my favorites for testing purposes and the 98 items were shifted towards page 6. So in fact it looks like there are still orphaned entries in the Flickr database that have not been removed correctly. It seems to be a Flickr bug to me too.

I'll keep an eye on this issue and look if it happens again and maybe can narrow down better in which situations it does happen.
Reply With Quote
  #32  
Old 15.09.2021, 08:28
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

Quote:
Originally Posted by pspzockerscene View Post
Also please provide some more example URLs for videos, especially highe quality ones.
I want to find out if it makes sense to add a "Preferred quality" option for videos too...
This is easily done you can look yourself for many videos on this search page:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

I have restricted the search for videos between 2020 and now. I guess there are many hd videos in there too.
Reply With Quote
  #33  
Old 15.09.2021, 12:22
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by EquiNox View Post
The file name I get in the LinkGrabber:
Code:
Flickr-20190421%20-%20清忠盃(許軒豪拍攝)_jj-wng_48336881542.jpg
The file name like it should be (you get when you parse the entire stream or the album):
Code:
Flickr-20190421 - 清忠盃(許軒豪拍攝)_jj-wng_48336881542.jpg
will be fixed (for new added links only) with next plugin update
__________________
JD-Dev & Server-Admin
Reply With Quote
  #34  
Old 15.09.2021, 13:33
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Quote:
Originally Posted by EquiNox View Post
I did some tests myself with favoring family + friend content that was not publicly available, checking if there was a gap on a page when you call that favorite page without an account. But no, the (non visible) item was missing on the page, the page was however counting exactly 100 it
Thanks for checking this.
If in the end you are sure it is a flickr bug, you may report that to their staff although it's nothing major.
Some applications using pagination may stop if a page contains less than max items so that's the worst that this bug can do.

Next update:
flickr video support:
- Added "preferred quality" selection along with correctly filling "quality" tag
- Enabled faster linkcheck for videos too
- Allow resume and chunkload for videos

I purposely left out "240p" which is listed internally as "700" and is pretty much the only available quality for old videos.
Same as for pictures:
If the selected quality is not found, the "best" will be used instead.
For videos added before this update, the quality tag may just return "original".
This is auto-corrected when adding them again.

Also, last update updated filename patterns for all users who were still using the default.
New default is:
Code:
*username_url*_*content_id*_*title**extension*
...which basically creates exactly the same filenames used before I started to work on these huge changes on the flickr plugin.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #35  
Old 16.09.2021, 07:54
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

Quote:
Originally Posted by pspzockerscene View Post
If in the end you are sure it is a flickr bug, you may report that to their staff although it's nothing major.
Some applications using pagination may stop if a page contains less than max items so that's the worst that this bug can do.
Reconsidering, this could also be the explanation for those discrepancies I was talking about, number of items stated on the user front page and number of items JD is getting in the end. I had not seen this before but now it obviously makes sense. We talked about it before...

Quote:
Originally Posted by pspzockerscene View Post
Quote:
Originally Posted by EquiNox View Post
I saw that already works very well. I did notice myself there can always be a certain discrepancy between the amount of pictures stated on the user front page at Flickr and the amount that actually shows up in the stream in the end. And I am not talking about any mature content that is not visible for JD since it does not yet have a working Flickr account support. In most these cases there is however only a difference of a few pictures. In this case I use the /archives/date-taken/ view to crosscheck because in most cases that shows the correct number of items in a stream.

Do you have example URLs for that?
As said I'll do some more tests but afaik the problem is not the mature content (anymore).
Also, are there URLs which require login but are not mature?
E.g. private photos of other users which are only unlocked for you or other specified users/friends?
If so, I'd need to get some examples to work on/check login support.



Quote:
Originally Posted by pspzockerscene View Post
Next update:
flickr video support:
- Added "preferred quality" selection along with correctly filling "quality" tag
- Enabled faster linkcheck for videos too
- Allow resume and chunkload for videos

I purposely left out "240p" which is listed internally as "700" and is pretty much the only available quality for old videos.
Same as for pictures:
If the selected quality is not found, the "best" will be used instead.
For videos added before this update, the quality tag may just return "original".
This is auto-corrected when adding them again.

Also, last update updated filename patterns for all users who were still using the default.
New default is:
Code:
*username_url*_*content_id*_*title**extension*
...which basically creates exactly the same filenames used before I started to work on these huge changes on the flickr plugin.
Looking forward to this new update!

One difference to the old Flickr plugin before the big update I noticed when downloading. When I want to stop downloads and resume them later, I am asked "Do you want to stop all running downloads? You would lose x MB due to y non resumable Downloads?" Did this change come with the new implementation? If I am not mistaken it was not like that before. Not a big issue but generating some unnecessary traffic.
Reply With Quote
  #36  
Old 16.09.2021, 08:43
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

Quote:
Originally Posted by pspzockerscene View Post
Hi dear users,
It has been a long time ago since you've reported the flickr.com login issues.
I'm working at the flickr plugin at this moment but I need some more information for you in order to decide how to proceed regarding the flickr login support.

- In which situations is a flickr.com account required (e.g. private content, mature content, content only available when befriended with a user?)
- I need example URLs for content only accessible via account.
I'll reply to this too, but later.
Reply With Quote
  #37  
Old 16.09.2021, 11:01
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

Quote:
Originally Posted by EquiNox View Post
One difference to the old Flickr plugin before the big update I noticed when downloading. When I want to stop downloads and resume them later, I am asked "Do you want to stop all running downloads? You would lose x MB due to y non resumable Downloads?" Did this change come with the new implementation? If I am not mistaken it was not like that before. Not a big issue but generating some unnecessary traffic.
Resume is currently only enabled for videos. Unsure why it's disabled for images, maybe pspzockerscene can check later
__________________
JD-Dev & Server-Admin
Reply With Quote
  #38  
Old 16.09.2021, 14:24
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

That would be great, thanks Jiaz!

There is one another option that I think would be useful, it's specifically Flickr related. The Flickr users are very creative once in a while and sometimes using extremely long titles for their pictures or videos, even whole stories. The *title* however is very useful imo to give a description for the items. But sometimes it's so long that JD refuses to download because the entire path+filename would break the limits of the filesystem (using NTFS here).

So I think it would be a good idea to be able to limit the length of the title. An option where you can enter the maximum length of the title. The last three characters of the title then maybe three dots "...", so you know the title is not complete. And when you enter the value "0" for this option the limitation is turned off.

Example, this site:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

Examples from this site with very long titles:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

And for this picture the entire filename is so long, it breaks the filesystem for me:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

I used my default custom filename definition:
Code:
Flickr-*title*_*username_url*_*content_id**extension*

Last edited by pspzockerscene; 20.09.2021 at 20:46. Reason: Edit by mistake -> I did not modify the post!
Reply With Quote
  #39  
Old 16.09.2021, 18:02
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Quote:
Originally Posted by EquiNox View Post
There is one another option that I think would be useful, it's specifically Flickr related. The Flickr users are very creative once in a while and sometimes using extremely long titles for their pictures or videos, even whole stories. The *title* however is very useful imo to give a description for the items. But sometimes it's so long that JD refuses to download because the entire path+filename would break the limits of the filesystem (using NTFS here).
Are you using Windows?
You can remove the path limit (scroll down to "Hint for Windows users") but it may cause other trouble.

Apart from that you can use our Packagizer feature to limit filename sizes.
As it has been requested multiple times in the past, I've just added an example on how to do that.

Please keep in mind that this example works only on the filename without checking the full path.
If you need it to act more intelligent you may need an EventScripter script.

Please understand that a plugin specific solution/setting just for flickr wouldn't be target-oriented as the "too long path" issue can happen with any source/file/path.

We do have tickets regarding this topic but it is more complex that you might think because:
- Which path to use if not only the filename but also the path needs to be shortened
- Shorten paths before- or after Packagizer actions?
- Always preserve file-type when shorting filenames: This implies that this is always correctly found and may fail for files without extension or with "unknown" extension

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #40  
Old 16.09.2021, 20:04
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Quote:
Originally Posted by EquiNox View Post
One difference to the old Flickr plugin before the big update I noticed when downloading. When I want to stop downloads and resume them later, I am asked "Do you want to stop all running downloads? You would lose x MB due to y non resumable Downloads?" Did this change come with the new implementation? If I am not mistaken it was not like that before. Not a big issue but generating some unnecessary traffic.
I've tested this again and I was unable to resume stopped image files (only videos).
I guess it simply wasn#t set correctly before which is why you did not get the warning.
I'm pretty sure picture downloads would also start from zero in the older flickr plugin revisions.

@Jiaz
If you want go ahead and test again but afaik they do not even return the Content-Length for images.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 12:07.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.