#61
|
||||
|
||||
@EquiNox:
1.) file extension should be fixed with next update 2.) item parser will be fixed with next update
__________________
JD-Dev & Server-Admin Last edited by Jiaz; 27.09.2021 at 17:10. |
#62
|
|||
|
|||
Quote:
Thank you very much Jiaz! |
#63
|
||||
|
||||
@EquiNox: Thanks for the feedback and you're welcome
__________________
JD-Dev & Server-Admin |
#64
|
||||
|
||||
Quote:
Existing "date_taken" values will be auto-overwritten with the new formatted value. Quote:
Please provide some URLs if you find some. I can imagine how this might happen as we do skip 1-2 frequently broken video qualities so there might be an edge case where only those ones are available... Please also check if those videos are playable via browser. Quote:
Yeah that plugin was really old and when I saw that old stuff I couldn't resist rewriting it. @Jiaz Thanks for fixing that last encoding issue! Wartest du auf einen angekündigten Bugfix oder ein neues Feature? Updates werden nicht immer sofort bereitgestellt! Bitte lies unser Update FAQ! | Please read our Update FAQ! --- Are you waiting for recently announced changes to get released? Updates to not necessarily get released immediately! Bitte lies unser Update FAQ! | Please read our Update FAQ! -psp- EDIT One more thing about date_taken: I've noticed that their werbsite/api can contain this: Code:
"isDateTakenUnknown":true My guess is that it is wrong in this case and that whenever "isDateTakenUnknown" is true, I rather should not set the value of "datedaken" as that will simply be the same as the upload date then. If you want you can do some tests on that e.g. find items where isDateTakenUnknown equals false. Please let me know your results and I'll update our plugin accordingly.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download Last edited by pspzockerscene; 28.09.2021 at 17:12. |
#65
|
|||
|
|||
Ok, many thanks for the last changes, psp!
Quote:
I'll do some futher tests with this too, but later. Busy at work these days. |
#66
|
||||
|
||||
After the next update, the "date_taken" property will only be set when given according to "isDateTakenUnknown" (website) or "takenunknown"/"datetakenunknown" (API).
Wartest du auf einen angekündigten Bugfix oder ein neues Feature? Updates werden nicht immer sofort bereitgestellt! Bitte lies unser Update FAQ! | Please read our Update FAQ! --- Are you waiting for recently announced changes to get released? Updates to not necessarily get released immediately! Bitte lies unser Update FAQ! | Please read our Update FAQ! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#67
|
|||
|
|||
That is probably more consistent. Thanks psp! But I think this issue is a little more complicated. When the date-taken is given but the uploader forbids to download the original files (_o) then you can download the 3k, 4k, 5k or whatever files but the exif-header of the file is (always?) empty. So in this case it's really helpful that you have the *date_taken* tag, so you can at least include this information in the filename when it's not stored within the file itself. Just an observation. When it's not that long ago you maybe think it's not so important to have this information, but when you look back 20, 30 years you will be grateful to have the date, when the photo was actually taken.
|
#68
|
||||
|
||||
Quote:
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#69
|
|||
|
|||
Quote:
Ok, I have tested the plugin little more over the weekend and must say it really works like a charm and so nicely fast and precise!! So thanks again for all the time and efforts you have put into this, psp and Jiaz! There is however one more issue I ran accross, not a bug but more a feature from Flickr that has not yet been covered adequately I'd say. I think the solution should be very simple and I think that was also the same behaviour of the JD-Flickr plugin before this big update. The problem is that as a Flickr user you can put the same items into different albums. I think for Flickr this is a very nice feature as you are not limited to put an item only in one album but it does only appear once in the stream. You can make for example theme albums like "sundown" or "nature" where you just combine all pictures from other albums that apply to the theme. Or sometimes users make favorite albums where they put their own favorite pictures together from their other albums. There are many possibilities. Anyway the current behaviour of JD is that items are only parsed once, and when the same item later appears in another album too it's simply left out, so that album would have less items in JD than stated on the Flickr site. Generally this behaviour is very helpful as you very nicely get exactly the correct number of items in the whole stream (first parse the albums, later the fotostream -> you have all items). But sometimes this behaviour is very unfortunate, when as I described above a user has put same items in more than one album. Then it could be that the original albums have big holes since those items are already part of a theme album and this theme albums was parsed first by JD. When there are only a few albums it's not big issue you just let JD parse the original albums first then the theme albums (copy paste every album singly one-by-one) but when there are hundreds or thousands of albums this is tedious work. Ok, to cut a long story short, it would be very nice to have a button on the Flickr plugin settings [x] Allow duplicate items, you can switch it on and off. This would solve all these problems that I described above. You then would maybe have a some duplicates in that Flickr collection but you'd know all albums are as complete as they appear on the Flickr site. Then later you can decide if you want to remove any duplicates and from which albums. The stream where I noticed this issue was this one: Code:
**External links are only visible to Support Staff** Code:
**External links are only visible to Support Staff** **External links are only visible to Support Staff** **External links are only visible to Support Staff** A link of this stream that is even part of three albums, original, fave-album and a camera album. Code:
**External links are only visible to Support Staff** |
#70
|
||||
|
||||
Quote:
It really doesn't matter which way you're doing it. Quote:
Quote:
It probably also worked with the old plugin revision but probably only partly as dupes were matched by URL so items as part opf an album had different URLÖs than single items -> No dupe detected. Quote:
Settings -> Advanced Settings -> Code:
LinkCollector.dupemanagerenabled An extra setting for flickr which I could imagine would be to set other unique IDs for albums e.g. a picture with ID 123456 gets the unique ID 123456. If it gets added in context of the album with albumID 55555, we could use the combination of both IDs (55555_123456) as unique ID thus the same item can be added unlimited times as long as it is in a different album then. Good idea? -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#71
|
|||
|
|||
Quote:
Example: The filename of an item parsed with an album or the fotostream looked like this: Code:
Flickr-DSCN1234_flickruser_1234567890.jpg Code:
Flickr-1234567890_DSCN1234_flickruser_1234567890.jpg Code:
*owner*-*title*_*username*_*photo_id**extension* Not a big deal, bulk renaming with regular expressions fixed this within seconds (removing the double id). Anyway thankfully this issue is now history and was fixed with the great update now. Quote:
But I think I need to change two options to get the desired result: Code:
Advanced Settings\LinkCollector: Dupe Manager -> deactivate Advanced Settings\LinkgrabberSettings: Default On Added Dupes Links Action -> add them, too Quote:
When I have deactivated dupe check in Advanced Settings and use this custom filename... Code:
Flickr-*date_taken*_*title*_*username_url*_*content_id*_*set_id*_*quality**extension* There is another question I have about the "Menu Manager". There is a very nice function "Copy Information" that offers various tags as pattern for links and packages (right click -> Open Menu Manager -> Copy Information). This function was already so tremendously helpful for me to update collections, finding missing items, sorting into albums. You can easily create scripts with this function. My question, are there more tags that can be used than those that are stated there ({path}, {name}, {url.content}, etc.)? It would be fantastic for example if I could also use the *content_id* tag from the Flickr plugin with this function but I am not sure if that would be possible. Then I could use for example this command: Code:
rename "*{flickr.content_id}*" "{name}" |
#72
|
||||
|
||||
Quote:
Code:
Advanced Settings\LinkgrabberSettings: Default On Added Dupes Links Action -> add them, too Not at all. content_id != dupe_id Our internal dupe_ids/"link ids" for flickr.com look like this: Code:
flickr.com://<content_id> Code:
flickr.com://123456 Nothing regarding filename properties would change even if I introduced the change mentioned in my last post. Quote:
Jiaz may also add his opinion on that. -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#73
|
|||
|
|||
Quote:
Quote:
Yes, that would be very helpful. Content_id probably the most useful tag but the other ones too. It would make so many tasks so much easier. And this is not necessarily limited to Flickr. I could imagine the unique YouTube íd would be very useful too, if we could have access to this in this dialog. |
#74
|
||||
|
||||
@pspzockerscene: contact me
__________________
JD-Dev & Server-Admin |
#75
|
||||
|
||||
Quote:
Quote:
Quote:
https://support.jdownloader.org/Know...the-packagizer --> See: Code:
<jd:prop:KEY>
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#76
|
||||
|
||||
Copying Downloadlink-specific properties via "copy information" will be available after the next CORE-update.
While the pattern may look a bit different, the properties/keys (such as "username_url") themselves are 1:1 the same as in Packagizer and (in this case) in the plugin filename settings. I haven't done anything regarding the dupe-check behavior. Just let me know your thoughts on that. I get that it actually would make sense that, even by default to allow duplicates as long as they were added in a different context...but then again on the other side it makes no sense because e.g. whats the difference between photo galleries and a user e.g. trying to add the same link from two differen other sources such as forum-posts. ...so either a complete new global setting for "fine tuning" would be needed so that we can basically set a "unique ID" and a "unique context ID" on every link...or just stick with the global dupe-check setting we already have. I still consider this and edge cases/rare issue but I do understand your remarks on this topic. Wartest du auf einen angekündigten Bugfix oder ein neues Feature? Updates werden nicht immer sofort bereitgestellt! Bitte lies unser Update FAQ! | Please read our Update FAQ! --- Are you waiting for recently announced changes to get released? Updates to not necessarily get released immediately! Bitte lies unser Update FAQ! | Please read our Update FAQ! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download Last edited by pspzockerscene; 06.10.2021 at 17:52. |
#77
|
|||
|
|||
Quote:
Maybe we should straigthen this out thoroughly because it's still not working for me the way I'd like it to be working. First of all a question about the "Advanced Settings", I filtered them with "dupe" and get these settings (among a few others): Code:
1. GeneralSettings: Dupe Manager (Default: true, activated) 2. LinkCollector: Dupe Manager (Default: true, activated) 3. LinkgrabberSettings: Default On Added Dupes Links Action (Default: ask me every time) 4. LinkgrabberSettings: Handle Dupes On Confirm Latest Selection (Default: add them, too) Now I tried to parse again this stream (Download and LinkGrabber list empty): Code:
**External links are only visible to Support Staff** My custom filename for this: Code:
Flickr-*date_taken*_*title*_*username_url*_*content_id*_*set_id*_*quality**extension* After one time parsing this is what I get in total (Group single files in a 'various package' deactivated): Code:
6997 links, 280 packages Now I counted the number of items and albums on the pages manually together and this is what I get, when all albums have the exact number of items stated on the site: Code:
8552 items, 288 albums As said before there are more items in all albums because some items are part of more than one album. When I parse the albums page a second time this is what I get in total: Code:
7964 links, 286 packages When I parse the albums page a third time this is what I get in total: Code:
8245 links, 286 packages When I now empty the LinkGrabber and do the same thing again, I get different results but I never get exactly 8552 links in 288 packages as I wanted them. The "Parse Clipboard" bubble always (or in most cases) says "Found link(s): 6997" which is of course the correct number of unique links in the albums but it does not parse the dupes correctly when I have deactivated the option in "Advanced Settings". Dupes show up but often just in the same album again the link was already part of before. Those won't be downloaded because they have exactly the same filename. What am I doing wrong here? I have added an attachment with a list of all the albums and the number of items in there. I'd just like to be able to download these albums exactly the way they are listed at Flickr. The main issue for me here is that I want to have the complete model albums and not any gaps in those when those items are already part of another fav-, camera- or whatever-album. So the best would be just to get all albums complete. When you want to remove duplicates you can decide afterwards which album you want to remove them from. Last edited by EquiNox; 07.10.2021 at 13:16. |
#78
|
|||
|
|||
Quote:
Quote:
|
#79
|
|||
|
|||
Quote:
Of course it is possible to get this done, even when you have activated dup check. When you just parse and download the albums one-by-one. But that's a lot of work. Last edited by EquiNox; 08.10.2021 at 08:59. |
#80
|
||||
|
||||
First response:
This one is only about adding dupes and the configurable behavior: Quote:
First of all, to simplify tests, I'm using an URL to a single album here: flickr.com/photos/derodeolifant/albums/72157635508464613 --> Contains 37 items Reason for this: Adding thousands of items for testing the dupes behavior simply takes too much time so it doesn't make sense Keep in mind that the first two dupes settings need a JDownloader restart to become active! First: Code:
LinkCollector.dupemanagerenabled --> If you add the above album twice, you will only get all items once. Disabled = allow duplicated items in linkgrabber --> If you add the above item twice, resulting in a single package containing 74 URLs. If you add it again and again, that package will get bigger each time. I've added a setting description for this setting for the next CORE-update. Now to: Code:
GeneralSettings.dupemanagerenabled Quote:
Now to the less important stuff: Code:
LinkgrabberSettings: Default On Added Dupes Links Action If you want JD to shut up and completely don't care about this, set it to "add them, too". I've added a setting description for this setting for the next CORE-update. Then to: Code:
LinkgrabberSettings.handledupesonconfirmlatestselection Code:
LinkgrabberSettings: Default On Added Dupes Links Action If you disabled the dialog and e.g. always allow JD to add dupes, this last setting is of course absolutely irrelevant for you. I've added a setting description for this setting for the next CORE-update. Now I've completely read your post but at this moment there is no point for me to go into details regarding your test results. When adding many items there is a lot of stuff that could go wrong such hitting one of flickrs rate-limits so better test with one link and whenever you experience "missing items" we can check it and see if we maybe have to slow down our crawler to avoid hitting such rate-limits...though this is just an assumption here. Until now I've never hit any of their rate-limits during testing. Conclusion: All you need to do is to disable: Code:
LinkCollector.dupemanagerenabled Wartest du auf einen angekündigten Bugfix oder ein neues Feature? Updates werden nicht immer sofort bereitgestellt! Bitte lies unser Update FAQ! | Please read our Update FAQ! --- Are you waiting for recently announced changes to get released? Updates to not necessarily get released immediately! Bitte lies unser Update FAQ! | Please read our Update FAQ! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
Thread Tools | |
Display Modes | |
|
|