#81
|
||||
|
||||
Second response, now answerung all of the rest:
Quote:
Quote:
Also you should give JD some time to process all found items - maybe you just didn't wait long enough? Quote:
I guess I should update the default packagenames to contain the username/ID so that this won't happen anymore. I mean even if you had two separate packages with the same name, they would probably end up in the same download folder so best would be to try to get unique packagenames by default. Quote:
-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#82
|
||||
|
||||
Next flickr pluginupdate will contain the following changes:
- Settings for custom packagenames - New default packagenames - Reset user defined custom tag for empty strings as I renamed the property (I rarely reset users' defaults but this time I did no care) Wartest du auf einen angekündigten Bugfix oder ein neues Feature? Updates werden nicht immer sofort bereitgestellt! Bitte lies unser Update FAQ! | Please read our Update FAQ! --- Are you waiting for recently announced changes to get released? Updates to not necessarily get released immediately! Bitte lies unser Update FAQ! | Please read our Update FAQ! -psp- EDIT I haven't included the set/gallery create-date (yet) so if that information is available somewhere, I could add a tag for that too.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download Last edited by pspzockerscene; 07.10.2021 at 17:19. |
#83
|
||||
|
||||
Ok, thanks for the very detailed explanation, psp!
And fully understood the meaning of all those settings now! Quote:
Quote:
Code:
**External links are only visible to Support Staff** **External links are only visible to Support Staff** **External links are only visible to Support Staff** **External links are only visible to Support Staff** **External links are only visible to Support Staff** **External links are only visible to Support Staff** **External links are only visible to Support Staff** **External links are only visible to Support Staff** **External links are only visible to Support Staff** **External links are only visible to Support Staff** ... In a batchscript a one-liner would do it for me, album-links.txt contains the links of all albums with their id like in the code box above: Code:
@for /f %%j in (album-links.txt) do echo Copy to clipboard: %%j && timeout /t 5 && echo %%j | clip Quote:
Quote:
Code:
**External links are only visible to Support Staff** **External links are only visible to Support Staff** Thanks for the big update, it's much appreciated! The one for Menu Manager too, still need to test. Thank you both again very much, psp and Jiaz! |
#84
|
||||||
|
||||||
Quote:
Also we do have our album crawler which should auto-crawl all albums of a user -> No need to do this manually As said I'd need to get a log of you to try to find the cause. If it is caused by rate-limiting the only thing I can do about this is to slow down our crawler Quote:
Again: I need a log to investigate this. Quote:
If there is no activity displayed, some URLs may continue to get added some seconds later but then the crawl process is done. Quote:
As said, you can easily see if duplicates are currently allowed or not. Adding hundreds of albums for testing is not necessary...only if that test it about finding out whether any flickr.com API limits were hit Quote:
Quote:
My test results using your example with these many albums: Dupe check enabled (= no dupes allowed): 6697 items 285 packages Dupe check disabled (= duplicates allowed): 6697 items 281 packages (Hm strange result) If you still think there is an issue with missing items, as said I will need a log and/or maybe you can find some kind of pattern e.g. images are only missing for specific albums/users... -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#85
|
|||
|
|||
Quote:
Quote:
My explanation for this would be that the Flickr album crawler receives the links from flickr in a random order -> therefore the different number of packages. When an item for example is part of a bigger album and another two very small albums and JD2 parsed that bigger album first, it maybe does not even find the smaller albums at once, when all items in there were already part of the bigger album. Quote:
An example link that is part of three albums: Code:
**External links are only visible to Support Staff** Code:
**External links are only visible to Support Staff** After now 6 time parsing I am getting close to the number of items I stated before: 8552 items, 288 albums, but still a few items are missing. Ok, will try to find a smaller stream as an example that is easier to test. Last edited by EquiNox; 08.10.2021 at 22:01. |
#86
|
|||
|
|||
@psp / @EquiNox: Although dupes are allowed in linkgrabber, parser would still filter them. So, if the same link appears multiple times in a crawler job, only one of them will be crawled on each pass.
|
#87
|
|||
|
|||
Quote:
Overall the plugin works really great, I am so happy. However, I tested more over the weekend the new packagename/gallery tags and noticed some issues, that should be fixed I'd say. Only a brief description now, I'll explain it more thoroughly later. - For the Menu Manager, would these new jd:prop: plugin tags also available for the "Pattern for the Packages"? The new "Pattern for the Links" tags already helped me a lot. - Titles and descriptions for Galleries/groups do not yet seem to work. *title* and *description* always empty. - *username_url* for Galleries/groups I need to check if they show up correctly. - About packagename names, are underscores not allowed? When I put them in the custom definition as a separator, they did not show up in the LinkGrabber list, were just removed I think. Flickr galleries to find here: Code:
**External links are only visible to Support Staff** More later... Last edited by EquiNox; 11.10.2021 at 12:25. |
#88
|
||||
|
||||
Quote:
e.g. same picture in multiple galleries may look like this: flickr.com/photos/1234@N56/11111/in/album-7777777777777777 flickr.com/photos/1234@N56/11111/in/album-8888888888888888 Quote:
- Custom properties for packages do exist but at this moment none are set for flickr.com packages - "copy information" does not (yet) have the functionality to copy custom package related tags Also most of all "package name tags" are already available via the properties of the individual URLs. Quote:
- Title worked fine for me for galleries - At this moment, the name of a group is not considered the "title" though I could change that (= *title* will be empty for groups atm.) Description: - groups: a "description" has never been built in and I wasn't able to find a group with a description so if there is one, please provide example URLs. - At this moment, this tag is only available for albums and only if there actually is a description Quote:
Yes but by default those are removed by the "packagename cleanup". You can disable it via the following advanced setting: Code:
GeneralSettings: cleanupfilenames https://board.jdownloader.org/showthread.php?t=71257 Regarding the duplicates stuff: Jiaz may have a look at it. -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#89
|
|||
|
|||
Quote:
Edit: The following three urls result in same content. The content will be added only once if all three urls are added simultaneously to linkgrabber. Code:
pastebin.com/EM08LaWL pastebin.com/RiGEXXtq pastebin.com/adnnRjJh This is based on my understanding of the problem, which could also be quite possibly wrong. Last edited by mgpai; 11.10.2021 at 17:01. |
#90
|
|||
|
|||
Are you sure?
Ok, used this long custom packagename: Code:
*type* + *set_or_gallery_id* + *title* + *description* + *username* + *username_internal* + *username_full* + *username_url* And empty tags will be replaced by this string: Code:
Unknown Now tested with this gallery: Code:
**External links are only visible to Support Staff** This is what I get as a packagename: Code:
Gallery + 72157719950870126 + Unknown + Unknown + nobby15 + Unknown + Unknown + naubrey I think the correct title should be: Code:
Other Night Sky Photography A description for this gallery is given too: Code:
Something to learn by. Description is not so important as it mostly does not make much sense to include long descriptions in a foldername. Why *username_internal* and *username_full* seem to be empty, I don't know but *username* and *username_url* look fine. At least it would be nice to have a title for the gallery, so you get an idea what it is about. |
#91
|
||||
|
||||
Quote:
Appearently you're right. Next flickr plugin update includes the following changes: - gallery: fixed missing tags: description, title, usernameInternal - gallery/set: added tags: date, date_update - groups: use same value which is user as "usernameFull" as "title" - groups: crawl group create date so it can be used in the package tags too Some sidenotes: Groups: Seems like the "description" is only displayed by their website and not available via API(?) Example: flickr.com/groups/germany HTML snippet containing group description: Code:
<span class="blast-message "> <div class="blast-message-display"> <span class="blast-message-secure"> Willkommen in Deutschland! </span> </div> <div class="blast-message-edit"> <textarea class="blast-message-edited" wrap="on">Willkommen in Deutschland!</textarea> <textarea tabIndex="-1" class="blast-message-edited-false" style="display:none;" wrap="on"></textarea> <button class="alt blast-cancel-button">Abbrechen</button> <button class="blast-done-button">Fertig</button> </div> </span> Wartest du auf einen angekündigten Bugfix oder ein neues Feature? Updates werden nicht immer sofort bereitgestellt! Bitte lies unser Update FAQ! | Please read our Update FAQ! --- Are you waiting for recently announced changes to get released? Updates to not necessarily get released immediately! Bitte lies unser Update FAQ! | Please read our Update FAQ! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#92
|
|||
|
|||
Quote:
Quote:
About the dupe-checker issue, yes, it would be nice if the LinkGrabber would find all dupes in all albums just with one crawling job. I had a look dupes (meaning unique items have been put into serveral albums) are more widespread at Flickr than you might think. They just never showed up in LinkGrabber when dupe check was activated by default. When the number of total items in the stream (albums + remaining fotostream) matches the number stated on the site you think everything is complete and fine although several albums can still have many gaps because items in there are already part of another albums. Another question I have, would it be possible to have a packagename tag with the number of items in a package/album? Not sure how Flickr is counting them or saving this value somewhere. I don't mean just counting the items in a package but extracting this value from the website. That way you could easily check if an album/gallery would be really complete or if it still has gaps. That was my idea. |
#93
|
||||
|
||||
Quote:
Quote:
Quote:
See changelog below. You could also use an EventScripter script to wait until the package is finished and then auto-compare that value and e.g. display a warning message if it looks like something is missing Flickr plugin changes for the next update: - groups: added date of last activity in group for "date_update" tag - added new packagename tag 'total_number_of_items' RE: requested by user RE forum 492420 Wartest du auf einen angekündigten Bugfix oder ein neues Feature? Updates werden nicht immer sofort bereitgestellt! Bitte lies unser Update FAQ! | Please read our Update FAQ! --- Are you waiting for recently announced changes to get released? Updates to not necessarily get released immediately! Bitte lies unser Update FAQ! | Please read our Update FAQ! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#94
|
|||
|
|||
Many thanks for the update, psp!!
Quote:
Btw I had just seen there are two columns in "Downloads" and "LinkGrabber" "Size" and "File Count". What is the difference between those two? As far as I have seen they always have the same values except that "Size" values are bracketed. Ok, when I have the dupe check deactivated and the number of total items is now shown on each packagename I see very nicely which albums are incomplete with the first album crawling job. Just comparing "File Count" with *total_number_of_items*. When I do the album crawling a second time, both numbers for "Size" and "File Count" increase in the same way since dupes are added to each package. This time the values in both columns are useless for me since the numbers more or less doubled and cannot anymore correspond with *total_number_of_items*. And these dupes within one package will never be downloaded because they have exactly the same filename (mirror). Maybe it would be nice to have a column like "unique items" or "unique filenames" that counts items in a package with the same filename only once. Not sure maybe you can configure JD2 so it also downloads these mirrors (dupe filenames) with "Auto rename" "If the file already exists". There might be cases when this is useful although I think it only happens very rarely. Just thinking aloud. Cool, looks JD2 is full of possibilities to manage almost anything. When this dupe issue is maybe solved, this would probably not be necessary but I'll keep this in mind. I have not used EventScripter before but now installed it. |
#95
|
||||
|
||||
Quote:
Only Jiaz will be able to answer this more detailed. EDIT: File Size = Filesize ONLY (for package and Links) Size = For links, filesize, for package "[NumberOfLinks] <fileSize>" See also the following advanced setting: Code:
GraphicalUserInterfaceSettings.filecountinsizecolumnvisible ...or are we talking about two different things here? Quote:
As explained you can realize such special wishes using EventScripter script (though no extra colums but you will find ways to compare new vs. existing items!). Quote:
If the files are 100% the same, why would you want to download them? If your question is about downloading files with the same name but different size: Yes, you can configure JD in a way so that it handles the mirror detection differently. I won't go into details for now but see advanced settings --> "mirror detection" Quote:
That's the first step -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download Last edited by pspzockerscene; 14.10.2021 at 17:21. |
#96
|
|||
|
|||
It's all ok, my fault I did not look closely enough!
Quote:
Not having tested everything, but already want to give a big thanks to everyone who worked on this plugin!! All the new updates, improvements, additions make it very powerful and so enjoyable. The most I am happy about the Exif-info on most pictures now and the *date_taken* and *quality* tag that are very helpful to get the collections better sorted. And of course the working account support now! Just wonderful! |
#97
|
|||
|
|||
And about the EventScripter script I'll have a look over the weekend and maybe ask for some help because I think such one would really be helpful.
|
Thread Tools | |
Display Modes | |
|
|