JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #81  
Old 07.10.2021, 14:18
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 59,687
Default

Second response, now answerung all of the rest:

Quote:
Originally Posted by EquiNox View Post
Looking forward to this new update. That sound great! Many thanks!
The update has just been released

Quote:
Originally Posted by EquiNox View Post
Ok, when the dupe check or deactivated dupe check would be working as I described in my previous post, then all would be fine for me. It only happens now and then, not too often.
As described in my last post, missing items are probably caused by rate-limiting and we should be able to find that in your logs.
Also you should give JD some time to process all found items - maybe you just didn't wait long enough?

Quote:
Originally Posted by EquiNox View Post
Ok, this is probably not entirely correct. When albums have the same name (this is allowed at Flickr), they are simply merged in one package in LinkGrabber.
Please provide some examples for that.
I guess I should update the default packagenames to contain the username/ID so that this won't happen anymore.
I mean even if you had two separate packages with the same name, they would probably end up in the same download folder so best would be to try to get unique packagenames by default.

Quote:
Originally Posted by EquiNox View Post
Of course it is possible to get this done, even when you have deactivated dup check. When you just parse and download the albums one-by-one. But that's a lot of work.
As explained this should not be needed...

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #82  
Old 07.10.2021, 18:05
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 59,687
Default

Next flickr pluginupdate will contain the following changes:
- Settings for custom packagenames
- New default packagenames
- Reset user defined custom tag for empty strings as I renamed the property (I rarely reset users' defaults but this time I did no care)

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
EDIT

I haven't included the set/gallery create-date (yet) so if that information is available somewhere, I could add a tag for that too.
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist

Last edited by pspzockerscene; 07.10.2021 at 18:19.
Reply With Quote
  #83  
Old 08.10.2021, 09:07
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

Ok, thanks for the very detailed explanation, psp!

And fully understood the meaning of all those settings now!

Quote:
Originally Posted by pspzockerscene View Post
Conclusion:
All you need to do is to disable:
Code:
LinkCollector.dupemanagerenabled
...and restart JD.
Yes, that's exactly what I have done.

Quote:
Originally Posted by pspzockerscene View Post
Now I've completely read your post but at this moment there is no point for me to go into details regarding your test results.
When adding many items there is a lot of stuff that could go wrong such hitting one of flickrs rate-limits so better test with one link and whenever you experience "missing items" we can check it and see if we maybe have to slow down our crawler to avoid hitting such rate-limits...though this is just an assumption here.
Until now I've never hit any of their rate-limits during testing.
The solution for this -> getting all albums with full number of items (when duplicate check has been deactivated the way we just talked about) is absolutely easy (in theory :-p). I only need the entire list of all album links...

Code:
https://flickr.com/photos/username/albums/0000000000001
https://flickr.com/photos/username/albums/0000000000002
https://flickr.com/photos/username/albums/0000000000003
https://flickr.com/photos/username/albums/0000000000004
https://flickr.com/photos/username/albums/0000000000005
https://flickr.com/photos/username/albums/0000000000006
https://flickr.com/photos/username/albums/0000000000007
https://flickr.com/photos/username/albums/0000000000008
https://flickr.com/photos/username/albums/0000000000009
https://flickr.com/photos/username/albums/0000000000010
...
..and copy paste them one-by-one (always waiting until the last album has been completely parsed!) and I get the desired result immediately because every album is allowed to have any item in the stream even when it's already part of another album. So would it be possible to automat this? Just curious. As said this only happens seldom and I do not want to slow down album parsing just because for such rare occasions.

In a batchscript a one-liner would do it for me, album-links.txt contains the links of all albums with their id like in the code box above:
Code:
@for /f %%j in (album-links.txt) do echo Copy to clipboard: %%j && timeout /t 5 && echo %%j | clip
Copy the link to the clipboard, wait 5 seconds and then copy the next to the clipboard. But after testing now this was maybe still too fast for JD, I don't know. At least in the LinkGrabber there did not yet show up all items and albums after one time parsing this way. Maybe I should slow it down little more.


Quote:
Originally Posted by pspzockerscene View Post
Without replying to all of your further tests, I'll try to add an explanation:
First of all, to simplify tests, I'm using an URL to a single album here:
flickr.com/photos/derodeolifant/albums/72157635508464613
--> Contains 37 items
Reason for this:
Adding thousands of items for testing the dupes behavior simply takes too much time so it doesn't make sense

Keep in mind that the first two dupes settings need a JDownloader restart to become active!

First:
Code:
LinkCollector.dupemanagerenabled
Enabled = Do not allow duplicated items in linkgrabber
--> If you add the above album twice, you will only get all items once.
Disabled = allow duplicated items in linkgrabber
--> If you add the above item twice, resulting in a single package containing 74 URLs. If you add it again and again, that package will get bigger each time.
That stream is also a good example as it has a lot of items that are part of more than one album. But one album does not help to test the issue. You need a few more at least that do have an overlap of items, so some items are part of more than one album, maybe the "Origami"-albums.


Quote:
Originally Posted by pspzockerscene View Post
Quote:
Originally Posted by EquiNox View Post
Ok, this is probably not entirely correct. When albums have the same name (this is allowed at Flickr), they are simply merged in one package in LinkGrabber. That could also have happened here.
Please provide some examples for that.
Code:
https://www.flickr.com/photos/134430854@N03/albums/72157665419969382
**External links are only visible to Support Staff**
Exactly the same album name, but now thanks to your great packagename change this issue is fixed and they both can get their own package!
Thanks for the big update, it's much appreciated!
The one for Menu Manager too, still need to test.

Thank you both again very much, psp and Jiaz!
Reply With Quote
  #84  
Old 08.10.2021, 14:11
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 59,687
Default

Quote:
Originally Posted by EquiNox View Post
..and copy paste them one-by-one (always waiting until the last album has been completely parsed!) and I get the desired result immediately because every album is allowed to have any item in the stream even when it's already part of another album.
I don't really understand...still the same issue with missing items?
Also we do have our album crawler which should auto-crawl all albums of a user -> No need to do this manually
As said I'd need to get a log of you to try to find the cause.
If it is caused by rate-limiting the only thing I can do about this is to slow down our crawler

Quote:
Originally Posted by EquiNox View Post
Copy the link to the clipboard, wait 5 seconds and then copy the next to the clipboard. But after testing now this was maybe still too fast for JD, I don't know.
No, not too fast for JD but maybe "too fast for flickr" as mentioned above.
Again: I need a log to investigate this.

Quote:
Originally Posted by EquiNox View Post
At least in the LinkGrabber there did not yet show up all items and albums after one time parsing this way. Maybe I should slow it down little more.
You can see the linkgrabber activity in the bottom right corner.
If there is no activity displayed, some URLs may continue to get added some seconds later but then the crawl process is done.

Quote:
Originally Posted by EquiNox View Post
That stream is also a good example as it has a lot of items that are part of more than one album. But one album does not help to test the issue.
Yes one album is enough to test the dupe setting itself.
As said, you can easily see if duplicates are currently allowed or not.
Adding hundreds of albums for testing is not necessary...only if that test it about finding out whether any flickr.com API limits were hit

Quote:
Originally Posted by EquiNox View Post
You need a few more at least that do have an overlap of items, so some items are part of more than one album, maybe the "Origami"-albums.
Even if it was about that, 2-3 URLs would be enough for testing.

Quote:
Originally Posted by EquiNox View Post
Exactly the same album name, but now thanks to your great packagename change this issue is fixed and they both can get their own package!
Thanks for your feedback!

My test results using your example with these many albums:
Dupe check enabled (= no dupes allowed):
6697 items
285 packages

Dupe check disabled (= duplicates allowed):
6697 items
281 packages
(Hm strange result)

If you still think there is an issue with missing items, as said I will need a log and/or maybe you can find some kind of pattern e.g. images are only missing for specific albums/users...

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #85  
Old 08.10.2021, 22:56
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

Quote:
Originally Posted by pspzockerscene View Post
I don't really understand...still the same issue with missing items?
Also we do have our album crawler which should auto-crawl all albums of a user -> No need to do this manually
Yes, I know and that is what I am doing here already all the time with unwanted results.

Quote:
Originally Posted by pspzockerscene View Post
My test results using your example with these many albums:
Dupe check enabled (= no dupes allowed):
6697 items
285 packages

Dupe check disabled (= duplicates allowed):
6697 items
281 packages
(Hm strange result)
That is what I was trying to explain over the last days, maybe it was not clear enough. It doesn't matter if I have dupe check activated or not, with the first time parsing the albums always have 6997 items (I think you just had some transposed digits in your post, it happened to me too before ) even though the number of album items is much higher when you simply sum up the amount of pictures stated on the album pages (8552 items, 288 albums I have counted). So parsing for the first time JD2 never finds any dupes to be in different albums when dupe check is deactivated. And when I parse a second and a third time the album crawler bubble always says "Found link(s): 6997". And about the number of albums (packages) JD2 finds in that stream, yes, it's also everytime little different.

My explanation for this would be that the Flickr album crawler receives the links from flickr in a random order -> therefore the different number of packages. When an item for example is part of a bigger album and another two very small albums and JD2 parsed that bigger album first, it maybe does not even find the smaller albums at once, when all items in there were already part of the bigger album.

Quote:
And when I parse a second and a third time the album crawler bubble always says "Found link(s): 6997".
^--- So this does mean to me that the Flickr album crawler process does never allow any dupes within this one crawling process. There are exactly 6997 unique items in the albums of this user. When an item for example is part of three albums with the first time parsing JD2 finds 1 link of this item in an album, with the second time parsing it does find one dupe link of this item, maybe in another album or the same like before (just randomly), with a third time parsing the same and so on.

An example link that is part of three albums:
Code:
**External links are only visible to Support Staff**
You can filter the LinkGrabber list by filename and use the content_id of this item as filter. Each time parsing you can see that there is exactly one more dupe of this item showing up but only one. With some luck it does show up in all three albums that it is part of when you parse the albums often enough (6 times here). Just copy pasted again and again the albums-link after one crawling process was finished of course:
Code:
**External links are only visible to Support Staff**
The way JD2 is counting the links is interesting too. When it finds a "real" dupe in another album with a new parsing it does increase the number of found Link(s) (on the bottom under "Overview"). The number of Flickr items (on the right under "Views") is always 6997 x number of total parsings.

After now 6 time parsing I am getting close to the number of items I stated before: 8552 items, 288 albums, but still a few items are missing.

Ok, will try to find a smaller stream as an example that is easier to test.

Last edited by EquiNox; 08.10.2021 at 23:01.
Reply With Quote
  #86  
Old 09.10.2021, 12:27
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 1,371
Default

@psp / @EquiNox: Although dupes are allowed in linkgrabber, parser would still filter them. So, if the same link appears multiple times in a crawler job, only one of them will be crawled on each pass.
Reply With Quote
  #87  
Old 11.10.2021, 08:57
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

Quote:
Originally Posted by mgpai View Post
@psp / @EquiNox: Although dupes are allowed in linkgrabber, parser would still filter them. So, if the same link appears multiple times in a crawler job, only one of them will be crawled on each pass.
Thanks, yes, I think that is how it appeared to me.

Overall the plugin works really great, I am so happy.

However, I tested more over the weekend the new packagename/gallery tags and noticed some issues, that should be fixed I'd say. Only a brief description now, I'll explain it more thoroughly later.

- For the Menu Manager, would these new jd:prop: plugin tags also available for the "Pattern for the Packages"? The new "Pattern for the Links" tags already helped me a lot.

- Titles and descriptions for Galleries/groups do not yet seem to work. *title* and *description* always empty.

- *username_url* for Galleries/groups I need to check if they show up correctly.

- About packagename names, are underscores not allowed? When I put them in the custom definition as a separator, they did not show up in the LinkGrabber list, were just removed I think.

Flickr galleries to find here:
Code:
**External links are only visible to Support Staff**

More later...

Last edited by EquiNox; 11.10.2021 at 13:25.
Reply With Quote
  #88  
Old 11.10.2021, 17:33
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 59,687
Default

Quote:
Originally Posted by mgpai View Post
@psp / @EquiNox: Although dupes are allowed in linkgrabber, parser would still filter them. So, if the same link appears multiple times in a crawler job, only one of them will be crawled on each pass.
The same link should not appear twice because those URLs then contain the gallery ID:
e.g. same picture in multiple galleries may look like this:
flickr.com/photos/1234@N56/11111/in/album-7777777777777777
flickr.com/photos/1234@N56/11111/in/album-8888888888888888

Quote:
Originally Posted by EquiNox View Post
For the Menu Manager, would these new jd:prop: plugin tags also available for the "Pattern for the Packages"? The new "Pattern for the Links" tags already helped me a lot
Well yes and no:
- Custom properties for packages do exist but at this moment none are set for flickr.com packages
- "copy information" does not (yet) have the functionality to copy custom package related tags

Also most of all "package name tags" are already available via the properties of the individual URLs.

Quote:
Originally Posted by EquiNox View Post
- Titles and descriptions for Galleries/groups do not yet seem to work. *title* and *description* always empty.
Title:
- Title worked fine for me for galleries
- At this moment, the name of a group is not considered the "title" though I could change that (= *title* will be empty for groups atm.)
Description:
- groups: a "description" has never been built in and I wasn't able to find a group with a description so if there is one, please provide example URLs.
- At this moment, this tag is only available for albums and only if there actually is a description

Quote:
Originally Posted by EquiNox View Post
- *username_url* for Galleries/groups I need to check if they show up correctly.
Should work fine as it's used in the new default patterns.

Quote:
Originally Posted by EquiNox View Post
About packagename names, are underscores not allowed?
Yes but by default those are removed by the "packagename cleanup".
You can disable it via the following advanced setting:
Code:
GeneralSettings: cleanupfilenames
There is also a forum thread where improvement suggestions to this feature were made:
https://board.jdownloader.org/showthread.php?t=71257

Regarding the duplicates stuff:
Jiaz may have a look at it.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #89  
Old 11.10.2021, 17:52
mgpai mgpai is offline
Script Master
 
Join Date: Sep 2013
Posts: 1,371
Default

Quote:
Originally Posted by pspzockerscene View Post
The same link should not appear twice because those URLs then contain the gallery ID:
Source urls might be unique, if resulting content urls are dupes I think only one of them will be added on each pass.

Edit: The following three urls result in same content. The content will be added only once if all three urls are added simultaneously to linkgrabber.

Code:
pastebin.com/EM08LaWL
pastebin.com/RiGEXXtq
pastebin.com/adnnRjJh

This is based on my understanding of the problem, which could also be quite possibly wrong.

Last edited by mgpai; 11.10.2021 at 18:01.
Reply With Quote
  #90  
Old 12.10.2021, 07:27
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

Quote:
Originally Posted by pspzockerscene View Post
Title:
- Title worked fine for me for galleries-
Are you sure?

Ok, used this long custom packagename:
Code:
*type* + *set_or_gallery_id* + *title* + *description* + *username* + *username_internal* + *username_full* + *username_url*

And empty tags will be replaced by this string:
Code:
Unknown

Now tested with this gallery:
Code:
**External links are only visible to Support Staff**

This is what I get as a packagename:
Code:
Gallery + 72157719950870126 + Unknown + Unknown + nobby15 + Unknown + Unknown + naubrey

I think the correct title should be:
Code:
Other Night Sky Photography

A description for this gallery is given too:
Code:
Something to learn by.

Description is not so important as it mostly does not make much sense to include long descriptions in a foldername. Why *username_internal* and *username_full* seem to be empty, I don't know but *username* and *username_url* look fine. At least it would be nice to have a title for the gallery, so you get an idea what it is about.
Reply With Quote
  #91  
Old 12.10.2021, 17:16
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 59,687
Default

Quote:
Originally Posted by mgpai View Post
This is based on my understanding of the problem, which could also be quite possibly wrong.
Yap looks like you're on the right path with your assumption.

Quote:
Originally Posted by EquiNox View Post
Are you sure?
Appearently you're right.
Next flickr plugin update includes the following changes:

- gallery: fixed missing tags: description, title, usernameInternal
- gallery/set: added tags: date, date_update
- groups: use same value which is user as "usernameFull" as "title"
- groups: crawl group create date so it can be used in the package tags too

Some sidenotes:
Groups:
Seems like the "description" is only displayed by their website and not available via API(?)
Example:
flickr.com/groups/germany
HTML snippet containing group description:
Code:
<span class="blast-message ">
			<div class="blast-message-display">
				<span class="blast-message-secure">
					Willkommen in Deutschland!
				</span>

			</div>
			<div class="blast-message-edit">
				<textarea class="blast-message-edited" wrap="on">Willkommen in Deutschland!</textarea>
				<textarea tabIndex="-1" class="blast-message-edited-false" style="display:none;" wrap="on"></textarea>

				<button class="alt blast-cancel-button">Abbrechen</button>
				<button class="blast-done-button">Fertig</button>
			</div>

		</span>
Also, if you want, I could grab the date of the last activity of a group ("dateactivity") and set that so it would be available via "date_update".

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #92  
Old 13.10.2021, 08:57
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

Quote:
Originally Posted by pspzockerscene View Post
Next flickr plugin update includes the following changes:

- gallery: fixed missing tags: description, title, usernameInternal
- gallery/set: added tags: date, date_update
- groups: use same value which is user as "usernameFull" as "title"
- groups: crawl group create date so it can be used in the package tags too
Ok, thanks for fixing, psp! I'll test this closer soon.


Quote:
Originally Posted by pspzockerscene View Post
Also, if you want, I could grab the date of the last activity of a group ("dateactivity") and set that so it would be available via "date_update".
Having the date when something was last updated I guess could be useful, good idea, I'd say.


About the dupe-checker issue, yes, it would be nice if the LinkGrabber would find all dupes in all albums just with one crawling job. I had a look dupes (meaning unique items have been put into serveral albums) are more widespread at Flickr than you might think. They just never showed up in LinkGrabber when dupe check was activated by default. When the number of total items in the stream (albums + remaining fotostream) matches the number stated on the site you think everything is complete and fine although several albums can still have many gaps because items in there are already part of another albums.


Another question I have, would it be possible to have a packagename tag with the number of items in a package/album? Not sure how Flickr is counting them or saving this value somewhere. I don't mean just counting the items in a package but extracting this value from the website. That way you could easily check if an album/gallery would be really complete or if it still has gaps. That was my idea.
Reply With Quote
  #93  
Old 13.10.2021, 16:31
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 59,687
Default

Quote:
Originally Posted by EquiNox View Post
Having the date when something was last updated I guess could be useful, good idea, I'd say.
Done. See changelog at the end of this post.

Quote:
Originally Posted by EquiNox View Post
About the dupe-checker issue, yes, it would be nice if the LinkGrabber would find all dupes in all albums just with one crawling job.
As explained you will have to wait for Jiaz to check this.

Quote:
Originally Posted by EquiNox View Post
Another question I have, would it be possible to have a packagename tag with the number of items in a package/album?
Yes.
See changelog below.
You could also use an EventScripter script to wait until the package is finished and then auto-compare that value and e.g. display a warning message if it looks like something is missing

Flickr plugin changes for the next update:
- groups: added date of last activity in group for "date_update" tag
- added new packagename tag 'total_number_of_items' RE: requested by user RE forum 492420

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #94  
Old 14.10.2021, 08:48
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

Many thanks for the update, psp!!


Quote:
Originally Posted by pspzockerscene View Post
Quote:
Originally Posted by EquiNox View Post
Another question I have, would it be possible to have a packagename tag with the number of items in a package/album?
Yes.
See changelog below.
Super cool! Works just perfectly! And definitely a help!

Btw I had just seen there are two columns in "Downloads" and "LinkGrabber" "Size" and "File Count". What is the difference between those two? As far as I have seen they always have the same values except that "Size" values are bracketed.

Ok, when I have the dupe check deactivated and the number of total items is now shown on each packagename I see very nicely which albums are incomplete with the first album crawling job. Just comparing "File Count" with *total_number_of_items*. When I do the album crawling a second time, both numbers for "Size" and "File Count" increase in the same way since dupes are added to each package. This time the values in both columns are useless for me since the numbers more or less doubled and cannot anymore correspond with *total_number_of_items*.
And these dupes within one package will never be downloaded because they have exactly the same filename (mirror). Maybe it would be nice to have a column like "unique items" or "unique filenames" that counts items in a package with the same filename only once. Not sure maybe you can configure JD2 so it also downloads these mirrors (dupe filenames) with "Auto rename" "If the file already exists". There might be cases when this is useful although I think it only happens very rarely. Just thinking aloud.

Quote:
Originally Posted by pspzockerscene View Post
You could also use an EventScripter script to wait until the package is finished and then auto-compare that value and e.g. display a warning message if it looks like something is missing
Cool, looks JD2 is full of possibilities to manage almost anything. When this dupe issue is maybe solved, this would probably not be necessary but I'll keep this in mind. I have not used EventScripter before but now installed it.
Reply With Quote
  #95  
Old 14.10.2021, 17:41
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 59,687
Default

Quote:
Originally Posted by EquiNox View Post
Btw I had just seen there are two columns in "Downloads" and "LinkGrabber" "Size" and "File Count". What is the difference between those two? As far as I have seen they always have the same values except that "Size" values are bracketed.
You mean "File Size" and "Size"?
Looks to be 100% the same here.
Only Jiaz will be able to answer this more detailed.

EDIT:
File Size = Filesize ONLY (for package and Links)

Size = For links, filesize, for package "[NumberOfLinks] <fileSize>"
See also the following advanced setting:
Code:
GraphicalUserInterfaceSettings.filecountinsizecolumnvisible
The "File count" cloumn is different though and only shows the number of items (for all files, "1" and for packages, the number of items in that package).
...or are we talking about two different things here?

Quote:
Originally Posted by EquiNox View Post
Maybe it would be nice to have a column like "unique items" or "unique filenames" that counts items in a package with the same filename only once.
I don't think we'll add this as this is very specific.
As explained you can realize such special wishes using EventScripter script (though no extra colums but you will find ways to compare new vs. existing items!).

Quote:
Originally Posted by EquiNox View Post
Not sure maybe you can configure JD2 so it also downloads these mirrors (dupe filenames) with "Auto rename" "If the file already exists".
I wouldn't know why.
If the files are 100% the same, why would you want to download them?
If your question is about downloading files with the same name but different size:
Yes, you can configure JD in a way so that it handles the mirror detection differently.
I won't go into details for now but see advanced settings --> "mirror detection"

Quote:
Originally Posted by EquiNox View Post
When this dupe issue is maybe solved, this would probably not be necessary but I'll keep this in mind.
I'm not even sure if this can be considered a bug but as said, you need to wait for Jiaz to answer on this.

Quote:
Originally Posted by EquiNox View Post
I have not used EventScripter before but now installed it.
That's the first step

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist

Last edited by pspzockerscene; 14.10.2021 at 18:21.
Reply With Quote
  #96  
Old 15.10.2021, 08:42
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

Quote:
Originally Posted by pspzockerscene View Post
You mean "File Size" and "Size"?
It's all ok, my fault I did not look closely enough!

Quote:
Originally Posted by pspzockerscene View Post
I'm not even sure if this can be considered a bug but as said, you need to wait for Jiaz to answer on this.
Right now I have just parsed the albums-link several times, then downloaded the albums and looked afterwards if they were complete or still gaps in there with this helpful new tag *total_number_of_items*. Big improvement I'd say. Not just in such situations. When there were still gaps just parsed that album again and added the missing items to the download queue.

Not having tested everything, but already want to give a big thanks to everyone who worked on this plugin!! All the new updates, improvements, additions make it very powerful and so enjoyable. The most I am happy about the Exif-info on most pictures now and the *date_taken* and *quality* tag that are very helpful to get the collections better sorted. And of course the working account support now! Just wonderful!
Reply With Quote
  #97  
Old 15.10.2021, 08:46
EquiNox EquiNox is offline
Wind Gust
 
Join Date: Sep 2021
Posts: 40
Default

And about the EventScripter script I'll have a look over the weekend and maybe ask for some help because I think such one would really be helpful.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 08:20.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.