[Solved] LinkGrabber missing some Flickr links

[User Name]

This is my first time using JDownloader, please be patient.

I'm trying to grab all images from this flickr account (not mine):
**External links are only visible to Support Staff****External links are only visible to Support Staff**

There are 285 albums, and an average of 66 photos per album. No login is needed to see the contents.

Flickr states there's a total 18,839 photos in the whole account, yet the LinkGrabber can only see 18,192. I've run it about 6 times, sometimes it creeps up by another 50 or so, but I still can't get it to see all links/images.

I've manually looked to see if some of the missing links are viewable in the regular way through a browser, and the ones I checked were all OK.

Why might I not be getting 100% of links, and is there anything I can do to improve my success?

[Jiaz]

Doing some tests. I can reproduce the issue. Trying to figure out what is happening

[Jiaz]

hmm, still working on it

[User Name]

Great, thank you.

Only just found out about JD2. It's amazing!

[Jiaz]

will continue next week

[User Name]

Danke.

[Jiaz]

Maybe you have smaller example links? I can reproduce it but because of the many links, debugging is not easy.

[User Name]

That was the first and only link I tried. But I have randomly found another:

**External links are only visible to Support Staff**...

That user has 419 photos in total, across 53 albums, however the LinkGrabber (when I ran it) only found 336 photos.

Also, this user has 2779 photos:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

But LinkGrabber only finds 2755.

[Jiaz]

Can you please send the links to support@jdownloader.org ?
The forum has *eaten* the first link

[User Name]

Ah, I think it's because I edited the post to include the 2nd link. Here they both are again:

1st link, 419 photos:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

2nd link, 2779 photos:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

:)

[Jiaz]

Thanks!

[Jiaz]

The answer is very simply
8337597@N08/albums -> has only 336 images

but the account itself
8337597@N08 has 419 images

Not every image is assigned to an album and therefore not available when adding /albums link

same for the other example links

[User Name]

Thanks for looking at the problem, however I don't think this is the answer for the first link I posted (which is the one I want to work).

**External links are only visible to Support Staff****External links are only visible to Support Staff**

For instance, in this sub-album...
**External links are only visible to Support Staff****External links are only visible to Support Staff**
...there are 55 photos, however LinkGrabber can only find 5, even if I ask it to search that specific album it doesn't find any more links.

In this album there are 43 photos:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
...but LinkGrabber only sees 3.

When I sort the LinkGrabber results by size I can see there are about 50 other albums in that user's account for which LinkGrabber can only see a very low number of photos per album (1-20). But when I check on flickr I see there should be about 40-60 per album.

[Jiaz]

There are many duplicates! and those are not added by JDownloader.
There are images in multiple albums but JDownloader only adds them once.

51106326@N00 has 18.853 overall images.
When you add '51106326@N00/albums/' link, then only the albums will be processed and there are about 16k unique images in those albums.
You've got to add the account link '51106326@N00/' so JDownloader will also add those images that are not assigned to any album. JDownloader does find all images for me with this link.

Adding the subalbum '51106326@N00/albums/72157631208817176' does find all 55 images ONLY if you haven't yet added any other albums/images from this account. Becuase there are 50 shared/non unique images in that album.
Also your other subalbum example, JDownloader does find all 43 images. 40 are shared/non unique.

[User Name]

Ahh, I understand now. (remember I haven't used JDownloader before)

Hmmm... I see there's an album called '1000+ views', which seems to be where the duplicates are and what LinkGrabber finds first.

Is there a way to force LinkGrabber to see duplicates when looking at /albums/ ? Or to ignore the 1000+ views folder?

Ideally I would like to just point LinkGrabber at /albums, delete the few albums I'm not interested in (including the '1000+ views'), and grab the rest, keeping all the photos in their respective sub-folders upon download.

The alternative is for me to manually add the url for each album I'm interested in, but that seems very long winded?

Or could I restrict LinkGrabber to only look for albums with a certain word/phrase in their title? That might help restrict the duplicates and get me the full set of albums I desire?

[Jiaz]

I could add support to treat duplicates in different albums as unique so you can add the same image multiple times from multiple albums. Would that be an option for you?
I can also try to add correct album information when adding the non album link

[User Name]

That would be awesome to have duplicates treated as unique. I think that would allow me to download sets of albums as I envisioned.

I'm not sure what what you mean by adding correct album information though?

[Jiaz]

Please try again with next plugin update and check if it works as wished

[User Name]

That's so cool Jiaz, thank you.

How will I know when the plugin is updated? Do I just have to keep clicking the Check Updates button?

[Jiaz]

The update is live since 24.08 Just hit update and restart JDownloader in doubt. You can also check about dialog and check build date. date after >24.08 contains the update.

with updated plugin you can now add album links and albums may contain images that are also part of other albums -> no longer filtered as duplicate

[User Name]

Hmmm... well I've updated and restarted twice, build date is 28.08, but it seems to behave as it did before?

I cleared all the previously found links from my list, ran LinkGrabber a couple of times, even did a deep scan, but still only getting 5 images in a folder of 55, etc...

Is there something else I need to do?

[Jiaz]

Can you please provide the exact link you're adding?
Send them to support@jdownloader.org

I can also offer you live help via Teamviewer, just contact me by given mail

[User Name]

It's still that first one I tried:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

[Jiaz]

finally found the little devil wait for next plugin update
your example link will not return over 20k links because of the no longer filtered links.
In case you want to download all links without duplicates, just add the photostream link

[User Name]

Awesome, thank you. Link Grabber is now finding all the files in /albums, but when I go to download them all the files have the error 'Try restarting this link (plugin outdated?)'

Is there something I need to do? Have restarted JD2 twice and tried adding the links twice.

[Jiaz]

sorry, caused by changes. will be fixed with next update

[Jiaz]

the update is live

[User Name]

Thank you. Links are working now.

But they're not getting packaged into seperate folders (by album name) as they were before? I've got 'Information overwrites packagizer rules' checked, I don't think I've changed any other settings??

Any ideas?

[Jiaz]

"Information overwrites packagizer rules" -> that overwrites *sub folder packagizer rule* and all links will land in set download folder.
You've got to disable that for *sub folder packagizer rule* to work

[User Name]

Ah, of course.

Amazing, thank you for all your help and assistance with this.

[Jiaz]

You're welcome