JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 15.08.2018, 21:14
User Name User Name is offline
Registered / Inactive
 
Join Date: Aug 2018
Posts: 14
Default LinkGrabber missing some Flickr links

This is my first time using JDownloader, please be patient.

I'm trying to grab all images from this flickr account (not mine):
**External links are only visible to Support Staff****External links are only visible to Support Staff**

There are 285 albums, and an average of 66 photos per album. No login is needed to see the contents.

Flickr states there's a total 18,839 photos in the whole account, yet the LinkGrabber can only see 18,192. I've run it about 6 times, sometimes it creeps up by another 50 or so, but I still can't get it to see all links/images.

I've manually looked to see if some of the missing links are viewable in the regular way through a browser, and the ones I checked were all OK.

Why might I not be getting 100% of links, and is there anything I can do to improve my success?

Last edited by User Name; 15.08.2018 at 21:21.
Reply With Quote
  #2  
Old 16.08.2018, 18:56
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,049
Default

Doing some tests. I can reproduce the issue. Trying to figure out what is happening
__________________
JD-Dev & Server-Admin
Reply With Quote
  #3  
Old 16.08.2018, 19:27
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,049
Default

hmm, still working on it
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 16.08.2018 at 19:40.
Reply With Quote
  #4  
Old 16.08.2018, 23:30
User Name User Name is offline
Registered / Inactive
 
Join Date: Aug 2018
Posts: 14
Default

Great, thank you.

Only just found out about JD2. It's amazing!
Reply With Quote
  #5  
Old 17.08.2018, 19:20
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,049
Default

will continue next week
__________________
JD-Dev & Server-Admin
Reply With Quote
  #6  
Old 20.08.2018, 14:56
User Name User Name is offline
Registered / Inactive
 
Join Date: Aug 2018
Posts: 14
Default

Danke.
Reply With Quote
  #7  
Old 20.08.2018, 19:52
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,049
Default

Maybe you have smaller example links? I can reproduce it but because of the many links, debugging is not easy.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #8  
Old 21.08.2018, 03:26
User Name User Name is offline
Registered / Inactive
 
Join Date: Aug 2018
Posts: 14
Default

That was the first and only link I tried. But I have randomly found another:

**External links are only visible to Support Staff**...

That user has 419 photos in total, across 53 albums, however the LinkGrabber (when I ran it) only found 336 photos.

Also, this user has 2779 photos:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

But LinkGrabber only finds 2755.

Last edited by User Name; 21.08.2018 at 03:31.
Reply With Quote
  #9  
Old 21.08.2018, 13:53
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,049
Default

Can you please send the links to support@jdownloader.org ?
The forum has *eaten* the first link
__________________
JD-Dev & Server-Admin
Reply With Quote
  #10  
Old 21.08.2018, 14:42
User Name User Name is offline
Registered / Inactive
 
Join Date: Aug 2018
Posts: 14
Default

Ah, I think it's because I edited the post to include the 2nd link. Here they both are again:

1st link, 419 photos:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

2nd link, 2779 photos:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

:)
Reply With Quote
  #11  
Old 21.08.2018, 20:02
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,049
Default

Thanks!
__________________
JD-Dev & Server-Admin
Reply With Quote
  #12  
Old 22.08.2018, 19:27
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,049
Default

The answer is very simply
8337597@N08/albums -> has only 336 images

but the account itself
8337597@N08 has 419 images

Not every image is assigned to an album and therefore not available when adding /albums link

same for the other example links
__________________
JD-Dev & Server-Admin
Reply With Quote
  #13  
Old 23.08.2018, 03:42
User Name User Name is offline
Registered / Inactive
 
Join Date: Aug 2018
Posts: 14
Default

Thanks for looking at the problem, however I don't think this is the answer for the first link I posted (which is the one I want to work).

**External links are only visible to Support Staff****External links are only visible to Support Staff**

For instance, in this sub-album...
**External links are only visible to Support Staff****External links are only visible to Support Staff**
...there are 55 photos, however LinkGrabber can only find 5, even if I ask it to search that specific album it doesn't find any more links.

In this album there are 43 photos:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
...but LinkGrabber only sees 3.

When I sort the LinkGrabber results by size I can see there are about 50 other albums in that user's account for which LinkGrabber can only see a very low number of photos per album (1-20). But when I check on flickr I see there should be about 40-60 per album.
Reply With Quote
  #14  
Old 23.08.2018, 12:53
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,049
Default

There are many duplicates! and those are not added by JDownloader.
There are images in multiple albums but JDownloader only adds them once.

51106326@N00 has 18.853 overall images.
When you add '51106326@N00/albums/' link, then only the albums will be processed and there are about 16k unique images in those albums.
You've got to add the account link '51106326@N00/' so JDownloader will also add those images that are not assigned to any album. JDownloader does find all images for me with this link.

Adding the subalbum '51106326@N00/albums/72157631208817176' does find all 55 images ONLY if you haven't yet added any other albums/images from this account. Becuase there are 50 shared/non unique images in that album.
Also your other subalbum example, JDownloader does find all 43 images. 40 are shared/non unique.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #15  
Old 23.08.2018, 16:27
User Name User Name is offline
Registered / Inactive
 
Join Date: Aug 2018
Posts: 14
Default

Ahh, I understand now. (remember I haven't used JDownloader before)

Hmmm... I see there's an album called '1000+ views', which seems to be where the duplicates are and what LinkGrabber finds first.

Is there a way to force LinkGrabber to see duplicates when looking at /albums/ ? Or to ignore the 1000+ views folder?

Ideally I would like to just point LinkGrabber at /albums, delete the few albums I'm not interested in (including the '1000+ views'), and grab the rest, keeping all the photos in their respective sub-folders upon download.

The alternative is for me to manually add the url for each album I'm interested in, but that seems very long winded?

Or could I restrict LinkGrabber to only look for albums with a certain word/phrase in their title? That might help restrict the duplicates and get me the full set of albums I desire?

Last edited by User Name; 23.08.2018 at 16:40.
Reply With Quote
  #16  
Old 23.08.2018, 19:39
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,049
Default

I could add support to treat duplicates in different albums as unique so you can add the same image multiple times from multiple albums. Would that be an option for you?
I can also try to add correct album information when adding the non album link
__________________
JD-Dev & Server-Admin
Reply With Quote
  #17  
Old 24.08.2018, 03:44
User Name User Name is offline
Registered / Inactive
 
Join Date: Aug 2018
Posts: 14
Default

That would be awesome to have duplicates treated as unique. I think that would allow me to download sets of albums as I envisioned.

I'm not sure what what you mean by adding correct album information though?
Reply With Quote
  #18  
Old 24.08.2018, 13:10
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,049
Default

Please try again with next plugin update and check if it works as wished
__________________
JD-Dev & Server-Admin
Reply With Quote
  #19  
Old 28.08.2018, 12:39
User Name User Name is offline
Registered / Inactive
 
Join Date: Aug 2018
Posts: 14
Default

That's so cool Jiaz, thank you.

How will I know when the plugin is updated? Do I just have to keep clicking the Check Updates button?
Reply With Quote
  #20  
Old 28.08.2018, 12:50
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,049
Default

The update is live since 24.08 Just hit update and restart JDownloader in doubt. You can also check about dialog and check build date. date after >24.08 contains the update.

with updated plugin you can now add album links and albums may contain images that are also part of other albums -> no longer filtered as duplicate
__________________
JD-Dev & Server-Admin
Reply With Quote
  #21  
Old 28.08.2018, 15:29
User Name User Name is offline
Registered / Inactive
 
Join Date: Aug 2018
Posts: 14
Default

Hmmm... well I've updated and restarted twice, build date is 28.08, but it seems to behave as it did before?

I cleared all the previously found links from my list, ran LinkGrabber a couple of times, even did a deep scan, but still only getting 5 images in a folder of 55, etc...

Is there something else I need to do?
Reply With Quote
  #22  
Old 28.08.2018, 15:43
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,049
Default

Can you please provide the exact link you're adding?
Send them to support@jdownloader.org

I can also offer you live help via Teamviewer, just contact me by given mail
__________________
JD-Dev & Server-Admin
Reply With Quote
  #23  
Old 28.08.2018, 18:22
User Name User Name is offline
Registered / Inactive
 
Join Date: Aug 2018
Posts: 14
Default

It's still that first one I tried:

**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #24  
Old 28.08.2018, 20:13
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,049
Default

finally found the little devil wait for next plugin update
your example link will not return over 20k links because of the no longer filtered links.
In case you want to download all links without duplicates, just add the photostream link
__________________
JD-Dev & Server-Admin
Reply With Quote
  #25  
Old 30.08.2018, 16:36
User Name User Name is offline
Registered / Inactive
 
Join Date: Aug 2018
Posts: 14
Default

Awesome, thank you. Link Grabber is now finding all the files in /albums, but when I go to download them all the files have the error 'Try restarting this link (plugin outdated?)'

Is there something I need to do? Have restarted JD2 twice and tried adding the links twice.
Reply With Quote
  #26  
Old 30.08.2018, 17:27
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,049
Default

sorry, caused by changes. will be fixed with next update
__________________
JD-Dev & Server-Admin
Reply With Quote
  #27  
Old 30.08.2018, 17:50
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,049
Default

the update is live
__________________
JD-Dev & Server-Admin
Reply With Quote
  #28  
Old 30.08.2018, 18:35
User Name User Name is offline
Registered / Inactive
 
Join Date: Aug 2018
Posts: 14
Default

Thank you. Links are working now.

But they're not getting packaged into seperate folders (by album name) as they were before? I've got 'Information overwrites packagizer rules' checked, I don't think I've changed any other settings??

Any ideas?
Reply With Quote
  #29  
Old 30.08.2018, 18:56
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,049
Default

"Information overwrites packagizer rules" -> that overwrites *sub folder packagizer rule* and all links will land in set download folder.
You've got to disable that for *sub folder packagizer rule* to work
__________________
JD-Dev & Server-Admin
Reply With Quote
  #30  
Old 09.09.2018, 02:09
User Name User Name is offline
Registered / Inactive
 
Join Date: Aug 2018
Posts: 14
Default

Ah, of course.

Amazing, thank you for all your help and assistance with this.
Reply With Quote
  #31  
Old 10.09.2018, 12:07
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 81,049
Default

You're welcome
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 22:29.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.