[Solved] instagram.com profile reels crawler does not find all items

[lalaupoa]

can't crawl all the links, only 107, and there are more than 800, even logged into my account


**External links are only visible to Support Staff****External links are only visible to Support Staff**

jdlog://1492311370661/

[pspzockerscene]

How can you know that this profile has 800 videos?
I get 216 items (without IG account) and according to the Instagram Web-API that's all.

In your log I can see that the crawler stopped after 107 items because according to the instagram API there are no more items.
So far I was unable to find any issues on our side.

[lalaupoa]

when I scroll down, I can't save the links that open with any extension, it only captures the first links, even the full screen screenshot extensions don't work, they only take the first page, I recorded a video of my screen


**External links are only visible to Support Staff****External links are only visible to Support Staff**

[pspzockerscene]

I'm sorry but your video only shows you scrolling down, not copying any links.

You can copy links ba highlighting parts of the website and hitting CTRL + C or using this method:
https://support.jdownloader.org/Know...orted-websites
--> Using the "Link Gopher" browser addon, I'am able to catch all links after scrolling down multiple times no problem.

[lalaupoa]

I couldn't copy all the links with any extension, neither from chrome nor firefox, I tried many extensions

The only way was manually with the help of "linkclump", but "linkclump" can only copy 24 links at a time, more than that it doesn't get the links

Then I copied 24 links, scrolled down and copied 24 more links and so on

In this profile there are 758 reels

look at the screenshot 2, when the red squares appear, it no longer copies the links stopping at 24 with linkclump

I send you the dlc file too

[pspzockerscene]

I got no explanation on why the "/reels" endpoint doesn't return more items but here is how I got it working:
Remove "/reels" from that profile URL and let JD crawl all posts of that profile.
You can simply deselect "Image" from "File Type" in the top right corner in your linkgrabber to only see all video results.

This way I get 1577 profile video items and another ~240 video items from all story- and story highlights items.

Is that fine or are those "reels" different items than the posted videos?

[lalaupoa]

yes that I already knew, but that's the problem, not all reels are on the main profile, the person can choose when they post to put the reel on the main profile or not, if it's on the main profile it works, if it's only on reels it doesn't

[pspzockerscene]

I've checked this again:
Indeed the parameter "include_feed_video" was missing in the request, JDownloader is doing.
This resulted in it not returning the same amount of items you would get via browser.

With this parameter added, I found 1513 items which is even more than you expected but that should be the emount of items you get via website.

Bitte auf das nächste CORE-Update warten!

Please wait for the next CORE-Update!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[lalaupoa]

I manually counted the thumbnails on the site, it has 818 reels, but I don't know if the user can post duplicate reels

[pspzockerscene]

I'm sorry but I still can't confirm that.
As explained, after the next update JD will be doing exactly the same requests that the browser is doing and that returned more than 818 items for me.

If you want to pre-test my announced changes, you can get the dev build here:
https://support.jdownloader.org/Know...up-ide-eclipse

[pspzockerscene]

CORE-Updates have been released!
All announced bugfixes and features are live!
Please update your JDownloader and report any issues you find asap.
If this thread gets marked as "[Solved]" by our forum staff you can still post in it and we will read- and reply to it!

CORE-Updates wurden released!
Alle angekündigten Bugfixes/Features sind nun verfügbar!
Bitte JDownloader updaten und eventuelle Bugs schnellstmöglich an uns melden.
Falls dieser Thread vom Team als "[Erledigt]" markiert wird, kannst du weiterhin darin antworten und wir lesen/beantworten auch solche Threads!

-psp-