[User feedback required] Instagram profile downloads are not in original size

[DenniSundaY]

Hi all,

I noticed that when downloading images crawled from an Instagram profile the files aren't in the original size most of the time while I have the option 'try to download original quality' enabled.

This happens mostly when downloading big batches, or so the entire profile at once. That way maybe a handful are in original quality but the latest ones are not, like it can't get to the bigger images after a while. Downloading in smaller batches seems to be a workaround. I just tried downloading 100 links from the 2100 crawled links and they were in original quality.

I can confirm that the images for example from post are 1400x1400 by grabbing it with the Safari developer tools, but JD only grabs the 1080x1080.

This is actually happening for over a month now as I discovered a package I downloaded on April 4th, which also has lower quality than I can grab from Instagram website.

Edit: Is there perhaps something I can do or aren't doing correctly? Like setting the max. simultaneous downloads lower or global request limit. I haven't touched any of those settings though.

[pspzockerscene]

Hi,
please provide example URLs and a log.

Please also test the following:
1. Go to Settings -> Plugins -> instagram.com -> Scroll down to "Profile crawler: Set API preference" --> Set this to "API Only"
2. Now delete- and re-add the previously crawled instagram profile and check if you now get all images in max. quality.

-psp-

[Jiaz]

@DenniSundaY: Please provide a debug log for this. Enable Settings->Advanced Settings->Log.debugmodeenabled and restart JDownloader, reproduce the issue and then create a log, see https://support.jdownloader.org/Know...d-session-logs and post shown logID here

[DenniSundaY]

Here is it: jdlog://7339925302851/

Maybe a little background information what I did, not sure what you guys can see in the logs:
First crawled profile and then downloaded the entire profile. Then I crawled again and downloaded only the last images. At first it was to close to the profile download so I got the lower quality images. Then I waited a while and downloaded the last image again. This one now is in higher quality.

@pspzockerscene: When I set it on API only it can't find anything when crawling. Though when crawled trough website it can still download the original quality, but after about 200 downloads it can only download the lower quality images. So I can't see how the crawl process is in solved as the links seem the same to me, but I'm no developer

Links:
Profile: **External links are only visible to Support Staff****External links are only visible to Support Staff**
latest post: **External links are only visible to Support Staff****External links are only visible to Support Staff**
Image link: **External links are only visible to Support Staff****External links are only visible to Support Staff**

[pspzockerscene]

I wasn't able to find any bugs related to the "Prefer original quality" handling.
Also I've burned my test-account for now (temp ban, Insta wants phone number verification) so I'm tired of this for today.

This is what happened according to your log:
- Those higher quality images are only available when downloading with an active instagram account which was given in your case [I've updated the wording of that "Download original quality" setting to clarify this]
- At some point of time you've reached the Instagram rate-limit in account mode which disabled the usage of your account and made JD fallback to downloading without account (default handling)
--> This caused JD to download the "lower quality" images

For now I recommend setting up an account usage rule for domain "instagram.com" to enforce JD to only perform Instagram downloads using any given account.

I have some ideas to improve the handling for Instagram:

1. Always display error if user prefers higher quality images but it's not possible at this moment for whatever reason
2. or: add a 2nd setting, something like "Allow only original quality downloads" which does exactly what I've described in #1.

What do you think about these ideas?

-psp-

[DenniSundaY]

Thank you for the clear information! I already suspected it had something to do with Instagram's awful rate limits as I already discovered downloading in small batches worked, but doing this manually is time consuming.

Those ideas, if they can be implemented, would be great! And for now I try the account usage rules, but what happens when I reach the rate limit? Will it stop automatically and retry after a given time? I can't use reconnect as I got a fixed IP from my ISP.

I was also thinking as I mention before, downloading in batches of about 100 links with a few minutes between them doesn't seem to trigger the rate limit. But is there a way I can set it up to do that automatically, like with the event scripter? I'm not the best making scripts :')

[Jiaz]

@pspzockerscene: maybe an additional setting in plugin that disables free download when *prefer highest quality* is enabled?

[Jiaz]

Originally Posted by pspzockerscene

[*]or: add a 2nd setting, something like "Allow only original quality downloads" which does exactly what I've described in #1.

I second this!

[Jiaz]

Originally Posted by DenniSundaY

I was also thinking as I mention before, downloading in batches of about 100 links with a few minutes between them doesn't seem to trigger the rate limit. But is there a way I can set it up to do that automatically, like with the event scripter? I'm not the best making scripts :')

@pspzockerscene: maybe customizable rate/request limits in the plugin?

[pspzockerscene]

Originally Posted by DenniSundaY

Those ideas, if they can be implemented, would be great!

I've put it on my TODO-list and will work on it as soon as I find the time.
In the meanwhile you can use an account-usage rule as it will do 100% of what that extra setting would do otherwise

Originally Posted by DenniSundaY

But what happens when I reach the rate limit? Will it stop automatically and retry after a given time?

It will stop and try again later.

Originally Posted by DenniSundaY

I can't use reconnect as I got a fixed IP from my ISP.

Even if you could a reconnect would probably not help.
As I said an account is required to download those images so Instagram would be pretty stupid to put that limit only on your IP.
It will probably be on your used Instagram Account and/or a mix/algorithm of other information.
By testing you might be able how to avoid it.
My assumption would be you can avoid it by either waiting or using another Instagram account.

Originally Posted by DenniSundaY

I was also thinking as I mention before, downloading in batches of about 100 links with a few minutes between them doesn't seem to trigger the rate limit.

Yes and no.
Downloading in batches of X: No
Setting custom rate-limits: Yes
See Settings -> Plugins -> instagram.com -> Scroll down -> "Define global request limit for domain 'instagram.com' in milliseconds (0 = no limit)" --> Set this e.g. to 2000 or 3000
You will have to experiment with that and find the sweet-spot on your own...

Originally Posted by Jiaz

@pspzockerscene: maybe customizable rate/request limits in the plugin?

As explained that is already possible...

-psp-
EDIT

You can find more posts by me about this topic by simply searching our forum (example).

[DenniSundaY]

Originally Posted by pspzockerscene

Even if you could a reconnect would probably not help.
As I said an account is required to download those images so Instagram would be pretty stupid to put that limit only on your IP.
It will probably be on your used Instagram Account and/or a mix/algorithm of other information.
By testing you might be able how to avoid it.
My assumption would be you can avoid it by either waiting or using another Instagram account.

Yeah that was stupid of me. Thought of it the minute I posted it, it is account related of course.

Anyways thanks for the tips, I'll tinker a bit with the global request limit (thought it was only for the crawl process at first) and have set up the usage rule as suggested. So far so good!

[uzarluzar]

Hi, thank you for all the work, I just wanted to let you know that since a few days ago (maybe when you started these changes) it has become impossible to download whole profiles. Putting in the general instagram profile link (for example **External links are only visible to Support Staff****External links are only visible to Support Staff**), instead of adding hundreds of pictures, just produces a couple of 'story highlights' folders, with 25 and 9 files respectively, and nothing else. This is both for 'cache' and using the 'add new links' option.

[Jiaz]

@uzarluzar: most likely same issue as here https://board.jdownloader.org/showpo...49&postcount=7
we will check it

[pspzockerscene]

@uzarluzar
Please try again after the next update and/or provide a log.

Instagram changelog for the next update

• Fixed possible crawler stop when "crawl story" is selected but a user currently doesn't have a story
• added errorhandling for when a user only wants all highlight stories of a profile but that profile has none
• changed "Original quality" setting. This will reset this particular setting once for all users as there are now 3 options instead of on/off! Use "Enforce original quality" if you want JD to download only the higher quality versions. Keep in mind that this all goes for the "attempt". If no higher quality version than on the Instagram website is available, that one will be downloaded instead no matter which option you select! Still it is recommended to use an account usage rule to disable free mode if you only want to download "original quality" media, more see here: https://support.jdownloader.org/Know...-free-download
• fixed mistake due to which an account was always mandatory to crawl user profiles - this is indeed not the case but still recommended as in many cases IG will only let you crawl the first X items when you're not logged in

Bitte auf das nächste CORE-Update warten!

Please wait for the next CORE-Update!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-