#1
|
|||
|
|||
Instagram Taking Very Long Time To Crawl Links
I know that this happens if too many API calls are made, but currently it is happening even though I copied around, maybe, 20 links? And no, I did not add all the links simulatenously to the clipboard, I added them one at a time. This should not be happening, right?
|
#2
|
||||
|
||||
We can't- and never will know all block-measures of Instagram.
I'm unable to reproduce this here. Here is what you can do/experiment with to remove the blocks: - Add/remove Instagram account - Enable/disable proxy/VPN (= change IP) - Check if the same happens via browser - maybe Instagram is slow in general at this moment -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#3
|
|||
|
|||
Quote:
**External links are only visible to Support Staff****External links are only visible to Support Staff** So, on my end, what exactly has happened, is that the crawler is not able to grab more than 70 items from the profile. If this happens on your end, the number of successfully grabbed files may vary. Also, the LinkGrabber remains stuck on, and I have to abort it manually. Another thing to note, is that this problem is not causing my account to get flagged and locked for bot activity. So, Instagram is not blocking JD in this instance. My guesses are, either there is a bug in JD, or a change in Instagram's internal code is causing this problem. |
#4
|
||||
|
||||
Hi again,
I've checked this and I noticed some things: 1. Without account, I'm not able to get all items as it will simply stop after X items saying account is needed for the rest -> Not a bug (expected behavior) as this is what Instagram can do to limit requests or at least to be able to track down accounts/users that download a lot more easily than without them using an account. 2. With account,. I get about 280 items very quickly but then run into their "rate limiting". In this case, our crawler will wait some time and then resume. At this point it is still making progress but really slowly. I'am now at about 450 crawled items but the wait in between increased to sometimes over 1 minute (again: normal behavior). Summary: I was unable to find any bugs/unexpected behavior so far. ...so maybe you just quit the process too early. Without a log I won't be able to tell you what happened: Please post your log-ID here | bitte poste deine Log-ID hier. -psp- EDIT Now it stopped at/after 474 items. EDIT2 Download successful -> 474 items -> 1.06GiB
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download Last edited by pspzockerscene; 18.11.2021 at 15:28. |
#5
|
|||
|
|||
I tried logging out of my account, deleting all Insta cookies, and logging in again. This time, trying to crawl the account "vikachekahair", I was able to grab 354 files, but now it has been over an hour, and JD still hasn't been able to grab the remaining files from the account.
Here is my log: 19.11.21 12.12.24 <--> 19.11.21 23.42.33 jdlog://6790925302851/ Edit:So, around 5:30 AM, I noticed that JD finally finished grabbing all the links. But, I cannot say exactly when the grabbing finished, because I wasn't paying attention. Regardless, taking several hours to grab only 475 items, is not a "rate-limiting" issue. Fast forward to the current time, and JD is taking forever to grab only 3 links, and this is a few hours after the previous grabbing was done. So I think rate limiting should be ruled out at this point. I'm thinking Instagram itself is having issues. Recently, I have seen reports of many Instagram users suffering major downtime with Instagram. Even though I am able to access the site on my browser, perhaps some of their other server components are experiencing difficulties. Hopefully, my log will give you better insight into what is going on. Last edited by goldensun87; 20.11.2021 at 16:54. |
#6
|
||||
|
||||
I can see about 40 minutes of waiting in your log but that's it.
No idea why it took so long. -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#7
|
|||
|
|||
Is there any way you can confirm whether it is rate limiting or not? I am thinking this is similar to when a change in Instagram's website code, was causing videos to download very slowly. Well, I'm not sure whether the cause of that was on Instagram's side or JD's side.
|
#8
|
||||
|
||||
Youg log is very big thus harder to analyze.
Please provide a new log and do the following in beforehand to make sure that it only contains relevant activities: - Disable all accounts you don't need (e.g. all except IG) - Disable all URLs you don't need (all except IG) - Restart JD -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#9
|
|||
|
|||
Quote:
|
#10
|
|||
|
|||
I'm having the exact same problem, and for FWIW, I have a temp fix until it gets resolved........
I grab my links, for instance 4 photos in one post, and if the crawler takes its time instead of grabbing them right away, I restart JD. I regrab and most of the time it will grab them right away. Then it seems as though I can grab a few more from other posts. When it keeps crawling again, I repeat the above. Just thought I'd point this out for the time being. |
#11
|
||||
|
||||
I'm sorry indeed that was confusing and unnecessary:
Just make sure JDownloader is not downloading anything and all it does is run that crawl process. Yes that's what we need. Then again when uploading logs, rather upload "too much" than not enough but make sure that you select timeframes in which your problem occured. Quote:
Any example URLs/logs available? As said I was not able to reproduce this issue. Currently I'am also waiting for Jiaz to provide feedback on another possible issue with our Instagram crawler... -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#12
|
|||
|
|||
Ok, I hope this log is not too big.
25.11.21 05.31.16 <--> 25.11.21 05.36.30 jdlog://0711925302851/ |
#13
|
||||
|
||||
Please wait for our developer Jiaz to find time to investigate this.
-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#14
|
||||
|
||||
I've just fixed another Instagram plugin issue.
Please re-check after the next set of CORE-updates, if that has also fixed the issue you were having. Bitte auf das nächste CORE-Update warten! Please wait for the next CORE-Update! Wartest du auf einen angekündigten Bugfix oder ein neues Feature? Updates werden nicht immer sofort bereitgestellt! Bitte lies unser Update FAQ! | Please read our Update FAQ! --- Are you waiting for recently announced changes to get released? Updates to not necessarily get released immediately! Bitte lies unser Update FAQ! | Please read our Update FAQ! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#15
|
|||
|
|||
This issue does seem to be fixed. Thank you.
|
#16
|
||||
|
||||
We haven't made any changes though...
Thanks for your feedback anyways! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
|
|