JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 18.11.2021, 10:48
goldensun87 goldensun87 is offline
Wind Gust
 
Join Date: Feb 2012
Posts: 44
Default Instagram Taking Very Long Time To Crawl Links

I know that this happens if too many API calls are made, but currently it is happening even though I copied around, maybe, 20 links? And no, I did not add all the links simulatenously to the clipboard, I added them one at a time. This should not be happening, right?
Reply With Quote
  #2  
Old 18.11.2021, 14:51
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 59,659
Default

We can't- and never will know all block-measures of Instagram.

I'm unable to reproduce this here.
Here is what you can do/experiment with to remove the blocks:
- Add/remove Instagram account
- Enable/disable proxy/VPN (= change IP)
- Check if the same happens via browser - maybe Instagram is slow in general at this moment

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #3  
Old 18.11.2021, 16:08
goldensun87 goldensun87 is offline
Wind Gust
 
Join Date: Feb 2012
Posts: 44
Default

Quote:
Originally Posted by pspzockerscene View Post
We can't- and never will know all block-measures of Instagram.

I'm unable to reproduce this here.
Here is what you can do/experiment with to remove the blocks:
- Add/remove Instagram account
- Enable/disable proxy/VPN (= change IP)
- Check if the same happens via browser - maybe Instagram is slow in general at this moment

-psp-
Ok, I'll provide a url for an entire profile.

**External links are only visible to Support Staff****External links are only visible to Support Staff**

So, on my end, what exactly has happened, is that the crawler is not able to grab more than 70 items from the profile. If this happens on your end, the number of successfully grabbed files may vary. Also, the LinkGrabber remains stuck on, and I have to abort it manually.

Another thing to note, is that this problem is not causing my account to get flagged and locked for bot activity. So, Instagram is not blocking JD in this instance. My guesses are, either there is a bug in JD, or a change in Instagram's internal code is causing this problem.
Reply With Quote
  #4  
Old 18.11.2021, 16:22
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 59,659
Default

Hi again,

I've checked this and I noticed some things:

1. Without account, I'm not able to get all items as it will simply stop after X items saying account is needed for the rest -> Not a bug (expected behavior) as this is what Instagram can do to limit requests or at least to be able to track down accounts/users that download a lot more easily than without them using an account.

2. With account,. I get about 280 items very quickly but then run into their "rate limiting".
In this case, our crawler will wait some time and then resume.
At this point it is still making progress but really slowly.
I'am now at about 450 crawled items but the wait in between increased to sometimes over 1 minute (again: normal behavior).

Summary:
I was unable to find any bugs/unexpected behavior so far.

...so maybe you just quit the process too early.

Without a log I won't be able to tell you what happened:
Please post your log-ID here | bitte poste deine Log-ID hier.

-psp-
EDIT

Now it stopped at/after 474 items.

EDIT2

Download successful -> 474 items -> 1.06GiB
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist

Last edited by pspzockerscene; 18.11.2021 at 16:28.
Reply With Quote
  #5  
Old 20.11.2021, 07:44
goldensun87 goldensun87 is offline
Wind Gust
 
Join Date: Feb 2012
Posts: 44
Default

I tried logging out of my account, deleting all Insta cookies, and logging in again. This time, trying to crawl the account "vikachekahair", I was able to grab 354 files, but now it has been over an hour, and JD still hasn't been able to grab the remaining files from the account.

Here is my log: 19.11.21 12.12.24 <--> 19.11.21 23.42.33 jdlog://6790925302851/

Edit:So, around 5:30 AM, I noticed that JD finally finished grabbing all the links. But, I cannot say exactly when the grabbing finished, because I wasn't paying attention. Regardless, taking several hours to grab only 475 items, is not a "rate-limiting" issue.

Fast forward to the current time, and JD is taking forever to grab only 3 links, and this is a few hours after the previous grabbing was done. So I think rate limiting should be ruled out at this point. I'm thinking Instagram itself is having issues. Recently, I have seen reports of many Instagram users suffering major downtime with Instagram. Even though I am able to access the site on my browser, perhaps some of their other server components are experiencing difficulties. Hopefully, my log will give you better insight into what is going on.

Last edited by goldensun87; 20.11.2021 at 17:54.
Reply With Quote
  #6  
Old 22.11.2021, 22:49
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 59,659
Default

I can see about 40 minutes of waiting in your log but that's it.

No idea why it took so long.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #7  
Old 23.11.2021, 16:03
goldensun87 goldensun87 is offline
Wind Gust
 
Join Date: Feb 2012
Posts: 44
Default

Quote:
Originally Posted by pspzockerscene View Post
I can see about 40 minutes of waiting in your log but that's it.

No idea why it took so long.

-psp-
Is there any way you can confirm whether it is rate limiting or not? I am thinking this is similar to when a change in Instagram's website code, was causing videos to download very slowly. Well, I'm not sure whether the cause of that was on Instagram's side or JD's side.
Reply With Quote
  #8  
Old 23.11.2021, 16:47
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 59,659
Default

Youg log is very big thus harder to analyze.
Please provide a new log and do the following in beforehand to make sure that it only contains relevant activities:
- Disable all accounts you don't need (e.g. all except IG)
- Disable all URLs you don't need (all except IG)
- Restart JD

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #9  
Old 23.11.2021, 22:48
goldensun87 goldensun87 is offline
Wind Gust
 
Join Date: Feb 2012
Posts: 44
Default

Quote:
Originally Posted by pspzockerscene View Post
Youg log is very big thus harder to analyze.
Please provide a new log and do the following in beforehand to make sure that it only contains relevant activities:
- Disable all accounts you don't need (e.g. all except IG)
- Disable all URLs you don't need (all except IG)
- Restart JD

-psp-
Disabling all other accounts is simple enough. Can you elaborate on the "disable all urls"? What exactly should I do regarding that? Should my "Downloads" tab be cleared of all downloaded files? Finally, when exactly should I capture a log? Will any Instagram grab attempt do? Or, should it only be after a lagging crawl?
Reply With Quote
  #10  
Old 24.11.2021, 00:25
TexasDude TexasDude is offline
Baby Loader
 
Join Date: Aug 2019
Posts: 5
Default

I'm having the exact same problem, and for FWIW, I have a temp fix until it gets resolved........

I grab my links, for instance 4 photos in one post, and if the crawler takes its time instead of grabbing them right away, I restart JD.

I regrab and most of the time it will grab them right away. Then it seems as though I can grab a few more from other posts. When it keeps crawling again, I repeat the above.

Just thought I'd point this out for the time being.
Reply With Quote
  #11  
Old 25.11.2021, 03:55
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 59,659
Default

Quote:
Originally Posted by goldensun87 View Post
Can you elaborate on the "disable all urls"?
I'm sorry indeed that was confusing and unnecessary:
Just make sure JDownloader is not downloading anything and all it does is run that crawl process.

Quote:
Originally Posted by goldensun87 View Post
Or, should it only be after a lagging crawl?
Yes that's what we need.
Then again when uploading logs, rather upload "too much" than not enough but make sure that you select timeframes in which your problem occured.

Quote:
Originally Posted by TexasDude
I grab my links, for instance 4 photos in one post, and if the crawler takes its time instead of grabbing them right away, I restart JD.
So you can even reproduce this with small "image albums"?
Any example URLs/logs available?

As said I was not able to reproduce this issue.
Currently I'am also waiting for Jiaz to provide feedback on another possible issue with our Instagram crawler...

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #12  
Old 25.11.2021, 13:36
goldensun87 goldensun87 is offline
Wind Gust
 
Join Date: Feb 2012
Posts: 44
Default

Ok, I hope this log is not too big.

25.11.21 05.31.16 <--> 25.11.21 05.36.30 jdlog://0711925302851/
Reply With Quote
  #13  
Old 25.11.2021, 17:04
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 59,659
Default

Please wait for our developer Jiaz to find time to investigate this.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 12:40.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.