JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 04.02.2019, 01:12
devdaring
Guest
 
Posts: n/a
Default Instagram accounts with up to/around 1500+ posts will stall the Linkgrabber

Most profiles with around 1500+ posts seem to freeze and bug out the linkgrabber. The best example will be any profile with 5,000+ posts.

Here is an example profile that replicates the stall and bug that will make grabbing other links glitched afterwards and only picks up 10-20 posts then freezes as well -
**External links are only visible to Support Staff****External links are only visible to Support Staff** in my case this account gave up after 2,662 files found when the account has over 5,800 posts. As another example, I've seen profiles with 2000 posts stop at sometimes 1000.

Thanks!

Last edited by devdaring; 05.02.2019 at 23:23. Reason: Via title - Above*, not "up to".
Reply With Quote
  #2  
Old 04.02.2019, 17:47
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

What exactly do you mean by *freeze and bug out* ?
I did some testing and inbetween the instagram server killed the connection that stopps the crawling process

Can you provide some more profiles for testing, so we can try to add some *pauses* to avoid server killing the connection
__________________
JD-Dev & Server-Admin
Reply With Quote
  #3  
Old 04.02.2019, 18:20
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,290
Default

__________________
JD-Dev & Server-Admin
Reply With Quote
  #4  
Old 05.02.2019, 00:50
devdaring
Guest
 
Posts: n/a
Default

Quote:
Originally Posted by Jiaz View Post
What exactly do you mean by *freeze and bug out* ?
I did some testing and inbetween the instagram server killed the connection that stopps the crawling process

Can you provide some more profiles for testing, so we can try to add some *pauses* to avoid server killing the connection
"Freezing" as in the "Crawling for downloads" continues forever without doing anything, so that's the server reset, I'm guessing. Then I need to abort it. After the "freeze", I find what seems like a "bug" where trying to crawl for a new instagram profile will only pick up 12 or so files from the user. It seems that the plugin doesn't respond properly after the server reset.

Accounts that freeze the crawling:
**External links are only visible to Support Staff****External links are only visible to Support Staff** (stops at ~70% finished)
**External links are only visible to Support Staff****External links are only visible to Support Staff** (stops at ~50% finished)
**External links are only visible to Support Staff****External links are only visible to Support Staff** (stops at ~40% finished)
**External links are only visible to Support Staff****External links are only visible to Support Staff** (Untested)
**External links are only visible to Support Staff****External links are only visible to Support Staff** (Untested)
**External links are only visible to Support Staff****External links are only visible to Support Staff** (stops at ~30% finished)
**External links are only visible to Support Staff****External links are only visible to Support Staff** (stops at ~75% finished)

Aborting before it server resets (freezes) stops it from bugging

Accounts that bug out after aborting the frozen crawl:
**External links are only visible to Support Staff****External links are only visible to Support Staff** (31 files found)
This happens to every profile. I'm just providing a quick link to show it happening

The steps I take to fix the bug are (they are completely random attempts to fix it):
1. Abort crawl
2. Delete the "frozen" links folder
3. Add something new to the downloadlist
4. Restarting JDownloader many times
5. Crawl for links from a site other than Instagram
6. Crawling for Instagram profiles works again, at least the ones without too many posts.

In no particular order. I don't know which of these steps or combination of steps fixes it. It does not always fix. Maybe all you have to do is wait a while?
Reply With Quote
  #5  
Old 21.01.2020, 03:37
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,922
Default

Hi devdaring,

this may happen once the rate limit is reached and Instagram returns error 429 on requests.
However, we do have handling for that in place and I just improved it.
Please keep in mind that once the rate limit is reached, JD will wait and retry and wait even more after each retry.
You cannot see this which is why you might think it's "stuck".

Also when adding other links afterwards, they might end up in the rate limit even faster which is why you might only get 30 entries and the wait happens right afterwards --> You think it is stuck.

My recommendation is to only add 1 profile to crawl - after all JD will only crawl one at the time but to avoid confusion it might be a good idea to only add one.

Then if you think it's stuck, memorize the number of added items, let it sit for a while and come back 5 minutes later.
It should have a few hundret more items by then!

I've successfully tested one of your test URLs which contained 6876 items --> Needed about 45 minutes with debugging to complete

After the next update, you should easily be able to crawl all of your testlinks without any issues.

Please keep in mind that Instagram could change their website at any time or even add other limits e.g. based on your current IP address.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 23:08.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.