#1
|
|||
|
|||
Instagram accounts with up to/around 1500+ posts will stall the Linkgrabber
Most profiles with around 1500+ posts seem to freeze and bug out the linkgrabber. The best example will be any profile with 5,000+ posts.
Here is an example profile that replicates the stall and bug that will make grabbing other links glitched afterwards and only picks up 10-20 posts then freezes as well - **External links are only visible to Support Staff****External links are only visible to Support Staff** in my case this account gave up after 2,662 files found when the account has over 5,800 posts. As another example, I've seen profiles with 2000 posts stop at sometimes 1000. Thanks! Last edited by devdaring; 05.02.2019 at 23:23. Reason: Via title - Above*, not "up to". |
#2
|
||||
|
||||
What exactly do you mean by *freeze and bug out* ?
I did some testing and inbetween the instagram server killed the connection that stopps the crawling process Can you provide some more profiles for testing, so we can try to add some *pauses* to avoid server killing the connection
__________________
JD-Dev & Server-Admin |
#3
|
||||
|
||||
__________________
JD-Dev & Server-Admin |
#4
|
|||
|
|||
Quote:
Accounts that freeze the crawling: **External links are only visible to Support Staff****External links are only visible to Support Staff** (stops at ~70% finished) **External links are only visible to Support Staff****External links are only visible to Support Staff** (stops at ~50% finished) **External links are only visible to Support Staff****External links are only visible to Support Staff** (stops at ~40% finished) **External links are only visible to Support Staff****External links are only visible to Support Staff** (Untested) **External links are only visible to Support Staff****External links are only visible to Support Staff** (Untested) **External links are only visible to Support Staff****External links are only visible to Support Staff** (stops at ~30% finished) **External links are only visible to Support Staff****External links are only visible to Support Staff** (stops at ~75% finished) Aborting before it server resets (freezes) stops it from bugging Accounts that bug out after aborting the frozen crawl: **External links are only visible to Support Staff****External links are only visible to Support Staff** (31 files found) This happens to every profile. I'm just providing a quick link to show it happening The steps I take to fix the bug are (they are completely random attempts to fix it): 1. Abort crawl 2. Delete the "frozen" links folder 3. Add something new to the downloadlist 4. Restarting JDownloader many times 5. Crawl for links from a site other than Instagram 6. Crawling for Instagram profiles works again, at least the ones without too many posts. In no particular order. I don't know which of these steps or combination of steps fixes it. It does not always fix. Maybe all you have to do is wait a while? |
#5
|
||||
|
||||
Hi devdaring,
this may happen once the rate limit is reached and Instagram returns error 429 on requests. However, we do have handling for that in place and I just improved it. Please keep in mind that once the rate limit is reached, JD will wait and retry and wait even more after each retry. You cannot see this which is why you might think it's "stuck". Also when adding other links afterwards, they might end up in the rate limit even faster which is why you might only get 30 entries and the wait happens right afterwards --> You think it is stuck. My recommendation is to only add 1 profile to crawl - after all JD will only crawl one at the time but to avoid confusion it might be a good idea to only add one. Then if you think it's stuck, memorize the number of added items, let it sit for a while and come back 5 minutes later. It should have a few hundret more items by then! I've successfully tested one of your test URLs which contained 6876 items --> Needed about 45 minutes with debugging to complete After the next update, you should easily be able to crawl all of your testlinks without any issues. Please keep in mind that Instagram could change their website at any time or even add other limits e.g. based on your current IP address. -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
Thread Tools | |
Display Modes | |
|
|