JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 23.05.2021, 04:19
goldensun87 goldensun87 is offline
Giga Loader
 
Join Date: Feb 2012
Posts: 90
Default JDownloader Program Issue - Right Side Of UI Becomes Unviewable If...

...There are too many Linkgrabber operations running simultaneously. When I say "too many", I mean hundreds. This occurs on all three tabs (Downloads, Linkgrabber, Settings).

In case you are wondering why, and how, I am copying so many links at once, the reason is due to this issue for which I have already started a thread:

https://board.jdownloader.org/showthread.php?t=87497

So, because that function is not working properly at the moment, I had to fall back on my previous solution for grabbing my saved posts. Which is, loading all of my saved posts on the web page, and then using a browser extension called "LinkClump" to copy 30 links at a time. At this moment, I had over 5,000 posts to copy. Which, thanks to multi-posts, come out to over 6,000 files, for this particular batch.

Now, aside from part of the UI becoming unviewable is a problem in itself, this problem creates another problem for me. The sidebar on the right of the Linkgrabber tab, shows the total number of files being grabbed. Now, since I cannot see these numbers, I cannot even be sure if the files are still being grabbed or not.

Is there a dedicated window that can be opened, which will let me view the ongoing progress of the Linkgrabber operations? Because currently all I can see on the Linkgrabber tabs are the green "clock" animations still spinning away. The first link was grabbed at 13:46, and now it is 20:18 and the Linkgrabber is still crawling. Are the files being grabbed, or has the crawler reached traffic limit or what?
Reply With Quote
  #2  
Old 23.05.2021, 16:27
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,293
Default

Quote:
Originally Posted by goldensun87 View Post
...There are too many Linkgrabber operations running simultaneously. When I say "too many", I mean hundreds. This occurs on all three tabs (Downloads, Linkgrabber, Settings).
Those jobs are queued and max Settings->Advanced Settings->LinkCrawler.maxthreads are running simultaneously, default 12 threads.

IG may throttle/block the API requests and thus the crawling may take longer/really really long because rate limiting will affect ALL jobs.

Adding many single link/posts will of course make you hit that limits very fast and also trigger their suspicious activity detection, see https://board.jdownloader.org/showthread.php?t=87496 Instead of adding the links one by one, you should wait for us to check/fix the "saved" support in IG.
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 23.05.2021 at 16:30.
Reply With Quote
  #3  
Old 23.05.2021, 16:35
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,293
Default

Quote:
Originally Posted by goldensun87 View Post
Now, aside from part of the UI becoming unviewable is a problem in itself, this problem creates another problem for me. The sidebar on the right of the Linkgrabber tab, shows the total number of files being grabbed.
What number exactly do you mean? Maybe can provide a screenshot?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #4  
Old 23.05.2021, 16:37
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,293
Default

Quote:
Originally Posted by goldensun87 View Post
The first link was grabbed at 13:46, and now it is 20:18 and the Linkgrabber is still crawling. Are the files being grabbed, or has the crawler reached traffic limit or what?
See my note about IG crawling/rate limits. Most likely you won't be able to finish those in any time at all. Once you've hit the rate limit it can easily result in 1-30 minutes for a single request. And the huge number of jobs you added will just cause you to hit the limits much faster and more often and more waiting time required
__________________
JD-Dev & Server-Admin
Reply With Quote
  #5  
Old 23.05.2021, 16:39
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,293
Default

I'm sorry but the solution is NOT to just add your links in small batches and add hundreds/thousands of links that way and *hope* the site/service won't detect your crawling. Wait for us to check the saved posts support and then you should be able to mass add them.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #6  
Old 24.05.2021, 22:29
goldensun87 goldensun87 is offline
Giga Loader
 
Join Date: Feb 2012
Posts: 90
Default

Yeah, I had decided to give up and aborted all the Linkgrabber operations. Once I did that, I could finally see that JD was only able to grab about 1,700 files out of the total 6,000+ it grabbed when using the "instagram.com/*username*/saved/" url.

Between the JD login issues that FB/Insta keep giving me, and this new issue with saved posts not being named properly, I decided those saved posts were not worth the trouble. I unsaved them all using a browser extension.

I will just wait to see if you guys can solve the saved post filename issue, so I can get back to grabbing them all with one url. Meanwhile, I will try to grab my future saved posts without letting them accumulate, so that the JD crawler does not get overloaded.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 03:39.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.