#1
|
|||
|
|||
JDownloader Program Issue - Right Side Of UI Becomes Unviewable If...
...There are too many Linkgrabber operations running simultaneously. When I say "too many", I mean hundreds. This occurs on all three tabs (Downloads, Linkgrabber, Settings).
In case you are wondering why, and how, I am copying so many links at once, the reason is due to this issue for which I have already started a thread: https://board.jdownloader.org/showthread.php?t=87497 So, because that function is not working properly at the moment, I had to fall back on my previous solution for grabbing my saved posts. Which is, loading all of my saved posts on the web page, and then using a browser extension called "LinkClump" to copy 30 links at a time. At this moment, I had over 5,000 posts to copy. Which, thanks to multi-posts, come out to over 6,000 files, for this particular batch. Now, aside from part of the UI becoming unviewable is a problem in itself, this problem creates another problem for me. The sidebar on the right of the Linkgrabber tab, shows the total number of files being grabbed. Now, since I cannot see these numbers, I cannot even be sure if the files are still being grabbed or not. Is there a dedicated window that can be opened, which will let me view the ongoing progress of the Linkgrabber operations? Because currently all I can see on the Linkgrabber tabs are the green "clock" animations still spinning away. The first link was grabbed at 13:46, and now it is 20:18 and the Linkgrabber is still crawling. Are the files being grabbed, or has the crawler reached traffic limit or what? |
#2
|
||||
|
||||
Quote:
IG may throttle/block the API requests and thus the crawling may take longer/really really long because rate limiting will affect ALL jobs. Adding many single link/posts will of course make you hit that limits very fast and also trigger their suspicious activity detection, see https://board.jdownloader.org/showthread.php?t=87496 Instead of adding the links one by one, you should wait for us to check/fix the "saved" support in IG.
__________________
JD-Dev & Server-Admin Last edited by Jiaz; 23.05.2021 at 16:30. |
#3
|
||||
|
||||
What number exactly do you mean? Maybe can provide a screenshot?
__________________
JD-Dev & Server-Admin |
#4
|
||||
|
||||
See my note about IG crawling/rate limits. Most likely you won't be able to finish those in any time at all. Once you've hit the rate limit it can easily result in 1-30 minutes for a single request. And the huge number of jobs you added will just cause you to hit the limits much faster and more often and more waiting time required
__________________
JD-Dev & Server-Admin |
#5
|
||||
|
||||
I'm sorry but the solution is NOT to just add your links in small batches and add hundreds/thousands of links that way and *hope* the site/service won't detect your crawling. Wait for us to check the saved posts support and then you should be able to mass add them.
__________________
JD-Dev & Server-Admin |
#6
|
|||
|
|||
Yeah, I had decided to give up and aborted all the Linkgrabber operations. Once I did that, I could finally see that JD was only able to grab about 1,700 files out of the total 6,000+ it grabbed when using the "instagram.com/*username*/saved/" url.
Between the JD login issues that FB/Insta keep giving me, and this new issue with saved posts not being named properly, I decided those saved posts were not worth the trouble. I unsaved them all using a browser extension. I will just wait to see if you guys can solve the saved post filename issue, so I can get back to grabbing them all with one url. Meanwhile, I will try to grab my future saved posts without letting them accumulate, so that the JD crawler does not get overloaded. |
Thread Tools | |
Display Modes | |
|
|