JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 25.11.2021, 17:33
woolf woolf is offline
Super Loader
 
Join Date: Jun 2021
Posts: 28
Default Checking the status of a crawler.

Hello,
when lots of links are added to clipboard, JD starts crawling and adding them to the link grabber, but sometimes it stucks on some of them and they are added in a much later time (or not added at all because JD processes them indefinitely). Is there any way to check what are those "stuck" links?
Is there any gui "queue" of links being processed?

For example I copy 100 links and 95 are already added to linkgrabber by crawler so I know there are still 5 missing, but I can't find out which exactly are those to copy and try download them manually.
Reply With Quote
  #2  
Old 25.11.2021, 17:43
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 62,017
Default

Hi,
Quote:
Originally Posted by woolf View Post
Is there any way to check what are those "stuck" links?
No.
All you can do is cancel crawl processes by rightclicking on the crawler icon -> Cancel

Quote:
Originally Posted by woolf View Post
Is there any gui "queue" of links being processed?
We had a progress bar a while back but it has been removed for various (good) reasons.
Some crawlers have good logging of their progress so you could check your "logs" folder manually (located in your JD install dir).

Quote:
Originally Posted by woolf View Post
For example I copy 100 links and 95 are already added to linkgrabber by crawler so I know there are still 5 missing, but I can't find out which exactly are those to copy and try download them manually.
How would a status help in this case?
JD will re-process all of them but it will not add duplicates to your linkgrabber by default.
So if you e.g. re-add a big twitter profile and you're only waiting for JD to find some items you haven't added last time, indeed you won't know the progress until new URLs show up in your linkgrabber.

While you can't do much here is what you can do:
1. When re-adding big items such as profiles, delete all previously added items from your linkgrabber (or at least move them to the downloadlist) so you can see the added links ("as a progress") live in your linkgrabber.
2. Some plugins have special settings to speed-up the crawl- and/or linkchecking process.
Check if that is possible for the website you're adding links from, see Settings -> Plugins -> Search for the website you're using (no result = no settings available for that website/plugin).
3. For some websites (rare case), you can e.g. define the order to add items or the page/date to start from.
There is no documentation for this so you will ask us separately for each website or look it up in our source code.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #3  
Old 25.11.2021, 18:09
woolf woolf is offline
Super Loader
 
Join Date: Jun 2021
Posts: 28
Default

Thanks for your reply. I didn't mean a status bar but a stack of copied links.

For example I copy following list of links:

link1. com
link2. com
link3. com

JD starts crawling and adding them to link grabber. It starts with link1 so if it's done with this link (doesn't matter if it's online or offline) it is added to link grabber and list for crawler looks now like this:

link2. com
link3. com

But now if crawler stucks on link2 it won't be added to link grabber at all. It will just show this spinning green circle in lower right corner all the time and nothing will happen untill I restart JD.

So I want too see the list of links that were copied but were not added to link grabber (even with the offline status).
Those "stuck" links are probably sites affected by this new cloudflare system - and I want JD to show me the list of these links.

PS
Right click -> abort linkgrabber, doesn't work - green circle keeps spinning (again probably when cloudflare links are involved).
Reply With Quote
  #4  
Old 25.11.2021, 18:43
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 62,017
Default

I'm sorry but that is not possible.

Also there is nu gurantee for the order of added links.
This means that the order links get added may differ from the order you've added links to JD.

Also, some crawlers will process multiple links at the same time while others are limited to 1-X simultaneous crawl processes.

Crawlers should not block/get stuck so it would be helpful if you provided example URLs and logs.

Please post your log-ID here | bitte poste deine Log-ID hier.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 15:39.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2022, Jelsoft Enterprises Ltd.