[navmesh]

Hi, i encounter this issue with **External links are only visible to Support Staff****External links are only visible to Support Staff**
i 'm using jdownloader java 17 64 bits ,
(i have 24 gb ram and 7gb heap size for jdownloader , and its heap still have spaces)
i add about 15 links (each links contains many images, about hundreds) and the duration display of grabber stuck after amount of times (usually 5-30 minutes), then it stuck and cannot crawl anymore (the duration near the clock icon of grabber stops)
if i add about <7 links, it works with no problems .
For me it's not critical bugs (i can add small amount of links instead of all at once) but i still report here in case the developer insterested in

here is the links example. bug reproduce rate about 70%
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

[notice]

@navmesh: when it's stuck, please create a log and post logID here
https://support.jdownloader.org/Know...d-session-logs

[navmesh]

Originally Posted by notice

@navmesh: when it's stuck, please create a log and post logID here
**External links are only visible to Support Staff**...

I see, thank you, is it needed to enable debug mode , like that tutorial ?

[notice]

debug mode is not required, just create log normal and post logID here. it should show where the plugins/JDownloader is stuck

[navmesh]

i see, thank you

[notice]

Also please check about dialog for free heap memory, if JDownloader runs at max memory or not

[navmesh]

sure, i check heap size frequently , i set max heap to 7 gb and always free more than half when i check

[notice]

@navmesh: I did check plugin and most likely you run into crawl/bot protection. In that case each the requests are no longer in parallel but one after another and whenever website blocks, JDownloader waits 30 seconds. This will *feel* like full stop, cause crawling speed is limited by how often the protection kicks in. When it is totally stuck, most likely they change changed this protection and JDownloader waits forever.

Originally Posted by navmesh

if i add about <7 links, it works with no problems .

Just a coincidence because the crawling plugin is imited to 2 concurrent executions
Most likely it's random hit of bot/crawling protection.

Trying to reproduce the block

[navmesh]

Originally Posted by notice

@navmesh: I did check plugin and most likely you run into crawl/bot protection. In that case each the requests are no longer in parallel but one after another and whenever website blocks, JDownloader waits 30 seconds. This will *feel* like full stop, cause crawling speed is limited by how often the protection kicks in. When it is totally stuck, most likely they change changed this protection and JDownloader waits forever.


Just a coincidence because the crawling plugin is imited to 2 concurrent executions
Most likely it's random hit of bot/crawling protection.

Trying to reproduce the block

i see, thank you for the info

[notice]

@navmesh: how many links do you get/expect?
I did add all your example links and resulted in 613 entries

Please provide a log when it's stuck, so I can confirm it's the bot/crawling protection or something else

[navmesh]

Originally Posted by notice

@navmesh: how many links do you get/expect?
I did add all your example links and resulted in 613 entries

Please provide a log when it's stuck, so I can confirm it's the bot/crawling protection or something else

Thank you very much, i sent you the logs and detail via PM

[navmesh]

Hi again,
the link grabber just gets stuck and i created the log
27.04.23 00.35.41 <--> 27.04.23 10.17.46 jdlog://1894311370661/
Thanks

[notice]

@navmesh: Thanks for the log. There is no sign of any running crawling plugin/process, so I guess you're encountering a rar issue where the *finished* signal of the crawler got lost and thus JDownloader still *thinks* there is ongoing crawler process. That would explain that the counters do not increase and I guess you also cannot abort the crawler? rightclick on the lower right crawler indicator icon and try to abort the process

[notice]

https://board.jdownloader.org/showth...842#post462842

[notice]

[navmesh]

Hi , Thank you for the info
actually i can abort the crawler , i click on the abort button and it stop crawling

[notice]

@navmesh: good to know, helps me to limit possible causes for this!

[navmesh]

I'm glad to hear that