JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 25.11.2020, 21:11
nathan1 nathan1 is offline
Black Hole
 
Join Date: Apr 2012
Posts: 274
Default too long on Waiting for Hashcheck

Dear Staff,
I wish to understand why JD2 it stays too long on Waiting for Hashcheck



LOG

Code:
25.11.20 14.10.55 <--> 25.11.20 18.41.27 jdlog://2927425302851/
I also think that JD2 has some problems to capture links because I need to restart JD different times to capture all links. For example, first time capture 30 gb, after restart and copy same links it update and capture 50 gb.

I post another complete LOG to explain this
Code:
25.11.20 14.10.55 <--> 23.11.20 14.35.31 jdlog://0037425302851/

Last edited by nathan1; 25.11.2020 at 21:51.
Reply With Quote
  #2  
Old 26.11.2020, 11:15
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,349
Default

HashChecks are done sequentially and not in parallel to avoid IO thrashing. Looks like your system/IO cannot handle download speed(write) and read(hash check) and thus take very long to check the files.
For example, if you normal hdd can handle 100Mb/s, then you can expect max 30-50Mb/s download speed before the IO will be overloaded.
Also know when you change download folder in DownloadList, then the files will be moved!
__________________
JD-Dev & Server-Admin
Reply With Quote
  #3  
Old 26.11.2020, 12:22
nathan1 nathan1 is offline
Black Hole
 
Join Date: Apr 2012
Posts: 274
Default

@♣Jiaz

Yes, this because I use a cloud service and I think that is about size resources.

But, for second problem, why JD2 can't fetch 1000 links or more correctly but I need to restart JD2 many times to catch them completely?

For example, I have 1000 links to crawl. First time it fetch, for example, 18 gb, but I know that this is not correct size because 1000 links has a real size about 100 gb.
Ok, I restart and then I re-crawl again. Now JD2 adds some gbs so I have 30gb.

But is still uncomplete fetching. I posted a log for that.
Have you idea about this?

Last edited by nathan1; 26.11.2020 at 12:25.
Reply With Quote
  #4  
Old 26.11.2020, 12:36
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,349
Default

Your JDownloader/Java is restricted to 256MB memory which is too little!
GUI+Downloads+Links in DownloadList/Linkgrabber and Plugins. Most likely your Java will run out-of-memory.
How much memory does the system have?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #5  
Old 26.11.2020, 12:45
nathan1 nathan1 is offline
Black Hole
 
Join Date: Apr 2012
Posts: 274
Default

My vps has

CPU:
1 vCore
RAM:
2048 MB
Storage:
55 GB SSD

I try now another vps size (4096, 2 vCore, 80 gb) and then I write you new log
Reply With Quote
  #6  
Old 26.11.2020, 13:01
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,349
Default

No need for newer VPS, you can tell JD to use more
In your JDownloader directory you'll find .vmoptions file
add new line with -Xmx768m to allow up to 768mb of ram
and restart JDownloader
__________________
JD-Dev & Server-Admin
Reply With Quote
  #7  
Old 26.11.2020, 14:37
nathan1 nathan1 is offline
Black Hole
 
Join Date: Apr 2012
Posts: 274
Default

mm... I test with your solution (add -Xmx768m line) but problem is the same

I change and use another vps to check for other problems.
Problems is about fetching

Before to change JDownloader2.vmoptions I try to crawl about 2400 metalarea links block (copy and paste 2400 links into JD2)

First time: it crawl 40 gb. This is not correct
Second time: about 51 gb. I add simply same 2400 links

Ok, I restart JD2 again but now I copy little blocks of links, for example 400, 300, 400.
After every block I restart JD because if simply I add links could happens that JD don't crawl, I don't know why.

Final size for same 2400 links is 124 gb and not 40 gb.


LOG before to edit JDownloader2.vmoptions
Code:
 26.11.20 12.20.17 <--> 26.11.20 10.51.12 jdlog://8847425302851/
LOG after edited JDownloader2.vmoptions
Code:
26.11.20 12.20.17 <--> 26.11.20 12.29.10 jdlog://9847425302851/
Reply With Quote
  #8  
Old 26.11.2020, 14:40
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 55,740
Default

If you add 2400 URLs leading to a forum, chances are high that they'll (temp.) ban you at some point of time.
Are you sure that this did not happen and your issue is happening for other reasons?
Also what you're describing here has nothing todo with your initial "hashcheck issue".

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #9  
Old 26.11.2020, 14:58
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,349
Default

Yes, I think the mass crawling of forum links will lead to ban/bot-detection and then crawler fails
__________________
JD-Dev & Server-Admin
Reply With Quote
  #10  
Old 26.11.2020, 17:12
nathan1 nathan1 is offline
Black Hole
 
Join Date: Apr 2012
Posts: 274
Default

@psp
@Jiaz

I can access on site everytime, also after this massive crawling. But maybe you can have right that Forum can consider crawler as a bot, but I'm not sure about this.
Ok, I follow a more low profile crawling to avoid potential interruptions.

Sorry if thread name is different, I wanted to insert 2 problems in same topic.
Reply With Quote
  #11  
Old 26.11.2020, 18:45
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,349
Default

@nathan1: I recommend smaller batches Also add next batch ones the previous batch is finished.
And browser is maybe different session, so you don't see any error/bot/protection site there.
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 22:53.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.