[nathan1]

Dear Staff,
I wish to understand why JD2 it stays too long on Waiting for Hashcheck



LOG

25.11.20 14.10.55 <--> 25.11.20 18.41.27 jdlog://2927425302851/

I also think that JD2 has some problems to capture links because I need to restart JD different times to capture all links. For example, first time capture 30 gb, after restart and copy same links it update and capture 50 gb.

I post another complete LOG to explain this

25.11.20 14.10.55 <--> 23.11.20 14.35.31 jdlog://0037425302851/

[Jiaz]

HashChecks are done sequentially and not in parallel to avoid IO thrashing. Looks like your system/IO cannot handle download speed(write) and read(hash check) and thus take very long to check the files.
For example, if you normal hdd can handle 100Mb/s, then you can expect max 30-50Mb/s download speed before the IO will be overloaded.
Also know when you change download folder in DownloadList, then the files will be moved!

[nathan1]

@♣Jiaz

Yes, this because I use a cloud service and I think that is about size resources.

But, for second problem, why JD2 can't fetch 1000 links or more correctly but I need to restart JD2 many times to catch them completely?

For example, I have 1000 links to crawl. First time it fetch, for example, 18 gb, but I know that this is not correct size because 1000 links has a real size about 100 gb.
Ok, I restart and then I re-crawl again. Now JD2 adds some gbs so I have 30gb.

But is still uncomplete fetching. I posted a log for that.
Have you idea about this?

[Jiaz]

Your JDownloader/Java is restricted to 256MB memory which is too little!
GUI+Downloads+Links in DownloadList/Linkgrabber and Plugins. Most likely your Java will run out-of-memory.
How much memory does the system have?

[nathan1]

My vps has

CPU:
1 vCore
RAM:
2048 MB
Storage:
55 GB SSD

I try now another vps size (4096, 2 vCore, 80 gb) and then I write you new log

[Jiaz]

No need for newer VPS, you can tell JD to use more
In your JDownloader directory you'll find .vmoptions file
add new line with -Xmx768m to allow up to 768mb of ram
and restart JDownloader

[nathan1]

mm... I test with your solution (add -Xmx768m line) but problem is the same

I change and use another vps to check for other problems.
Problems is about fetching

Before to change JDownloader2.vmoptions I try to crawl about 2400 metalarea links block (copy and paste 2400 links into JD2)

First time: it crawl 40 gb. This is not correct
Second time: about 51 gb. I add simply same 2400 links

Ok, I restart JD2 again but now I copy little blocks of links, for example 400, 300, 400.
After every block I restart JD because if simply I add links could happens that JD don't crawl, I don't know why.

Final size for same 2400 links is 124 gb and not 40 gb.


LOG before to edit JDownloader2.vmoptions

 26.11.20 12.20.17 <--> 26.11.20 10.51.12 jdlog://8847425302851/

LOG after edited JDownloader2.vmoptions

26.11.20 12.20.17 <--> 26.11.20 12.29.10 jdlog://9847425302851/

[pspzockerscene]

If you add 2400 URLs leading to a forum, chances are high that they'll (temp.) ban you at some point of time.
Are you sure that this did not happen and your issue is happening for other reasons?
Also what you're describing here has nothing todo with your initial "hashcheck issue".

-psp-

[Jiaz]

Yes, I think the mass crawling of forum links will lead to ban/bot-detection and then crawler fails

[nathan1]

@psp
@Jiaz

I can access on site everytime, also after this massive crawling. But maybe you can have right that Forum can consider crawler as a bot, but I'm not sure about this.
Ok, I follow a more low profile crawling to avoid potential interruptions.

Sorry if thread name is different, I wanted to insert 2 problems in same topic.

[Jiaz]

@nathan1: I recommend smaller batches Also add next batch ones the previous batch is finished.
And browser is maybe different session, so you don't see any error/bot/protection site there.