[Jiaz]

Wait for next update, then request limit will be *limited* to api only and mirror handling also be be working correct (you have to re-add the links for this to work)

[Jiaz]

Limiting the request interval to api only will be available with next core update next week. I'm not finished yet

[Derp]

Originally Posted by pspzockerscene

"Fast linkcheck" is already always used.

You can still see the filesizes right away because this information is right away for 4chan content.

Oh right, forgot about that.

Originally Posted by Jiaz

Limiting the request interval to api only will be available with next core update next week. I'm not finished yet

Several years ago the authors of the 4chanX-userscript were asked by 4chan-admins to set the minimum interval of hitting the API to 30 seconds or so, the same as the native inline-extension. But that was before it was cloudflare'd, IIRC.
Considering the JSON already contains the filesizes and JD2 doesn't need to hit the image-server to get that info, it shouldn't be a problem at all.

-----
Another question: Is JD2 somehow limiting the concurrent downloads from cloudflare-protected links or something?
When I start to download a package with ~30 images (~11MiB total) from 4chan with 5 concurrent downloads, it takes about 3 minutes for it to finish, with most of them being stuck at "Starting..." for 10-15 seconds.
Or are you seeing the same behavior and it's being limited server-side? I think it changed to this about a week ago.
EDIT:
Another try yesterday: 140 files, 75 MiB, 3 concurrent downloads, took 8 minutes.

[Jiaz]

@Derp: Has nothing to do with cloudflare. At the moment the request limit also affects the downloads. Once the next core update is live, the request limit only limits api requests, current limiter doesn't support different subdomains.
I will release core update later this day, see
https://board.jdownloader.org/showpo...1&postcount=22

[pspzockerscene]

* this update has not been released yet.
Please be patient.

-psp-

[Derp]

Originally Posted by Jiaz

@Derp: Has nothing to do with cloudflare. At the moment the request limit also affects the downloads. Once the next core update is live, the request limit only limits api requests, current limiter doesn't support different subdomains.
I will release core update later this day, see
**External links are only visible to Support Staff**...

Was this change part of the update to Build Sat Nov 28 14:28:09 CET 2020 ?

[Jiaz]

@Derp: Yes, change is live and working fine here

[Derp]

Too bad, download is just as slow as it was before.
But now the downloads aren't stuck at "Starting..." for a while, but at "Download".
Many files have to be retried due to timeout, like this: **External links are only visible to Support Staff****External links are only visible to Support Staff**

Did another test with a set of 50 URLs and 1 simultaneous download per host: JD2 took 160 seconds, the wget-batchfile 47.
Raising the number of simultaneous downloads helps a bit: 30s when setting to 5 downloads.
I guess that's as good as it can get?

------

Found another bug with regard to filenames:

**External links are only visible to Support Staff****External links are only visible to Support Staff**
Post #104266878 has a file with the uploaded filename ".jpg", just the extension, no name-part.
JD2 doesn't add this file to the Linkgrabber-list when copying the 4chan-thread-url, regardless of the state of the filename-setting.

[Jiaz]

your log shows
1.) very slow server response times
Request-Time: 1758ms
2.) read timeouts
Caused by: java.net.SocketTimeoutException: Read timed out

I can reproduce both and looks like server side throttling. Same happens for me in browser too
Fast start and then takes ages for the last percent or end up in read timeout as well.

[Jiaz]

Originally Posted by Derp

Found another bug with regard to filenames:

**External links are only visible to Support Staff****External links are only visible to Support Staff**
Post #104266878 has a file with the uploaded filename ".jpg", just the extension, no name-part.
JD2 doesn't add this file to the Linkgrabber-list when copying the 4chan-thread-url, regardless of the state of the filename-setting.

Will be fixed with next plugin update

[Derp]

Originally Posted by Jiaz

your log shows
1.) very slow server response times
Request-Time: 1758ms
2.) read timeouts
Caused by: java.net.SocketTimeoutException: Read timed out

I can reproduce both and looks like server side throttling. Same happens for me in browser too
Fast start and then takes ages for the last percent or end up in read timeout as well.

Ok, thanks for testing.

-----

Another minor issue:
Since the change, JD2 is adding the MD5-hash to every image file, except WEBM-files.
You can use the URL from post #28 for an example.

[Jiaz]

@Derp:
MD5 Hash is available to every link for me. Plugin doesn't process/add items without md5.
Example link from #28 worked fine for me. Is it missing on linkcrawler or after download?

[Derp]

Originally Posted by Jiaz

@Derp:
MD5 Hash is available to every link for me. Plugin doesn't process/add items without md5.
Example link from #28 worked fine for me. Is it missing on linkcrawler or after download?

You are correct.
In a new JD2-install, it works for me, too.
But in my existing install, the WEBMs don't get the hash added, see pic attached.
Both updated to Build Thu Dec 03 19:21:05 CET 2020
Any ideas?

[Jiaz]

Any Eventscripter Scripts active? I cannot reproduce the issue.
Can you try to provide a log?
Is this screenshot from download or linkcrawler?

[Derp]

Originally Posted by Jiaz

Any Eventscripter Scripts active?

No.

Is this screenshot from download or linkcrawler?

Linkgrabber.

Found the culprit: A "learned file extension" Linkcrawler-rule from over five years ago.

{
  "enabled" : true,
  "cookies" : null,
  "updateCookies" : true,
  "logging" : false,
  "maxDecryptDepth" : 1,
  "id" : 1434728570583,
  "name" : "Learned file extension:webm",
  "pattern" : "(?i).*\\.webm($|\\?.*$)",
  "rule" : "DIRECTHTTP",
  "packageNamePattern" : null,
  "passwordPattern" : null,
  "formPattern" : null,
  "deepPattern" : null,
  "rewriteReplaceWith" : null
}

Adding it to the new install also leads to missing hashes.
No idea why the rule is even in there, because webm was added to the default supported extensions nine years ago.

Shouldn't be happening even with that rule in place, right?

[Jiaz]

Thanks for taking deeper look.
Normally this is expected behaviour because the rule has higher *order* but for this type of rule I've updated JD to forward the meta information correctly. Wait for next core update