JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #21  
Old 20.11.2020, 16:40
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,349
Default

Wait for next update, then request limit will be *limited* to api only and mirror handling also be be working correct (you have to re-add the links for this to work)
__________________
JD-Dev & Server-Admin
Reply With Quote
  #22  
Old 20.11.2020, 20:27
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,349
Default

Limiting the request interval to api only will be available with next core update next week. I'm not finished yet
__________________
JD-Dev & Server-Admin
Reply With Quote
  #23  
Old 22.11.2020, 14:20
Derp Derp is offline
JD VIP
 
Join Date: Oct 2011
Posts: 311
Default

Quote:
Originally Posted by pspzockerscene View Post
"Fast linkcheck" is already always used.

You can still see the filesizes right away because this information is right away for 4chan content.
Oh right, forgot about that.

Quote:
Originally Posted by Jiaz View Post
Limiting the request interval to api only will be available with next core update next week. I'm not finished yet
Several years ago the authors of the 4chanX-userscript were asked by 4chan-admins to set the minimum interval of hitting the API to 30 seconds or so, the same as the native inline-extension. But that was before it was cloudflare'd, IIRC.
Considering the JSON already contains the filesizes and JD2 doesn't need to hit the image-server to get that info, it shouldn't be a problem at all.

-----
Another question: Is JD2 somehow limiting the concurrent downloads from cloudflare-protected links or something?
When I start to download a package with ~30 images (~11MiB total) from 4chan with 5 concurrent downloads, it takes about 3 minutes for it to finish, with most of them being stuck at "Starting..." for 10-15 seconds.
Or are you seeing the same behavior and it's being limited server-side? I think it changed to this about a week ago.
EDIT:
Another try yesterday: 140 files, 75 MiB, 3 concurrent downloads, took 8 minutes.

Last edited by Derp; 23.11.2020 at 09:29.
Reply With Quote
  #24  
Old 23.11.2020, 10:13
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,349
Default

@Derp: Has nothing to do with cloudflare. At the moment the request limit also affects the downloads. Once the next core update is live, the request limit only limits api requests, current limiter doesn't support different subdomains.
I will release core update later this day, see
https://board.jdownloader.org/showpo...1&postcount=22
__________________
JD-Dev & Server-Admin
Reply With Quote
  #25  
Old 24.11.2020, 00:28
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 56,432
Default

* this update has not been released yet.
Please be patient.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #26  
Old 29.11.2020, 12:05
Derp Derp is offline
JD VIP
 
Join Date: Oct 2011
Posts: 311
Default

Quote:
Originally Posted by Jiaz View Post
@Derp: Has nothing to do with cloudflare. At the moment the request limit also affects the downloads. Once the next core update is live, the request limit only limits api requests, current limiter doesn't support different subdomains.
I will release core update later this day, see
**External links are only visible to Support Staff**...
Was this change part of the update to Build Sat Nov 28 14:28:09 CET 2020 ?
Reply With Quote
  #27  
Old 30.11.2020, 10:48
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,349
Default

@Derp: Yes, change is live and working fine here
__________________
JD-Dev & Server-Admin
Reply With Quote
  #28  
Old 01.12.2020, 13:51
Derp Derp is offline
JD VIP
 
Join Date: Oct 2011
Posts: 311
Default

Too bad, download is just as slow as it was before.
But now the downloads aren't stuck at "Starting..." for a while, but at "Download".
Many files have to be retried due to timeout, like this: **External links are only visible to Support Staff****External links are only visible to Support Staff**

Did another test with a set of 50 URLs and 1 simultaneous download per host: JD2 took 160 seconds, the wget-batchfile 47.
Raising the number of simultaneous downloads helps a bit: 30s when setting to 5 downloads.
I guess that's as good as it can get?

------

Found another bug with regard to filenames:

**External links are only visible to Support Staff****External links are only visible to Support Staff**
Post #104266878 has a file with the uploaded filename ".jpg", just the extension, no name-part.
JD2 doesn't add this file to the Linkgrabber-list when copying the 4chan-thread-url, regardless of the state of the filename-setting.
Reply With Quote
  #29  
Old 01.12.2020, 14:06
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,349
Default

your log shows
1.) very slow server response times
Request-Time: 1758ms
2.) read timeouts
Caused by: java.net.SocketTimeoutException: Read timed out

I can reproduce both and looks like server side throttling. Same happens for me in browser too
Fast start and then takes ages for the last percent or end up in read timeout as well.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #30  
Old 01.12.2020, 14:17
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,349
Default

Quote:
Originally Posted by Derp View Post
Found another bug with regard to filenames:

**External links are only visible to Support Staff****External links are only visible to Support Staff**
Post #104266878 has a file with the uploaded filename ".jpg", just the extension, no name-part.
JD2 doesn't add this file to the Linkgrabber-list when copying the 4chan-thread-url, regardless of the state of the filename-setting.
Will be fixed with next plugin update
__________________
JD-Dev & Server-Admin
Reply With Quote
  #31  
Old 01.12.2020, 15:40
Derp Derp is offline
JD VIP
 
Join Date: Oct 2011
Posts: 311
Default

Quote:
Originally Posted by Jiaz View Post
your log shows
1.) very slow server response times
Request-Time: 1758ms
2.) read timeouts
Caused by: java.net.SocketTimeoutException: Read timed out

I can reproduce both and looks like server side throttling. Same happens for me in browser too
Fast start and then takes ages for the last percent or end up in read timeout as well.
Ok, thanks for testing.

-----

Another minor issue:
Since the change, JD2 is adding the MD5-hash to every image file, except WEBM-files.
You can use the URL from post #28 for an example.

Last edited by Derp; 01.12.2020 at 15:43.
Reply With Quote
  #32  
Old 01.12.2020, 15:54
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,349
Default

@Derp:
MD5 Hash is available to every link for me. Plugin doesn't process/add items without md5.
Example link from #28 worked fine for me. Is it missing on linkcrawler or after download?
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 01.12.2020 at 16:10.
Reply With Quote
  #33  
Old 04.12.2020, 10:20
Derp Derp is offline
JD VIP
 
Join Date: Oct 2011
Posts: 311
Default

Quote:
Originally Posted by Jiaz View Post
@Derp:
MD5 Hash is available to every link for me. Plugin doesn't process/add items without md5.
Example link from #28 worked fine for me. Is it missing on linkcrawler or after download?
You are correct.
In a new JD2-install, it works for me, too.
But in my existing install, the WEBMs don't get the hash added, see pic attached.
Both updated to Build Thu Dec 03 19:21:05 CET 2020
Any ideas?
Attached Images
File Type: png 4chan_webm_checksum.png (18.0 KB, 4 views)
Reply With Quote
  #34  
Old 04.12.2020, 10:49
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,349
Default

Any Eventscripter Scripts active? I cannot reproduce the issue.
Can you try to provide a log?
Is this screenshot from download or linkcrawler?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #35  
Old 04.12.2020, 12:03
Derp Derp is offline
JD VIP
 
Join Date: Oct 2011
Posts: 311
Default

Quote:
Originally Posted by Jiaz View Post
Any Eventscripter Scripts active?
No.
Quote:
Is this screenshot from download or linkcrawler?
Linkgrabber.

Found the culprit: A "learned file extension" Linkcrawler-rule from over five years ago.
Code:
{
  "enabled" : true,
  "cookies" : null,
  "updateCookies" : true,
  "logging" : false,
  "maxDecryptDepth" : 1,
  "id" : 1434728570583,
  "name" : "Learned file extension:webm",
  "pattern" : "(?i).*\\.webm($|\\?.*$)",
  "rule" : "DIRECTHTTP",
  "packageNamePattern" : null,
  "passwordPattern" : null,
  "formPattern" : null,
  "deepPattern" : null,
  "rewriteReplaceWith" : null
}
Adding it to the new install also leads to missing hashes.
No idea why the rule is even in there, because webm was added to the default supported extensions nine years ago.

Shouldn't be happening even with that rule in place, right?

Last edited by Derp; 04.12.2020 at 12:18.
Reply With Quote
  #36  
Old 04.12.2020, 13:20
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 69,349
Default

Thanks for taking deeper look.
Normally this is expected behaviour because the rule has higher *order* but for this type of rule I've updated JD to forward the meta information correctly. Wait for next core update
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 04.12.2020 at 13:24.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 20:27.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.