JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 04.04.2020, 17:02
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 229
Default [rapidgator] jd2 prone to timeouts relative to browser

so in a previous thread https://board.jdownloader.org/showthread.php?t=83565 I discovered a problem with rapidgator (not jd2) that jd2 couldn't resume a download because of something on rapidgator side.

I had come to depend on resuming as my downloads would regularly time out (we are talking 12+ hour downloads). At first I thought it was an issue with my proxies, but it turns out even if I don't use a proxy, it happens just as often. So, I thought it might be rapidgator, but to test, I first tried downloading the same file I had difficulty with in firefox and it completed on its first try.

I dont know what's going on, but to me it seems that jd2 is prone to timing out in a way that firefox is not. While this was going on, I had a 30 hour rclone push to google drive going on and none of those connections broke either.

any good way of debugging this? if I had to throw a dart blindly, I'd wonder if it has anything to do with garbage collection causing hiccups and freezing things long enough for connections to die?
Reply With Quote
  #2  
Old 06.04.2020, 15:44
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,921
Default

A good way would be to post a log.
Please keep in mind that the initial resume issues were NOT caused by any timeout issue but I understood that your report is ment differently - it is you suspecting downloads to stop e.g. in the middle because of a too small timeout value in JD.

Please post your log-ID here | bitte poste deine Log-ID hier.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #3  
Old 08.04.2020, 23:48
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 229
Default

just happened again

08.04.20 00.45.33 <--> 08.04.20 23.47.57 jdlog://6644815302851/
Reply With Quote
  #4  
Old 09.04.2020, 11:47
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,921
Default

In your log I can see that the read timeout of 60 seconds was reached.

I've increased it to 120 seconds for the next update.
This should really be enough ...

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #5  
Old 09.04.2020, 12:22
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

I'm not sure of firefox defaults (or your proxies), putting that aside you can adjust timeout settings yourself globally in advances settings which might help your setup.
InternetConnectionSettings.httpconnecttimeout
InternetConnectionSettings.httpreadtimeout

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #6  
Old 09.04.2020, 12:29
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,921
Default

Aside from that rapidgator has had a lot of issues in the past so a higher timeout in our plugin might be a good idea but to be honest I've never seen any website needing > 60 seconds to continue responding ...

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #7  
Old 09.04.2020, 12:40
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

rapidshare did burst rate stuff, then nothing for a longer period which timed out most ppls downloads. That is the worst one that comes to memory.
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #8  
Old 09.04.2020, 14:01
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 229
Default

Quote:
Originally Posted by raztoki View Post
I'm not sure of firefox defaults (or your proxies), putting that aside you can adjust timeout settings yourself globally in advances settings which might help your setup.
InternetConnectionSettings.httpconnecttimeout
InternetConnectionSettings.httpreadtimeout

raztoki
i already had httpreadtimeout set to 600000 (i.e. max number) which should be 10 minutes. so my quesiton would be, is it not repecting that value (hence 60s timeout) or is the comment of 60s timeout bad assumption and it really a 10 minute timeout? (which would indicate some other problem?)

Last edited by thetoad; 09.04.2020 at 14:11.
Reply With Quote
  #9  
Old 09.04.2020, 14:14
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,921
Default

Your log shows:
Code:
Connection-Timeout: 60000ms
Read-Timeout: 60000ms
-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #10  
Old 09.04.2020, 14:19
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

the timestamps should indicate when request was requested vs received.
on the data side of thing its not logged the last time a data packet was received, and how long the transmission between packetless causing the timeout. I guess you can assume via the http request preheader (its present) there, and work backwards from the timeout error exception.

from one of my logs with default settings
Code:
----------------Request Information-------------
...
Connection-Timeout: 20000ms
Read-Timeout: 60000ms
----------------Request-------------------------
please note that I do remember some operating systems at the TCP/IP stack prevent sockets from been open without data transfer for extended periods, regardless of settings within application/client.
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #11  
Old 09.04.2020, 17:18
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 229
Default

my log might show that, but this is what the settings are in the app when I look at them

https://imgur.com/a/thZ6iQ0
Reply With Quote
  #12  
Old 09.04.2020, 19:38
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 229
Default

so I changed the value and restarted and now I see it updated.

Connection-Timeout: 60000ms
Read-Timeout: 120000ms

don't know why it showed a different thing in the UI before (still show a different value for connection timeout, but I didn't change that one)
Reply With Quote
  #13  
Old 10.04.2020, 12:49
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 229
Default

still getting occasional timeouts on my proxies, but seemed to work a little better.
Reply With Quote
  #14  
Old 11.04.2020, 14:41
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 229
Default

so experience shows 2 minutes works better, but not perfect. hard to determine if this is because connection is actually dying or just braindeadness on their side.

recommendation: every connection should keep track of the longest delta between bytes, when a connection dies after a timeout, can print the longest delta that didn't cause a timeout. if said delta is nowhere near the timeout length, indicative that its actually a problem, if the longest delta recorded is somewhat near the timeout length, perhaps value in increasing timeout.

thoughts?
Reply With Quote
  #15  
Old 14.04.2020, 14:19
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,921
Default

It is very unlikely that this will be changed soon.
Please keep in mind that we also cannopt update all of our default timeouts just for you.
Please either stick to what's there or use EventScripter Scripts to modify your JD's behavior however you'd like.
There is also a big "proxy helper" script linked which may help you.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #16  
Old 25.04.2020, 13:57
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 229
Default

I think there was confusion. I wasn't saying you should increase the timeout just for my experience. I was trying to make a 2 prong proposal.

1) the ability for users to set per hoster timeouts

2) the ability for users to judge if there's a value to increase per hoster timeout (i.e.on timeout can tell if this hoster has long deltas per packets, as rapidgator seems to have)

but mostly working for me now, so not a huge deal
Reply With Quote
  #17  
Old 26.04.2020, 12:16
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

What timeout have you set/increased it to at the moment? Also you should switch to newer Java version as you are still using heavily old Java 1.7 (seems to be very old JDownloader installation) Just rename the java/jre folder within your JDownloader folder and launcher should use the system installed java instead. Due to the age of used Java version and ongoing kernel/network changes, this might also be caused by combination of old java/modern kernel
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 26.04.2020 at 12:22.
Reply With Quote
  #18  
Old 26.04.2020, 12:24
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 229
Default

120K (i.e. 2 minutes) as mentioned above for changing rapidgator and as I said, its working better. Though recaptcha has gotten really annoying of late making it difficult to use but that's not a jd2 isuse. I see the annoyance even in web browser (not logged into google) on my jd2 box)
Reply With Quote
  #19  
Old 26.04.2020, 12:36
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

Thanks for the feedback!
Please try a newer Java version, eg latest 1.8 from adoptopenjdk.net
or from your OS repositories
__________________
JD-Dev & Server-Admin
Reply With Quote
  #20  
Old 01.05.2020, 21:56
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 229
Default

ok, didn't even realize running on java8. updated all java alternatives to openjdk11, so next time I restart it should be running on that.
Reply With Quote
  #21  
Old 02.05.2020, 11:05
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

In case your JDownloader is installed with bundled one. You've got rename the java/jre folder so the launcher will use the system installed java
__________________
JD-Dev & Server-Admin
Reply With Quote
  #22  
Old 23.06.2020, 12:39
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 229
Default longer read timeout?

so as we know, rapidgato has an issue due to the way they rate limit and delays between "packets" get longer and longer as time goes on.

I have my read timeout said to the max (10 minutes I believe) and after 13-14GB it consistently errors out with a read timeout.

any chance the max can be increased?
Reply With Quote
  #23  
Old 23.06.2020, 14:07
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,921
Default

Hi,

we currently got a readtimeout of 120 seconds for rapidgator - that should be more than enough ...

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #24  
Old 23.06.2020, 18:02
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 229
Default

I thought that was 640KB?

it's not. for me. as I wrote, having consistent issues on an 18GB file that is always dying around the 13-14GB mark (yes, in free mode, have a work around and lots of patience) and per my observation its gaps between "packets" grows longer and longer as the file gets larger.

I've been forcing those large ones to only be downlaoded without a proxy.

14.06.20 17.15.59 <--> 23.06.20 18.01.29 jdlog://9457025302851/
Reply With Quote
  #25  
Old 24.06.2020, 13:09
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,921
Default

Quote:
Originally Posted by thetoad View Post
I thought that was 640KB?
??

Youg log contains a lot of these exceptions:
Code:
0:Socks(SOCKS5:)null|EndPoint:rapidgator.net:443|Exception:org.appwork.utils.net.socketconnection.Socks5SocketConnection$Socks5AuthException: ERROR:1
Also your log does indeed contain read timeouts but again:
Why do you think it makes sense to allow higher timeouts than 120 seconds?
Would 180 seconds be better for you?
Have you ever tried it via browser using the same proxies?
Maybe rapidgator just stops sending data sometimes ...

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #26  
Old 25.06.2020, 09:16
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 229
Default

yes, my logs are huge, because I download a lot. I'm not complaining about the proxy connections (I think the proxy connections worsen the rapidgator issue, as no traffic for extended periods of time cause the proxy side to timeout itself).

This is without proxies.

My observation is that the longer a free rapidgator download goes, the longer it pauses and hence induces a timeout.

i.e. before I motivated you to extend the default timeout, I'd get timeouts on even 1-2GB files constantly, once you changed that, it stopped.

When I moved it to 5 minutes, I was able to download 5GB files consistently, but ~10GB were causing problems

when I moved it to 10 minutes I was then able to download ~10GB files consistently.

I can't easily move it beyond that (perhaps hacking the json config directly?), but its now always failing in the 13-14GB range. I'd like to try it with a 20 minute timeout and see what happens.

you maybe right that it doesn't change anything, but I'd like to be able to prove that. I'm not asking to change the rapidgator setting, I'm asking to change the limit in the advance settings.

it could also be that I'm totally wrong about the advance setting and my analysis is off
Reply With Quote
  #27  
Old 25.06.2020, 14:17
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,921
Default

I've added a plugin setting for the next update so you can define the read-timeout yourself:

Please also keep in mind that we're open source so you can simply grab our code, do all kinds of experiments and check back if you found something we should officially change and release for all users

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?

Last edited by pspzockerscene; 25.06.2020 at 14:20.
Reply With Quote
  #28  
Old 25.06.2020, 14:31
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 229
Default

so from what I understand, the the http read timeout doesn't impact this? does that do anything (or does this override it?)
Reply With Quote
  #29  
Old 25.06.2020, 14:42
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,921
Default

What do you mean?
The alread existing advanced setting?

No - the plugin overrides this as it is already using a non-default value.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #30  
Old 25.06.2020, 14:49
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 229
Default

ok, then perhaps my analysis was wrong. I'll experiment with the setting and see if I can complete a the ~19GB file once its available. will take a few days at at 100kbps
Reply With Quote
  #31  
Old 25.06.2020, 15:00
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,921
Default

I've updated it to allow max. 5 minutes - serverside it is uncommon to allow connections to be open for such huge amounts of time e.g. >5 minutes.

Your issue might either be your proxy or RG itself is killing the connection.
Also, so far you're the only user who reported this kind of issue.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #32  
Old 25.06.2020, 16:18
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 229
Default

I might be the only user who is able to download 10-20GB links over a free account regularly from rapidgator
Reply With Quote
  #33  
Old 25.06.2020, 16:19
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,921
Default

Quote:
Originally Posted by thetoad View Post
I might be the only user who is able to download 10-20GB links over a free account regularly from rapidgator
I doubt it.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #34  
Old 25.06.2020, 16:28
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 229
Default

and here I thought I was the only one crazy enough to try and download a 20GB at 100kbps (which is somewhat the equivalent of what I did 2+ decades ago of download 1+GB over a 56k modem)
Reply With Quote
  #35  
Old 07.07.2020, 13:22
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 229
Default

wondering what happened? do plugin updates not happen semi regularly anymore? old experince was that a few days after a commit was made to a plugin, jd2 would get upgraded.
Reply With Quote
  #36  
Old 07.07.2020, 13:26
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,921
Default

Sorry, we're busy with other stuff currently - please be patient and wait.

This has been the only disruption for a long time now.

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #37  
Old 08.07.2020, 12:53
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 229
Default

no problem, was just wondering.
Reply With Quote
  #38  
Old 12.07.2020, 01:19
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 229
Default

so, I'm about 12+GB into my download and my stop watch measured delays between "packets" has increased to about 2m15s (at aroud the 6GB mark it was about half that). about 16 hours to go. will check in the morning to see what the gap is. hopefully it will be able to complete now. but perhaps something else will impact it. (18GB at 100kbps takes a really really long time)
Reply With Quote
  #39  
Old 12.07.2020, 09:57
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 229
Default

now 15+GB through and the delay between "packets" is clos to 2m50s (when I say [packets, I don't mean individual pakcets, its more like bursts, i.e. gets the average download bac k to 100KBps).

also, after some period of time (maybe 1 day?) it seems to allow me to download from the same IP again (i.e. was able to start another download in browser), while JD2 won't let me start another one.

Last edited by thetoad; 12.07.2020 at 10:00.
Reply With Quote
  #40  
Old 12.07.2020, 16:42
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 229
Default

close to 17.5 GB in, and the delay between bursts is now about 3m15s

I think this indicates that my hypothesis was correct, and one should either make the timeout dynamic (i.e. start off with a timeout of 60s, constantly readjust to delta between bursts +60s or make the max even longer than the 300s it currently maxes at. (i.e. I'd expect at 40GB to need longer than 300s between burstsat the growth rate I'm seeing)
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 19:29.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.