#1
|
|||
|
|||
[rapidgator] jd2 prone to timeouts relative to browser
so in a previous thread https://board.jdownloader.org/showthread.php?t=83565 I discovered a problem with rapidgator (not jd2) that jd2 couldn't resume a download because of something on rapidgator side.
I had come to depend on resuming as my downloads would regularly time out (we are talking 12+ hour downloads). At first I thought it was an issue with my proxies, but it turns out even if I don't use a proxy, it happens just as often. So, I thought it might be rapidgator, but to test, I first tried downloading the same file I had difficulty with in firefox and it completed on its first try. I dont know what's going on, but to me it seems that jd2 is prone to timing out in a way that firefox is not. While this was going on, I had a 30 hour rclone push to google drive going on and none of those connections broke either. any good way of debugging this? if I had to throw a dart blindly, I'd wonder if it has anything to do with garbage collection causing hiccups and freezing things long enough for connections to die? |
#2
|
||||
|
||||
A good way would be to post a log.
Please keep in mind that the initial resume issues were NOT caused by any timeout issue but I understood that your report is ment differently - it is you suspecting downloads to stop e.g. in the middle because of a too small timeout value in JD. Please post your log-ID here | bitte poste deine Log-ID hier. -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#3
|
|||
|
|||
just happened again
08.04.20 00.45.33 <--> 08.04.20 23.47.57 jdlog://6644815302851/ |
#4
|
||||
|
||||
In your log I can see that the read timeout of 60 seconds was reached.
I've increased it to 120 seconds for the next update. This should really be enough ... Wartest du auf einen angekündigten Bugfix oder ein neues Feature? Updates werden nicht immer sofort bereitgestellt! Bitte lies unser Update FAQ! | Please read our Update FAQ! --- Are you waiting for recently announced changes to get released? Updates to not necessarily get released immediately! Bitte lies unser Update FAQ! | Please read our Update FAQ! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#5
|
||||
|
||||
I'm not sure of firefox defaults (or your proxies), putting that aside you can adjust timeout settings yourself globally in advances settings which might help your setup.
InternetConnectionSettings.httpconnecttimeout InternetConnectionSettings.httpreadtimeout raztoki
__________________
raztoki @ jDownloader reporter/developer http://svn.jdownloader.org/users/170 Don't fight the system, use it to your advantage. :] |
#6
|
||||
|
||||
Aside from that rapidgator has had a lot of issues in the past so a higher timeout in our plugin might be a good idea but to be honest I've never seen any website needing > 60 seconds to continue responding ...
-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#7
|
||||
|
||||
rapidshare did burst rate stuff, then nothing for a longer period which timed out most ppls downloads. That is the worst one that comes to memory.
__________________
raztoki @ jDownloader reporter/developer http://svn.jdownloader.org/users/170 Don't fight the system, use it to your advantage. :] |
#8
|
|||
|
|||
Quote:
Last edited by thetoad; 09.04.2020 at 13:11. |
#9
|
||||
|
||||
Your log shows:
Code:
Connection-Timeout: 60000ms Read-Timeout: 60000ms
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#10
|
||||
|
||||
the timestamps should indicate when request was requested vs received.
on the data side of thing its not logged the last time a data packet was received, and how long the transmission between packetless causing the timeout. I guess you can assume via the http request preheader (its present) there, and work backwards from the timeout error exception. from one of my logs with default settings Code:
----------------Request Information------------- ... Connection-Timeout: 20000ms Read-Timeout: 60000ms ----------------Request-------------------------
__________________
raztoki @ jDownloader reporter/developer http://svn.jdownloader.org/users/170 Don't fight the system, use it to your advantage. :] |
#11
|
|||
|
|||
my log might show that, but this is what the settings are in the app when I look at them
https://imgur.com/a/thZ6iQ0 |
#12
|
|||
|
|||
so I changed the value and restarted and now I see it updated.
Connection-Timeout: 60000ms Read-Timeout: 120000ms don't know why it showed a different thing in the UI before (still show a different value for connection timeout, but I didn't change that one) |
#13
|
|||
|
|||
still getting occasional timeouts on my proxies, but seemed to work a little better.
|
#14
|
|||
|
|||
so experience shows 2 minutes works better, but not perfect. hard to determine if this is because connection is actually dying or just braindeadness on their side.
recommendation: every connection should keep track of the longest delta between bytes, when a connection dies after a timeout, can print the longest delta that didn't cause a timeout. if said delta is nowhere near the timeout length, indicative that its actually a problem, if the longest delta recorded is somewhat near the timeout length, perhaps value in increasing timeout. thoughts? |
#15
|
||||
|
||||
It is very unlikely that this will be changed soon.
Please keep in mind that we also cannopt update all of our default timeouts just for you. Please either stick to what's there or use EventScripter Scripts to modify your JD's behavior however you'd like. There is also a big "proxy helper" script linked which may help you. -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#16
|
|||
|
|||
I think there was confusion. I wasn't saying you should increase the timeout just for my experience. I was trying to make a 2 prong proposal.
1) the ability for users to set per hoster timeouts 2) the ability for users to judge if there's a value to increase per hoster timeout (i.e.on timeout can tell if this hoster has long deltas per packets, as rapidgator seems to have) but mostly working for me now, so not a huge deal |
#17
|
||||
|
||||
What timeout have you set/increased it to at the moment? Also you should switch to newer Java version as you are still using heavily old Java 1.7 (seems to be very old JDownloader installation) Just rename the java/jre folder within your JDownloader folder and launcher should use the system installed java instead. Due to the age of used Java version and ongoing kernel/network changes, this might also be caused by combination of old java/modern kernel
__________________
JD-Dev & Server-Admin Last edited by Jiaz; 26.04.2020 at 11:22. |
#18
|
|||
|
|||
120K (i.e. 2 minutes) as mentioned above for changing rapidgator and as I said, its working better. Though recaptcha has gotten really annoying of late making it difficult to use but that's not a jd2 isuse. I see the annoyance even in web browser (not logged into google) on my jd2 box)
|
#19
|
||||
|
||||
Thanks for the feedback!
Please try a newer Java version, eg latest 1.8 from adoptopenjdk.net or from your OS repositories
__________________
JD-Dev & Server-Admin |
#20
|
|||
|
|||
ok, didn't even realize running on java8. updated all java alternatives to openjdk11, so next time I restart it should be running on that.
|
#21
|
||||
|
||||
In case your JDownloader is installed with bundled one. You've got rename the java/jre folder so the launcher will use the system installed java
__________________
JD-Dev & Server-Admin |
#22
|
|||
|
|||
longer read timeout?
so as we know, rapidgato has an issue due to the way they rate limit and delays between "packets" get longer and longer as time goes on.
I have my read timeout said to the max (10 minutes I believe) and after 13-14GB it consistently errors out with a read timeout. any chance the max can be increased? |
#23
|
||||
|
||||
Hi,
we currently got a readtimeout of 120 seconds for rapidgator - that should be more than enough ... -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#24
|
|||
|
|||
I thought that was 640KB?
it's not. for me. as I wrote, having consistent issues on an 18GB file that is always dying around the 13-14GB mark (yes, in free mode, have a work around and lots of patience) and per my observation its gaps between "packets" grows longer and longer as the file gets larger. I've been forcing those large ones to only be downlaoded without a proxy. 14.06.20 17.15.59 <--> 23.06.20 18.01.29 jdlog://9457025302851/ |
#25
|
||||
|
||||
??
Youg log contains a lot of these exceptions: Code:
0:Socks(SOCKS5:)null|EndPoint:rapidgator.net:443|Exception:org.appwork.utils.net.socketconnection.Socks5SocketConnection$Socks5AuthException: ERROR:1 Why do you think it makes sense to allow higher timeouts than 120 seconds? Would 180 seconds be better for you? Have you ever tried it via browser using the same proxies? Maybe rapidgator just stops sending data sometimes ... -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#26
|
|||
|
|||
yes, my logs are huge, because I download a lot. I'm not complaining about the proxy connections (I think the proxy connections worsen the rapidgator issue, as no traffic for extended periods of time cause the proxy side to timeout itself).
This is without proxies. My observation is that the longer a free rapidgator download goes, the longer it pauses and hence induces a timeout. i.e. before I motivated you to extend the default timeout, I'd get timeouts on even 1-2GB files constantly, once you changed that, it stopped. When I moved it to 5 minutes, I was able to download 5GB files consistently, but ~10GB were causing problems when I moved it to 10 minutes I was then able to download ~10GB files consistently. I can't easily move it beyond that (perhaps hacking the json config directly?), but its now always failing in the 13-14GB range. I'd like to try it with a 20 minute timeout and see what happens. you maybe right that it doesn't change anything, but I'd like to be able to prove that. I'm not asking to change the rapidgator setting, I'm asking to change the limit in the advance settings. it could also be that I'm totally wrong about the advance setting and my analysis is off |
#27
|
||||
|
||||
I've added a plugin setting for the next update so you can define the read-timeout yourself:
Please also keep in mind that we're open source so you can simply grab our code, do all kinds of experiments and check back if you found something we should officially change and release for all users Wartest du auf einen angekündigten Bugfix oder ein neues Feature? Updates werden nicht immer sofort bereitgestellt! Bitte lies unser Update FAQ! | Please read our Update FAQ! --- Are you waiting for recently announced changes to get released? Updates to not necessarily get released immediately! Bitte lies unser Update FAQ! | Please read our Update FAQ! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download Last edited by pspzockerscene; 25.06.2020 at 13:20. |
#28
|
|||
|
|||
so from what I understand, the the http read timeout doesn't impact this? does that do anything (or does this override it?)
|
#29
|
||||
|
||||
What do you mean?
The alread existing advanced setting? No - the plugin overrides this as it is already using a non-default value. -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#30
|
|||
|
|||
ok, then perhaps my analysis was wrong. I'll experiment with the setting and see if I can complete a the ~19GB file once its available. will take a few days at at 100kbps
|
#31
|
||||
|
||||
I've updated it to allow max. 5 minutes - serverside it is uncommon to allow connections to be open for such huge amounts of time e.g. >5 minutes.
Your issue might either be your proxy or RG itself is killing the connection. Also, so far you're the only user who reported this kind of issue. Wartest du auf einen angekündigten Bugfix oder ein neues Feature? Updates werden nicht immer sofort bereitgestellt! Bitte lies unser Update FAQ! | Please read our Update FAQ! --- Are you waiting for recently announced changes to get released? Updates to not necessarily get released immediately! Bitte lies unser Update FAQ! | Please read our Update FAQ! -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#32
|
|||
|
|||
I might be the only user who is able to download 10-20GB links over a free account regularly from rapidgator
|
#33
|
||||
|
||||
I doubt it.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#34
|
|||
|
|||
and here I thought I was the only one crazy enough to try and download a 20GB at 100kbps (which is somewhat the equivalent of what I did 2+ decades ago of download 1+GB over a 56k modem)
|
#35
|
|||
|
|||
wondering what happened? do plugin updates not happen semi regularly anymore? old experince was that a few days after a commit was made to a plugin, jd2 would get upgraded.
|
#36
|
||||
|
||||
Sorry, we're busy with other stuff currently - please be patient and wait.
This has been the only disruption for a long time now. -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#37
|
|||
|
|||
no problem, was just wondering.
|
#38
|
|||
|
|||
so, I'm about 12+GB into my download and my stop watch measured delays between "packets" has increased to about 2m15s (at aroud the 6GB mark it was about half that). about 16 hours to go. will check in the morning to see what the gap is. hopefully it will be able to complete now. but perhaps something else will impact it. (18GB at 100kbps takes a really really long time)
|
#39
|
|||
|
|||
now 15+GB through and the delay between "packets" is clos to 2m50s (when I say [packets, I don't mean individual pakcets, its more like bursts, i.e. gets the average download bac k to 100KBps).
also, after some period of time (maybe 1 day?) it seems to allow me to download from the same IP again (i.e. was able to start another download in browser), while JD2 won't let me start another one. Last edited by thetoad; 12.07.2020 at 09:00. |
#40
|
|||
|
|||
close to 17.5 GB in, and the delay between bursts is now about 3m15s
I think this indicates that my hypothesis was correct, and one should either make the timeout dynamic (i.e. start off with a timeout of 60s, constantly readjust to delta between bursts +60s or make the max even longer than the 300s it currently maxes at. (i.e. I'd expect at 40GB to need longer than 300s between burstsat the growth rate I'm seeing) |
|
|