#1
|
|||
|
|||
trying to understand the download process (rapidgator)
so as noted in another thread, I've stopped being able to resume downloads from rapidgator. at one point it seemed random, but now its gone to all the files I try to download.
This combined with the fact that they don't send bytes consistently, but in bursts probably plays havoc with the proxies I use (or the proxies I use have trouble keeping connections alive for 12-24+ hours that it takes to download some of these files). so I've wanted to dig in a little and perhaps some light could be shed one why things are happeing. from the rapidgator log file, here's the initial download I'm obfuscating things a little i.e. what I'm downloading and the socks ip Code:
BrowserID:3739|BrowserParentID:3723|RequestID:1|URL:**External links are only visible to Support Staff** ->SOCKS5:/server1:1080 ----------------CONNECTRequest(SOCKS5)---------- ->SOCKS5 Hello to:/server1:1080 ->SOCKS5 Offer Plain Authentication ->AUTH user:pass <-AUTH Valid! ->SEND tcp connect request by domain:s18.rapidgator.net|port:80 <-BOUND IPv4:/server1:34140 ------------------------------------------------ ----------------Request Information------------- URL: **External links are only visible to Support Staff** ConnectIP: /server1:1080 Connection-Timeout: 60000ms Read-Timeout: 120000ms ----------------Request------------------------- GET /download/<rapidgator uuid> HTTP/1.1 Host: s18.rapidgator.net User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-US,en;q=0.8 Accept-Encoding: gzip, deflate, br Referer: **External links are only visible to Support Staff** Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3 Cookie: lang=en; download_url=<url>; buyed_from_dl=1; file_id=<id>; sdata__=<whatver sdata is> ----------------Response Information------------ Connection-Time: 5271ms Request-Time: 88ms ----------------Response------------------------ HTTP/1.1 200 OK Server: nginx Date: Mon, 25 May 2020 09:34:20 GMT Content-Type: application/octet-stream Content-Length: 4831838608 Last-Modified: Mon, 27 Apr 2020 03:50:08 GMT Connection: keep-alive Cache-Control: no-cache, private Content-Disposition: attachment; filename="filename" ETag: "5ea65670-120000190" ------------------------------------------------ so we can see that it connected to socks server that I call server1 and everything seems to be good. it then started downloading the 4.5GB file but this connection died, so it tried to reconnect, but on a different IP (could make sense, rapidgator won't allow it to use the same IP I guess, even with same cookies, but as we'll see, it didn't use the cookies Code:
BrowserID:4234|BrowserParentID:4233|RequestID:1|URL:**External links are only visible to Support Staff** -->SOCKS5:/server2:1080 ----------------CONNECTRequest(SOCKS5)---------- ->SOCKS5 Hello to:/server2:1080 ->SOCKS5 Offer Plain Authentication ->AUTH user:pass <-AUTH Valid! ->SEND tcp connect request by domain:s18.rapidgator.net|port:80 <-BOUND IPv4:/server2:45534 ------------------------------------------------ ----------------Request Information------------- URL: **External links are only visible to Support Staff** ConnectIP: /server2:1080 Connection-Timeout: 60000ms Read-Timeout: 120000ms ----------------Request------------------------- GET /download/<rapidgator uuid> HTTP/1.1 Host: s18.rapidgator.net User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-US,en;q=0.8 Accept-Encoding: gzip, deflate, br Range: bytes=3387685064- Referer: **External links are only visible to Support Staff** Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3 Cookie: lang=en; download_url=<rapidgator url>; buyed_from_dl=1; file_id=<file id> ----------------Response Information------------ Connection-Time: 5268ms Request-Time: 66ms ----------------Response------------------------ HTTP/1.1 404 Not Found Server: nginx Content-Type: application/json Transfer-Encoding: chunked Connection: keep-alive Cache-Control: no-cache, private Date: Mon, 25 May 2020 18:54:41 GMT X-Error: Download session unexpected state - 10 ------------------------------------------------ 1) if the proxy changes, I'm guessing there's a good chance it wont work 2) i don't see how it could ever work without the cookie as a different example where I went through the process (not the same file as above) Code:
BrowserID:4080|BrowserParentID:4065|RequestID:1|URL:**External links are only visible to Support Staff** -->SOCKS5:/server3:1080 ----------------CONNECTRequest(SOCKS5)---------- ->SOCKS5 Hello to:/server3:1080 ->SOCKS5 Offer Plain Authentication ->AUTH user:pass <-AUTH Valid! ->SEND tcp connect request by domain:s22.rapidgator.net|port:80 <-BOUND IPv4:/server3:49510 ------------------------------------------------ ----------------Request Information------------- URL: **External links are only visible to Support Staff** ConnectIP: /server3:1080 Connection-Timeout: 60000ms Read-Timeout: 120000ms ----------------Request------------------------- GET /download/<uuid 2> HTTP/1.1 Host: s22.rapidgator.net User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-US,en;q=0.8 Accept-Encoding: gzip, deflate, br Range: bytes=3811225600- Referer: **External links are only visible to Support Staff** Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3 Cookie: lang=en; download_url=<url>; buyed_from_dl=1; file_id=<file id>; sdata__=<whatever sdata is> ----------------Response Information------------ Connection-Time: 403ms Request-Time: 66ms ----------------Response------------------------ HTTP/1.1 500 Internal Server Error Server: nginx Content-Type: application/json Transfer-Encoding: chunked Connection: keep-alive Cache-Control: no-cache, private Date: Mon, 25 May 2020 15:53:52 GMT X-Error: Unexpected range request ------------------------------------------------ |
#2
|
||||
|
||||
most generated final links (session links) are bound to ip address. They have a ttl which you most likely wont know timeframe for. Most providers do not bind to a cookie session, or even require it. Classic example to test this is, to generate a link in your browser then copy url and load it in JD it will download.
If you look at the method calls (in the log also) you can see what part of the code sequence a call is made at. This should give you a better understanding of whats happening. raztoki
__________________
raztoki @ jDownloader reporter/developer http://svn.jdownloader.org/users/170 Don't fight the system, use it to your advantage. :] |
#3
|
|||
|
|||
Wondering if this would be a valid feature request. for us that have multiple connections, it might be hard to determine which ones are reliable and which ones are not, would it be possible to have a log file for connections that is something like
<connection_description>:<url>:<success/failure description (ala success, timeout, auth failure...)> thoughts? |
#4
|
||||
|
||||
logs might be useful as could see errors easier.
The connection manager does have green/red type of thing about connections that are unstable in the problems column You can see info within the status column in download tab, though this does clear when you restart client. Maybe Jiaz could indicate on how connection selection happens with unstable connections
__________________
raztoki @ jDownloader reporter/developer http://svn.jdownloader.org/users/170 Don't fight the system, use it to your advantage. :] Last edited by raztoki; 26.05.2020 at 16:07. |
#5
|
|||
|
|||
yes, I use that occasionally, but haven't found a good workflow to use it to determine as too many variables (i.e. proxy could be done, proxy could be locked out from the server temporarily...)
|
#6
|
||||
|
||||
If I remember correctly, someone also made a big EventScripter proxy script which had more precise errorhandling ...
-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#7
|
|||
|
|||
that's a big thread to dig through :( it needs an index
|
#8
|
||||
|
||||
Just go ahead and ask - the user "mgpai" will often point you to the right direction
-psp- EDIT It is user generated content that's why there is no index. If you find it useful, you're free to add one^^
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#9
|
||||
|
||||
you realise you can you can search in thread right? Assuming its mentioned within that thread. I find google search easier keywords site:board.jdownloader.org
__________________
raztoki @ jDownloader reporter/developer http://svn.jdownloader.org/users/170 Don't fight the system, use it to your advantage. :] |
#10
|
||||
|
||||
Looks like Rapidgator has disabled resume for free users
https://board.jdownloader.org/showthread.php?t=84521
__________________
JD-Dev & Server-Admin |
#11
|
|||
|
|||
that's my thread too!
|
#12
|
||||
|
||||
You see...not enough coffee today
__________________
JD-Dev & Server-Admin |
Thread Tools | |
Display Modes | |
|
|