JDownloader Community - Appwork GmbH
 

 
 
Thread Tools Display Modes
Prev Previous Post   Next Post Next
  #1  
Old 25.05.2020, 22:36
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 228
Default trying to understand the download process (rapidgator)

so as noted in another thread, I've stopped being able to resume downloads from rapidgator. at one point it seemed random, but now its gone to all the files I try to download.

This combined with the fact that they don't send bytes consistently, but in bursts probably plays havoc with the proxies I use (or the proxies I use have trouble keeping connections alive for 12-24+ hours that it takes to download some of these files).

so I've wanted to dig in a little and perhaps some light could be shed one why things are happeing.

from the rapidgator log file, here's the initial download

I'm obfuscating things a little i.e. what I'm downloading and the socks ip

Code:
BrowserID:3739|BrowserParentID:3723|RequestID:1|URL:http://s18.rapidgator.net/download/<rapidgator uuid>
->SOCKS5:/server1:1080
----------------CONNECTRequest(SOCKS5)----------
->SOCKS5 Hello to:/server1:1080
->SOCKS5 Offer Plain Authentication
->AUTH user:pass
<-AUTH Valid!
->SEND tcp connect request by domain:s18.rapidgator.net|port:80
<-BOUND IPv4:/server1:34140
------------------------------------------------
----------------Request Information-------------
URL: http://s18.rapidgator.net/download/<rapidgator uuid>
ConnectIP: /server1:1080
Connection-Timeout: 60000ms
Read-Timeout: 120000ms
----------------Request-------------------------
GET /download/<rapidgator uuid> HTTP/1.1
Host: s18.rapidgator.net
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.8
Accept-Encoding: gzip, deflate, br
Referer: https://rapidgator.net/download/captcha
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: lang=en; download_url=<url>; buyed_from_dl=1; file_id=<id>; sdata__=<whatver sdata is>
----------------Response Information------------
Connection-Time: 5271ms
Request-Time: 88ms
----------------Response------------------------
HTTP/1.1 200 OK
Server: nginx
Date: Mon, 25 May 2020 09:34:20 GMT
Content-Type: application/octet-stream
Content-Length: 4831838608
Last-Modified: Mon, 27 Apr 2020 03:50:08 GMT
Connection: keep-alive
Cache-Control: no-cache, private
Content-Disposition: attachment; filename="filename"
ETag: "5ea65670-120000190"
------------------------------------------------
[

so we can see that it connected to socks server that I call server1 and everything seems to be good. it then started downloading the 4.5GB file

but this connection died, so it tried to reconnect, but on a different IP (could make sense, rapidgator won't allow it to use the same IP I guess, even with same cookies, but as we'll see, it didn't use the cookies

Code:
BrowserID:4234|BrowserParentID:4233|RequestID:1|URL:http://s18.rapidgator.net/download/<rapidgator uuid>
-->SOCKS5:/server2:1080
----------------CONNECTRequest(SOCKS5)----------
->SOCKS5 Hello to:/server2:1080
->SOCKS5 Offer Plain Authentication
->AUTH user:pass
<-AUTH Valid!
->SEND tcp connect request by domain:s18.rapidgator.net|port:80
<-BOUND IPv4:/server2:45534
------------------------------------------------
----------------Request Information-------------
URL: http://s18.rapidgator.net/download/<rapidgator uuid>
ConnectIP: /server2:1080
Connection-Timeout: 60000ms
Read-Timeout: 120000ms
----------------Request-------------------------
GET /download/<rapidgator uuid> HTTP/1.1
Host: s18.rapidgator.net
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.8
Accept-Encoding: gzip, deflate, br
Range: bytes=3387685064-
Referer: https://rapidgator.net/file/<rapigator file>
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: lang=en; download_url=<rapidgator url>; buyed_from_dl=1; file_id=<file id>

----------------Response Information------------
Connection-Time: 5268ms
Request-Time: 66ms
----------------Response------------------------
HTTP/1.1 404 Not Found
Server: nginx
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
Cache-Control: no-cache, private
Date: Mon, 25 May 2020 18:54:41 GMT
X-Error: Download session unexpected state - 10
------------------------------------------------
in this example I didn't try to download with it going through the whole process again (as I know it fails), this seems to be somewhat automatic, but seems to be possibly broken on 2 levels

1) if the proxy changes, I'm guessing there's a good chance it wont work
2) i don't see how it could ever work without the cookie

as a different example where I went through the process (not the same file as above)

Code:
BrowserID:4080|BrowserParentID:4065|RequestID:1|URL:http://s22.rapidgator.net/download/<uuid>
-->SOCKS5:/server3:1080
----------------CONNECTRequest(SOCKS5)----------
->SOCKS5 Hello to:/server3:1080
->SOCKS5 Offer Plain Authentication
->AUTH user:pass
<-AUTH Valid!
->SEND tcp connect request by domain:s22.rapidgator.net|port:80
<-BOUND IPv4:/server3:49510
------------------------------------------------
----------------Request Information-------------
URL: http://s22.rapidgator.net/download/<uuid>
ConnectIP: /server3:1080
Connection-Timeout: 60000ms
Read-Timeout: 120000ms
----------------Request-------------------------
GET /download/<uuid 2> HTTP/1.1
Host: s22.rapidgator.net
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.8
Accept-Encoding: gzip, deflate, br
Range: bytes=3811225600-
Referer: https://rapidgator.net/download/captcha
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: lang=en; download_url=<url>; buyed_from_dl=1; file_id=<file id>; sdata__=<whatever sdata is>

----------------Response Information------------
Connection-Time: 403ms
Request-Time: 66ms
----------------Response------------------------
HTTP/1.1 500 Internal Server Error
Server: nginx
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
Cache-Control: no-cache, private
Date: Mon, 25 May 2020 15:53:52 GMT
X-Error: Unexpected range request
------------------------------------------------
thoughts?
Reply With Quote
 

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 20:29.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.