JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 25.05.2020, 22:36
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 228
Default trying to understand the download process (rapidgator)

so as noted in another thread, I've stopped being able to resume downloads from rapidgator. at one point it seemed random, but now its gone to all the files I try to download.

This combined with the fact that they don't send bytes consistently, but in bursts probably plays havoc with the proxies I use (or the proxies I use have trouble keeping connections alive for 12-24+ hours that it takes to download some of these files).

so I've wanted to dig in a little and perhaps some light could be shed one why things are happeing.

from the rapidgator log file, here's the initial download

I'm obfuscating things a little i.e. what I'm downloading and the socks ip

Code:
BrowserID:3739|BrowserParentID:3723|RequestID:1|URL:http://s18.rapidgator.net/download/<rapidgator uuid>
->SOCKS5:/server1:1080
----------------CONNECTRequest(SOCKS5)----------
->SOCKS5 Hello to:/server1:1080
->SOCKS5 Offer Plain Authentication
->AUTH user:pass
<-AUTH Valid!
->SEND tcp connect request by domain:s18.rapidgator.net|port:80
<-BOUND IPv4:/server1:34140
------------------------------------------------
----------------Request Information-------------
URL: http://s18.rapidgator.net/download/<rapidgator uuid>
ConnectIP: /server1:1080
Connection-Timeout: 60000ms
Read-Timeout: 120000ms
----------------Request-------------------------
GET /download/<rapidgator uuid> HTTP/1.1
Host: s18.rapidgator.net
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.8
Accept-Encoding: gzip, deflate, br
Referer: https://rapidgator.net/download/captcha
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: lang=en; download_url=<url>; buyed_from_dl=1; file_id=<id>; sdata__=<whatver sdata is>
----------------Response Information------------
Connection-Time: 5271ms
Request-Time: 88ms
----------------Response------------------------
HTTP/1.1 200 OK
Server: nginx
Date: Mon, 25 May 2020 09:34:20 GMT
Content-Type: application/octet-stream
Content-Length: 4831838608
Last-Modified: Mon, 27 Apr 2020 03:50:08 GMT
Connection: keep-alive
Cache-Control: no-cache, private
Content-Disposition: attachment; filename="filename"
ETag: "5ea65670-120000190"
------------------------------------------------
[

so we can see that it connected to socks server that I call server1 and everything seems to be good. it then started downloading the 4.5GB file

but this connection died, so it tried to reconnect, but on a different IP (could make sense, rapidgator won't allow it to use the same IP I guess, even with same cookies, but as we'll see, it didn't use the cookies

Code:
BrowserID:4234|BrowserParentID:4233|RequestID:1|URL:http://s18.rapidgator.net/download/<rapidgator uuid>
-->SOCKS5:/server2:1080
----------------CONNECTRequest(SOCKS5)----------
->SOCKS5 Hello to:/server2:1080
->SOCKS5 Offer Plain Authentication
->AUTH user:pass
<-AUTH Valid!
->SEND tcp connect request by domain:s18.rapidgator.net|port:80
<-BOUND IPv4:/server2:45534
------------------------------------------------
----------------Request Information-------------
URL: http://s18.rapidgator.net/download/<rapidgator uuid>
ConnectIP: /server2:1080
Connection-Timeout: 60000ms
Read-Timeout: 120000ms
----------------Request-------------------------
GET /download/<rapidgator uuid> HTTP/1.1
Host: s18.rapidgator.net
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.8
Accept-Encoding: gzip, deflate, br
Range: bytes=3387685064-
Referer: https://rapidgator.net/file/<rapigator file>
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: lang=en; download_url=<rapidgator url>; buyed_from_dl=1; file_id=<file id>

----------------Response Information------------
Connection-Time: 5268ms
Request-Time: 66ms
----------------Response------------------------
HTTP/1.1 404 Not Found
Server: nginx
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
Cache-Control: no-cache, private
Date: Mon, 25 May 2020 18:54:41 GMT
X-Error: Download session unexpected state - 10
------------------------------------------------
in this example I didn't try to download with it going through the whole process again (as I know it fails), this seems to be somewhat automatic, but seems to be possibly broken on 2 levels

1) if the proxy changes, I'm guessing there's a good chance it wont work
2) i don't see how it could ever work without the cookie

as a different example where I went through the process (not the same file as above)

Code:
BrowserID:4080|BrowserParentID:4065|RequestID:1|URL:http://s22.rapidgator.net/download/<uuid>
-->SOCKS5:/server3:1080
----------------CONNECTRequest(SOCKS5)----------
->SOCKS5 Hello to:/server3:1080
->SOCKS5 Offer Plain Authentication
->AUTH user:pass
<-AUTH Valid!
->SEND tcp connect request by domain:s22.rapidgator.net|port:80
<-BOUND IPv4:/server3:49510
------------------------------------------------
----------------Request Information-------------
URL: http://s22.rapidgator.net/download/<uuid>
ConnectIP: /server3:1080
Connection-Timeout: 60000ms
Read-Timeout: 120000ms
----------------Request-------------------------
GET /download/<uuid 2> HTTP/1.1
Host: s22.rapidgator.net
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.8
Accept-Encoding: gzip, deflate, br
Range: bytes=3811225600-
Referer: https://rapidgator.net/download/captcha
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: lang=en; download_url=<url>; buyed_from_dl=1; file_id=<file id>; sdata__=<whatever sdata is>

----------------Response Information------------
Connection-Time: 403ms
Request-Time: 66ms
----------------Response------------------------
HTTP/1.1 500 Internal Server Error
Server: nginx
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
Cache-Control: no-cache, private
Date: Mon, 25 May 2020 15:53:52 GMT
X-Error: Unexpected range request
------------------------------------------------
thoughts?
Reply With Quote
  #2  
Old 26.05.2020, 03:23
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,226
Default

most generated final links (session links) are bound to ip address. They have a ttl which you most likely wont know timeframe for. Most providers do not bind to a cookie session, or even require it. Classic example to test this is, to generate a link in your browser then copy url and load it in JD it will download.

If you look at the method calls (in the log also) you can see what part of the code sequence a call is made at. This should give you a better understanding of whats happening.

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #3  
Old 26.05.2020, 15:45
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 228
Default

Wondering if this would be a valid feature request. for us that have multiple connections, it might be hard to determine which ones are reliable and which ones are not, would it be possible to have a log file for connections that is something like

<connection_description>:<url>:<success/failure description (ala success, timeout, auth failure...)>

thoughts?
Reply With Quote
  #4  
Old 26.05.2020, 15:59
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,226
Default

logs might be useful as could see errors easier.

The connection manager does have green/red type of thing about connections that are unstable in the problems column

You can see info within the status column in download tab, though this does clear when you restart client.

Maybe Jiaz could indicate on how connection selection happens with unstable connections
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]

Last edited by raztoki; 26.05.2020 at 16:07.
Reply With Quote
  #5  
Old 26.05.2020, 16:32
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 228
Default

yes, I use that occasionally, but haven't found a good workflow to use it to determine as too many variables (i.e. proxy could be done, proxy could be locked out from the server temporarily...)
Reply With Quote
  #6  
Old 26.05.2020, 16:47
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 55,195
Default

If I remember correctly, someone also made a big EventScripter proxy script which had more precise errorhandling ...

-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #7  
Old 26.05.2020, 19:55
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 228
Default

that's a big thread to dig through :( it needs an index
Reply With Quote
  #8  
Old 26.05.2020, 20:04
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 55,195
Default

Just go ahead and ask - the user "mgpai" will often point you to the right direction

-psp-
EDIT

It is user generated content that's why there is no index.
If you find it useful, you're free to add one^^
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #9  
Old 26.05.2020, 20:07
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,226
Default

you realise you can you can search in thread right? Assuming its mentioned within that thread. I find google search easier keywords site:board.jdownloader.org
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #10  
Old 27.05.2020, 12:59
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 68,214
Default

Looks like Rapidgator has disabled resume for free users
https://board.jdownloader.org/showthread.php?t=84521
__________________
JD-Dev & Server-Admin
Reply With Quote
  #11  
Old 27.05.2020, 18:00
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 228
Default

that's my thread too!
Reply With Quote
  #12  
Old 27.05.2020, 18:03
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 68,214
Default

You see...not enough coffee today
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 23:03.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.