JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 01.04.2015, 00:14
Llamatron
Guest
 
Posts: n/a
Default IP checks should happen on the connection being checked

I want JD2 to use multiple IP addresses for parallel downloads. I run JD2 on a computer behind a NAT firewall with a fixed IP. So my "Direct" connection always has the same external IP.

I give JD2 access to other IPs via SOCKS proxies. They should all be working at once.

With settings -> advanced -> "IPCheck uses proxy" disabled:
When trying to check the IP of a connection, JD2 does it via direct, so it always thinks all connections have the same IP.

With settings -> advanced -> "IPCheck uses proxy" enabled: (I had been using this successfully with just one proxy.)
IP checks all happened on the lowest priority connection in the connection manager. It's easy to see from the logs of the proxies that only k2s.cc connections were made on proxy0 and proxy1, while proxy2 had k2s.cc and ipcheck*.jdownloader.org connections.

Shouldn't JD2 always do IP checks using the connection that's getting checked? Why is JD2 designed the way it currently is, and why? Is there some problem with checking IP through a proxy in some cases?


PS, in case anyone's curious about testing this: I got a few IPs from my ISP by changing my ethernet MAC address and DHCPing a new IP. then
Code:
sudo ifconfig eth0:0 x.x.x.x
sudo ifconfig eth0:1 y.y.y.y
sudo ifconfig eth0:2 z.z.z.z
 # each in a separate terminal, with logs printing to stderr:
3proxy/src/socks -i10.0.0.19 -l -p9080 -ex.x.x.x
3proxy/src/socks -i10.0.0.19 -l -p1081 -ey.y.y.y
3proxy/src/socks -i10.0.0.19 -l -p2080 -ez.z.z.z
# port numbers chosen to have the first number different,
# for limited column width in the "Gateway" column
EDIT: This doesn't work. All your connections will have the IP of eth0:0 as a source address

Probably this isn't viable long term, since the DHCP lease for the other IPs will eventually time out and then those connections might stop working.

This probably doesn't work on very many ISPs. If "Reconnect" gives you a new IP, it prob. won't work well, as your ISP prob. recycles IP addresses too fast.

Last edited by Llamatron; 01.04.2015 at 18:26.
Reply With Quote
  #2  
Old 01.04.2015, 18:16
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,044
Default

1.) No need to create proxies. JDownloader is able to use network interfaces/gateways directly. Just setup a DirectIP and either enter the IP of it or name of the network device (eg eth0:0). Routing should do the rest.

2.) JDownloader is not able to *reconnect* a proxy/connection. Because normally a proxy has a static ip and when you have multiple ip's they will not change either. So JDownloader only supports reconnect via
2.1.) non proxy (first gateway given by os)
2.2.) first proxy that is whitelisted/allowed to be used for ipcheck
2.3.) Also note that some plugins do ip check themselves to check for waiting limitations. so this might be the reason why you see k2s and ipcheck on proxy 2. download starts on proxy 2, plugin checks ip (via proxy2).

3.) JDownloader is currently unable to detect ip changes from connections other that none/first proxy(ipceck uses proxy)

By default JDownloader will balance free downloads over all available connections and automatically handle waittime /limits on each connection. You say k2s uses all 3 proxies, so where is the problem) ?

Can you explain what exactly the problem is?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #3  
Old 01.04.2015, 19:36
Llamatron
Guest
 
Posts: n/a
Default

Quote:
Can you explain what exactly the problem is?
The main problem was with my setup. All my proxies were actually getting the same source IP. I thought JD2 was deciding not to start downloads from all proxies, based on incorrect IP checks, but in fact it was trying and getting blocked because there was already a DL running on that IP. In Linux, the source IP for a TCP connection is always the IP of eth0:0, regardless of bind()ing a socket to the IP of another interface. (I checked by connecting to a server that printed the IP that connected to it.)

There might still be something weird going on, because when I tested earlier, I saw connections to k2s on all the proxies, but ipchecks on only one proxy. If the k2s plugin is checking IPs, shouldn't I see ip checks on all the proxies?

I'll test this further. Another thing is that the "socks" server from 3proxy only handles one connection at a time, it seems. So if there's already one connection open via that proxy, other SOCKS requests are refused, which is pretty silly.

So basically there was a lot wrong with my setup; don't waste time on this until I fix it and find anything that might actually be a JD2 problem.




Quote:
Originally Posted by Jiaz View Post
1.) No need to create proxies. JDownloader is able to use network interfaces/gateways directly. Just setup a DirectIP and either enter the IP of it or name of the network device (eg eth0:0). Routing should do the rest.
So IPs put into DirectIP get used as the source address JD2 bind(2)s sockets to before connect(2)? That makes sense.

I was confused because the "proxy type" field says "Direct Gateway" for that type. I thought JD2 was going to get connections routed through an alternate (NAT or normal) first-hop gateway, not the default route. (Although I'm not sure this is even possible through the normal socket API.)

A better name might be "set source IP". Anything without the word "gateway", anyway. The popup dialog for adding one is good, now that I look at it. The problem is the "proxy type" column on the connection table. Using the dropdown there, you just see "direct gateway" as one of the options.

I'm running JD2 on my desktop, which only has one IP (10.xxx). My router is the default gateway, and there's another machine on the internal network that also has external IPs. I want to use this second gateway via SOCKS or an HTTP proxy.


Quote:
2.3.) Also note that some plugins do ip check themselves to check for waiting limitations. so this might be the reason why you see k2s and ipcheck on proxy 2. download starts on proxy 2, plugin checks ip (via proxy2).
Ahhhh, ok. I didn't realize that plugin IP checks were separate from reconnects or JD2 core knowing the IP of its connections.

I was assuming that plugins would ask the core what the IP of a connection was, and it was JD2 core stuff deciding which connection to use to check the current IP.
Reply With Quote
  #4  
Old 01.04.2015, 19:38
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 66,538
Default

Quote:
Originally Posted by Llamatron View Post
Ahhhh, ok. I didn't realize that plugin IP checks were separate from reconnects or JD2 core knowing the IP of its connections.

I was assuming that plugins would ask the core what the IP of a connection was, and it was JD2 core stuff deciding which connection to use to check the current IP.
As stated by jiaz, only SOME plugins do that.
At the moment its just a hand full - the ones I could tell you right now are: rapidgator.net, datafile.com, uploaded.net
Maybe some more but that's all then.
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #5  
Old 02.04.2015, 04:27
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,549
Default

we went down that path of plugins doing IP checks because calling core functions which do not exist in earlier version will prevent feature from working. So when I originally coded the concept in rapidgator (Which is also used in uploaded and datafile) needs to run at the plugin level to give backward capability. K2S doesn't at this stage have any IP checking within plugin source.
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #6  
Old 02.04.2015, 09:41
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,044
Default

In Linux, the source IP for a TCP connection is always the IP of eth0:0, regardless of bind()ing a socket to the IP of another interface. (I checked by connecting to a server that printed the IP that connected to it.)
That is not true. You can bind to specific IP/Interface. What you have to keep in mind is that the IP that is used for outgoing is decided by routing! So you have to customize your routing to use different gateway depending on its source IP/device. By default linux will route all through default gateway even when you have multiple outgoing connections.

So IPs put into DirectIP get used as the source address JD2 bind(2)s sockets to before connect(2)? That makes sense.
-> Yes, but check your routing. it must choose gateway depending on ip/device instead of always returning default gateway -> google should help fast on this one

I'm running JD2 on my desktop, which only has one IP (10.xx. My router is the default gateway, and there's another machine on the internal network that also has external IPs. I want to use this second gateway via SOCKS or an HTTP proxy.

->I suggest to setup dante(socks5 server) on other computer and setup x socks5 proxies. 1 for each outgoing external IPs and then setup those proxies in JDownloader. That should work fine.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #7  
Old 02.04.2015, 22:33
Llamatron
Guest
 
Posts: n/a
Default

Quote:
Originally Posted by Jiaz View Post
That is not true. You can bind to specific IP/Interface.
Thanks, yeah I'd already started to figure this out. It seems the way I was adding alias interfaces was creating /8 entries in the "local" routing table, and one was getting used. The source IP you bind() can be overriden by the src in the routing table entry used for the first hop.

Code:
ip route list table main
ip route list table local
This means you can't be sloppy with routing tables when you care about sending traffic from multiple IPs in the same subnet, not just receiving traffic to all of your aliases.
Reply With Quote
  #8  
Old 03.04.2015, 03:55
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,044
Default

As I dont have multiple gateways myself, its hard for me to help with correct routing settings. I bet google will provide help much faster here

So, everything works now?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #9  
Old 12.04.2015, 06:58
Llamatron
Guest
 
Posts: n/a
Default

I posted a guide / HOWTO on setting this up, since there was some stuff I had to figure out that even the LARTC HOWTO didn't cover. (e.g. modifying the "local" routing table so it works even for multiple IPs that are in the same subnet.)

https://board.jdownloader.org/showth...667#post330667

And yeah, everything seems to work. All the problems I've found and reported so far after getting this working properly are plugin-specific.
Reply With Quote
  #10  
Old 12.04.2015, 09:23
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,044
Default

Thanks for the tutorial
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 08:12.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2023, Jelsoft Enterprises Ltd.