JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 06.01.2010, 16:50
nzelan
Guest
 
Posts: n/a
Default multiple proxy support / automatic rotating list

Hi guys,
I've read in the different threads that you're currently working on the topic.

Could you please give use some status about an eventual release date ?

In addition, I'd just like to share that:
1) you should not limit the size of the proxy list : why ? most of public proxies won't allow download features, those who does support this have poor performance, so only dedicated or non public proxy are worth using, and there is absolutely no issue of performance or whatever with it. Think about worldwide company proxies for instance.
2) it may be possible to use multiple proxies at the same time, with a configurable "concurrency" option for maximum occurences.

3) using proxies will never harm file hosters more than before !

some other soft downloaders support point1), or point 2).

Would be good if JD could support both !

Thank you anyway for your great job !
nz
Reply With Quote
  #2  
Old 07.01.2010, 12:38
remi
Guest
 
Posts: n/a
Cool

I completely agree with your post. I think it was not possible to implement it in the current download system. For its status, please look at Feature #688 : ProxyList support

Since this "rotating proxy list" feature is one of the most wanted features on this board I wouldn't understand it can't be included in the next version of the download engine.
Reply With Quote
  #3  
Old 08.01.2010, 12:14
nzelan
Guest
 
Posts: n/a
Default

thanks for the information,
and let's cross the fingers
nz
Reply With Quote
  #4  
Old 08.01.2010, 12:58
remi
Guest
 
Posts: n/a
Cool

In the mean time and if you've enough RAM memory, you can run multiple instances of jD with a different proxy server address. Just make copies of your jD folder and start the exe or jar files. Don't forget to split your link lists as well.
Reply With Quote
  #5  
Old 08.01.2010, 13:40
drbits's Avatar
drbits drbits is offline
JD English Support (inactive)
 
Join Date: Sep 2009
Location: Physically in Los Angeles, CA, USA
Posts: 4,434
Default

@remi:

Have you looked at the Tor network (or Vidalia)? That might help, but it would have to be set up to not change proxies in the middle of a download.
Reply With Quote
  #6  
Old 09.01.2010, 12:21
remi
Guest
 
Posts: n/a
Cool

If you use jD on the TOR network, be careful for bandwidth hogging if you transfer massive amounts of data. Tor enables Internet anonymity by thwarting network traffic analysis. It should be reserved for people suffering from extremely communist dictatorships.
Reply With Quote
  #7  
Old 09.01.2010, 12:29
drbits's Avatar
drbits drbits is offline
JD English Support (inactive)
 
Join Date: Sep 2009
Location: Physically in Los Angeles, CA, USA
Posts: 4,434
Default

Like the US under the Bush administration?

Actually, I think you can use Tor with a level of 1 (the default is 3), so that you hog less of the resources. The best solution I have heard about is that people work together to lease a virtual server and a Premium account. But RS and HF are adding further restrictions that make such use difficult.
__________________
Please, in each Forum, Read the Rules!.Helpful Links. Read before posting.
Reply With Quote
  #8  
Old 10.01.2010, 15:30
remi
Guest
 
Posts: n/a
Cool

Quote:
Originally Posted by drbits View Post
Like the US under the Bush administration?
I don't see many differences between Bush and Obama's policies, but the USA are still not an extremely communist country, but it's getting worse every year. Do you remember Michael Moore's documentary Sicko on health care? Do you know which country/isle he cited as having the "best and cheapest" health care system?

I think foreign policy is much more inspired by communism, but I don't see any difference between Bush, Obama or whatever US administration in the last decades. The USA are actively supporting and creating more and more extremely corrupt and communist regimes. Most of the USA's gigantic national debt is covered by China - I guess you don't need a picture about who's controlling who?
Reply With Quote
  #9  
Old 14.01.2010, 01:57
hippofactory
Guest
 
Posts: n/a
Default Multiple simultaneous HTTP proxies

Hi,

What'd you think of this idea:

Have JDownloader take a list of HTTP or SOCKS proxies, and it will be able to do multiple downloads from hosts that restrict connections per IP.
Reply With Quote
  #10  
Old 14.01.2010, 13:54
remi
Guest
 
Posts: n/a
Cool

.

Last edited by remi; 22.01.2010 at 11:03. Reason: redundant because of merge
Reply With Quote
  #11  
Old 14.01.2010, 15:17
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,920
Default

Merged these 2 threads...

GreeZ pspzockerscene
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #12  
Old 21.01.2010, 15:39
little little is offline
DSL User
 
Join Date: Mar 2009
Posts: 39
Default

Any news on this ?
I heard that someone wrote an updated proxy support which should just be included :-)
Reply With Quote
  #13  
Old 22.01.2010, 11:05
remi
Guest
 
Posts: n/a
Default

I Feature #688 I see that "Done" is equal to 0% and the priority is Low.
Reply With Quote
  #14  
Old 25.01.2010, 20:15
E71
Guest
 
Posts: n/a
Default

That's definitely my most wanted feature too.

I have dedicated hosting I normally use as SOCKS proxy for browsing the net.
Would love to have jD rotate between my direct internet connection and the proxy when download limits are reached.

Also, I too am hoping there are no proxy limits... at least no less than 4 slots (including direct connection as an option).
Reply With Quote
  #15  
Old 26.01.2010, 01:05
drbits's Avatar
drbits drbits is offline
JD English Support (inactive)
 
Join Date: Sep 2009
Location: Physically in Los Angeles, CA, USA
Posts: 4,434
Default

I believe that the developers have the incorrect notion that proxies are not for downloading unless your ISP requires it.

They don't want people using Tor, etc. for downloading.

There are proxy manager programs that you can use. You would set JD to use a proxy on Localhost:8080 and the proxy manager program would rotate your proxies from a list. You download a new list daily from Google.
Reply With Quote
  #16  
Old 26.01.2010, 01:27
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,920
Default

Using tor to download something is indeed a VERY VERY bad idea because it's very slow (also because many people use it for downloading -.-) and well i's just not made for downloading...

GreeZ pspzockerscene
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #17  
Old 26.01.2010, 08:25
E71
Guest
 
Posts: n/a
Unhappy

Quote:
Originally Posted by drbits View Post
I believe that the developers have the incorrect notion that proxies are not for downloading unless your ISP requires it.

They don't want people using Tor, etc. for downloading.

There are proxy manager programs that you can use. You would set JD to use a proxy on Localhost:8080 and the proxy manager program would rotate your proxies from a list. You download a new list daily from Google.
Aww shoot. Embarrassingly enough, I haven't actually used Tor before... like ever so I don't know but I'm assuming there's no way of blocking access to Tor so that private proxies can be used?

I'm afraid an external proxy app won't be much help since it's not aware of when a new IP is required (in other words, is a fully manual process) otherwise I have a tool called Proxifier that can do that.

Damn, that is really bad news.
Reply With Quote
  #18  
Old 26.01.2010, 13:03
remi
Guest
 
Posts: n/a
Default

When the event manager comes back an external proxy changer might be a solution.
Reply With Quote
  #19  
Old 26.01.2010, 19:25
drbits's Avatar
drbits drbits is offline
JD English Support (inactive)
 
Join Date: Sep 2009
Location: Physically in Los Angeles, CA, USA
Posts: 4,434
Default

How about a batch script for Reconnection that tells Proxifier to change proxies?

I bet that would be very popular.
Reply With Quote
  #20  
Old 27.01.2010, 12:50
remi
Guest
 
Posts: n/a
Default

Wow, good idea. Can you write that?
Reply With Quote
  #21  
Old 05.02.2010, 10:46
animebuster
Guest
 
Posts: n/a
Default

the Status of this has been changed two days ago

http://svn.jdownloader.org/issues/show/688

Botzi edited it and said "Already implemented in the ProxyRotation addon."

Anyone know where you could get this addon? I cant find it in the Plugins section...
Reply With Quote
  #22  
Old 05.02.2010, 13:47
remi
Guest
 
Posts: n/a
Default

It must exist, because there is this recent thread in the Polish forum :- ProxyRotation

This is all I could find. It might be included in the Nightly though.
Reply With Quote
  #23  
Old 06.02.2010, 00:25
drbits's Avatar
drbits drbits is offline
JD English Support (inactive)
 
Join Date: Sep 2009
Location: Physically in Los Angeles, CA, USA
Posts: 4,434
Default

Yes, it is in Nightly (next release). You specify a list of ploxies as hxxp://user:password@ip:port . It also supports SOCKS proxies.

I don't have anymore information about it. I assume that the proxy will be changed each time a reconnect would take place.
Reply With Quote
  #24  
Old 16.02.2010, 22:57
zzz
Guest
 
Posts: n/a
Default

has it been released yet?

Last edited by zzz; 16.02.2010 at 23:37.
Reply With Quote
  #25  
Old 17.02.2010, 00:15
pspzockerscene's Avatar
pspzockerscene pspzockerscene is online now
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,920
Default

Quote:
Originally Posted by drbits View Post
Yes, it is in Nightly
Boardsearch->"Nightly":rolleyes:

GreeZ psp
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #26  
Old 17.02.2010, 01:53
zzz
Guest
 
Posts: n/a
Default

Thanks sir!

I've downloaded nightly and added some proxy to proxy rotation addon, but it doesn't seem to be using this list (still waiting for new IP), do I need some reconnect script to make it work?
Reply With Quote
  #27  
Old 18.02.2010, 13:57
remi
Guest
 
Posts: n/a
Default

Can you find something in your log?
Reply With Quote
  #28  
Old 19.02.2010, 22:56
shawn8888
Guest
 
Posts: n/a
Default

Proxy list is finally supported?! I have not used JD for a long time just because of this feature. I would love to try it out!

Can anyone share me a documentation about how to use it?
Reply With Quote
  #29  
Old 20.02.2010, 03:38
drbits's Avatar
drbits drbits is offline
JD English Support (inactive)
 
Join Date: Sep 2009
Location: Physically in Los Angeles, CA, USA
Posts: 4,434
Default

Shawn,

It is in the test (Nightly version). This is only for experienced users, as there is no guarantee it will work from day to day. On the other hand, I have used Nightly for three months and had only one major problem.

To install Nightly (on Windows), exit JDownloader, backup that directory, and set a system restore point (yes, I am being overly cautious). Then execute the windows_ALPHA.bat script. It takes a while, but it upgrades each of your JD files that need to change. Too change back, you use windows_STABLE.bat.

Proxy rotation is one of the addons. You check the box next to it and then an item appears in the settings tree (under addons) for Proxy rotation. There is a page for you to list your proxies, one per line.

You have to supply your own list of proxies. You can search for "google proxy list" in Google. One of the groups keeps a rotating list of proxy addresses, but it changes daily. The # in the examples are to make those lines comments, don't put a # in your own.

Remember, you are testing this for the JDownloader team and users. If it doesn't work at first, please submit a description of the problem, a log, your list of proxies, and a list of URLs you were trying to download.

I think it rotates proxies as if it were an Automatic Reconnection, but I do not know whether you have to set Automatic Reconnection to ON or OFF (You tell us here).

Thank You and Good Luck
__________________
Please, in each Forum, Read the Rules!.Helpful Links. Read before posting.
Reply With Quote
  #30  
Old 20.02.2010, 06:14
shawn8888
Guest
 
Posts: n/a
Default

Quote:
Originally Posted by drbits View Post
Thank You and Good Luck
Thank you so much for your great help, drbits!
I will definitely give it a try, but I can not find "windows_ALPHA.bat". The only thing I found is windows_restore.bat and windows_createlog.bat, in jd folder.

So, I re-download the zip jD from homepage, but still can not find the bat file.

Last edited by shawn8888; 20.02.2010 at 06:22.
Reply With Quote
  #31  
Old 20.02.2010, 12:52
zzz
Guest
 
Posts: n/a
Default

Quote:
Originally Posted by shawn8888 View Post
Thank you so much for your great help, drbits!
I will definitely give it a try, but I can not find "windows_ALPHA.bat". The only thing I found is windows_restore.bat and windows_createlog.bat, in jd folder.

So, I re-download the zip jD from homepage, but still can not find the bat file.
try this: http://board.jdownloader.org/showpos...9&postcount=33
it worked for me
Reply With Quote
  #32  
Old 23.05.2010, 12:30
remi
Guest
 
Posts: n/a
Default

Reading the posts on this forum, I wonder whether a sufficient number of people would ever become aware of the possibilities of proxy servers.

I wonder how hosts could ever detect that a particular downloader is using multiple proxy server addresses to download.
I think well before the DoS attack would come, hosts will have assembled lists of these proxy server addresses and probably block them.
Reply With Quote
  #33  
Old 24.05.2010, 05:09
drbits's Avatar
drbits drbits is offline
JD English Support (inactive)
 
Join Date: Sep 2009
Location: Physically in Los Angeles, CA, USA
Posts: 4,434
Default

As I wrote above, Proxy Rotation will be part of the next major release of JDownloader. The release is behind the schedule I was given 2 months ago, but I have not heard a new estimate.

As far as I know, the support of multiple concurrent proxies will not be in this release or the next one.

Redacted: I still haven't even seen a request for that in Suggestions and Wishes. Nor have I seen a use example (a scenario) for this feature that is compelling.

Last edited by drbits; 24.05.2010 at 23:51. Reason: Redact a statement.
Reply With Quote
  #34  
Old 24.05.2010, 14:14
remi
Guest
 
Posts: n/a
Default

Quote:
Originally Posted by drbits View Post
As far as I know, the support of multiple concurrent proxies will not be in this release or the next one. I still haven't even seen a request for that in Suggestions and Wishes. Nor have I seen a use example (a scenario) for this feature that is compelling.
Read post #32.

The advantage is to download more in less time, using free download slots.

Of course the software's architecture must be able to support this and as most "insiders" know, this isn't possible yet.
Reply With Quote
  #35  
Old 25.05.2010, 00:37
drbits's Avatar
drbits drbits is offline
JD English Support (inactive)
 
Join Date: Sep 2009
Location: Physically in Los Angeles, CA, USA
Posts: 4,434
Default

Messages 32-40 accidentally deleted in attempt to split thread.

I am sorry for the inconvenience.
------------------------------------------------------------
Multiple Concurrent Proxies has been added as a feature request in the bugtracker for a future version. It is expected that several major projects must be completed before JD is ready for this feature.

The request was expanded to include support for multiple network cards (each with its own IP address).

Last edited by drbits; 25.05.2010 at 01:10.
Reply With Quote
  #36  
Old 13.07.2010, 18:11
shawn8888
Guest
 
Posts: n/a
Default

"multiple concurrent proxies" feature is SO useful and that's why I use another tool most of the time instead of jD.

PLEASE consider it in the next release!
Reply With Quote
  #37  
Old 14.07.2010, 13:39
drbits's Avatar
drbits drbits is offline
JD English Support (inactive)
 
Join Date: Sep 2009
Location: Physically in Los Angeles, CA, USA
Posts: 4,434
Default

It will not be in the next release. It probably won't be in the release after that. However, the proxy rotate addon and the rest of JDownloader are open source. Have at it!

Please consider that every free user that uses multiple concurrent proxies is taking the resources away form other free users. If you are using 10 proxies, then you are attempting to use 5-10 times the bandwidth normally allowed for a free user, thus reducing the slots and/or bandwidth available for everybody else. Add the burden on the proxy network, and you are making a mess.
Reply With Quote
  #38  
Old 14.07.2010, 14:10
remi
Guest
 
Posts: n/a
Default

@drbits

Please stop your altruism towards hosts. Are you working for jD or are you representing the file hosts?

If I would have the means (mainly disk space) I would use a billion simultaneous proxy servers.

@shawn8888

What's the tool you're using?
Reply With Quote
  #39  
Old 15.07.2010, 13:16
drbits's Avatar
drbits drbits is offline
JD English Support (inactive)
 
Join Date: Sep 2009
Location: Physically in Los Angeles, CA, USA
Posts: 4,434
Default

I am not altruistic toward Hosts and the companies that run them. I am just being realistic.

You have been provided with instructions about how to use the Nightly release of JD to do multiple proxies that rotate through separate lists (which is what is required).

To implement multiple concurrent rotating proxies correctly, JD would have to keep a list of which proxies had been used with which hosts during the previous 28 (or so) hours. Then when a file is to be downloaded, JD would have to select a proxy at random, check the host list (and possibly repeat until there are no duplicates). This is not a massive job, but it means that each download has to be defined with its own proxy (and make the calls to change to the correct proxy before opening a port). This requires work on the download engine.

The other considerations are to detect when there is no proxy available (for example, 20 proxies and 20 downloads from the host). Determining when a proxy has been used or has just made contact requires fixing the status monitoring.

Finally, there is an easier way (and the way most of the tools probably do this), that is to just use the proxies in order for each download (starting at different places in the list). This is not as flexible, is easily detected, and does strange things to the proxy loads.

The easiest way is to not worry about choosing the proxy, and just keep trying proxies until the host stops complaining that you are out of quota. This is probably unacceptable.
Reply With Quote
  #40  
Old 15.07.2010, 15:12
remi
Guest
 
Posts: n/a
Default

The reality I see is that file hosts are operating in a free market. Every week there are hosts that die and new ones are created. jD's business is to ease downloading from them. How they survive is their business. We should not give any advice to them nor warn jD's customers that too much downloading might harm them. As long as they allow downloads I don't see any problem with automating the process. Some people pay for several IP addresses or several proxy servers. They could take advantage of a parallel proxy feature in jD.

As to a technical solution, Bill's tabbed approach might work but I must admit that I do not fully understand it. I've looked at Videophile II, but it doesn't seem to have tabs (as I understand what they are) and it's not a downloading tool neither. FDM's tabs are used for something completely different.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 18:16.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.