JDownloader Community - Appwork GmbH
 

 
 
Thread Tools Display Modes
  #1  
Old 03.10.2009, 20:19
Gweilo's Avatar
Gweilo Gweilo is offline
JD Legend
 
Join Date: Mar 2009
Posts: 716
Default RS: "Too many users" error

I was downloading several multi-part files on Rapidshare.

Checking progress I notice that I have 5 out of 7 parts of the first file, and jD is now downloading part 3 of the third file, having skipped all the ones in between.

Checking the log I see that the other files had all been aborted:
Code:
4/10/09 1:46:59 AM - INFO [java_downloader] -> Start working on x1.part7.rar
4/10/09 1:47:06 AM - INFO [java_downloader] -> Too many users are currently downloading this file
4/10/09 1:47:06 AM - WARNING [java_downloader] -> 
Error occurred- latest:ERROR_TEMPORARILY_UNAVAILABLE
Now while the error says "Too many users are currently downloading this file", actually RS is just busy, full stop. every file I try is blocked with the same message.

However, this only lasts a few minutes and then RS will give me a slot.

So JD basically starts a random download, whichever one it happened to try when RS came back.

I really want JD to just try the files in the first package and finish that before trying any others.

Setting priorities has no effect on this behaviour.

If I'm monitoring it I can just disable all the other packages. But then obviously I have to be there to enable them again.

This matters to me because I can't do anything with three parts of one file, two parts of a second and two of a third. But if they were all the same package it would be complete and I could be processing it.

Last edited by Jiaz; 07.10.2009 at 13:26.
  #2  
Old 03.10.2009, 22:29
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 48,180
Default

If the status doesn't change and jd tries to download 1 file for hours it makes no sense...

GreeZ pspzockerscene
__________________

Ad-free installers || Werbefreie Installer
Windows Setup<--JD2 BETA-->Linux Setup x86 || Linux Setup x64 || Mac Setup
-----=>Support Chat<=-----
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
  #3  
Old 04.10.2009, 02:42
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 61,796
Default

im sure we will not add such a feature at all (or in near future ) because

1.) jd cannot know what files do belong together (at least not for sure)
2.) why would someone want NOT to download when he can only to wait for some files that are still *server busy*
3.) JD cannot know in advance how long the file will be busy , thus it would have to wait for it and not downloading any other file (see 2)

so i think if you really want to have one package finished first then
1.) disable the other ones
2.) add interchangable links (multiple hosters)
3.) buy premium
4.) use non rs links
__________________
JD-Dev & Server-Admin
  #4  
Old 04.10.2009, 06:29
Gweilo's Avatar
Gweilo Gweilo is offline
JD Legend
 
Join Date: Mar 2009
Posts: 716
Default

Quote:
Originally Posted by pspzockerscene View Post
If the status doesn't change and jd tries to download 1 file for hours it makes no sense...

GreeZ pspzockerscene
For two minutes it does make sense.
That is the case I'm talking about.
If it was strill "UNAVAILABLE" then it could try other packages.

Or better, once it gets a positive response, go back and check if the highest priority download is now available, not start on one 20 places down the list.
  #5  
Old 04.10.2009, 06:39
Gweilo's Avatar
Gweilo Gweilo is offline
JD Legend
 
Join Date: Mar 2009
Posts: 716
Default

Quote:
Originally Posted by Jiaz View Post
1.) jd cannot know what files do belong together (at least not for sure)
That's what packages are for.


Quote:
Originally Posted by Jiaz View Post
2.) why would someone want NOT to download when he can only to wait for some files that are still *server busy*
Because all the RS links give the same error.

Once one link becomes available again, then they ALL are in this situation. But JD has already given up on the top links. If it could retry the top link it would be able to download it.

I know this because when I'm monitoring it I can disable other links, reset the "aborted" ones and JD will then continue with the top package.

I would like JD to do that for itself.

Quote:
Originally Posted by Jiaz View Post
3.) JD cannot know in advance how long the file will be busy , thus it would have to wait for it and not downloading any other file (see 2)
As above, it could give up after a couple of minutes and try other packages.



Quote:
Originally Posted by Jiaz View Post
so i think if you really want to have one package finished first then
1.) disable the other ones
2.) add interchangable links (multiple hosters)
3.) buy premium
4.) use non rs links
Thanks, but I worked that out for myself.
If it were possible to find mirrors, I do.
  #6  
Old 04.10.2009, 11:31
remi
Guest
 
Posts: n/a
Cool

I agree with ... Gweilo.

This issue is related to the old automatic retry (of failed downloads) feature.
  #7  
Old 04.10.2009, 18:12
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 61,796
Default

Quote:
Originally Posted by Gweilo View Post
That's what packages are for.
Edit by Jiaz: ppl also use packages for other things. image a complete season, then it will again be useless because all files are in one package but it might happen that they are in different states


Because all the RS links give the same error.
Once one link becomes available again, then they ALL are in this situation. But JD has already given up on the top links. If it could retry the top link it would be able to download it.
Edit by Jiaz: im sorry, that is wrong, every file can be hosted on different servers and can have different load. i saw often that only part 7 of 14 is working, then 9 , then 2....its just wrong that all files are having the same waittime or download usage

I know this because when I'm monitoring it I can disable other links, reset the "aborted" ones and JD will then continue with the top package.
Edit by Jiaz: thats luck
I would like JD to do that for itself.



As above, it could give up after a couple of minutes and try other packages.




Thanks, but I worked that out for myself.
If it were possible to find mirrors, I do.
see comments
__________________
JD-Dev & Server-Admin
  #8  
Old 04.10.2009, 20:07
Gweilo's Avatar
Gweilo Gweilo is offline
JD Legend
 
Join Date: Mar 2009
Posts: 716
Default

Quote:
Originally Posted by Jiaz View Post
Because all the RS links give the same error.
Once one link becomes available again, then they ALL are in this situation. But JD has already given up on the top links. If it could retry the top link it would be able to download it.
Edit by Jiaz: im sorry, that is wrong, every file can be hosted on different servers and can have different load. i saw often that only part 7 of 14 is working, then 9 , then 2....its just wrong that all files are having the same waittime or download usage
The situation I am talking about is when I have dozens of RS links enabled; and EVERY SINGLE ONE aborts for the same reason. Then 2 minutes later they are ALL active again. But JD starts some random one way down the list instead if the one at the top, which is equally as available now.

You may just never see this where you are. I think you're in Germany; those of us in Asia are much more likely to to be frozen out of all RS downloads when they're busy in Europe or the US.

And even when it starts up again, downloads are likely to be very slow. So it can delay the completion of the top package by hours while it slowly gets parts from lower priority packages before finally going back to the top package.

I can live with it, but it is annoying as I have either to wait much longer or manually enable and disable links to guide it.
  #9  
Old 06.10.2009, 17:48
james3
Guest
 
Posts: n/a
Default

it seems to me that there should be an independent Priority set on packages that is completely separate from the file priority and take president over file priority, then once inside that package the priority on a file would take over.

sound simple but...

say you have 20 packages and each has multiple files say 10 in each and all are from the same host, now if JD has received an error on package 1 file 6 and then proceeds down through the packages still receiving the same error and in package 15 file 3 it starts to connect and download, should JD...

A: immediately stop and go to P1 F6
B: complete the file in P15 F3 and then resume in P1 F6
C: complete the file in P15 F3 and proceed to P15 F4

for this error message I'm assuming B and that would be fine if all files had indeed become available but very annoying if they haven't and JD was forced by package priority to recheck the previous packages/files only to end up downloading P15 F4 and then the scenario starts again for P15 F5 and so on.

of course in an ideal world JD could be programed to behave differently on receiving different error codes from the host in question.
  #10  
Old 06.10.2009, 18:51
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 61,796
Default

A: would be not a good idea, because you have to wait or reconnect in order to download the file, because you stopped a running download in order to try the other file ;9

B: not doable with the current implementation because jd would have to manage those error messages to know that all the files are having the same error

C: that the current implementation
__________________
JD-Dev & Server-Admin
  #11  
Old 06.10.2009, 19:32
james3
Guest
 
Posts: n/a
Default

the only other way I can think of would be to disable/pause P2 through P20 and once P1 F10 had completed JD could enable/unpause P2 and so on... but of course this would lead back to JD having to manage error messages once again
  #12  
Old 06.10.2009, 20:28
Gweilo's Avatar
Gweilo Gweilo is offline
JD Legend
 
Join Date: Mar 2009
Posts: 716
Default

Quote:
Originally Posted by Jiaz View Post
A: would be not a good idea, because you have to wait or reconnect in order to download the file, because you stopped a running download in order to try the other file ;9
This happens BEFORE there is a running download.
When I'm monitoring this I see JD trying one after another, only a few seconds between trying the next download. (See my error log in the first post, it took 7 seconds.) It does not go through the 1-2 minute wait that RS has before starting a free download. I would not want to abort a running download.


Quote:
Originally Posted by Jiaz View Post
B: not doable with the current implementation because jd would have to manage those error messages to know that all the files are having the same error
JD already does manage the error messages. It marks each failed download with the error. So the data is available. JD can use this to notice that a host has failed 100% of downloads with "Temporarily unavailable" since XXXX, and thus is likely a global outage/block (same result). And so when a file is "Available" maybe the block is over; so it can (quickly) check availability of previously "unavailable" files before starting the next download.


Quote:
Originally Posted by Jiaz View Post
C: that the current implementation
With respect, that's the option I like the least.

Last edited by Gweilo; 06.10.2009 at 20:31.
  #13  
Old 07.10.2009, 10:36
remi
Guest
 
Posts: n/a
Cool

If jD wants to become a real download "manager", I think this sort of intelligence is needed. Like Gweilo says, the data about the failing links is available in those links. jD should "think" a little and decide that it isn't worthwhile to check all the links from the same host or even other hosts. It should simply check or retry the first link at regular intervals.

This probably means that jD should have a categorisation of types of errors it receives from the different hosts or equipment between the PC and the Internet. Two categories I see immediately are: "no connection to the internet", "no connection to the host". You, jD networking experts must have other categories in mind as well.

Today I had a a problem with my connection. Azureus continued normally, although almost all tracker connections were lost (blue icons). My browser, jD and FDM lost their connections entirely. I reset my router and everything came back again. Such a general connection problem should be detectable by jD.
  #14  
Old 07.10.2009, 10:53
Gweilo's Avatar
Gweilo Gweilo is offline
JD Legend
 
Join Date: Mar 2009
Posts: 716
Default

Quote:
Originally Posted by remi View Post
I jD should "think" a little and decide that it isn't worthwhile to check all the links from the same host or even other hosts. It should simply check or retry the first link at regular intervals.
That would not be a good idea; sometimes it is only one link, or one server that is out because of some server problem. If it's just checking, it's actually better for it to try different links. Only when it finally strikes an "available" one it might (quickly) recheck the top link from the same host before committing to a download.

But it certainly should be able to recognise when there is no conectivity at all and pop up an error while waiting for the connection to be restored. (It could just ping some standard sites.) I mentioned this some time ago, when I was offline for whatever reason JD would keep trying links and marking them "unavailable" and I had to manually reset them to get JD to try again.
  #15  
Old 07.10.2009, 13:26
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 61,796
Default

http://svn.jdownloader.org/issues/show/505
__________________
JD-Dev & Server-Admin
  #16  
Old 07.10.2009, 14:15
remi
Guest
 
Posts: n/a
Cool

Quote:
Originally Posted by Gweilo View Post
That would not be a good idea; sometimes it is only one link, or one server that is out because of some server problem.
That was not what I meant. I meant jD should gather the information it gets from the failed links and reason about the causes of these failures. If there is a general connection problem it should not continue trying new links, but it should at regular intervals check the first link that is the result of the general problem. I hope this is a better explanation.


@Jiaz : thanks for adding the feature/bug to the tracker.
  #17  
Old 07.10.2009, 16:35
Gweilo's Avatar
Gweilo Gweilo is offline
JD Legend
 
Join Date: Mar 2009
Posts: 716
Default

Quote:
Originally Posted by remi View Post
That was not what I meant. I meant jD should gather the information it gets from the failed links and reason about the causes of these failures. If there is a general connection problem it should not continue trying new links, but it should at regular intervals check the first link that is the result of the general problem. I hope this is a better explanation.
I understand, but it's still safer to check different links (assuming you have several on the same host) just in case the problem is specific to a particular link or server. Why not?
  #18  
Old 08.10.2009, 12:21
remi
Guest
 
Posts: n/a
Cool

I'm still not clear enough

There are categories of problems where you don't need to check more than one link. It doesn't make sense to test several links when jD detects that there's no Internet connection for whatever reason.

If jD has a problem with only one server, chances are high the next link will work. There will be no cascade effect for the other links in the download queue.

If there's a problem with a host, jD should only test one link of that host. It doesn't need to test the other links for the same host. The "Too many users" problem is a typical example for this type of problems as it is host related.

That's what I wanted to express with the problem categorisation. jD diagnoses the problem and then some implemented rules of thumb in its "problem reasoning engine" will lead to a more intelligent download management.
It'll retry links on a regular basis, depending on the problem type. The customer does no longer need to intervene, resetting or resuming links as is currently the case.
  #19  
Old 08.10.2009, 14:47
Gweilo's Avatar
Gweilo Gweilo is offline
JD Legend
 
Join Date: Mar 2009
Posts: 716
Default

Quote:
Originally Posted by remi View Post
I'm still not clear enough
There are categories of problems where you don't need to check more than one link. It doesn't make sense to test several links when jD detects that there's no Internet connection for whatever reason.
Obviously.
In that case it should just ping some standard, reliable, sites.


Quote:
Originally Posted by remi View Post
If there's a problem with a host, jD should only test one link of that host. It doesn't need to test the other links for the same host. The "Too many users" problem is a typical example for this type of problems as it is host related.
Your point is clear, but I still disagree.

Why not test a different link each time, if they are all in the queue? It doesn't take any longer.

You shouldnt trust the host's error messages too much. (For instance, MegaUpload has been giving a "temporarily unavailable" message to users in selected countries for almost a year.) It really might be just one bad link, or one server, not all.


Quote:
Originally Posted by remi View Post
retry links on a regular basis, depending on the problem type. The customer does no longer need to intervene, resetting or resuming links as is currently the case.
Agreed, just a small difference in methodology.
  #20  
Old 08.10.2009, 17:12
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 61,796
Default

http://svn.jdownloader.org/issues/show/676
these changes need a rewrite of the current downloadcontroller and that takes time
__________________
JD-Dev & Server-Admin
 

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 17:52.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.