[her34]

Suggestion:
option to exclude mirror from calculations

What it would do:
Total size of queue would not include mirrors, which would give more accurate information about download.

When downloading, the status eta of a packaged is based on all files in the package even though there are mirrors. This causes inaccurately long eta times.

[her34]

perhaps mirrors could be listed as indented under file.

Currently status of two identical file from different hosts looks like this:
[Package name]
.....[filename].....[host1]
.....[filename].....[host2]


Suggestion is to change to somethings like this:
[Package name]
.....[filename]
..........["mirror"].....[host1]
..........["mirror"].....[host2]

Please, see .

If this feature is implemented, what you ask will be solved as well.

Note that I would like to see the mirrors of one file in one row, instead of several rows.

The ETA is another issue but I've read that it's working better in the test version of jD.

[Gweilo]

ETA is completely bogus now in any case. I see it swing wildly from minutes to days with each momentary change of download speed. A year ago we were asking for this to be based on an average over some period, which would have a closer relation to the real amount of time, but no one seems interested in implementing this.

The estimated size of packages with mirrors has has been obviously wrong for years too.

For both cases you just have to write things down with a pencil and calculate it for yourself, it's apparently too hard for JD.

[drbits]

JD will eventually have a "Mirror controller" that identifies files that are Mirrors (the files are exactly the same Hash and length - no Hash, not a mirror) or Collisions (not mirrors, but save to the exact same place and name),

This has to wait for some other fixes. However, there are major plans for dealing with Mirrors and Collisions. It is not clear how Mirrors will be displayed. I favour a single line, but Copy URL returns one URL for each Mirror.

Collisions are already handled by the Download & Connections -> If file already exists options (skip (default), rename, overwrite, or ask). These will probably continue to be displayed on their own lines.

Calculation of Package size and completion also requires using Assemblies (either a link, mirror list, or list of items that compose an archive). A MirrorList with different file names complicates identifying an Assembly (the mirror with the correct name must be used). Assemblies are already determined in LinkGrabber if all links in the assembly are added at the same time.

Once you are dealing with Assemblies and Mirrors, there are actually several different ways to calculate the length of a package. Only one copy of a mirror counts for downloading (even if they are in different Assemblies), but mirrors in different assemblies would be stored for each assembly (unless Hard-Linking is implemented, but that is a different story).

How to deal with collisions is a problem once you identify Assemblies. Only one of the files can be in the assembly, but there is no automated way to determine which one if the assembly is not all added to the LinkGrabber as a unit.

This issue is further discussed in discussions about the future user interface (mostly in Nightly).
___________________________________

ETA

We have discussed ETA extensively. It is a little better in the next release. There are three problems with estimating download speed:
1) The servers send data at an inconsistent speed (especially for Free Users on sites like RapidShare, which can stop sending for 20 seconds at a time).
2) The operating system changes the connection priority (for instance in Visa) and program priority dynamically.
3) For programs that have already been partially downloaded, only the downloaded bytes for this session can be used in the calculation.

Of course, the ETA for a link is the length/speed. When Waits are involved, they make the computation of a package ETA very difficult to estimate (especially if the links are not all from the same host) and reconnection just complicates things more.

One solution is to report (total bytes processed)/(downloading time). This tends to smooth the data too much and under-reports the speed, because of connection overhead.

Another solution is to report based on a sliding window (for instance, the past 3 seconds or past 20 seconds). This does not give enough weight to what is currently happening. I believe that this is the current approach, but the time is short to make a compromise between current download speed and overall download speed (perhaps we should report both).

A third common approach is called the "Geometric mean". Basically, one starts with the speed as the instantaneous speed, then update the speed by a fraction every X amount of time.

If the last reported speed was Sa and the instantaneous speed is Sb (must measure time at least to the millisecond), the average speed Sg = (Sa*(X-1)+Sb)/X. X is the factor that adjusts how important recent events are compared to earlier events. However, all samples count, right back to the beginning. If the sample time 1 second and X=20, the most recent sample is worth 1/20 of the total.

X=20 to X=40 are good choices, because Rapidshare (and some other sites) use a 20 second cycle to download to free users.

Statisticians often use a modified version of this formula: Sg = sqr-root((sqr(Sa*(X-1))+sqr(Sb))/sqr(X))

Regardless of the way that file download speed is calculated, the package download speed must estimate the wait time using one of these methods.


Bugtracker Tickets related to this thread:

[Gweilo]

Originally Posted by drbits

Bugtracker Tickets related to this thread:

All with "Progress: 0%".

I think at the moment the download engine (kernel) of jD is being redesigned/reprogrammed. I hope these features will then be easier to implement.

@drbits

I think you forget to mention one type of mirror file handling. The customer should be able to indicate the mirrors. jD cannot know everything about mirror files and sites.

I like your ETA 'paper'. I think estimates based on event data stamps and download volume will be the easiest to implement.

[drbits]

@ remi,

You are right. The user has to be able to move links into and out of mirror lists (if they are the same size) or conflict lists or Assemblies. Just as a link in an Assembly can be moved into a nested mirror list.

Probably no copying. Eventually, Mirror Lists should be shareable (one download, multiple copies of the file or multiple directory entries for the file).

I'm not sure I understand your last paragraph, but there's a generic XML format for that already and jD supports it. It's called Metalink ("http://en.wikipedia.org/wiki/Metalink").