JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 17.09.2014, 23:31
mikebell mikebell is offline
Giga Loader
 
Join Date: Oct 2009
Posts: 99
Default Bug: Check whether the file exists BEFORE the download begins

Hello jD team!

I found a bug in latest jD 2. If you have a list of files to download, jD2 tries to download the file before it checks whether the file already exits. That means that you'll get a captcha dialog (and you will get blocked by some file hosting sites) even if the file exits and you won't download anything.

jD2 should check whether the file exits before it contacts the server, presents you with a captcha dialog ( and refuses to download if the file exits and blocks you from the server for some amount of time).
Reply With Quote
  #2  
Old 17.09.2014, 23:44
Lram32's Avatar
Lram32 Lram32 is offline
Spanish Supporter
 
Join Date: Jul 2012
Location: Dark side of the moon
Posts: 9,908
Default

Please post some example links, also include your log.
__________________
» Setup JD2 / Instalador de JD2 «
Spoiler:

Installer for Windows XP/Vista/Seven/Eight
|| JD2 x86 - x64 (Beta) ||

Installer for Mac
|| JD2 (Beta) ||

Installers for Linux
|| JD2 (Beta) x86 || <---> || JD2 (Beta) x64 ||

How to Create a Log
-» Click Here «-


¿Cómo crear un registro?
-» Click Aquí «-


Support Chat / Chat de Soporte
-» Click Here / Click Aquí «-


Reply With Quote
  #3  
Old 18.09.2014, 10:28
MrVideo's Avatar
MrVideo MrVideo is offline
Black Hole
 
Join Date: Nov 2013
Location: Madison, WI
Posts: 274
Default

@Mikebell

It is "exists" not "exits"

JD2 has to contact the server in order to determine if the files exists, or not.

Last edited by MrVideo; 18.09.2014 at 10:33.
Reply With Quote
  #4  
Old 18.09.2014, 12:45
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 16,194
Default

Depending on your user settings when file exists AND if the hoster doesn't provide VERIFIED FILESIZE its not trusted source. Download routine is needed in order to verify that the file you're about to download is the same/smaller/larger in size.
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #5  
Old 18.09.2014, 23:03
mikebell mikebell is offline
Giga Loader
 
Join Date: Oct 2009
Posts: 99
Default

Quote:
Originally Posted by raztoki View Post
Depending on your user settings when file exists AND if the hoster doesn't provide VERIFIED FILESIZE its not trusted source. Download routine is needed in order to verify that the file you're about to download is the same/smaller/larger in size.
None of those things are required to check that the file exists.

If jD sees that a file exists in the specified downloads directory, it should not connect to the file server, present you with a captcha, and then refuse to download because the file exists. jD does NOT download anything from the server right now to verify that the files are the same (i.e. it doesn't download the file and then check the hash to verify they're the same) yet jD does connect to the server, initiates the download transfer and then just refuses to download because the file already exists.

If the file exists, it should let you know ahead of time so you can decide what to do. It shouldn’t connect to these sites and waste your time by "poisoning" the IP and initiating wait times on file sharing sites.

As for links, it will work with ANY link. I've used NitoFlare link but it's the same behavior on all links. It's just that NitroFlare flags you as "downloaded file" even if you don't download anything and just solve captcha.
Reply With Quote
  #6  
Old 18.09.2014, 23:14
Lram32's Avatar
Lram32 Lram32 is offline
Spanish Supporter
 
Join Date: Jul 2012
Location: Dark side of the moon
Posts: 9,908
Default

As said before, post the links to try to reproduce the problem.
__________________
» Setup JD2 / Instalador de JD2 «
Spoiler:

Installer for Windows XP/Vista/Seven/Eight
|| JD2 x86 - x64 (Beta) ||

Installer for Mac
|| JD2 (Beta) ||

Installers for Linux
|| JD2 (Beta) x86 || <---> || JD2 (Beta) x64 ||

How to Create a Log
-» Click Here «-


¿Cómo crear un registro?
-» Click Aquí «-


Support Chat / Chat de Soporte
-» Click Here / Click Aquí «-


Reply With Quote
  #7  
Old 19.09.2014, 04:30
MrVideo's Avatar
MrVideo MrVideo is offline
Black Hole
 
Join Date: Nov 2013
Location: Madison, WI
Posts: 274
Default

I think I misunderstood the OP's original question. By file exists, I was thinking about the file existing on the server. After the above postings, I now believe that the OP meant the file already existing on his/her computer.

My bad.
Reply With Quote
  #8  
Old 19.09.2014, 21:35
mikebell mikebell is offline
Giga Loader
 
Join Date: Oct 2009
Posts: 99
Default

Quote:
Originally Posted by Lram32 View Post
As said before, post the links to try to reproduce the problem.
Here's a link but links are completely irrelevant. It happens with every hoster.

**External links are only visible to Support Staff****External links are only visible to Support Staff**

Quote:
Originally Posted by MrVideo View Post
I think I misunderstood the OP's original question. By file exists, I was thinking about the file existing on the server. After the above postings, I now believe that the OP meant the file already existing on his/her computer.

My bad.

That's right. If the file is available locally, jD2 still connects and tries to download and then it stops. "File exits" check should happen a lot sooner in the process.
Reply With Quote
  #9  
Old 21.09.2014, 17:43
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 48,686
Default

Its complicated because of the mirror handling...

GreeZ psp
__________________

Ad-free installers || Werbefreie Installer
Windows Setup<--JD2 BETA-->Linux Setup x86 || Linux Setup x64 || Mac Setup
-----=>Support Chat<=-----
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #10  
Old 21.09.2014, 17:53
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 48,686
Default

Advanced settings:
GeneralSettings: Mirror Detection Decision
--> FILENAME
GeneralSettings: Force Mirror Detection File Size Check
--> DISABLE

GreeZ psp
__________________

Ad-free installers || Werbefreie Installer
Windows Setup<--JD2 BETA-->Linux Setup x86 || Linux Setup x64 || Mac Setup
-----=>Support Chat<=-----
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #11  
Old 22.11.2014, 03:15
CuF
Guest
 
Posts: n/a
Default

@pspzockerscene - Can you clarify what those two settings do? They only seem to effect how JD2 decides if something is a duplicate (filename and/or file size). There is no indication that it effects when it checks for duplicates.

I agree this behavior is undesirable, the default settings should always check if there is a file with the same name in the designated directory before any attempt is made to contact a file service. If there is, regardless of size, boom, done.

File size and name are already checked when links are added so rechecking is extraneous either way.

Finally, filesize is too fluid between different services to be a useful identifier. Besides, if I've got a file by the same name and a different size, I'm still not going to want it over-written. I can't image anyone else would either.

Last edited by Lram32; 22.11.2014 at 03:18. Reason: removed Caps
Reply With Quote
  #12  
Old 22.11.2014, 04:42
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 16,194
Default

@CuF

Those settings basically switch from verified checking to filename only. 98% hosts do not provide a verified way of linkchecking, its all estimated (they say 2.3GiB or even worse 2GiB, yet you don't know if they have rounded up or down) each mirror could be done differently.

Switching it off means if you have a file that's 1MiB downloaded on your disk, will prevent another download from trying to download even though new download is larger.
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #13  
Old 22.11.2014, 06:12
CuF
Guest
 
Posts: n/a
Default

Quote:
Originally Posted by raztoki View Post
@CuF

Those settings basically switch from verified checking to filename only. 98% hosts do not provide a verified way of linkchecking, its all estimated (they say 2.3GiB or even worse 2GiB, yet you don't know if they have rounded up or down) each mirror could be done differently.

Switching it off means if you have a file that's 1MiB downloaded on your disk, will prevent another download from trying to download even though new download is larger.
Sounds good. I hate having to enter a CAPTCHA and maybe even being force to wait again because JD2 just wanted to check file details. (It of course kills a free download sessions.)

Edit: I should point out that skipped files don't get counted in percentage completed.
E.g. You have a 3 part file, 1 part you download manually, JD2 knows to skip the manually downloaded part, but percent completed won't pass 66% when you are finished.

I'll try those settings out.

Edit 2: Tested with settings found in post #10. It does not change this behavior.
Even if a file exists of the same name, JD2 still contacts the file service before skipping.

Last edited by CuF; 24.11.2014 at 17:18.
Reply With Quote
  #14  
Old 10.12.2014, 21:12
1971ti 1971ti is offline
Baby Loader
 
Join Date: Dec 2014
Location: Australia
Posts: 8
Default

This is a big change from the old JD.

I have just given JD2 a list of files to download from fileboom, using premium bought through JD. I have most of the files and want to complete the set so I was relying on JD2 bypassing the files that exist on my hard disk and downloading the missing ones.

It started to do this but then I ran out of download allowance.

The problem is that each of the contacts made before bypassing an existing file debited my download allowance and the result is I consumed my 20 GB daily download allowance on files I did not download.

Working around this problem looks like it will be a lot of work for me. Any suggestions would be welcome.
Reply With Quote
  #15  
Old 10.12.2014, 21:58
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 48,686
Default

Ticket:


GreeZ psp
__________________

Ad-free installers || Werbefreie Installer
Windows Setup<--JD2 BETA-->Linux Setup x86 || Linux Setup x64 || Mac Setup
-----=>Support Chat<=-----
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #16  
Old 15.12.2014, 03:51
1971ti 1971ti is offline
Baby Loader
 
Join Date: Dec 2014
Location: Australia
Posts: 8
Default

JD team.

Thanks for all your efforts.

I thought that the goal in resolving this problem was to bypass downloads where a file of the same name exists in the download directory without querying the hosting service. That problem is not fixed.

If the file already exists then querying the hosting service in unnecessary and and that unnecessary step caused the problem i described in my earlier post.

I understand the the design of JD2 has been greatly improved by having a more sophisticated ways of detecting duplicate files and mirrors. The implementation is creating problems and is still worthy of a bug report.
Reply With Quote
  #17  
Old 16.12.2014, 23:31
1971ti 1971ti is offline
Baby Loader
 
Join Date: Dec 2014
Location: Australia
Posts: 8
Default Error

or putting it less politely, marking this problem as fixed without modifying JD2 is an error, incorrect and simply wrong. I love your work but you have screwed up on this one.
Reply With Quote
  #18  
Old 17.12.2014, 03:52
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 48,686
Default

@1971ti
This is simply wrong!
My mistake was that I did not realize that it is already possible and yours that you only read the "closed" status but did not read the response of our developer jiaz:
Quote:
already possible!

1.) MirrorDecision
SAFE (100% sure)
AUTO (filesize,hash,name)
FILENAME_FILESIZE(filesize,name)
FILENAME (very unsafe! as filesize is not compared at all!)
2.) MirrorDetectionFileSizeEquality
10000 -> 100%
lower the value to allow differences in filesizes(eg filesize from an API vs parsed filesize from website)
3.) ForceMirrorDetectionFileSizeCheck
So, what does this mean:
Select a more unsafe mirrorDecision and a less than 100% MirrorDetectionFileSizeEquality.
Disable ForceMirrorDetectionFileSizeCheck (if enabled).
I did not check this but this is what I assume from the developers' comment on this!

GreeZ pspzockerscene
__________________

Ad-free installers || Werbefreie Installer
Windows Setup<--JD2 BETA-->Linux Setup x86 || Linux Setup x64 || Mac Setup
-----=>Support Chat<=-----
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #19  
Old 18.12.2014, 06:56
1971ti 1971ti is offline
Baby Loader
 
Join Date: Dec 2014
Location: Australia
Posts: 8
Default Bugs

I made the advanced settings changes as you interpreted them:mirror decision = filename and force mirror detection file size check = not ticked.

Even with this setting, JD2 queries the hoster before skipping the download. I have not done a controlled test but with this setting it looks like that query initiates a download and debits my fileboom daily download allowance. This is the problem I reported in an earlier post where I consumed 20GB download allowance without downloading a file.

Given that the file name is discovered in link grabber and can be compared to file names in the download directory, there is no need to query the hoster (consuming my download quota) before skipping the download.

The setting mostly works but the implementation is flawed.

I say mostly because I grabbed a couple of mega links where the files were already in the download directory. Mega deleted the files after I grabbed the links but before JD2 got to downloading them. Instead of bypassing the files JD2 went into loop retrying the downloads at 15 minute intervals. I would say this indicates a small problem in the mega plugin as well.
Reply With Quote
  #20  
Old 18.12.2014, 08:33
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 16,194
Default

I'll let Jiaz know, and ask for some feedback.

I would personally take up the fileboom data usage issue with fileboom, the file hoster should only debit your useage on download of data (http server logs of throughput vs generating final download link with 0 data transfer). They shouldn't take data off your account, unless you actually physically received such data.
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 11:54.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.