#1
|
|||
|
|||
Bug: Check whether the file exists BEFORE the download begins
Hello jD team!
I found a bug in latest jD 2. If you have a list of files to download, jD2 tries to download the file before it checks whether the file already exits. That means that you'll get a captcha dialog (and you will get blocked by some file hosting sites) even if the file exits and you won't download anything. jD2 should check whether the file exits before it contacts the server, presents you with a captcha dialog ( and refuses to download if the file exits and blocks you from the server for some amount of time). |
#2
|
||||
|
||||
Please post some example links, also include your log.
__________________
» Setup JD2 / Instalador de JD2 «
Spoiler:
Installer for Windows XP/Vista/Seven/Eight || JD2 x86 - x64 (Beta) || Installer for Mac || JD2 (Beta) || Installers for Linux || JD2 (Beta) x86 || <---> || JD2 (Beta) x64 || How to Create a Log -» Click Here «- ¿Cómo crear un registro? -» Click Aquí «- Support Chat / Chat de Soporte -» Click Here / Click Aquí «- |
#3
|
||||
|
||||
@Mikebell
It is "exists" not "exits" JD2 has to contact the server in order to determine if the files exists, or not. Last edited by MrVideo; 18.09.2014 at 10:33. |
#4
|
||||
|
||||
Depending on your user settings when file exists AND if the hoster doesn't provide VERIFIED FILESIZE its not trusted source. Download routine is needed in order to verify that the file you're about to download is the same/smaller/larger in size.
__________________
raztoki @ jDownloader reporter/developer http://svn.jdownloader.org/users/170 Don't fight the system, use it to your advantage. :] |
#5
|
|||
|
|||
Quote:
If jD sees that a file exists in the specified downloads directory, it should not connect to the file server, present you with a captcha, and then refuse to download because the file exists. jD does NOT download anything from the server right now to verify that the files are the same (i.e. it doesn't download the file and then check the hash to verify they're the same) yet jD does connect to the server, initiates the download transfer and then just refuses to download because the file already exists. If the file exists, it should let you know ahead of time so you can decide what to do. It shouldn’t connect to these sites and waste your time by "poisoning" the IP and initiating wait times on file sharing sites. As for links, it will work with ANY link. I've used NitoFlare link but it's the same behavior on all links. It's just that NitroFlare flags you as "downloaded file" even if you don't download anything and just solve captcha. |
#6
|
||||
|
||||
As said before, post the links to try to reproduce the problem.
__________________
» Setup JD2 / Instalador de JD2 «
Spoiler:
Installer for Windows XP/Vista/Seven/Eight || JD2 x86 - x64 (Beta) || Installer for Mac || JD2 (Beta) || Installers for Linux || JD2 (Beta) x86 || <---> || JD2 (Beta) x64 || How to Create a Log -» Click Here «- ¿Cómo crear un registro? -» Click Aquí «- Support Chat / Chat de Soporte -» Click Here / Click Aquí «- |
#7
|
||||
|
||||
I think I misunderstood the OP's original question. By file exists, I was thinking about the file existing on the server. After the above postings, I now believe that the OP meant the file already existing on his/her computer.
My bad. |
#8
|
|||
|
|||
Here's a link but links are completely irrelevant. It happens with every hoster.
**External links are only visible to Support Staff****External links are only visible to Support Staff** Quote:
That's right. If the file is available locally, jD2 still connects and tries to download and then it stops. "File exits" check should happen a lot sooner in the process. |
#9
|
||||
|
||||
Its complicated because of the mirror handling...
GreeZ psp
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#10
|
||||
|
||||
Advanced settings:
GeneralSettings: Mirror Detection Decision --> FILENAME GeneralSettings: Force Mirror Detection File Size Check --> DISABLE GreeZ psp
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#11
|
|||
|
|||
@pspzockerscene - Can you clarify what those two settings do? They only seem to effect how JD2 decides if something is a duplicate (filename and/or file size). There is no indication that it effects when it checks for duplicates.
I agree this behavior is undesirable, the default settings should always check if there is a file with the same name in the designated directory before any attempt is made to contact a file service. If there is, regardless of size, boom, done. File size and name are already checked when links are added so rechecking is extraneous either way. Finally, filesize is too fluid between different services to be a useful identifier. Besides, if I've got a file by the same name and a different size, I'm still not going to want it over-written. I can't image anyone else would either. Last edited by Lram32; 22.11.2014 at 03:18. Reason: removed Caps |
#12
|
||||
|
||||
@CuF
Those settings basically switch from verified checking to filename only. 98% hosts do not provide a verified way of linkchecking, its all estimated (they say 2.3GiB or even worse 2GiB, yet you don't know if they have rounded up or down) each mirror could be done differently. Switching it off means if you have a file that's 1MiB downloaded on your disk, will prevent another download from trying to download even though new download is larger.
__________________
raztoki @ jDownloader reporter/developer http://svn.jdownloader.org/users/170 Don't fight the system, use it to your advantage. :] |
#13
|
|||
|
|||
Quote:
Edit: I should point out that skipped files don't get counted in percentage completed. E.g. You have a 3 part file, 1 part you download manually, JD2 knows to skip the manually downloaded part, but percent completed won't pass 66% when you are finished. I'll try those settings out. Edit 2: Tested with settings found in post #10. It does not change this behavior. Even if a file exists of the same name, JD2 still contacts the file service before skipping. Last edited by CuF; 24.11.2014 at 17:18. |
#14
|
|||
|
|||
This is a big change from the old JD.
I have just given JD2 a list of files to download from fileboom, using premium bought through JD. I have most of the files and want to complete the set so I was relying on JD2 bypassing the files that exist on my hard disk and downloading the missing ones. It started to do this but then I ran out of download allowance. The problem is that each of the contacts made before bypassing an existing file debited my download allowance and the result is I consumed my 20 GB daily download allowance on files I did not download. Working around this problem looks like it will be a lot of work for me. Any suggestions would be welcome. |
#15
|
||||
|
||||
Ticket:
GreeZ psp
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#16
|
|||
|
|||
JD team.
Thanks for all your efforts. I thought that the goal in resolving this problem was to bypass downloads where a file of the same name exists in the download directory without querying the hosting service. That problem is not fixed. If the file already exists then querying the hosting service in unnecessary and and that unnecessary step caused the problem i described in my earlier post. I understand the the design of JD2 has been greatly improved by having a more sophisticated ways of detecting duplicate files and mirrors. The implementation is creating problems and is still worthy of a bug report. |
#17
|
|||
|
|||
Error
or putting it less politely, marking this problem as fixed without modifying JD2 is an error, incorrect and simply wrong. I love your work but you have screwed up on this one.
|
#18
|
||||
|
||||
@1971ti
This is simply wrong! My mistake was that I did not realize that it is already possible and yours that you only read the "closed" status but did not read the response of our developer jiaz: Quote:
Select a more unsafe mirrorDecision and a less than 100% MirrorDetectionFileSizeEquality. Disable ForceMirrorDetectionFileSizeCheck (if enabled). I did not check this but this is what I assume from the developers' comment on this! GreeZ pspzockerscene
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#19
|
|||
|
|||
Bugs
I made the advanced settings changes as you interpreted them:mirror decision = filename and force mirror detection file size check = not ticked.
Even with this setting, JD2 queries the hoster before skipping the download. I have not done a controlled test but with this setting it looks like that query initiates a download and debits my fileboom daily download allowance. This is the problem I reported in an earlier post where I consumed 20GB download allowance without downloading a file. Given that the file name is discovered in link grabber and can be compared to file names in the download directory, there is no need to query the hoster (consuming my download quota) before skipping the download. The setting mostly works but the implementation is flawed. I say mostly because I grabbed a couple of mega links where the files were already in the download directory. Mega deleted the files after I grabbed the links but before JD2 got to downloading them. Instead of bypassing the files JD2 went into loop retrying the downloads at 15 minute intervals. I would say this indicates a small problem in the mega plugin as well. |
#20
|
||||
|
||||
I'll let Jiaz know, and ask for some feedback.
I would personally take up the fileboom data usage issue with fileboom, the file hoster should only debit your useage on download of data (http server logs of throughput vs generating final download link with 0 data transfer). They shouldn't take data off your account, unless you actually physically received such data.
__________________
raztoki @ jDownloader reporter/developer http://svn.jdownloader.org/users/170 Don't fight the system, use it to your advantage. :] |
#21
|
|||
|
|||
Thanks for your reply.
I have had a number of premium accounts over the years and have 4 now (bought through Jdownloader links, so I hope you got your share). It is not uncommon or unreasonable for a file hoster to debit your download quota when you initiate the download rather than metering actual traffic. They, and I would ask, why initiate a download if if you are not going to proceed with it. The problem here is an unnecessary querying of the file hoster. I understand that sophisticated duplicate checking and mirror detection requires such a query but when you turn off the sophistication, you should also turn off the query. This is the bug in a nutshell. Personally, I audit my download list manually for files skipped where the file in the download directory is different. For example, a human can make the judgement that 63.rar is a likely candidate for erroneous slip and BeardedPriateFolkSongsNo12.mp3 is not. Last edited by 1971ti; 18.12.2014 at 21:37. |
#22
|
||||
|
||||
@1971ti
Understandable - anyways, we already explained how to change the behavior and we will fix bugs in case you find some in these functions! GreeZ psp
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#23
|
||||
|
||||
I remember jiaz said (via IRC) he made some changes/improvements, I read quickly his feedback so I can not remember now what he wrote. When I come back home, will post the most important information of his message.
__________________
» Setup JD2 / Instalador de JD2 «
Spoiler:
Installer for Windows XP/Vista/Seven/Eight || JD2 x86 - x64 (Beta) || Installer for Mac || JD2 (Beta) || Installers for Linux || JD2 (Beta) x86 || <---> || JD2 (Beta) x64 || How to Create a Log -» Click Here «- ¿Cómo crear un registro? -» Click Aquí «- Support Chat / Chat de Soporte -» Click Here / Click Aquí «- |
#24
|
||||
|
||||
<+jiaz> raztoki: I've added new config (AllowUnsafeFileNameForFileExistsCheck)
<+jiaz> file exists normally only works on safe (verified,forced)filenames <+jiaz> I guess his hoster only sets this value ondownload start <+jiaz> so ive added an option to also trust unsafe names for file exists checks changes not committed yet, as I understand.
__________________
raztoki @ jDownloader reporter/developer http://svn.jdownloader.org/users/170 Don't fight the system, use it to your advantage. :] |
#25
|
|||
|
|||
I agree with 1971ti, JD2 doesn't check if the file exists before querying with captcha, even after changing advanced settings, it's still ask captcha before checking filename.
Is there a way to create a plug-in ? I would like to implement this checking with a right click or something else. |
#26
|
||||
|
||||
Please make tests after the next core update (Core #28434 or higher), jiaz's changes hasn't been released yet.
__________________
» Setup JD2 / Instalador de JD2 «
Spoiler:
Installer for Windows XP/Vista/Seven/Eight || JD2 x86 - x64 (Beta) || Installer for Mac || JD2 (Beta) || Installers for Linux || JD2 (Beta) x86 || <---> || JD2 (Beta) x64 || How to Create a Log -» Click Here «- ¿Cómo crear un registro? -» Click Aquí «- Support Chat / Chat de Soporte -» Click Here / Click Aquí «- |
#27
|
|||
|
|||
I have been looking forward to a resolution of this issue.
We are now at core 28633 and I haven't been able the find the new setting (AllowUnsafeFileNameForFileExistsCheck). Am i missing something? |
#28
|
||||
|
||||
Indeed I can't find it either.
Will have to ask jiaz again where it is - maybe it s just been renamed. GreeZ psp
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#29
|
|||
|
|||
Quote:
Is there any news on the change proposed in this thread to provide an option that works like the original Jdownloader? I have have had a few disappointments with the keep2share/fileboom businesses and decided to dispense with their premium service so the problem I reported in this thread is not an issue for me anymore. I am now in the position or the original poster where you need to enter a capture and waste a free download slot before bypassing an existing file. Working around this is a manual task and, after all, what is the purpose of an application other than to automate tasks which are done more accurately and quickly by machine. |
#30
|
||||
|
||||
jiaz said the option will be there but he is busy with server maintenance at the moment.
as you might have noticed the forum is much faster now Anyways, this can take some time as he will probably make a huge committ with a lot of changes and we do not want to break the whole JD to get flooded with bugreports by you guys :D GreeZ psp
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#31
|
|||
|
|||
Bug: Check whether the file exists BEFORE the download begins
Any idea when or whether the AllowUnsafeFileNameForFileExistsCheck might be implemented?
I am intrigued as to why this unsolved bug has not been fixed and would welcome any insights from anyone can enlighten me. I am imagining that I am the only one on the planet who misses the behavior of the original JD and asking for its restoration reveals me as bad person completely out of touch of the needs of the JD user base. Anyway, I love your work but, if you cannot oblige me, I understand and will seek a solution to my problem by other means. Thanks |
#32
|
||||
|
||||
Please just wait.
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
#33
|
||||
|
||||
It will come with my next commits next week. Sorry for long delay but needs more testing
__________________
JD-Dev & Server-Admin |
#34
|
|||
|
|||
So, is the function implemented now ?
Because I still can't find it :/ |
#35
|
|||
|
|||
I tried to find the answer to my problem here and changed my mirror settings but the problem still is there. In JD1 if the same file name was in the download directory it just skipped the file. As a free user on sites now I have to wait the timeout and enter capachas and only then it tells me the file already exists. The problem is file hosts count this as a download and you have to wait another timeout. Pretty frustrating for the hosts that only allow one download a day. I don't care if the file size is an exact match and I unchecked the box.
|
#36
|
||||
|
||||
@Oldgeezer: I'm sorry for long delay. I will give this higher priority!
__________________
JD-Dev & Server-Admin |
#37
|
|||
|
|||
I didn't tell you that this is only a problem on certain sites. Some still do skip automatically without contacting the host. This may make it difficult for you to test a fix. I have not kept track of which sites do or do not work right with existing files. Here is some Uploadable.ch files that show that they count these files against your limit. Sometimes when the timer runs down they ask for a capacha and after you enter it they start another countdown even 2 or 3 more times. I knew these files already existed but as I said before I have accidentally had it happen where I have had to wait up to 24 hours to get another file.
|
#38
|
||||
|
||||
Best would be that you send me example links to support@jdownloader.org, so I can use them for testing my changes.
__________________
JD-Dev & Server-Admin |
#39
|
|||
|
|||
Quote:
|
#40
|
||||
|
||||
Thx, i will try to work on this as soon as possible
__________________
JD-Dev & Server-Admin |
|
|