JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 08.08.2011, 08:20
mikebell mikebell is offline
JD Adviser
 
Join Date: Oct 2009
Posts: 104
Angry JD is losing my files after they're downloaded :(

Guys, I'm using the latest JD, all updates applied etc and I'm running it under Mac OS X Lion. Everything goes fine through the download but then JD loses the file when it renames it to proper filename.

Over the past few months, I must have lost 4-5Gb worth of files this way.

My computer is brand new and have 400GB of free space.

Here's the relevant part from the logs.

Quote:
------------------------ Thread: 49 -----------------------
49 8/8/11 12:26:35 AM - FINER [java_downloader] -> onCHunksReady
49 8/8/11 12:26:35 AM - INFO [java_downloader] -> Close connections if the are not closed yet
49 8/8/11 12:26:35 AM - INFO [java_downloader] -> Closed connection before closing file
49 8/8/11 12:26:35 AM - INFO [java_downloader] -> Close File. Let AV programs run
49 8/8/11 12:26:35 AM - FINEST [java_downloader] -> no errors : rename
49 8/8/11 12:26:35 AM - SEVERE [java_downloader] -> Could not rename file /Users/mike/Downloads/poker-tutorial.wmv.part to /Users/mike/Downloads/poker-tutorial.wmv
49 8/8/11 12:26:35 AM - SEVERE [java_downloader] -> Error occured: ERROR_LOCAL_IO
49 8/8/11 12:26:35 AM - SEVERE [java_downloader] -> A critical Downloaderror occured. Terminate...
49 8/8/11 12:26:35 AM - WARNING [java_downloader] ->
Error occured- latest: ERROR_LOCAL_IO
00000000001000000000000000000010 <Statuscode
00000000000000000000000000000010 |FINISHED
00000000001000000000000000000000 |ERROR_LOCAL_IO
ErrorMessage: Could not rename Part File

What is going on and what can be done?

Last edited by Jiaz; 12.08.2011 at 09:54.
Reply With Quote
  #2  
Old 08.08.2011, 11:14
remi
Guest
 
Posts: n/a
Default

This is almost certainly caused by a virus. Disable (or better, un-install) the port 80/html/web scanner in your virus software.
Reply With Quote
  #3  
Old 08.08.2011, 11:54
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,232
Default

Could not rename file /Users/mike/Downloads/poker-tutorial.wmv.part to /Users/mike/Downloads/poker-tutorial.wmv
the file is blocked or something like that.
java cannot rename the file because it is still locked/blocked.
the file is not lost, you have to rename it yourself, cause java could not do it
__________________
JD-Dev & Server-Admin
Reply With Quote
  #4  
Old 09.08.2011, 21:01
mikebell mikebell is offline
JD Adviser
 
Join Date: Oct 2009
Posts: 104
Default

Quote:
Originally Posted by remi View Post
This is almost certainly caused by a virus. Disable (or better, un-install) the port 80/html/web scanner in your virus software.
Viruses on a Mac


Quote:
Originally Posted by Jiaz View Post
Could not rename file /Users/mike/Downloads/poker-tutorial.wmv.part to /Users/mike/Downloads/poker-tutorial.wmv
the file is blocked or something like that.
java cannot rename the file because it is still locked/blocked.
the file is not lost, you have to rename it yourself, cause java could not do it
That's the thing. It is lost :( After the rename fails, for whatever reason, the file is gone. The way that JD renames the files is dangerous and not fool-proof. Maybe JD should do an atomic rename or just pop an error message box and instruct the user to rename it manually. Right now it's just deleting the file :(
Reply With Quote
  #5  
Old 10.08.2011, 09:17
remi
Guest
 
Posts: n/a
Default

What about a complete and detailed log?
Reply With Quote
  #6  
Old 10.08.2011, 10:13
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,232
Default

Quote:
Originally Posted by mikebell View Post
That's the thing. It is lost :( After the rename fails, for whatever reason, the file is gone. The way that JD renames the files is dangerous and not fool-proof. Maybe JD should do an atomic rename or just pop an error message box and instruct the user to rename it manually. Right now it's just deleting the file :(
jd does an atomic rename :p but i guess you are downloading from a hoster which does not allow resume. in that case jd thinks following
download failed = because rename still happens in download core and fails
then no resume possible = so lets delete the part file again because we cannot resume it.

you should check/find out why the file is blocked! the file must not be opened when the rename is going to take place
__________________
JD-Dev & Server-Admin
Reply With Quote
  #7  
Old 11.08.2011, 08:25
mikebell mikebell is offline
JD Adviser
 
Join Date: Oct 2009
Posts: 104
Default

Quote:
Originally Posted by remi View Post
OK. Logging was set to ALL. It happens once every 15-20 downloads. But how useful is that really? The failure is clearly happening in rename function when some other file has a lock on it. It could be QuickView or Finder that's generating a thumbnail of the file and that won't be captured in a log. I doubt the log will be more detailed than what I've provided you above. But I will post the whole thing once it happens again.

Quote:
Originally Posted by Jiaz View Post
jd does an atomic rename :p but i guess you are downloading from a hoster which does not allow resume. in that case jd thinks following
download failed = because rename still happens in download core and fails
then no resume possible = so lets delete the part file again because we cannot resume it.

you should check/find out why the file is blocked! the file must not be opened when the rename is going to take place
Jiaz, yes, I do use some hosts with no resume (i.e. free mode) and from what I've seen when this happens, the file is downloaded 100% and there's a checkmark icon beside the filename and then when it tries to rename, the whole entry (checkmark et all) is grayed out and the file's gone and you have to re-download it.

What I don't understand is this: if the rename fails, why is JD deleting the file? Why doesn't it fail gracefully and say: "Rename failed of file.part, rename it manually". Why does the failure of rename operation result in such a catastrophic failure of whole file being deleted? That makes no sense to me if it's an atomic operation. If it were atomic, wouldn't I be left with two files: file.wmv and file.wmv.part?

Even if some other app has a lock on a file (you open partially downloading file in VLC for example), a failure of rename should not result in a file being deleted.

Last edited by mikebell; 11.08.2011 at 08:30.
Reply With Quote
  #8  
Old 11.08.2011, 09:41
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,232
Default

atomic rename = only one file exists, not 2 so rename fails = only the .part file exists because the rename did not work
and as i said. the old download core could not know what exactly happened, it only knows
1.) something bad happened
2.) resume not possible
3.) so lets delete part file because we cannot resume it anyway

i agree with your on rename error, but what should happen then?
inform the user that he has to rename the file himself?
another way would be fallback to copy instead of rename
issue of that would be twice disk use and user has to delete part file himself then.

im good with both solutions so which one do you prefer
__________________
JD-Dev & Server-Admin
Reply With Quote
  #9  
Old 11.08.2011, 09:59
remi
Guest
 
Posts: n/a
Default

Another solution might be that jD opens a downloading file in exclusive mode. No other application would be able to read or lock the file. When the file is complete or downloading is topped for some reason the lock is removed.

The advantage is that other tools and viruses have no access to the file as long as it isn't complete/stopped.

A disadvantage might be that if jD crashes the files cannot be accessed easily.
Reply With Quote
  #10  
Old 11.08.2011, 10:03
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,232
Default

locking works good or none at all on different os
locks are getting released when jvm closes, so no issues there but locking itself does not help with this issue
it stops from read/write
but to rename the file you have to free the locks too
__________________
JD-Dev & Server-Admin
Reply With Quote
  #11  
Old 11.08.2011, 10:17
remi
Guest
 
Posts: n/a
Default

Thanks professor Jiaz. I'm always glad to learn something from a great java expert.

Does it mean that some OSs don't allow the exclusive locking of files?

Renaming happens when the file is complete, so the lock has already been released.
Reply With Quote
  #12  
Old 11.08.2011, 10:25
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,232
Default

yes, and its the renaming part that is causing this
rename can only be done when no lock is held but there are many reasons why renaming fails then (other apps creating thumbnail, virusscanner, filesystem...)

so i guess the best way would be to do the *copy file* approach?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #13  
Old 11.08.2011, 10:45
remi
Guest
 
Posts: n/a
Default

What I understand now is that some other application locks the file (for reading for instance) as soon as jD's exclusive lock is released and then the rename cannot be done.

I don't like that idea of copying, but who am I - I'm just a simple customer. You're the technical person.

The disadvantages are that :-

1) it's not atomic and that in some cases you'll get the file twice; with large files this might be problematic for the remaining disk space;

2) copying large files on a busy disk might take a lot of time.

The advantages are that :-

1) the copied file might be less fragmented; but this is not an advantage if the file is an archive that needs to be decompressed;

2) problems with other applications/viruses locking the file might be circumvented.

I think that if the customer knows why these stupid locks happen the renaming is the best method. These problems happen rarely and the disadvantages of copying have more weight than the advantages.
Reply With Quote
  #14  
Old 11.08.2011, 10:58
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,232
Default

its just a fallback and fallbacks never are main solutions, else they would not be fallbacks
__________________
JD-Dev & Server-Admin
Reply With Quote
  #15  
Old 11.08.2011, 11:09
remi
Guest
 
Posts: n/a
Default

Ah yes. Then do it and give a warning. It's better than having to re-download a large file.
Reply With Quote
  #16  
Old 12.08.2011, 06:21
mikebell mikebell is offline
JD Adviser
 
Join Date: Oct 2009
Posts: 104
Default

EXCELLENT discussion guys! Thank you Jiaz and remi! I've learned a lot.

Jiaz, to answer your question, I'd prefer any other solution than what's happening right now: file being deleted. If I'm downloading a 1Gb file for an hour, last thing I want is to download it again Manually renaming or copying would be preferable to waiting another few hours to be able to grab the file again.

Maybe giving people an option to download a file without the extension .part? That way there would be no need to rename the file (and Java seems to have a rename operation that's ridiculously bad and is deleting files when it fails). If you create and lock file for writing, no other app could get a lock for the file so it couldn't overwrite it while it's being downloaded. Would that fix things?
Reply With Quote
  #17  
Old 12.08.2011, 09:54
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,232
Default

the deleting is done by jdownloader and not by java i already explained internal tech infos.

and the locking does not help because to rename the file you must release all locks! the error does NOT happen while writing into the file! its in the rename process because some other app opened the file.

i already added the copy workaround to source
__________________
JD-Dev & Server-Admin
Reply With Quote
  #18  
Old 12.08.2011, 19:48
mikebell mikebell is offline
JD Adviser
 
Join Date: Oct 2009
Posts: 104
Default

Thank you!
Reply With Quote
  #19  
Old 13.08.2011, 02:03
Statter Statter is offline
JD Legend
 
Join Date: Aug 2010
Posts: 541
Default

If one does not mind I'll throw my 2 cents in here.

@ mikebell and others of course, the .part naming is typical of many programs while a fle is being downloaded, some apps will name it as .download, etc. this is done for several reasons ... one is so that the application and user knows without a doubt that the file in question is not complete. Two, if an internet connection is broken for any reason then both the application and the user will know right away and the application will try to continue where it left off if the hoster has allowed it. Else the file will need to be restarted from scratch. Three some applications will either auto rename them or move them to the new file name where they belong. In the past often these downloaded files were not even visible to most as they were put in a hidden system folder to begin with while downloading. they wold then be renamed and moved to a uses default folder were they either set or left as default by the program. So giving people an option to download a file without the extension .part can be dangerous and end up giving more corrupted files. Also as mentioned when a file is downloaded from one place to another the file(s) may get fragmented and when they are done and copied out of the original download area(s) some applications actually will copy the file out and then rename it also before it is available to use. If something happens during this process on the local machine, one can or will be screwed and the file may be lost into oblivion ...

Copying extremely large files is often a bit trickier as the file first should be compressed to keep the real file from being accidentally broken with intermittent download networking glitches which can happen from the hoster to you through the various IP routes it may need to take. this is one reason many places only allow for certain file sizes and files often need to be compressed and split.

This of course makes it a bit difficult for companies that do huge client layouts for such things as Photos (I have one client that often creates a PSD type file that is more than 50 MB in size with many layers to get what their client wants and needs, he has to share them with his clients at full size for approvals etc. before going to production etc.), then there are Musicians and studio technicians that often need to send full individual or full set tracks back and forth or from one recording engineer to another before a final product is ready for release. Then there are those in the Movie type fields that also often have to send huge video files back and forth too for individual editing touches such as 3D, animation etc. before a final cut can be finished. A lot of times unless one is a major studio or multi roll backed they do not have the best fastest internet connections to the outside world for the file transfers, nor do they have the best fastest servers for such transfers of multiple clients accessing their shared areas to upload and download continuously. Thus they will often use external file hosters for such too.

Hope this helps some understand a little better on such problems. I suspect it is something internal on your machine ....
I suggest that you have at least twice if not a bit more disk space available for the download drive and the resulting output drive for all files one is trying to dl and every once in a while wen problems are noticed ...
Try to turn off AV for such application file download areas and for network monitoring the app, only grab from trusted sources also. One can always run AV on a directory after everything has been verified as complete...
Run a disk check now and then to ensure no bad blocks and or directory structure corruption.
Be sure one does not have any other applications and or settings that conflict with the application having problems.
When and only when truly needed then uninstall the application totally including registor data (winblows), support files, etc. and reinstall the application .. and try again.
Worst case scenario is needing to reformat the OS drive and starting over.
__________________
OS X !0.6.8 Mac Pro Intel (Workhorse)
OS X 10.13.6 MBP Intel 17" (Secondary)
OS X MBP Intel 15" Dual boot 10.6.8 and 10.13.6 (as needed)
Reply With Quote
  #20  
Old 13.08.2011, 11:19
remi
Guest
 
Posts: n/a
Cool

Quote:
Originally Posted by Statter View Post
Also as mentioned when a file is downloaded from one place to another the file(s) may get fragmented and when they are done and copied out of the original download area(s) some applications actually will copy the file out and then rename it also before it is available to use. If something happens during this process on the local machine, one can or will be screwed and the file may be lost into oblivion ...
Moving a file from one disk to another should be an atomic transaction. Either the file is moved or not. Losing a file during this type of transaction means you're using a bad OS.

Yesterday I was moving some GB files from one disk to another and the process stopped because there was an error on the other disk. The move transactions were stopped and Windoze is now checking/repairing (it's in stage 4 of five now) that WD disk for more than a day now.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 06:25.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.