JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #21  
Old 13.08.2011, 12:01
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 66,134
Default

moving a file from a to b is NEVER ATOMIC! as long as its not on the same disk/partition!
__________________
JD-Dev & Server-Admin
Reply With Quote
  #22  
Old 13.08.2011, 13:01
remi
Guest
 
Posts: n/a
Default

I've never lost a file by moving it to a bad disk. Why is that?

I thought files are only removed when the copy operation is successful.
Reply With Quote
  #23  
Old 14.08.2011, 09:55
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 66,134
Default

why should you?
moving to a different disk = copy, delete old one after that
its not the rename/copy/java who is deleting the file. i explained it several times, it is jd that deletes the part file
__________________
JD-Dev & Server-Admin
Reply With Quote
  #24  
Old 14.08.2011, 10:43
remi
Guest
 
Posts: n/a
Default

The OS moves its file system from one consistent state to another consistent state (if all fails then it's the same state). Isn't that called an atomic transaction?

Note that this principle doesn't hold for folders even when it's on the same disk. Sometimes a folder can't be moved and an empty folder is created but the old one is locked while the files aren't moved.
Reply With Quote
  #25  
Old 14.08.2011, 10:51
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 66,134
Default

atomic also means that the file is also only in one state
and when we dig deeper, it also means that its so atomic that is not breakable.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #26  
Old 14.08.2011, 10:52
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 16,304
Default

If windows failed before the removal of source data, you shouldn't loose anything. Windows does not crc check transfers so if the data wrote to disk and windows removed file from source and your destination data is corrupted you have effectively lost your data. Unless you of course use some data recovery software. Windows never removes source data if the transmission is broken/stopped/halted..

Personally I use 3rd party software to copy/move data on my systems as windows lacks features! CRC check source (on copy) and an additional read CRC check on destination after write. This process has saved me in the past and worth the extra time required (additional as in it needs to read destination data). More modern filesystems have checksums built in, one of these days this should all be built into OS operations based on filesystem figures over software ontop of operating systems.
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #27  
Old 14.08.2011, 11:13
remi
Guest
 
Posts: n/a
Default

It's ridiculous that a modern OS doesn't do these basic checks as file operations certainly belong to its core business.

If I understand you well this means that Windoze writes files to bad sectors that haven't been detected/defined as bad sectors yet.
Reply With Quote
  #28  
Old 14.08.2011, 12:09
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 16,304
Default

Windows will try, it will only know something's wrong if the devices reports back to the OS! It assumes data integrity on disk is fine unless reported otherwise, as I understand it. Not all storage devices are equal on features/functions.
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #29  
Old 14.08.2011, 13:07
remi
Guest
 
Posts: n/a
Default

Windoze is a stupid OS and many people, who know it, including me, are still using it. What does it say about those people?

Anyway, XP will be my last Windoze version. I hope Linux are another OS will not be that stupid.
Reply With Quote
  #30  
Old 14.08.2011, 13:22
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 16,304
Default

lol, in unix/linux its generally the users that's stupid (rtfm) :]
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #31  
Old 18.08.2011, 03:50
Statter Statter is offline
JD VIP
 
Join Date: Aug 2010
Posts: 458
Default

Quote:
Originally Posted by remi View Post
It's ridiculous that a modern OS doesn't do these basic checks as file operations certainly belong to its core business.

If I understand you well this means that Windoze writes files to bad sectors that haven't been detected/defined as bad sectors yet.
Many OS'es will often write to bad blocks sectors if they have not been marked bad previously. Hard drives deteriorate over time also and other things like heads hitting the platters can cause bad sectors to form etc. Why do you thing even the Windows tools for checking drives have the options to check for bad blocks and try to move data from bad blocks? of course this is done during a reboot before windows has a chance to start and if the correct switches have been set.
__________________
OS X !0.6.8 Mac Pro Intel (Workhorse)
OS X 10.10.5 MBP Intel (Secondary)
Reply With Quote
  #32  
Old 18.08.2011, 10:46
remi
Guest
 
Posts: n/a
Default

Windoze is based on DOS, i.e., Disk Operating System. I would expect that the disk should be the kernel of the OS. I would also expect that Windoze tests whether a file has been written/copied/moved properly. What's the sense of writing a file if you can't read it back a minute later?

Instead I as a customer have to check my disks on a regular basis and I do that while I'm working with the OS. I need the detailed log of all the errors and repair actions. I can't let my computer idle during these checks because these repair actions sometimes take more than a day.

Note that one of my last checks resulted in a diagnosis (made by the manufacturer) that the disk controller was kaput. Fortunately the disk was still under warranty.
Reply With Quote
  #33  
Old 13.09.2011, 01:15
edible
Guest
 
Posts: n/a
Default

Quote:
Originally Posted by Jiaz View Post
the deleting is done by jdownloader and not by java i already explained internal tech infos.

and the locking does not help because to rename the file you must release all locks! the error does NOT happen while writing into the file! its in the rename process because some other app opened the file.

i already added the copy workaround to source
When should users expect to this see this in the live build? Because this bug is still happening to me, on both ubuntu two weeks ago (haven't used since), and mac os x lion presently. This has happened both on a filevault secured system and not, as well as with the firewall on and off. I don't have an anti-virus installed. Here is the relevant log output, with some file name censoring. Seems to randomly affect all file hosters. As an aside, I never witnessed this bug using the Windows version, and have only been using the mac version recently.

------------------------ Thread: 127 -----------------------
127 9/12/11 3:03:24 PM - FINER [java_downloader] -> onCHunksReady
127 9/12/11 3:03:24 PM - INFO [java_downloader] -> Close connections if the are not closed yet
127 9/12/11 3:03:24 PM - INFO [java_downloader] -> Closed connection before closing file
127 9/12/11 3:03:24 PM - INFO [java_downloader] -> Close File. Let AV programs run
127 9/12/11 3:03:24 PM - FINEST [java_downloader] -> no errors : rename
127 9/12/11 3:03:24 PM - SEVERE [java_downloader] -> Could not rename file /Users/edible/Downloads/FILENAME.part3.rar.part to /Users/edible/Downloads/FILENAME.part3.rar
127 9/12/11 3:03:24 PM - SEVERE [java_downloader] -> Error occured: ERROR_LOCAL_IO
127 9/12/11 3:03:24 PM - SEVERE [java_downloader] -> A critical Downloaderror occured. Terminate...
127 9/12/11 3:03:24 PM - WARNING [java_downloader] ->
Error occured- latest: ERROR_LOCAL_IO
00000000001000000000000000000010 <Statuscode
00000000000000000000000000000010 |FINISHED
00000000001000000000000000000000 |ERROR_LOCAL_IO
ErrorMessage: Could not rename Part File
Reply With Quote
  #34  
Old 13.09.2011, 10:36
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 66,134
Default

you will have to wait for next major update as this was done deep in core and core is not ready/stable yet for next major release. we are currently working on it.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #35  
Old 14.09.2011, 15:34
scoob
Guest
 
Posts: n/a
Default

First of all, it's shameful that some so-called developer programmed jdownloader to delete completed downloads, UNDER ANY CIRCUMSTANCE. Were you out of your brilliant mind?!

Second of all, all this talk about atomic semantics is nonsense. You are not a database or a file system, you are a simple JAVA APP whose purpose is to DOWNLOAD FILES. The common-sense "fallback" when a rename fails would be to POP UP A DIALOG and tell the user he has to do the renaming himself.

Thirdly, this has been around forever, and it's only now being discussed seriously. I've previously seen other excuses for why parts get deleted.

Unfortunately it's not the first or last gross stupidity that JD does. I'm looking for alternatives myself. There are too many major problems, and also too many little annoyances that make the life of a JD user miserable.

Oh yeah, sorry for nice guyin'. I know it's free and all that. But it doesn't have to be THIS crappy.
Reply With Quote
  #36  
Old 15.09.2011, 10:15
remi
Guest
 
Posts: n/a
Default

An alternative to using another product is to help jD's developers as you might be one of those brilliant java programmers who're standing at the sideline.

Please read "Getting started as a JD-Developer".

BTW, also let me know the product you're going to use. I'm a very curious person.
Reply With Quote
  #37  
Old 01.10.2011, 12:32
weaverslodge
Guest
 
Posts: n/a
Default

Not sure I've posted here before but was checking out reasons for my files not being saved correctly and I came across this thread.

Firstly, a comment

Atomic - This has been overused and badly used. The renaming of a file that is within the same filesystem is an atomic operation due to the fact that the function is just to update the filesystem header with the new information. This will be handled by the FS code in the operating system. It'll be the same for Windoze aswell as Unix (Mac). If you are crossing filesystems then the rename is not a rename, it is a copy and then delete (or unlink system call if you're picky :-)) which is definitely not atomic. Th original idea of an atomic operation is an operation that cannot be stopped in the sense that it happens during one clock cycle and cannot be interrupted. Not quite sure how the advent of dual or quad core chips impacts on atomic operations, but I think that the concept is the same.

One thing that I cannot understand that one of the devs said was that if they know something bad has happened and they can't restart the download then they allow the part file to be removed. This is bizarre !!! You must know at that point that the file has completed the download as the code for the rename wouldn't even be accessed until that had happenned. So if you do get an error from the rename then there is no reason to remove the file. It was said that it was JD that was removing the part file. Why would JD even get involved in removing the part file? If the rename works then there is no part fle to remove anyway. If it didn't work then the error is there and the user can investigate for themselves.

If I were to hazard a guess I would think that you had an error function that you dropped into that cleaned up in the event of an error and that this error shouldn't fall into that category but has unfortunately done so.

Cheers

Dave
Reply With Quote
  #38  
Old 01.10.2011, 13:17
remi
Guest
 
Posts: n/a
Default

I think the way Jiaz uses the word atomic has nothing to do with clock cycles. An atomic operation executes all actions necessary for the operation to succeed in a consistent way, or rolls back the whole operation, i.e., all its separate actions. The goal is to keep a system in a consistent state. The term 'atomicity' is used in database systems and refers to database transactions. It's the first time I see it being used for file operations.

In this way, and I go further than Jiaz, moving a file to another disk can be made atomic if at least the file is written properly and can be read back (and compared), otherwise the file shouldn't be written to the other disk. The OS must also keep a log of all operations so that it can roll back actions in case of a crash or hardware failure during such an operation. I think such systems exist in mission critical environments.
Writing a file to several disks/networks/OSs probably requires a two-phase commit protocol, but this can become very complex.

If a download has failed and it cannot be resumed I think it's better to delete the file because otherwise you'll end up with corrupt/incomplete files on your disks.
Reply With Quote
  #39  
Old 01.10.2011, 13:53
weaverslodge
Guest
 
Posts: n/a
Default

I agree that if a download fails and cannot be resumed that the file should be deleted, no problem with that at all. But if the download is complete and the final stage is to rename the file then that is different.

I appreciate what you say about atomic operations, it's something that I always associate with databases also and in a database context it's fine as you have recovery daemons and the like in the event of a DB crash. For journaled filesystems I guess you could make the rename across filesystems clean up should a crash occur although there is only one stage of the process that you have to worry about. If it crashes during the copy then it will clean up, but if it crashes after the copy but before the unlink then I'm not sure it would be able to deal with it.

The checking of copied files is another matter. From a hardware point of view you have to assume that if you make a request for a file to be moved then it will be and the data will be transferred faithfully. If there is a problem then the hardware will report the problem. From an OS point of view, the fundamental commands do the job that they have been doing for years. They will issue reads and writes and check the return codes and that's all. If you want them to do more than that then you have to look at third party software. If you can't trust the OS to copy a file I wouldn't use that OS (probably why I use Mac's :-))

In reality though if you were to check or compare then every copy would at least double in duration.

I think my main point though was about the removal of a perfectly good file just because it failed to rename it !
Reply With Quote
  #40  
Old 01.10.2011, 14:08
remi
Guest
 
Posts: n/a
Cool

Quote:
Originally Posted by weaverslodge View Post
If you can't trust the OS to copy a file I wouldn't use that OS (probably why I use Mac's :-))
I know Windoze can't be trusted. That's why XP will be my last Windoze version.

Quote:
Originally Posted by weaverslodge View Post
I think my main point though was about the removal of a perfectly good file just because it failed to rename it !
Can you please quote the text where you found this?
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 04:33.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.