JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 07.03.2010, 05:34
Gweilo's Avatar
Gweilo Gweilo is offline
JD Legend
 
Join Date: Mar 2009
Posts: 716
Default Extract when unselected

I set JD to download some files last night.
These were a kind of segmented zip file:
file.zip.001
file.zip.002
etc.

JD can't extract these (they're not HJsplit, despite having a similar naming system), but I can extract using Winrar or 7-zip, so I unset the "Extract" on the package.

However I found that JD had gone ahead and tried to combine these files using HJSplit, created a large .zip file, and in the process filled up the hard disk (space is tight, but I had enough to download and was going to extract to another drive), and so could not finish the other packages.

Also, despite there being zero space available, JD kept trying to download other files, of course failing every time.

It had the error "ErrorMessage: I O: There is not enough space on the disk" but then just tried to download another file to the same location regardless.


I can't see anything in the log about HJSplit or extract status, but here it is anyway: log.

Last edited by Jiaz; 10.03.2010 at 11:02.
Reply With Quote
  #2  
Old 07.03.2010, 12:35
remi
Guest
 
Posts: n/a
Default

You probably already know that there's a Feature #712 : Check free disk space before unpacking in the bug tracker. Just wait for the next major upgrade.

I'm not sure it will also check disk space before downloading a new file. This has been asked many times as well.
Reply With Quote
  #3  
Old 07.03.2010, 13:15
Gweilo's Avatar
Gweilo Gweilo is offline
JD Legend
 
Join Date: Mar 2009
Posts: 716
Default

Quote:
Originally Posted by remi View Post
You probably already know that there's a Feature #712 : Check free disk space before unpacking in the bug tracker. Just wait for the next major upgrade.

I'm not sure it will also check disk space before downloading a new file. This has been asked many times as well.
Yes, but the major issue here was that JD extracted (or tried to) despite that being unset. The disk space problem was a result of that, as it made a 2 GB zip file but was unable to go further, so did not delete the original segments and I didn't have 2 GB spare.
Reply With Quote
  #4  
Old 07.03.2010, 15:29
remi
Guest
 
Posts: n/a
Default

I know what your major issue is, but I have no solution for it. I prefer to solve the easiest problems first. You happen to attract unique problems.

BTW, are you the guy who was operating H.A.L. in Kubrick's "2001, A Space Odyssey"?
Reply With Quote
  #5  
Old 07.03.2010, 18:26
Gweilo's Avatar
Gweilo Gweilo is offline
JD Legend
 
Join Date: Mar 2009
Posts: 716
Default

Quote:
Originally Posted by remi View Post
I know what your major issue is, but I have no solution for it. I prefer to solve the easiest problems first. You happen to attract unique problems.
I don't think I'm unique in having problems, but maybe I'm more likely to complain about them...

Anyway, it does seem odd that while when JDunrar does its thing I see the command line and some output in the log, but nothing at all is shown for HJSPlit.

If it would help I can give the URLs I was downloading, though it is a large file (2 GB is big for me anyway).


Quote:
Originally Posted by remi View Post
BTW, are you the guy who was operating H.A.L. in Kubrick's "2001, A Space Odyssey"?
Dave Bowman (Keir Dullea). From the Greatest Movie Of All Time.


Last edited by Gweilo; 07.03.2010 at 18:37.
Reply With Quote
  #6  
Old 08.03.2010, 06:18
drbits's Avatar
drbits drbits is offline
JD English Support (inactive)
 
Join Date: Sep 2009
Location: Physically in Los Angeles, CA, USA
Posts: 4,437
Default

BUGTRACKER

The Zip part will be fixed when unZip capability is added to JDownloader.

JD is supposed to check the file signature before combining files, so this is a bug (the leading PK should have been a tip-off). Currently, the check box is ignored for Merging.
Reply With Quote
  #7  
Old 08.03.2010, 10:34
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 66,134
Default

because of unsupported split format we temporarily disabled the part deletion of hjmerge files! also the part files only get deleted if merge is complete! not while merging (so merging of 2gb files needs 2gb empty space)! it would be bad idea to delete part while merging because if something goes wrong you lost the other parts

also jd already checks if its allowed to merge the files or not and its working okay as far as we know. we need example links that cause problems!
zip files can be merged because they use the same technique!
__________________
JD-Dev & Server-Admin
Reply With Quote
  #8  
Old 08.03.2010, 10:37
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 66,134
Default

extract checkbox is not for hjmerge! please write feature request if you want this
__________________
JD-Dev & Server-Admin
Reply With Quote
  #9  
Old 08.03.2010, 10:56
remi
Guest
 
Posts: n/a
Cool

Quote:
Originally Posted by Gweilo View Post
Dave Bowman (Keir Dullea). From the Greatest Movie Of All Time.[/IMG]
Since you don't allow PMs I would like to add to that statement "By the Greatest Director Of All Time".
Reply With Quote
  #10  
Old 08.03.2010, 11:39
Gweilo's Avatar
Gweilo Gweilo is offline
JD Legend
 
Join Date: Mar 2009
Posts: 716
Default

Quote:
Originally Posted by Jiaz View Post
because of unsupported split format we temporarily disabled the part deletion of hjmerge files! also the part files only get deleted if merge is complete! not while merging (so merging of 2gb files needs 2gb empty space)! it would be bad idea to delete part while merging because if something goes wrong you lost the other parts
I disabled the HJ deletion a long time ago because it created an invalid file and deleted the parts, I had to retrieve the whole set again.

Quote:
Originally Posted by Jiaz View Post
also jd already checks if its allowed to merge the files or not and its working okay as far as we know. we need example links that cause problems!
zip files can be merged because they use the same technique!
If JD is actually supposed to combine regardless of "extract", then it was technically correct after all.

But even if zip.00x can be merged to a valid zip (which I'm not sure they can, always -- I think I had problems when it was a passworded 7-zip archive, for instance) there is no point, it just wastes time and space as JD can't complete the extract. 7-Zip and Winrar can extract without combining first.

So I want it to only combine data files like *.avi.00x, not split archives.
The rest I will choose the application myself.

Here's my links:
Spoiler:
Code:
http://rapidshare.com/files/157962824/Tom_Burlinson_-_Man_From_Snowy_River__1__1982_.zip.001
http://rapidshare.com/files/157975996/Tom_Burlinson_-_Man_From_Snowy_River__1__1982_.zip.002
http://rapidshare.com/files/157985649/Tom_Burlinson_-_Man_From_Snowy_River__1__1982_.zip.003
http://rapidshare.com/files/157996695/Tom_Burlinson_-_Man_From_Snowy_River__1__1982_.zip.004
http://rapidshare.com/files/158008780/Tom_Burlinson_-_Man_From_Snowy_River__1__1982_.zip.005
http://rapidshare.com/files/158029153/Tom_Burlinson_-_Man_From_Snowy_River__1__1982_.zip.006
http://rapidshare.com/files/158122340/Tom_Burlinson_-_Man_From_Snowy_River__1__1982_.zip.007
http://rapidshare.com/files/158147034/Tom_Burlinson_-_Man_From_Snowy_River__1__1982_.zip.008
http://rapidshare.com/files/158179704/Tom_Burlinson_-_Man_From_Snowy_River__1__1982_.zip.009
http://rapidshare.com/files/158198047/Tom_Burlinson_-_Man_From_Snowy_River__1__1982_.zip.010
http://rapidshare.com/files/158211659/Tom_Burlinson_-_Man_From_Snowy_River__1__1982_.zip.011
http://rapidshare.com/files/158224284/Tom_Burlinson_-_Man_From_Snowy_River__1__1982_.zip.012
http://rapidshare.com/files/158232157/Tom_Burlinson_-_Man_From_Snowy_River__1__1982_.zip.013
http://rapidshare.com/files/158244363/Tom_Burlinson_-_Man_From_Snowy_River__1__1982_.zip.014
http://rapidshare.com/files/158251464/Tom_Burlinson_-_Man_From_Snowy_River__1__1982_.zip.015
http://rapidshare.com/files/158265688/Tom_Burlinson_-_Man_From_Snowy_River__1__1982_.zip.016
http://rapidshare.com/files/158280258/Tom_Burlinson_-_Man_From_Snowy_River__1__1982_.zip.017
http://rapidshare.com/files/158290990/Tom_Burlinson_-_Man_From_Snowy_River__1__1982_.zip.018
http://rapidshare.com/files/158305261/Tom_Burlinson_-_Man_From_Snowy_River__1__1982_.zip.019
http://rapidshare.com/files/158319246/Tom_Burlinson_-_Man_From_Snowy_River__1__1982_.zip.020
**External links are only visible to Support Staff**


These may be able to be combined to a valid zip, I didn't check that, there is no password so maybe it works.



Quote:
Originally Posted by Jiaz View Post
extract checkbox is not for hjmerge! please write feature request if you want this
It isn't really obvious that there should be a distinction between "merging" and "extracting". I know they are different, but both involve processing a set of downloaded files to get ("extract") the files you actually want.

I for one did expect this to already be the behaviour; and I don't think this was an unreasonable assumption. Maybe you could relabel the option as "process archives".

Now I know how you classify this, I've found and unchecked "Enabled" on the HJSplit Addons menu. By the way, surely if not "Enabled" it should toggle to "Disabled".

Last edited by Gweilo; 08.03.2010 at 12:00.
Reply With Quote
  #11  
Old 08.03.2010, 13:31
remi
Guest
 
Posts: n/a
Cool

Quote:
Originally Posted by Gweilo View Post
7-Zip and Winrar can extract without combining first.
I'm a simple customer and I've never understood why extracting files was made so complex, while with Z-Zip I can always extract files.

Thanks to 7-Zip's knowledge about archiving formats all the needless complications are simply avoided.
Reply With Quote
Old 09.03.2010, 04:51
drbits
Message deleted by drbits. Reason: duplicate
  #12  
Old 09.03.2010, 04:54
drbits's Avatar
drbits drbits is offline
JD English Support (inactive)
 
Join Date: Sep 2009
Location: Physically in Los Angeles, CA, USA
Posts: 4,437
Default

And 7za (the API version) expands most compressed archives and merges most split files. The archive format that 7za does not handle (natively) is RAR. The unrar library is proprietary to the unrar author.

The collection of split files needed to restore a whole file is a form of archive. The merging is a form of extraction as far as users are concerned. JD users rarely split files or create archives, so the distinction is not relevant to the users.

Since none of the archive processors that deal with split archives merge the files first, archives should not be merged. The exception is a split tgz. A tar file can be split with HJsplit (or equivalent) and then each part can be zipped. In that case, the file has to be uncompressed before merging the split portions.

There is already a feature request that includes merging the functionality of Merge and Extract. This is necessary from a user's point of view to do a deep extraction.
Reply With Quote
  #13  
Old 09.03.2010, 14:06
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 66,134
Default

jd already handles this correct

you can split a rar archive with normal rar splitter eg rar.r01 rar.r02 and so

but you can also split an rar.rar archive with hjsplit and result in rar.001 rar.002 and so on. this needs to be merged first!
__________________
JD-Dev & Server-Admin
Reply With Quote
  #14  
Old 09.03.2010, 15:47
Gweilo's Avatar
Gweilo Gweilo is offline
JD Legend
 
Join Date: Mar 2009
Posts: 716
Default

Quote:
Originally Posted by Jiaz View Post
jd already handles this correct
Sorry, no it does not.

In the case I reported, all JD did was waste time and 2 GB of file space by combining the zip.00x files to a giant zip file which it could not extract. The utilities that can extract it can just as easily use the original zip.00x fileset.

Only if you don't care about wasting time and disk space is it "correct" to create such a useless file.


Quote:
Originally Posted by Jiaz View Post
you can split a rar archive with normal rar splitter eg rar.r01 rar.r02 and so

but you can also split an rar.rar archive with hjsplit and result in rar.001 rar.002 and so on. this needs to be merged first!
Theoretically, but why would anyone split a rar archive like that instead of doing it within WinRar?

If someone was clueless enough to make such a file archive, I could deal with it as a special case. I don't want JD to do anything with it, and that is why I unselected "Extract" for that package.

HJsplit should only run automatically on things like .avi.00x.

Last edited by Gweilo; 09.03.2010 at 15:49.
Reply With Quote
  #15  
Old 09.03.2010, 15:52
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 66,134
Default

Quote:
Originally Posted by Gweilo View Post
Sorry, no it does not.

In the case I reported, all JD did was waste time and 2 GB of file space by combining the zip.00x files to a giant zip file which it could not extract. The utilities that can extract it can just as easily use the original zip.00x fileset.
the behaviour of jd was correct, the resulted 2gb zip file can be extracted. how should jdhjmerge know that jd cannot extract zip files. for the enduser the result is the same (no difference between 1 2gb file or 20 100mb files )
jdhjmerge merges files that need to get merged! thats its job and thats what it does
__________________
JD-Dev & Server-Admin
Reply With Quote
  #16  
Old 09.03.2010, 18:06
Gweilo's Avatar
Gweilo Gweilo is offline
JD Legend
 
Join Date: Mar 2009
Posts: 716
Default

Quote:
Originally Posted by Jiaz View Post
the behaviour of jd was correct, the resulted 2gb zip file can be extracted. how should jdhjmerge know that jd cannot extract zip files.
What? Because you (or whoever) wrote it and you (they) know it can't.
If it was a .xyz file you might claim ignorance, but zip is not a mystery format.


Quote:
Originally Posted by Jiaz View Post
for the enduser the result is the same (no difference between 1 2gb file or 20 100mb files )
Yes there is a difference, because the original files are not deleted, so it takes up twice as much space, which was critical in this case, as it consumed all the filespace I had allocated for the other downloads. And I'm still not convinced that in every case the merged files is usable. At any rare, it's not more usable than the original set of files, as they can be extracted without merging. The 2 GB file is at best redundant, and at worst, as in my case, a problem.

Quote:
Originally Posted by Jiaz View Post
jdhjmerge merges files that need to get merged! thats its job and thats what it does
Right. But these files DO NOT need to be merged.
There is no point or use in merging them.
I specifically did not want JD to even try.

Anyway, I know now how to prevent this from happening (now I know that despite my expectation that "merging" and "extracting" are treated differently) so my own problem is solved.

It seems that you are satisfied with the logic of the program in this respect so no point suggesting changes.
Reply With Quote
  #17  
Old 09.03.2010, 20:33
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 66,134
Default

1.) merging them is fine (as zip uses the same technique)
1.1) why should we start blacklisting if the merged file also works fine and with enabled deletion its the same?
2.) as i already told the deletion of the parts is temporarily disabled because of an unsupported split format (enabled deletion would kill your download in that case)
__________________
JD-Dev & Server-Admin
Reply With Quote
  #18  
Old 09.03.2010, 20:39
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 66,134
Default

blacklisting a filetype that works fine when merged correctly is advantage in my eyes. as i said before...i dont think there is a difference betweend 1 big file or many small files

okay the non deletion of the part files sucks at the moment, but that will be working again with next update!
__________________
JD-Dev & Server-Admin
Reply With Quote
  #19  
Old 09.03.2010, 21:42
drbits's Avatar
drbits drbits is offline
JD English Support (inactive)
 
Join Date: Sep 2009
Location: Physically in Los Angeles, CA, USA
Posts: 4,437
Default

It really should only merge when it knows the split signature. That would mean that the unsupported split would not be merged (or that the signature has to be more specific).

If the file was split by a Zip program, the signature should be different from a zip file that was split by a splitter. The file name regex is a prerequisite for checking for a corresponding signature (header that indicates the split). I know there are at least four places you have to check for signatures, depending on the splitter.

I think in this case, either PK was included in the signatures (bad decision) or no signature was found, so a merge was done based on the regex of the name.

I am currently downloading to a USB 2.0 drive. This uses PIO, not DMA. That means that the processor is involved in the copy from external bus to file. When it is two way, it is much slower. That extra post-process would take a long time.

Quote:
but why would anyone split a rar archive like that instead of doing it within WinRar?
As archives are moved from host to host, they sometimes have to be smaller size pieces. If it is copied to ugotfile, there is no limit to file size, it just splits the file into 100MiB pieces. They cannot unrar and rerar because of the passwords on many files (and they are less smart than JD).
Reply With Quote
  #20  
Old 09.03.2010, 21:49
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 66,134
Default

hjsplit files DONT have a signature

JD already checks for any known signatures and do not merge parts if each file has a valid signature!

if more than one part contains a fileheader then it will not be merged as its done by other software. if a valid split header or only one header is found then merging is okay
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 01:56.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.