JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #21  
Old 10.03.2010, 01:42
Gweilo's Avatar
Gweilo Gweilo is offline
JD Legend
 
Join Date: Mar 2009
Posts: 716
Default

Quote:
Originally Posted by Jiaz View Post
1.) merging them is fine (as zip uses the same technique)
1.1) why should we start blacklisting if the merged file also works fine and with enabled deletion its the same?
I am not so confident that merging will always be fine.
But the question is not why not merge every fileset you (think) you can, but why merge at all if it only gives you a (very large) intermediate file that JD cannot do anything with?

What is the advantage?
What good is this big zip file? What can you do with it that you couldn't do with the original segments?
And the cost is 1) filespace 2) some minutes intensive processing to create the file.

Rather than a blacklist, it should use a whitelist of formats that it can safely and completely merge and extract.

Quote:
Originally Posted by Jiaz View Post
2.) as i already told the deletion of the parts is temporarily disabled because of an unsupported split format (enabled deletion would kill your download in that case)
But the creation of that (unwanted and unnecessary) merged file blocked the next download, causing a delay of several hours.

Last edited by Gweilo; 10.03.2010 at 01:49.
Reply With Quote
  #22  
Old 10.03.2010, 06:31
drbits's Avatar
drbits drbits is offline
JD English Support (inactive)
 
Join Date: Sep 2009
Location: Physically in Los Angeles, CA, USA
Posts: 4,437
Default

@ Jiaz,
I am sorry, I misunderstood you about HJsplit in the past.

@ Gweilo,
You were inconvenienced. I think it is getting blown out of proportion. I have seen Zip files that were split. Your disk space problem and the subsequent delay in downloading other files is your own fault. It is important to keep at least 3 times the empty space as you will use for one program (for most editors and most archive programs).

@ Jiaz,
It is possible to distinguish a multi-volume Zip archive from an archive that has been split. multi-volume archives have the size of each volume in the index.

If you reassemble a multi-volume rar archive, it probably will be broken. Rar adds checksums and other data to the end of each volume.

Reassembly before extraction is an old approach for Zip. It has not been used by PKzip or WinZip for years, because with fixed sized volumes (except the last), extracting from multiple volumes is much faster. 7za never reassembles volumes to extract files. However, it can reassemble most split files.
Reply With Quote
  #23  
Old 10.03.2010, 08:07
Gweilo's Avatar
Gweilo Gweilo is offline
JD Legend
 
Join Date: Mar 2009
Posts: 716
Default

Quote:
Originally Posted by drbits View Post
You were inconvenienced. I think it is getting blown out of proportion. I have seen Zip files that were split. Your disk space problem and the subsequent delay in downloading other files is your own fault. It is important to keep at least 3 times the empty space as you will use for one program (for most editors and most archive programs).
I realise you are trying to be helpful, but you can also be very patronising and dismissive.
Implying that I'm a clueless idiot and remarks like "it's your own fault" are offensive.

I did not "blow anything out of proportion".
I simply reported exactly what happened.

I knew exactly how much disk space I needed and allowed for that.
I know how long it takes to download files and I planned to deal with them the next day, using another drive.

No, it's not a big deal that I lost 8 hours of download time.
Jiaz couldn't see any downside and I was replying to that.

And from your other posts it seems you too assumed that the "Extract" option for a package also governed HJSpit, and you certainly agree that the policy of combining any fileset just because it is possible is not a good idea.

But please restrain yourself from lecturing me on the ABCs of how to use a computer. If you think the issue is trivial, just ignore it.
Reply With Quote
  #24  
Old 10.03.2010, 11:01
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 66,134
Default

i will think about a blacklist what not to merge, okay?
whitelist would be too long blacklist is much smaller!

jdhjmerge already can differ between hjsplit splitted rar archives or normal rar splitted archives! same way goes with other fileformats!
as hjsplit and many other formats do not have a fileheader to tell that the file had been splitted, we added our own check. only if 1 valid fileheader/signature is found in all parts, jd starts to merge, as rar/7zip and other formats add fileheader/signature to every part and this is recognized by jd!
__________________
JD-Dev & Server-Admin
Reply With Quote
  #25  
Old 10.03.2010, 11:02
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 66,134
Default

http://svn.jdownloader.org/issues/show/1428
__________________
JD-Dev & Server-Admin
Reply With Quote
  #26  
Old 10.03.2010, 11:26
Gweilo's Avatar
Gweilo Gweilo is offline
JD Legend
 
Join Date: Mar 2009
Posts: 716
Default

Quote:
Originally Posted by Jiaz View Post
i will think about a blacklist what not to merge, okay?
whitelist would be too long blacklist is much smaller!
My personal whitelist would have two or three entries.

But, whichever you do, it would be nice if this list could be edited by users, not hard wired.

Accepting regex would let users make it either a black or white list.
But I'd settle for a simple list if that's hard.

Until then I've just turned it off and will merge manually.
Reply With Quote
  #27  
Old 10.03.2010, 12:00
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 66,134
Default

what more filetypes do you want not to merge?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #28  
Old 10.03.2010, 13:51
Gweilo's Avatar
Gweilo Gweilo is offline
JD Legend
 
Join Date: Mar 2009
Posts: 716
Default

Quote:
Originally Posted by Jiaz View Post
Excellent, thanks.


Quote:
Originally Posted by Jiaz View Post
what more filetypes do you want not to merge?
.zip, .rar for a start.
The appropriate archive apps can work as easily with the segments as a monolithic file.
drbits says he sees HJSplit RAR files, but I never have.
Thus my suggestion of a user-definable list to accommodate both of us.

And as I said, my whitelist would be shorter.
.avi
.mkv
and that's all.
Others might add
.iso
.flac
perhaps, but I rarely get that kind of file myself, and if I do they seem to be in RAR .r0x format rather than HJSplit.

Last edited by Gweilo; 10.03.2010 at 13:54.
Reply With Quote
  #29  
Old 11.03.2010, 06:12
drbits's Avatar
drbits drbits is offline
JD English Support (inactive)
 
Join Date: Sep 2009
Location: Physically in Los Angeles, CA, USA
Posts: 4,437
Default

I am sorry Gweilo, I do not mean to be condescending.

The three times the disk space you think you need rule of thumb is not as important as it used to be, but it is always a good guideline.

JD will eventually check disk space before performing most tasks. It just has not gotten there yet. I think that will come after some of the major internal changes.

I have not seen any other archive types than Zip in which a segmented archive looks so much like a split file.

@ Jiaz,
I like the idea of a blacklist better than a whitelist.

The number of extensions in use is at least in the tens of thousands. A large percentage of them grow big enough to split (over 1.44 MB, but realistically 15,000,000 bytes). Fortunately, the traditional Unix and MacOS used tar and sit, which are not multivolume. The OSX and Linux installers use many different extensions, some of them are zip or 7z volumes and some are splits. Even Jar files come both ways (but are more likely splits, unless it is part of a scene release). Because of tar, the Unix/Linux users got used to splits. Windows users are not used to splits.

The way to distinguish a split Zip from a multivolume Zip is to look at the first two bytes of each file. If they are all PK it is a multivolume. Since you cannot tell a Zip file from its extension, you always have to check this. I believe the same applies to RAR and 7z (different signatures). With early zip files, you could merge volumes and unzip. More modern versions do not always merge into a usable file (an index is allowed only at the end of a volume and they have started distributing the index in some cases).

Whatever you use has to check again after extraction/Merging. Scene releases of music or a small group of files are normally 15,000,000 bytes. These are usually rar (a few use zip). If the data is large enough the scene uses 50,000,000 bytes (15 files per full CD). Many RS or HF files are rar containers (no compression), containing a scene release. The RS and scene sizes do not divide evenly.

Sorry I got carried away.
drbits

Last edited by drbits; 11.03.2010 at 06:17.
Reply With Quote
  #30  
Old 11.03.2010, 07:03
Gweilo's Avatar
Gweilo Gweilo is offline
JD Legend
 
Join Date: Mar 2009
Posts: 716
Default

Quote:
Originally Posted by drbits View Post
I am sorry Gweilo, I do not mean to be condescending.

The three times the disk space you think you need rule of thumb is not as important as it used to be, but it is always a good guideline.
I can see that you really can't stop yourself from lecturing. So while others might do this as a veiled insult, I will assume you just have no idea what the effect is of your "computing for dummies" homilies.

Otherwise:
Quote:
Originally Posted by drbits View Post
The way to distinguish a split Zip from a multivolume Zip
Which misses the point that JD cannot decode a zip of any kind, whereas the programs that can don't need the file to be merged. So JD doesn't need to distinguish, it should just leave them all alone.
Reply With Quote
  #31  
Old 11.03.2010, 07:23
drbits's Avatar
drbits drbits is offline
JD English Support (inactive)
 
Join Date: Sep 2009
Location: Physically in Los Angeles, CA, USA
Posts: 4,437
Default

Quote:
So JD doesn't need to distinguish, it should just leave them all alone.
You are right about multivolume Zip archives.

However, Zips and Rars are often split after archiving (moving from MU to RS and many Linux users).

Since JD offers a Merge functionality, single archives that have been split should be merged. The user cannot tell the difference between a multivolume archive and a split archive.

One of the changes over the years is that M$ assumption that file extensions identify the type of contents has become less and less true.

Spliter programs really should add a signature, length, and checksum to each file. On the other hand, Windows users who can use the command line (10%?) can just type
copy MyArk.* MyArk to reassemble an HJsplit, because they have added no validation data. You can do the same thing in Emacs or GreaseMonkey with the right script.
Reply With Quote
  #32  
Old 11.03.2010, 11:38
remi
Guest
 
Posts: n/a
Cool

Quote:
Originally Posted by Gweilo View Post
"computing for dummies" homilies.
:biggrin: But I like these homilies because I'm a simple customer.
Reply With Quote
  #33  
Old 11.03.2010, 12:00
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 66,134
Default

i hope jd7zip will be worked on soon, so we dont need to continue this discussion
__________________
JD-Dev & Server-Admin
Reply With Quote
  #34  
Old 23.03.2010, 05:42
cityguy cityguy is offline
JD Fan
 
Join Date: Mar 2009
Posts: 73
Default

Just had disk full errors that sound very similar to those described above. HJSPLIT is not able to fully re-create the file due to lack of space but since HJSPLIT is not deleting the individual files as it goes along and hasn't been for some time (which I reported months ago) I have to clean-up after it. But I don't always realize that the file has not been reconstructed properly because of the lack of space and I delete the individual files only to find that when I try to extract the archive or play the file that there is a problem.

When will the HJSPLIT cleanup problem be fixed? And can't a general error message be displayed when a combine function fails and stop the rest of the program from trying to continue on creating multiple unknown failures? Unless you're extracting/combining to a different location there is no reason to continue with downloads, extracts or combining.
Reply With Quote
  #35  
Old 23.03.2010, 11:02
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 66,134
Default

http://svn.jdownloader.org/issues/show/1476

auto deletion will come back once
http://svn.jdownloader.org/issues/show/968
is done, because they use the same file extensions!
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 04:24.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.