JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 26.05.2016, 20:40
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 229
Default usenet support and incomplete files

Currently, jd2 marks files that are incomplete as offline and hence wont download them. In a world with par2 files, these are still valuable (and sometimes necessary), so wondering why these aren't downloaded?
Reply With Quote
  #2  
Old 27.05.2016, 03:28
raztoki's Avatar
raztoki raztoki is offline
English Supporter
 
Join Date: Apr 2010
Location: Australia
Posts: 17,611
Default

there is a package customiser rule to disable par/par2 files by default. You can disable this rule, settings > packagiser >

raztoki
__________________
raztoki @ jDownloader reporter/developer
http://svn.jdownloader.org/users/170

Don't fight the system, use it to your advantage. :]
Reply With Quote
  #3  
Old 27.05.2016, 03:48
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 229
Default

I think you misunderstood my Q.

If a file has 100 parts, and 10 parts are missing (i.e. not in the nzb), jd2 won't let you download the file at all. it will be marked offline (not disabled). The parts (of the file) that are available to download shoudl still be able to be downloaded as cant help in par2 recovery
Reply With Quote
  #4  
Old 27.05.2016, 10:45
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

At the moment JDownloader does not support par2 repair and therefore it would not make any sense to mark the file as complete. Can you please provide example nzb files to support@jdownloader.org and I will check par2 support

As an alternative I can provide an option to disable this = download incomplete files too
__________________
JD-Dev & Server-Admin
Reply With Quote
  #5  
Old 27.05.2016, 19:44
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 229
Default

I'm not asking to mark the file complete, I understand that I have to par2 repair on my own.

I think I'm not being clear

there are 2 sets of parts

there are the parts of the rar archive (which I am not talking about)

part001.rar
part002.rar

and the segments parts that combine to form a single file

i.e. if one has

<segment number="170" bytes="398683">OifkK5ENBCyfSt1L03h9@JBinUp.local</segment>
<segment number="172" bytes="398683">eiCLnTRiM7w6Qr6wgTS5@JBinUp.local</segment>


I'm talking about the segment parts

if a segment part is missing in the nzb file (or errors out), JD2 doesn't handle the situation (in my opinion correctly).

If the segment part is missing in the nzb file, it wont allow the file to download at all. If a part errors out, it stops downloading the file.

in both cases we want to download as much segment parts as possible as otherwise we might not have enough data to par2 rebuild (manually, I understand as stated above)
Reply With Quote
  #6  
Old 27.05.2016, 20:47
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

I know what parts/segments you were talking about. I'm the one who did the usenet support that's why I asked for example NZB that contains files with missing segments and par2 files.
I'm going to add par2 support to JDownloader so it can process them/repair files.
I will also add a "keep incomplete usenet files when x % segements are missing" option. So you can enable it and use external par2 repair or wait till JDownloader supports it nativ
__________________
JD-Dev & Server-Admin
Reply With Quote
  #7  
Old 27.05.2016, 22:12
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 229
Default

sent: Ticket ID: YVS-402-77955
Reply With Quote
  #8  
Old 28.05.2016, 03:40
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 229
Default

another thing I find weird about the usenet support is the crc32 performed at the end. It might be a function of jd2, but it seems that should be done inline, as bad for caching to do it at the end. As each yenc part can be verified on its own (I think) it just seems a waste to do it at the end. And if its being done inline, unsure what is happening at the end.

basically. I see disk thrashing when using it.
Reply With Quote
  #9  
Old 28.05.2016, 09:40
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

CRC32 is done for each segment AND at the end for the complete file, so JDownloader does know the complete CRC32. So it already verifies each yenc segment during download and the final CRC32 is only to get the CRC32 of the final complete file
__________________
JD-Dev & Server-Admin
Reply With Quote
  #10  
Old 29.05.2016, 08:20
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 229
Default

ok, I guess I never noticed other clients doing a crc32 at the end, maybe just didn't notice. With that said, the caching problem I mentioned is exacerbated by the fact that to get parallel downloads one can download the X threads in a single file, but are split between many files, so if want to download 20 at a time that could be 20 1GB files as opposed to a single 1GB file. in the later case it be still in memory so would crc32 quickly, while in the former io becomes a big performance hit
Reply With Quote
  #11  
Old 29.05.2016, 11:27
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

Hashchecks are done sequentielly to give OS more time to optimize write/read ops on disk. In the end the limiting factor is the IOPS of the device where SSD >>>> HDD
__________________
JD-Dev & Server-Admin
Reply With Quote
  #12  
Old 29.05.2016, 20:31
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 229
Default

I'm talking about the downloading of the files.

currently if we have 100 files to download, each with 100 parts, we can't say download 20 parts of a single file at a time, we have to say download 20 files,. 1 part at a time.

which is hard on the page cache when complete file hash check time comes. If we were able to download a single file a time (using all the possible download threads). the hash check would be much cheaper IO wise as it most likely be cached.
Reply With Quote
  #13  
Old 30.05.2016, 08:32
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 229
Default

Anyways to make this thread clearer I see 3 issues that make using jd2 as my only nzb loader difficult right now

1) doesn't download an incomplete file
2) stops downloading a file on a yenc error (and possibly others)
3) can't download a single file with multiple threads, to use multiple threads must download multiple files in parallel which is cache/iops inefficient (especially if something else that uses lots of iops is running at the same time)
Reply With Quote
  #14  
Old 30.05.2016, 13:13
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,286
Default

@thetoad: please let us use *part* for a single file (eg part01.rar) and *segment*s for the usenet segments for this *part*, helps to clear up what we mean

1.) will add option to enable this feature. you specify a ration of % of allowed missing segments
2.) will be fixed with solution of 1
3.) multi segment downloading is not implemented yet. as crc32 is down during download this has no influence on iops or caching stuff. CRC32 at the end happens sequentially and even high iops/disk usage is no problem as long as your disk is faster than your download speed. It's job of the OS to optimize disk access patterns and caches.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #15  
Old 30.05.2016, 19:35
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 229
Default

another thing I remember (And have encountered). yenc segment errors can be invalid on occasion, i.e. if something fails a yenc segment check, I wouldn't throw it away (or treat it as all zeroes), store it as is, it still might pass par2 check as the yenc error was invalid.

i.e. can give you an nzb that newsbin pro fails some files yenc checks, but the file still passes par2 check.

re #3 let me give an example.

machine has 8GB of ram.

downloading multiple 40 part NZBs where each part is 1GB (so 40GBs per NZB), if we have a 300Mbps connection we are going to max out JD2 (so lets say 20 connections). That means we are going to be finishing about 20 1GB at the same time, they can't be cached so the crc-32 of the complete file is slow.

this isn't so bad, except for when auto extract is happening at the same time (and worse if extracting to same media, I try to avoid that). both the auto extract and the crc 32 will want to use as many iops as you can feed them.

but it sounds like you want to implement threaded multi-segement downloading so that wont be a problem in future.
Reply With Quote
  #16  
Old 31.05.2016, 09:15
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 229
Default

my previous comment is in regards to this code

Code:
if (bodyInputStream instanceof YEncInputStream) {
    final YEncInputStream yEnc = (YEncInputStream) bodyInputStream;
    if (yEnc.getPartCRC32() != null && meteredThrottledInputStream.getInputStream() instanceof CheckedInputStream) {
        final HashInfo hashInfo = new HashInfo(yEnc.getPartCRC32(), HashInfo.TYPE.CRC32);
        final long checksum = ((CheckedInputStream) meteredThrottledInputStream.getInputStream()).getChecksum().getValue();
        if (!new HashResult(hashInfo, HexFormatter.byteArrayToHex(new byte[] { (byte) (checksum >>> 24), (byte) (checksum >>> 16), (byte) (checksum >>> 8), (byte) checksum })).match()) {
            throw new PluginException(LinkStatus.ERROR_DOWNLOAD_FAILED, _JDT.T.system_download_doCRC2_failed(HashInfo.TYPE.CRC32));
        }
        segment._setHashInfo(hashInfo);
        writeUsenetFile = true;
    }
    if (usenetFile.getHash() == null && yEnc.getFileCRC32() != null) {
        usenetFile._setHashInfo(new HashInfo(yEnc.getFileCRC32(), HashInfo.TYPE.CRC32, true));
        writeUsenetFile = true;
    }
}
I'm not convinced that any yenc crc error should be treated as a permanent error, at best it can be used to avoid par2 error checking (everything passed, so can assume par2 would pass as well), but it can't be trusted (as will have false negatives due to lots of bad uploading software).

The real trust should only be in the par2 file. At least this is the behavior of other nzb downloaded (newsbin pro, sabnzbd....).

It's even more complicated as many times will still have to par2 repair the set even if everything crc32'd ok, as its not usable as is, but a par2 will make it a usable download as it will rename all the files.

anyways, just my 2c.

Last edited by thetoad; 31.05.2016 at 09:18.
Reply With Quote
  #17  
Old 01.06.2016, 01:11
thetoad thetoad is offline
Tornado
 
Join Date: Jan 2014
Posts: 229
Default

another issue that I notice is that connections get "stuck". i.e. they stop downloading any data but jd2 thinks they are still connected (so average speed goes down, eta goes up, and total download stays constant), many times the only thing I can do is restart jd2 (or perhaps wait a really long time).
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 19:48.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.