JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 04.10.2016, 18:07
madcowpig's Avatar
madcowpig madcowpig is offline
Storm
 
Join Date: Aug 2016
Posts: 210
Default mega.nz and youtube.com questions and possible requests

Welcome back, Jiaz. Hope you had a happy, German Unity Day. I had a couple of questions about these two plugins, that I was told, you would know the answers to.



mega.nz
  1. SHA-1 checksums don't work. I asked Tony about this and he said, there was no hash check in the plugin. I know that MEGA uses AES 128-bit when decrypting, so my question is, does this count as the hash check? If so, is AES 128-bit as good as SHA-1? If not, I would like to request hash check to MEGA plugin.

  2. Also there is a tiny bug when renaming. If you add a file to download list and rename it while it's still downloading, when the file finishes downloading it says, "Host problem?" and you have to wait ten minutes for decrypting to start. A simple work around is, if you disable and enable it itwill start decrypting without waiting ten minutes.

Spoiler:




youtube.com
  1. Does the YouTube plugin losslesly remux the two dash files? Because when it's doing the process in JD it says, "converting/demuxing" and converting usually doesn't mean lossless. Also, does this process cause any audio sync issues, by milliseconds?

  2. When I download using FlashGot, I get different size and size on disk, than JD.

    Here's the results downloading a video using these, "(720p_30fps_H264-192kbit_AAC)" settings. Size on disk for JD is bigger.

    JD on the left and FlashGot on the right.



    When doing the same test with an .m4a file, it gets worse. The file size as well is different.

    Here's the results downloading an audio file using these, (128kbit_AAC) settings.

    JD on the left and FlashGot on the right.



  3. There's a speed issue when downloading 2 dash files for remux in JD. When downloading a 1080p .mp4 file and 125k .m4a file, in Mozilla Firefox, using FlashGot, I can download at 21 MB/s with 1 connection. In JD downloading the same file, which is these, "(1080p_30fps_H264-128kbit_AAC)" settings, I can only get about 1 MB/s using 1 connection, so with 20 connections I was just under the FlashGot speed, which only used 1 connection. The reason I say this issue is only for the 2 dash files for remux, is because downloading these, "(720p_30fps_H264-192kbit_AAC)" settings, which require no dash files or remuxing, I get 21 MB/s in FlashGot and JD with 1 connection each.

  4. Finally, is there a way to get the YouTube plugin to put the real title of the video in the title properties? For example, in the new UFC video the title is, "UFC 204 Free Fight: Michael Bisping vs Luke Rockhold", but since windows doesn't allow ":" in file names, JD renames it to, "UFC 204 Free Fight_ Michael Bisping vs Luke Rockhold". It changes the ":" to a "_". I want a way to put the real/full video title in the title properties automatically, so it displays properly in programs. I circled in red, where I'm talking about.



Here's the link I used for testing, in all of the above, **External links are only visible to Support Staff****External links are only visible to Support Staff** .
__________________
I don't fight the system, but I do play fighting games on it. PS4 to be exact. On second thought, sometimes it won't work right and I have to hit it a couple of times before it's fixed. So technically, I do fight the system.
Reply With Quote
  #2  
Old 04.10.2016, 18:34
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 65,456
Default

As I'm now on my way home, I will respond tomorrow morning
__________________
JD-Dev & Server-Admin
Reply With Quote
  #3  
Old 05.10.2016, 09:54
madcowpig's Avatar
madcowpig madcowpig is offline
Storm
 
Join Date: Aug 2016
Posts: 210
Default

It's morning now. *Sniffle, sniffle.* "Where are you, Jiaz!!!" *Cries uncontrollably.* "I'm scared and alone." *Hears twig snap and turns head swiftly.* "AHHHH!!! I think they're here." *Runs away, fast as hell.*
__________________
I don't fight the system, but I do play fighting games on it. PS4 to be exact. On second thought, sometimes it won't work right and I have to hit it a couple of times before it's fixed. So technically, I do fight the system.
Reply With Quote
  #4  
Old 05.10.2016, 12:14
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 65,456
Default

mega.nz:
1.) do you set the sha1 yourself or where does it come from? I could not find any hash information from their api. so I guess you set it manually and wonder why it is not checked at the end, right?
2.) will check/fix it


youtube:
1.) it is simple mux/demux, no transcoding
2.) Size on disk heavily depends on many filesystem factors.
2.1.) read more here https://board.jdownloader.org/showthread.php?t=70676
2.2.) Flashgot simply downloads the files but no mux/demux happening, as far as I know

3.) please provide example links, can't reproduce

4.) not supported yet
__________________
JD-Dev & Server-Admin
Reply With Quote
  #5  
Old 05.10.2016, 17:18
madcowpig's Avatar
madcowpig madcowpig is offline
Storm
 
Join Date: Aug 2016
Posts: 210
Default

Quote:
Originally Posted by Jiaz View Post
mega.nz:
1.) do you set the sha1 yourself or where does it come from? I could not find any hash information from their api. so I guess you set it manually and wonder why it is not checked at the end, right?
Yes, usually websites put multiple hosters, they almost always use Uploaded (SHA-1) and Rapidgator (MD5), and I use Adopt Checksums, to give it to the websites that have better free speeds.

Quote:
Originally Posted by Jiaz View Post
2.) will check/fix it
Thank you, very much.

Quote:
Originally Posted by Jiaz View Post
youtube:
1.) it is simple mux/demux, no transcoding
I assumed this, but just wanted to make sure, because JD was using the word, "converting". Let me ask you one more thing then, does JD automatically update FFmpeg? If not could you please add a feature to. Also is it for 32bit or 64bit system?

Quote:
Originally Posted by Jiaz View Post
2.) Size on disk heavily depends on many filesystem factors.
Okay.

Quote:
Originally Posted by Jiaz View Post
Just read it. So what does it mean to use a sparse file and sparse flag? What's the benefit of JD using it? When you add the advanced option to disable it, if I do, will there be any negative effects?

Quote:
Originally Posted by Jiaz View Post
2.2.) Flashgot simply downloads the files but no mux/demux happening, as far as I know
Correct, but YouTube does offer some files that are already muxed,. For example the JD YouTube plugin setting, "(720p_30fps_H264-192kbit_AAC)" is already muxed and does not need FFmpeg, so when I download in Flashgot, the video and audio is already together. For the separate audio and video DASH files, I use MKVToolnix to mux them together after downloading with FlashGot.

Quote:
Originally Posted by Jiaz View Post
3.) please provide example links, can't reproduce
I gave you one in my first post. Here it is again, **External links are only visible to Support Staff****External links are only visible to Support Staff** .

Like I said earlier, it only happens when I'm downloading the DASH files, that use FFmpeg to mux. When I download the 720p .mp4 that YouTube already has muxed, "(720p_30fps_H264-192kbit_AAC)" it downloads fast as hell. Here are example pictures,


Here's where the problem happens, it's downloading two separate DASH files that are going to be muxed, at 1 MB/s, with 1 connection,

(1080p_30fps_H264-128kbit_AAC)

Spoiler:


Here's what the file looks like in my Downloads folder, while downloading, "7_dKY3_vleU_MP4_H264_1080P_30FPS_AAC_128KBIT_DASH_58fc4d43e4353c3f81e1691f3107a4d4.dashVideo.pa rt".

It took 8 minutes to finish downloading, which is ridiculous for a, 318 MB file.

Spoiler:



Here's where it works fine, it's downloading 1 file, at 22.5 MB/s, with 1 connection,

(720p_30fps_H264-192kbit_AAC)

Spoiler:


Here's what the file looks like in my Downloads folder, while downloading, "UFC 204 Free Fight_ Michael Bisping vs Luke Rockhold (720p_30fps_H264-192kbit_AAC).mp4.part".

Just to leave no confusion, here's the log for, "(1080p_30fps_H264-128kbit_AAC)", where the problem happens.

05.10.16 10.49.50 <--> 05.10.16 11.00.50 jdlog://1227581887641/

Quote:
Originally Posted by Jiaz View Post
4.) not supported yet
Thank you, very, very much.
__________________
I don't fight the system, but I do play fighting games on it. PS4 to be exact. On second thought, sometimes it won't work right and I have to hit it a couple of times before it's fixed. So technically, I do fight the system.
Reply With Quote
  #6  
Old 07.10.2016, 12:24
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 65,456
Default

I've added hash support in mega with next plugin update. I'm sorry but I was not able to reproduce the second issue you've described (rename it while it's still downloading, when the file finishes downloading it says, "Host problem?"..) Can you please provide a logfile for that
__________________
JD-Dev & Server-Admin
Reply With Quote
  #7  
Old 07.10.2016, 12:33
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 65,456
Default

I've changed plugin to avoid that "Host Problem" but without a way to reproduce or a logfile I'm really trampling in the dark :(
__________________
JD-Dev & Server-Admin
Reply With Quote
  #8  
Old 07.10.2016, 13:53
madcowpig's Avatar
madcowpig madcowpig is offline
Storm
 
Join Date: Aug 2016
Posts: 210
Default

Quote:
Originally Posted by Jiaz View Post
I've added hash support in mega with next plugin update. I'm sorry but I was not able to reproduce the second issue you've described (rename it while it's still downloading, when the file finishes downloading it says, "Host problem?"..) Can you please provide a logfile for that
Sorry, I didn't provide a log file before, because I thought it would be easily reproduced. Here you go,

07.10.16 07.28.23 <--> 07.10.16 07.30.28 jdlog://1398581887641/

Also, if you could please answer these questions I asked before, when you have time.

Spoiler:
Quote:
Originally Posted by madcowpig View Post
Quote:
Originally Posted by Jiaz View Post
Just read it. So what does it mean to use a sparse file and sparse flag? What's the benefit of JD using it? When you add the advanced option to disable it, if I do, will there be any negative effects?


Last thing, you asked me to provide example links, for the YT download speed issue and I gave you one, plus more info and a log, but you gave no response. Is this because you're busy or just didn't see it? If you're busy, then I understand.

Spoiler:
Quote:
Originally Posted by madcowpig View Post
Quote:
Originally Posted by Jiaz View Post
3.) please provide example links, can't reproduce
I gave you one in my first post. Here it is again, **External links are only visible to Support Staff****External links are only visible to Support Staff** .

Like I said earlier, it only happens when I'm downloading the DASH files, that use FFmpeg to mux. When I download the 720p .mp4 that YouTube already has muxed, "(720p_30fps_H264-192kbit_AAC)" it downloads fast as hell. Here are example pictures,


Here's where the problem happens, it's downloading two separate DASH files that are going to be muxed, at 1 MB/s, with 1 connection,

(1080p_30fps_H264-128kbit_AAC)

Spoiler:


Here's what the file looks like in my Downloads folder, while downloading, "7_dKY3_vleU_MP4_H264_1080P_30FPS_AAC_128KBIT_DASH_58fc4d43e4353c3f81e1691f3107a4d4.dashVideo.pa rt".

It took 8 minutes to finish downloading, which is ridiculous for a, 318 MB file.

Spoiler:



Here's where it works fine, it's downloading 1 file, at 22.5 MB/s, with 1 connection,

(720p_30fps_H264-192kbit_AAC)

Spoiler:


Here's what the file looks like in my Downloads folder, while downloading, "UFC 204 Free Fight_ Michael Bisping vs Luke Rockhold (720p_30fps_H264-192kbit_AAC).mp4.part".

Just to leave no confusion, here's the log for, "(1080p_30fps_H264-128kbit_AAC)", where the problem happens.

05.10.16 10.49.50 <--> 05.10.16 11.00.50 jdlog://1227581887641/
__________________
I don't fight the system, but I do play fighting games on it. PS4 to be exact. On second thought, sometimes it won't work right and I have to hit it a couple of times before it's fixed. So technically, I do fight the system.
Reply With Quote
  #9  
Old 07.10.2016, 14:12
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 65,456
Default

I'm working one issue , then the next one
Thanks for the logs.
Sparse File: Normally the OS fills empty *gaps* with 0. For example you download a 1GB file with 2 connections, then one connection starts at beginning and the other one in the middle of the file. That causes the OS to fill the *gap* between beginning and middle with 0 so it can continue to write data to it. That can take a moment (longer time). In that time the tcp/http connection could timeout and cause download abort and need to retry/resume. With sparse flag enabled, the OS does NOT fill the gap. for sparse files only *used* regions are written to disk. A sparse file can easily be several TB/PB in size yet only use some bytes on the disk. Wikipedia and google also provides further information about this. That's why we make use of it to avoid timeouts with large files and max connections > 1.

Youtube: will take look at it after mega
__________________
JD-Dev & Server-Admin
Reply With Quote
  #10  
Old 07.10.2016, 14:20
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 65,456
Default

Rename issue should happen *less common* with next plugin update. But it requires more complex proper bugfixing, see
__________________
JD-Dev & Server-Admin
Reply With Quote
  #11  
Old 07.10.2016, 15:02
madcowpig's Avatar
madcowpig madcowpig is offline
Storm
 
Join Date: Aug 2016
Posts: 210
Default

Quote:
Originally Posted by Jiaz View Post
Thanks for the logs.
You're welcome.

Quote:
Originally Posted by Jiaz View Post
I'm working one issue , then the next one

Youtube: will take look at it after mega
Okay, that makes sense. I can finally sleep comfortably at night. Thank you.

Quote:
Originally Posted by Jiaz View Post
Sparse File: Normally the OS fills empty *gaps* with 0. For example you download a 1GB file with 2 connections, then one connection starts at beginning and the other one in the middle of the file. That causes the OS to fill the *gap* between beginning and middle with 0 so it can continue to write data to it. That can take a moment (longer time). In that time the tcp/http connection could timeout and cause download abort and need to retry/resume. With sparse flag enabled, the OS does NOT fill the gap. for sparse files only *used* regions are written to disk. A sparse file can easily be several TB/PB in size yet only use some bytes on the disk. Wikipedia and google also provides further information about this. That's why we make use of it to avoid timeouts with large files and max connections > 1.
Wow, this explanation was perfect. If you make an advanced option to turn off sparse files, this means I'll probably get timeouts with big files and multiple connections, which I don't want. Or am I misunderstanding, because after reading the ticket, it says, "Controlling: remove sparse flag when finshed". Does this mean that JD will download with sparse files to avoid timeouts, but when downloads finish it will remove sparse flag, to fix size and size on disk discrepancies?
__________________
I don't fight the system, but I do play fighting games on it. PS4 to be exact. On second thought, sometimes it won't work right and I have to hit it a couple of times before it's fixed. So technically, I do fight the system.
Reply With Quote
  #12  
Old 07.10.2016, 15:29
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 65,456
Default

I will first add an option to disable use of sparse files completely. Yes, that *can* cause issues in specific situations. For example download a single 10gb file with 2 connections, so the second connection will have to wait before the OS finished filling up 5GB of zeros in the file. In times of SSDs no issue but an HDD will take some time. No imagine downloading to slow USB2.0 drive or even external NAS network drive with slow connection.
To be honest I don't care about the difference in filesize/disk size. It will go away in the moment when you copy/not moving to another place. It also only affects downloaded files. Extraction dont use sparse files. Removing the sparse flag after download will not *fix* the discrepancies. Only a copy of the file can do it. But I also plan to remove the flag after file is finished because some tools don't *like* the flag enabled.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #13  
Old 07.10.2016, 17:46
madcowpig's Avatar
madcowpig madcowpig is offline
Storm
 
Join Date: Aug 2016
Posts: 210
Default

Quote:
Originally Posted by Jiaz View Post
I will first add an option to disable use of sparse files completely. Yes, that *can* cause issues in specific situations. For example download a single 10gb file with 2 connections, so the second connection will have to wait before the OS finished filling up 5GB of zeros in the file. In times of SSDs no issue but an HDD will take some time. No imagine downloading to slow USB2.0 drive or even external NAS network drive with slow connection.
To be honest I don't care about the difference in filesize/disk size. It will go away in the moment when you copy/not moving to another place. It also only affects downloaded files. Extraction dont use sparse files. Removing the sparse flag after download will not *fix* the discrepancies. Only a copy of the file can do it. But I also plan to remove the flag after file is finished because some tools don't *like* the flag enabled.
Thanks for the response. Now I fully understand. Why does the system write zeros by default, instead of the sparse method? Is there a benefit to that?

Quote:
Originally Posted by Jiaz View Post
It will go away in the moment when you copy
Wow, that worked. Is there a way you could please add an advanced settings option, to automatically copy sparse files and delete the original, so we can still download using sparse files, without the size and size on disc discrepancy? Example:


File_Name.mkv (Sparse file.)

to,

File_Name.mkv (Sparse file.)

File_Name_2.mkv (No sparse.)

to,

File_Name_2.mkv (No sparse.)

to,

File_Name.mkv (No sparse.)
__________________
I don't fight the system, but I do play fighting games on it. PS4 to be exact. On second thought, sometimes it won't work right and I have to hit it a couple of times before it's fixed. So technically, I do fight the system.
Reply With Quote
  #14  
Old 07.10.2016, 17:59
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 65,456
Default

SparseFlag must be set during creation of the file. Not every filesystem supports sparse files and it can cause *higher* fragmentation because of the missing zeros inbetween different file positions.
I've added *copy* as an addtional method to this ticket
__________________
JD-Dev & Server-Admin
Reply With Quote
  #15  
Old 07.10.2016, 18:19
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 65,456
Default

With next core update- Settings-Advanced Settings-Youtube.ratebypassenabled enable it
__________________
JD-Dev & Server-Admin
Reply With Quote
  #16  
Old 10.10.2016, 14:14
madcowpig's Avatar
madcowpig madcowpig is offline
Storm
 
Join Date: Aug 2016
Posts: 210
Default

Quote:
Originally Posted by Jiaz View Post
With next core update- Settings-Advanced Settings-Youtube.ratebypassenabled enable it
Jiaz, you m.o.t.h.e.r f****** liar!!! What happened to,

Quote:
Originally Posted by Jiaz View Post
I'm working one issue , then the next one

Youtube: will take look at it after mega
I can't trust you anymore.

HAHAHA Thank you, it works great. YT is downloading fast as hell with the dash files. What did you change?

Quote:
Originally Posted by Jiaz View Post
I've added hash support in mega with next plugin update.
It says,



Same file on other hoster works fine.



Size on disk for MEGA was slightly smaller.

MEGA on the left.



Also, another difference in the properties was,

MEGA on the left.



I don't know if that means anything, but I thought I'd mention it anyways, just in case.

Last thing, when I go into the context menu, "Other", "Start CRC task!", it won't start with MEGA files, but it does start with other hosters.

Here is the same file with different hosters, for testing purposes.
__________________
I don't fight the system, but I do play fighting games on it. PS4 to be exact. On second thought, sometimes it won't work right and I have to hit it a couple of times before it's fixed. So technically, I do fight the system.

Last edited by madcowpig; 13.10.2016 at 19:30. Reason: The forum changes the word m.o.t.h.e.r, to woman.
Reply With Quote
  #17  
Old 10.10.2016, 14:31
madcowpig's Avatar
madcowpig madcowpig is offline
Storm
 
Join Date: Aug 2016
Posts: 210
Default

Just tested the MEGA renaming issue and there's something wrong. When I download a file named, 1.part14.rar, start download, then rename it to, 2.part14.rar, when it finishes downloading, then decrypting, it gets automatically renamed back to, 1.part14.rar.

Edit: Also, after it renames it back to, 1.part14.rar, it says, "0 B" for file size in JD.
__________________
I don't fight the system, but I do play fighting games on it. PS4 to be exact. On second thought, sometimes it won't work right and I have to hit it a couple of times before it's fixed. So technically, I do fight the system.

Last edited by madcowpig; 10.10.2016 at 14:36.
Reply With Quote
  #18  
Old 10.10.2016, 15:36
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 65,456
Default

Fixed "Start CRC Task" issue. Please note that it will only work with fresh added files. Wait for next plugin update
__________________
JD-Dev & Server-Admin
Reply With Quote
  #19  
Old 10.10.2016, 15:40
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 65,456
Default

Quote:
Originally Posted by madcowpig View Post
Just tested the MEGA renaming issue and there's something wrong. When I download a file named, 1.part14.rar, start download, then rename it to, 2.part14.rar, when it finishes downloading, then decrypting, it gets automatically renamed back to, 1.part14.rar.

Edit: Also, after it renames it back to, 1.part14.rar, it says, "0 B" for file size in JD.
Yes, that's what I meant by

It no longer causes "Host problem" error but still does not work correct. A proper fix will take longer time
__________________
JD-Dev & Server-Admin
Reply With Quote
  #20  
Old 10.10.2016, 15:49
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 65,456
Default

I've tested your example file and worked fine for me every link. JDownloader shows CRC okay. So you should compare those files locally (eg via hexeditor, I'm sure they are not identical)
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 15:45.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.