[See Bugtracker] mega.nz and youtube.com questions and possible requests

Welcome back, Jiaz. Hope you had a happy, German Unity Day. I had a couple of questions about these two plugins, that I was told, you would know the answers to.



mega.nz

1. SHA-1 checksums don't work. I asked Tony about this and he said, there was no hash check in the plugin. I know that MEGA uses AES 128-bit when decrypting, so my question is, does this count as the hash check? If so, is AES 128-bit as good as SHA-1? If not, I would like to request hash check to MEGA plugin.
   
   
2. Also there is a tiny bug when renaming. If you add a file to download list and rename it while it's still downloading, when the file finishes downloading it says, "Host problem?" and you have to wait ten minutes for decrypting to start. A simple work around is, if you disable and enable it itwill start decrypting without waiting ten minutes.

Spoiler:

youtube.com

1. Does the YouTube plugin losslesly remux the two dash files? Because when it's doing the process in JD it says, "converting/demuxing" and converting usually doesn't mean lossless. Also, does this process cause any audio sync issues, by milliseconds?
   
   
2. When I download using FlashGot, I get different size and size on disk, than JD.
   
   Here's the results downloading a video using these, "(720p_30fps_H264-192kbit_AAC)" settings. Size on disk for JD is bigger.
   
   JD on the left and FlashGot on the right.
   
   
   
   When doing the same test with an .m4a file, it gets worse. The file size as well is different.
   
   Here's the results downloading an audio file using these, (128kbit_AAC) settings.
   
   JD on the left and FlashGot on the right.
   
   
   
   
3. There's a speed issue when downloading 2 dash files for remux in JD. When downloading a 1080p .mp4 file and 125k .m4a file, in Mozilla Firefox, using FlashGot, I can download at 21 MB/s with 1 connection. In JD downloading the same file, which is these, "(1080p_30fps_H264-128kbit_AAC)" settings, I can only get about 1 MB/s using 1 connection, so with 20 connections I was just under the FlashGot speed, which only used 1 connection. The reason I say this issue is only for the 2 dash files for remux, is because downloading these, "(720p_30fps_H264-192kbit_AAC)" settings, which require no dash files or remuxing, I get 21 MB/s in FlashGot and JD with 1 connection each.
   
   
4. Finally, is there a way to get the YouTube plugin to put the real title of the video in the title properties? For example, in the new UFC video the title is, "UFC 204 Free Fight: Michael Bisping vs Luke Rockhold", but since windows doesn't allow ":" in file names, JD renames it to, "UFC 204 Free Fight_ Michael Bisping vs Luke Rockhold". It changes the ":" to a "_". I want a way to put the real/full video title in the title properties automatically, so it displays properly in programs. I circled in red, where I'm talking about.

Here's the link I used for testing, in all of the above, **External links are only visible to Support Staff****External links are only visible to Support Staff** .

[Jiaz]

As I'm now on my way home, I will respond tomorrow morning

It's morning now. *Sniffle, sniffle.* "Where are you, Jiaz!!!" *Cries uncontrollably.* "I'm scared and alone." *Hears twig snap and turns head swiftly.* "AHHHH!!! I think they're here." *Runs away, fast as hell.*

[Jiaz]

mega.nz:
1.) do you set the sha1 yourself or where does it come from? I could not find any hash information from their api. so I guess you set it manually and wonder why it is not checked at the end, right?
2.) will check/fix it


youtube:
1.) it is simple mux/demux, no transcoding
2.) Size on disk heavily depends on many filesystem factors.
2.1.) read more here https://board.jdownloader.org/showthread.php?t=70676
2.2.) Flashgot simply downloads the files but no mux/demux happening, as far as I know

3.) please provide example links, can't reproduce

4.) not supported yet

Originally Posted by Jiaz

mega.nz:
1.) do you set the sha1 yourself or where does it come from? I could not find any hash information from their api. so I guess you set it manually and wonder why it is not checked at the end, right?

Yes, usually websites put multiple hosters, they almost always use Uploaded (SHA-1) and Rapidgator (MD5), and I use Adopt Checksums, to give it to the websites that have better free speeds.

Originally Posted by Jiaz

2.) will check/fix it

Thank you, very much.

Originally Posted by Jiaz

youtube:
1.) it is simple mux/demux, no transcoding

I assumed this, but just wanted to make sure, because JD was using the word, "converting". Let me ask you one more thing then, does JD automatically update FFmpeg? If not could you please add a feature to. Also is it for 32bit or 64bit system?

Originally Posted by Jiaz

2.) Size on disk heavily depends on many filesystem factors.

Okay.

Originally Posted by Jiaz

2.1.) read more here https://board.jdownloader.org/showthread.php?t=70676

Just read it. So what does it mean to use a sparse file and sparse flag? What's the benefit of JD using it? When you add the advanced option to disable it, if I do, will there be any negative effects?

Originally Posted by Jiaz

2.2.) Flashgot simply downloads the files but no mux/demux happening, as far as I know

Correct, but YouTube does offer some files that are already muxed,. For example the JD YouTube plugin setting, "(720p_30fps_H264-192kbit_AAC)" is already muxed and does not need FFmpeg, so when I download in Flashgot, the video and audio is already together. For the separate audio and video DASH files, I use MKVToolnix to mux them together after downloading with FlashGot.

Originally Posted by Jiaz

3.) please provide example links, can't reproduce

I gave you one in my first post. Here it is again, **External links are only visible to Support Staff****External links are only visible to Support Staff** .

Like I said earlier, it only happens when I'm downloading the DASH files, that use FFmpeg to mux. When I download the 720p .mp4 that YouTube already has muxed, "(720p_30fps_H264-192kbit_AAC)" it downloads fast as hell. Here are example pictures,


Here's where the problem happens, it's downloading two separate DASH files that are going to be muxed, at 1 MB/s, with 1 connection,

(1080p_30fps_H264-128kbit_AAC)

Spoiler:

Here's what the file looks like in my Downloads folder, while downloading, "7_dKY3_vleU_MP4_H264_1080P_30FPS_AAC_128KBIT_DASH_58fc4d43e4353c3f81e1691f3107a4d4.dashVideo.pa rt".

It took 8 minutes to finish downloading, which is ridiculous for a, 318 MB file.

Spoiler:

Here's where it works fine, it's downloading 1 file, at 22.5 MB/s, with 1 connection,

(720p_30fps_H264-192kbit_AAC)

Spoiler:

Here's what the file looks like in my Downloads folder, while downloading, "UFC 204 Free Fight_ Michael Bisping vs Luke Rockhold (720p_30fps_H264-192kbit_AAC).mp4.part".

Just to leave no confusion, here's the log for, "(1080p_30fps_H264-128kbit_AAC)", where the problem happens.

05.10.16 10.49.50 <--> 05.10.16 11.00.50 jdlog://1227581887641/

Originally Posted by Jiaz

4.) not supported yet

Thank you, very, very much.

[Jiaz]

I've added hash support in mega with next plugin update. I'm sorry but I was not able to reproduce the second issue you've described (rename it while it's still downloading, when the file finishes downloading it says, "Host problem?"..) Can you please provide a logfile for that

[Jiaz]

I've changed plugin to avoid that "Host Problem" but without a way to reproduce or a logfile I'm really trampling in the dark :(

Originally Posted by Jiaz

I've added hash support in mega with next plugin update. I'm sorry but I was not able to reproduce the second issue you've described (rename it while it's still downloading, when the file finishes downloading it says, "Host problem?"..) Can you please provide a logfile for that

Sorry, I didn't provide a log file before, because I thought it would be easily reproduced. Here you go,

07.10.16 07.28.23 <--> 07.10.16 07.30.28 jdlog://1398581887641/

Also, if you could please answer these questions I asked before, when you have time.

Spoiler:

Originally Posted by madcowpig

Just read it. So what does it mean to use a sparse file and sparse flag? What's the benefit of JD using it? When you add the advanced option to disable it, if I do, will there be any negative effects?

Last thing, you asked me to provide example links, for the YT download speed issue and I gave you one, plus more info and a log, but you gave no response. Is this because you're busy or just didn't see it? If you're busy, then I understand.

Spoiler:

Originally Posted by madcowpig

I gave you one in my first post. Here it is again, **External links are only visible to Support Staff****External links are only visible to Support Staff** .

Like I said earlier, it only happens when I'm downloading the DASH files, that use FFmpeg to mux. When I download the 720p .mp4 that YouTube already has muxed, "(720p_30fps_H264-192kbit_AAC)" it downloads fast as hell. Here are example pictures,


Here's where the problem happens, it's downloading two separate DASH files that are going to be muxed, at 1 MB/s, with 1 connection,

(1080p_30fps_H264-128kbit_AAC)

Spoiler:

Here's what the file looks like in my Downloads folder, while downloading, "7_dKY3_vleU_MP4_H264_1080P_30FPS_AAC_128KBIT_DASH_58fc4d43e4353c3f81e1691f3107a4d4.dashVideo.pa rt".

It took 8 minutes to finish downloading, which is ridiculous for a, 318 MB file.

Spoiler:

Here's where it works fine, it's downloading 1 file, at 22.5 MB/s, with 1 connection,

(720p_30fps_H264-192kbit_AAC)

Spoiler:

Here's what the file looks like in my Downloads folder, while downloading, "UFC 204 Free Fight_ Michael Bisping vs Luke Rockhold (720p_30fps_H264-192kbit_AAC).mp4.part".

Just to leave no confusion, here's the log for, "(1080p_30fps_H264-128kbit_AAC)", where the problem happens.

05.10.16 10.49.50 <--> 05.10.16 11.00.50 jdlog://1227581887641/

[Jiaz]

I'm working one issue , then the next one
Thanks for the logs.
Sparse File: Normally the OS fills empty *gaps* with 0. For example you download a 1GB file with 2 connections, then one connection starts at beginning and the other one in the middle of the file. That causes the OS to fill the *gap* between beginning and middle with 0 so it can continue to write data to it. That can take a moment (longer time). In that time the tcp/http connection could timeout and cause download abort and need to retry/resume. With sparse flag enabled, the OS does NOT fill the gap. for sparse files only *used* regions are written to disk. A sparse file can easily be several TB/PB in size yet only use some bytes on the disk. Wikipedia and google also provides further information about this. That's why we make use of it to avoid timeouts with large files and max connections > 1.

Youtube: will take look at it after mega

[Jiaz]

Rename issue should happen *less common* with next plugin update. But it requires more complex proper bugfixing, see

Originally Posted by Jiaz

Thanks for the logs.

You're welcome.

Originally Posted by Jiaz

I'm working one issue , then the next one

Youtube: will take look at it after mega

Okay, that makes sense. I can finally sleep comfortably at night. Thank you.

Originally Posted by Jiaz

Sparse File: Normally the OS fills empty *gaps* with 0. For example you download a 1GB file with 2 connections, then one connection starts at beginning and the other one in the middle of the file. That causes the OS to fill the *gap* between beginning and middle with 0 so it can continue to write data to it. That can take a moment (longer time). In that time the tcp/http connection could timeout and cause download abort and need to retry/resume. With sparse flag enabled, the OS does NOT fill the gap. for sparse files only *used* regions are written to disk. A sparse file can easily be several TB/PB in size yet only use some bytes on the disk. Wikipedia and google also provides further information about this. That's why we make use of it to avoid timeouts with large files and max connections > 1.

Wow, this explanation was perfect. If you make an advanced option to turn off sparse files, this means I'll probably get timeouts with big files and multiple connections, which I don't want. Or am I misunderstanding, because after reading the ticket, it says, "Controlling: remove sparse flag when finshed". Does this mean that JD will download with sparse files to avoid timeouts, but when downloads finish it will remove sparse flag, to fix size and size on disk discrepancies?

[Jiaz]

I will first add an option to disable use of sparse files completely. Yes, that *can* cause issues in specific situations. For example download a single 10gb file with 2 connections, so the second connection will have to wait before the OS finished filling up 5GB of zeros in the file. In times of SSDs no issue but an HDD will take some time. No imagine downloading to slow USB2.0 drive or even external NAS network drive with slow connection.
To be honest I don't care about the difference in filesize/disk size. It will go away in the moment when you copy/not moving to another place. It also only affects downloaded files. Extraction dont use sparse files. Removing the sparse flag after download will not *fix* the discrepancies. Only a copy of the file can do it. But I also plan to remove the flag after file is finished because some tools don't *like* the flag enabled.

Originally Posted by Jiaz

I will first add an option to disable use of sparse files completely. Yes, that *can* cause issues in specific situations. For example download a single 10gb file with 2 connections, so the second connection will have to wait before the OS finished filling up 5GB of zeros in the file. In times of SSDs no issue but an HDD will take some time. No imagine downloading to slow USB2.0 drive or even external NAS network drive with slow connection.
To be honest I don't care about the difference in filesize/disk size. It will go away in the moment when you copy/not moving to another place. It also only affects downloaded files. Extraction dont use sparse files. Removing the sparse flag after download will not *fix* the discrepancies. Only a copy of the file can do it. But I also plan to remove the flag after file is finished because some tools don't *like* the flag enabled.

Thanks for the response. Now I fully understand. Why does the system write zeros by default, instead of the sparse method? Is there a benefit to that?

Originally Posted by Jiaz

It will go away in the moment when you copy

Wow, that worked. Is there a way you could please add an advanced settings option, to automatically copy sparse files and delete the original, so we can still download using sparse files, without the size and size on disc discrepancy? Example:


File_Name.mkv (Sparse file.)

to,

File_Name.mkv (Sparse file.)

File_Name_2.mkv (No sparse.)

to,

File_Name_2.mkv (No sparse.)

to,

File_Name.mkv (No sparse.)

[Jiaz]

SparseFlag must be set during creation of the file. Not every filesystem supports sparse files and it can cause *higher* fragmentation because of the missing zeros inbetween different file positions.
I've added *copy* as an addtional method to this ticket

[Jiaz]

With next core update- Settings-Advanced Settings-Youtube.ratebypassenabled enable it

Originally Posted by Jiaz

With next core update- Settings-Advanced Settings-Youtube.ratebypassenabled enable it

Jiaz, you m.o.t.h.e.r f****** liar!!! What happened to,

Originally Posted by Jiaz

I'm working one issue , then the next one

Youtube: will take look at it after mega

I can't trust you anymore.

HAHAHA Thank you, it works great. YT is downloading fast as hell with the dash files. What did you change?

Originally Posted by Jiaz

I've added hash support in mega with next plugin update.

It says,



Same file on other hoster works fine.



Size on disk for MEGA was slightly smaller.

MEGA on the left.



Also, another difference in the properties was,

MEGA on the left.



I don't know if that means anything, but I thought I'd mention it anyways, just in case.

Last thing, when I go into the context menu, "Other", "Start CRC task!", it won't start with MEGA files, but it does start with other hosters.

Here is the same file with different hosters, for testing purposes.

Just tested the MEGA renaming issue and there's something wrong. When I download a file named, 1.part14.rar, start download, then rename it to, 2.part14.rar, when it finishes downloading, then decrypting, it gets automatically renamed back to, 1.part14.rar.

Edit: Also, after it renames it back to, 1.part14.rar, it says, "0 B" for file size in JD.

[Jiaz]

Fixed "Start CRC Task" issue. Please note that it will only work with fresh added files. Wait for next plugin update

[Jiaz]

Originally Posted by madcowpig

Just tested the MEGA renaming issue and there's something wrong. When I download a file named, 1.part14.rar, start download, then rename it to, 2.part14.rar, when it finishes downloading, then decrypting, it gets automatically renamed back to, 1.part14.rar.

Edit: Also, after it renames it back to, 1.part14.rar, it says, "0 B" for file size in JD.

Yes, that's what I meant by

It no longer causes "Host problem" error but still does not work correct. A proper fix will take longer time

[Jiaz]

I've tested your example file and worked fine for me every link. JDownloader shows CRC okay. So you should compare those files locally (eg via hexeditor, I'm sure they are not identical)

[Jiaz]

File on disk discrepancies comes from sparse and the Attributes signals Sparse flag (P attrib is for a SPARSE file)

Originally Posted by Jiaz

Fixed "Start CRC Task" issue. Please note that it will only work with fresh added files. Wait for next plugin update

Thank you.

Originally Posted by Jiaz

Yes, that's what I meant by

It no longer causes "Host problem" error but still does not work correct. A proper fix will take longer time

Okay, patiently waiting.

Originally Posted by Jiaz

I've tested your example file and worked fine for me every link. JDownloader shows CRC okay.

Same for me, including MEGA, when I start hash check AFTER downloads finish.

Originally Posted by Jiaz

So you should compare those files locally (eg via hexeditor, I'm sure they are not identical)

You're wrong on this one, they're definitely identical. Here you go,



After further testing, I know what the problem is. JD starts hash check before decryption happens, when you set JD to automatically start hash checks. It starts hash check on this file, "1.part14.rar.encrypted.part".

[Jiaz]

Found the issue with hashcheck. Caused by old vs new download core. Will be fixed with next update

Originally Posted by Jiaz

File on disk discrepancies comes from sparse and the Attributes signals Sparse flag (P attrib is for a SPARSE file)

I'm a fool, now I gotta Google all attribute letters meanings.

Originally Posted by Jiaz

Found the issue with hashcheck. Caused by old vs new download core. Will be fixed with next update

Cool, thank you.

Last thing, why did you edit my previous post? Did I break a forum rule? I only put a single .rar file of fourteen, so it couldn't even be extracted. Is there a different way I'm supposed to give you guys testing links?

[Jiaz]

I always remove links if they are no longer needed. That was the only reason

Wait for next core update and then mega hash check should work fine

Originally Posted by Jiaz

I always remove links if they are no longer needed. That was the only reason

Phew, I was worried for a second there.

Originally Posted by Jiaz

Wait for next core update and then mega hash check should work fine

Cool, thank you.

[Jiaz]

Just to let you know, still working on next update (which will include the mega hashcheck fix)

Originally Posted by Jiaz

Just to let you know, still working on next update (which will include the mega hashcheck fix)

Hey man, thank you very much. It always feels good to know what's going on, currently.

[Jiaz]

Update is now live

Originally Posted by Jiaz

Update is now live

Thanks. After testing there's still a minor issue. In the MEGA plugin there is a setting for tmp decryption file, which I use. When having this setting enabled, CRC still fails, because JD runs CRC on this file,

1.mkv.decrypted

Which automatically gets renamed and turns into this file,

1.mkv

If I manually run CRC on the final file, JD says CRC OK. In the next update, can you please create a few seconds of delay for CRC to start, so JD runs CRC on the final file.

[Jiaz]

I will have to test that. For me it worked in all cases, even with .decrypted.
Did you update and restart JDownloader? Just to be sure ?

Originally Posted by Jiaz

I will have to test that. For me it worked in all cases, even with .decrypted.
Did you update and restart JDownloader? Just to be sure ?

Yes, I always restart. I even restarted my computer. Keep in mind, I'm stilling using the previously shared links, for testing.

Spoiler:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

**External links are only visible to Support Staff****External links are only visible to Support Staff**

Here's the log where it fails and then I manually start CRC and it passes.

13.10.16 13.14.34 <--> 13.10.16 13.17.42 jdlog://8684681887641/

[Jiaz]

Found and fixed, Wait for next plugin update. Sorry that it took me several tries :(

Originally Posted by Jiaz

Found and fixed, Wait for next plugin update.

Thank you, very much.

Originally Posted by Jiaz

Sorry that it took me several tries :(

No man, no apology needed. This is part of the game. I don't mind waiting a year, as long as there's progress happening somewhere, no matter how slow.

Originally Posted by Jiaz

Found and fixed, Wait for next plugin update.

After testing, all hash checks work fine now. Thank you.

Now, the prefix of this thread should be changed to, "[See Bugtracker]", since I'm just waiting on these two tickets to get done.

[Jiaz]

thanks for the feedback!
next one will be metadata for yt links

Originally Posted by Jiaz

thanks for the feedback!
next one will be metadata for yt links

Yes!!! After that, I can finally bulk download from YouTube, all in one. Download, mux, correct file names and correct names in metadata. I used to do this all manually, for hundreds of links. Super headaches.

[Jiaz]

Youtube Meta + Twitch Meta will come with next update tomorrow

Originally Posted by Jiaz

Youtube Meta + Twitch Meta will come with next update tomorrow

I swear, right when I was about to ask for the status, then BOOM, you say it's ready. I've started my countdown timer. Thank you, very much.

[Jiaz]

Update is live