[Solved] Not removing .part file leads to invalid files

[googlebear]

When I download (on a mac) from mega.nz (I haven't tested it on other sites) and the download finishes, the .part file is still in the folder. For example, when I download a video and look in the package folder, there are two files: video.mp4 and video.mp4.part. When I try to play the video file in VLC or other media players, it won't play at all. I even tried renaming the .part file to the .mp4 extension, but that did not help.

Here is the log file:

30.04.18 10.17.09 <--> 30.04.18 10.17.09 jdlog://2856754433151/

Thank you!

[googlebear]

Looking further, it looks like it doesn't happen for PDF files.

Also, it might be an issue with the premium host "**External links are only visible to Support Staff** even though it says it's downloading the files from mega.nz. When I turned off that premium host, the files downloaded correctly.

[Jiaz]

Issue is caused by linkifier
For example the error message is:
members.linkifier.com/home/dlError?ErrDesc=Your download could not be started. An error occured while trying to fetch your file from the hosting website. Please try again. If the problem persists please contact our email support! Error Code: The method or operation is not implemented., Prior: 1
Another time the download is incomplete because of server issues:
java.net.SocketTimeoutException: Read timed out
The file is incomplete -> VerifiedFilesize: 83240311|FileSize: 83240311|Loaded: 83060109
Therefore the file is still .part

JDownloader does not remove .part files until you reset the download or the download is finished.

[Jiaz]

You can setup mega not to use linkifier, see https://support.jdownloader.org/Know...-free-download
or try to set max connections to 1 and see if linkifier service works more stable then

[Keagel]

I have had that happen to me a few times.
I downloaded a big video file (7.5 GB) overnight from mega.co.nz through real-debrid, woke up to a folder that contained a part file (7.23 GB) and the normal file (7.57 GB - the exact size of what it should be, down to the byte). File is marked as finished in JDownloader. The video file can't be opened despite being apparently complete. When renaming the part file to the actual video extension, it opens but obviously is not complete.

When looking through the logs, it appears that the download timed out due to real-debrid's service going down, which resulted in JDownloader attempting to reconnect a few times. When it couldn't connect through real-debrid, it fell back to trying to download through the normal Mega plugin, and it worked. The download resumed from where it was left using real-debrid, and it completed 10 minutes later without errors. I assume this is the step that caused the problem as the part file created before this step works, while the completed one doesn't.

The logs:
**External links are only visible to Support Staff**mega.co.nz_jd.plugins.hoster.MegaConz_real-debrid.com.log.0
**External links are only visible to Support Staff**mega.co.nz_jd.plugins.hoster.MegaConz.log.0

I am now left with a part file that's only missing 300 MB that seemingly can't be resumed even though mega.co.nz supports resuming downloads. What can I do?

PS: Apologies if I'm not supposed to post here (since it is marked as resolved), the rules state we shouldn't create a new thread if one mentioning the issue already exists.

[pspzockerscene]

Hm your log looks fine.

Please keep in mind that using MEGA with multihosts is never a good idea although it should not lead to such issues.

Does this only happend for MEGA files for you or also when downloading from other sources?

-psp-

[Keagel]

Please keep in mind that using MEGA with multihosts is never a good idea although it should not lead to such issues.

Why is it not a good idea for MEGA specifically? Except for this part file problem, I've never had anything else happen.

Does this only happend for MEGA files for you or also when downloading from other sources?

I'm pretty sure this has only happened with MEGA as I almost never download from any other source.

Either way I may have found the cause on a hunch. What I assume happened: when it resumed, JDownloader successfully downloaded the last part of the file but the first chunk wasn't correctly appended to the result file.

In a way, the file with the full size is filled with zeros from bytes 0 to 7768344639 in my case, then the newly downloaded data is present - so it only contains bytes 7768344640 to 8126680994; whereas the .part file contains bytes 0 to 7768344639.

So what I did was skip the first 7768344640 bytes from the "corrupted" file using dd:

dd bs=4096 skip=7768344640 iflag=skip_bytes if=full_video.mkv of=last_part.mkv

Then merged the part file with the newly created file using cat:

cat full_video.mkv.part last_part.mkv > full_video_merged.mkv

And just like that, the entire file plays normally.
I'm not sure what you can do about that, but I assume this is a bug on JDownloader's side since the resume would have happened on the computer.

[pspzockerscene]

Originally Posted by Keagel

Why is it not a good idea for MEGA specifically? Except for this part file problem, I've never had anything else happen.

Multihosts also have to decrypt the file (serverside).
This has caused some issues in the past.

Interesting findings.

Maybe Jiaz will find time to look into this the next week.
We got no similar reports as far as I know.

-psp-

[Jiaz]

JDownloader downloads the encrypted file from mega and decrypts AFTER the download is complete vs Multihoster handling the decryption on the fly. Starting with mega and continue with Multihoster or vice versa will result in broken file because decryption of JDownloader won't work and file is mix of raw/encrypted content.

[Keagel]

Thank you for the additional information.

I think this is what happened:

- Multihost plugin downloads from byte 0 to byte 7768344639 (decrypted on the fly, file is readable even though it's incomplete).
- Multihost times out.
- Mega plugin takes over and resumes download from byte 7768344640 to byte 8126680994.
- After 7 minutes, download is over.
- Decryption starts for the ENTIRE file which means:

=> Bytes 0 to 7768344639 get decrypted twice (first from multihost, then JD), resulting in incorrect first 7768344640 bytes
=> Bytes 7768344640 to 8126680994 get decrypted once: these 358336354 bytes are correct.

Thus, the .part file that wasn't touched by JD after the download timed out is correct and readable, the final file isn't correct for the first 7768344640 bytes.
So, removing those first bytes and appending what remains to the part file produces the correct file.

This does seem like something that could be fixed by decrypting the chunks individually, then merging them instead of merging them first and decrypting the entire file after.

That would work both ways:

- Multihost downloads entirely, do nothing.
- Multihost downloads partly then resumes on Mega plugin: decrypt the second chunk, then merge.
- Mega downloads partly then resumes on Multihost: decrypt the first chunk, then merge.
- Mega downloads then times out, resumes and times out, then again. You can detect that the three chunks are from Mega and you can merge them all, then decrypt.

Sorry if I'm not very clear.

[Jiaz]

Originally Posted by Keagel

This does seem like something that could be fixed by decrypting the chunks individually, then merging them instead of merging them first and decrypting the entire file after.

Not that easy because you have to start/stop at the correct positions/chunks and this also requires additional meta information so JDownloader knows which chunks/ranges are complete/decrypted or pending/encryption state.

[Jiaz]

I could update mega plugin to auto detect this and either auto reset when resuming from different service or do not allow resume with mega/multihoster. That way JDownloader would not break the download but will only resume with same service. eg complete file with multihoster or with mega.

[raztoki]

hmm thought we already had prevention of cross download (h->mh|mh->h) once the download had already been started.

[Jiaz]

@raztoki: yes, but only when mega came first, not when different multihoster was first in round. I've updated plugin to handle this

[raztoki]

ok, i thought we originally blocked it like youtube complete block by mh, and then switched.

neway good that you fixed =]

[Keagel]

Thank you for the fix!

[Keagel]

Sorry for the bump, just wondering if the update is already rolled out? The topic still says "wait for plugin update".

[Jiaz]

Update is already live and mega should no longer continue downloads that began from other hosts.

[Keagel]

Perfect, thank you.