JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 16.05.2018, 03:19
plip plip is offline
Vacuum Cleaner
 
Join Date: Jun 2016
Posts: 16
Default Did JD2 always update Bytes Loaded in 1/4th mb increments (Twitch), or is that new?

I don't remember seeing this before the JD2 update today. Since then I've noticed Bytes Loaded only updates every 1/4th mb (250 kb) for Twitch streams. Only Twitch seems to be doing this. Other sites update Bytes Loaded in realtime from what I saw.

On a slow connection like I have that can make bytes loaded on Twitch stay still for upwards of thirty seconds at a time or more - giving the illusion that the download may have stalled even though Speed still says something is happening.

This is a problem in particular because I did actually have two Twitch downloads stall today - they got stuck at less than 50mb downloaded and stayed there indefinitely, despite Speed saying they were getting full download speed for several minutes, until I cancelled them.

Those two streams where strange - chunked was much smaller than 720p 30fps, so I was trying to get chunked. (Edit: I thought at first that the chunked streams themselves were glitchy, but now I'm not sure that was the problem, see edit at bottom).

This issue is that now when Bytes Loaded freezes for a while because of the 250kb increments (which it is doing all the time because of my slow connection), I keep worrying that the download may have stalled like the other two did.

Is there any setting to make Bytes Loaded update in realtime for Twitch too, instead of those increments?

EDIT: I think this is messing up the Twitch downloads. Twitch downloads I've tried since seeing this have been staying at low amounts downloaded for very long times.

I mean something started 2 hours ago shouldn't still be at 57mb downloaded. That's not right.

vod where that was happening:
**External links are only visible to Supporters****External links are only visible to Supporters**

in case comparison is helpful, vod where bytes loaded/progress was going up much faster even though it was downloading at the same speed as the other one:
**External links are only visible to Supporters****External links are only visible to Supporters**

Restarting JD2 and even the PC doesn't change anything.

As a test I tried downloading a really short vod which was only a few minutes long (and actually under 50mb).
**External links are only visible to Supporters****External links are only visible to Supporters**

It plays back very glitchy and messed up, so I tried redownloading it. Same result. I have no idea what is going on here.

log:
15.05.18 19.50.46 <--> 15.05.18 21.30.45 jdlog://6073854433151/

Maybe this last update messed up Twitch somehow (or ffmpeg?). Or maybe the update just went wrong somehow on my end.

I have to go to sleep so unfortunately I can't test anything longer.

I have an recent backup of my JD2 installation (entire folder) from ten days ago. Is there a way to go back to that installation for the time being, but with the Downloads list and Linkgrabber list from now?.

Unfortunately there is a number of Twitch vods which I want to get that are going to be deleted within a week. I don't have time to wait for a fix, which is why I want to revert.

But it would be much nicer if I could transfer my current download/linkgrabber list to the backup installation.

Last edited by plip; 16.05.2018 at 04:53.
Reply With Quote
  #2  
Old 16.05.2018, 12:00
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 57,425
Default

I've updated ffmpeg binary yesterday to newer version. That is the only change. So there is no need to revert or use older version as you can easily switch/use to different ffmpeg version in case it causes issues for you.
Maybe just the output parser for ffmpeg needs an update, I will check. As long as the download is happening and the file is growing, that are signs that download still is in progress, so I guess it's just a parsing/gui issue
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 16.05.2018 at 12:05.
Reply With Quote
  #3  
Old 16.05.2018, 12:04
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 57,425
Default

I will look into the logs and try to reproduce the issue but I guess it might be a bug in ffmpeg or something else specific to your setup.
You can either use the old binaries from your backup (they are in tools/windows/ffmpeg folder) and replace the updated ones or just change the full path in Settings-Advanced Settings-search for ffmpeg and ffprobe
The one that is used is from here, ffmpeg.zeranoe.com/builds/

I suggest not to downgrade/revert the whole JDownloader but just ffmpeg. Using outdated version will only increase support effort on our side and not help you in any way.

All links/settings/accounts are stored within the cfg folder. You can just copy/replace this one
__________________
JD-Dev & Server-Admin
Reply With Quote
  #4  
Old 16.05.2018, 12:07
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 57,425
Default

Also please update your JDownloader as I've applied a workaround for the strange network errors I've encountered on AmazonS3 downloads
__________________
JD-Dev & Server-Admin
Reply With Quote
  #5  
Old 16.05.2018, 13:13
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 57,425
Default

I can reproduce the issue. Working on it
__________________
JD-Dev & Server-Admin
Reply With Quote
  #6  
Old 16.05.2018, 13:45
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 57,425
Default

Core update just went live, the issue should be fixed with it. Newer ffmpeg has new parallel download support that was not yet supported and caused data corruption for hls downloads
__________________
JD-Dev & Server-Admin
Reply With Quote
  #7  
Old 16.05.2018, 16:44
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 57,425
Default

Please update your JDownloader and the addressed issues should be fixed. In case you still have issues, please let us know
__________________
JD-Dev & Server-Admin
Reply With Quote
  #8  
Old 16.05.2018, 17:02
plip plip is offline
Vacuum Cleaner
 
Join Date: Jun 2016
Posts: 16
Default

After testing a couple tiny vods (after getting the update from a few minutes ago) I can confirm Twitch downloads are working properly again. Thank you so much for the fast fix. It was a very nice start to the day to see that fixed already

Thanks for describing what was going wrong too. It's interesting to see what the cause was.

I do see the Bytes Loaded for Twitch downloads is still updating in 256kb increments, although it's now updating every second instead of every ten to thirty seconds.

I'm guessing that has something to due with the ffmpeg update too? (I doubt anything's wrong, I'm just trying to verify that 256kb incrementing in the gui isn't indicative of possibly some other bug somewhere. I'm not seeing any other unusual behavior).

Oh, one other thing I've noticed is that linkgrabber can't grab videos from a Twitch collection. And using "analyse text and links", all you get is a single jpg and no vods.

Example:
**External links are only visible to Supporters****External links are only visible to Supporters**

I hadn't tried to get vods from a Twitch collection before so I don't know why that's happening.


This part is just for amusement. When I got up this morning, that vod I mentioned which had only 50mb after two hours, that was still going. What's bizarre is that I also noticed JD2 was using massively more ram then I had ever seen it use before (close to 6GB!).

When I canceled that vod, JD2's ram usage went down by a huge amount. Somehow that single download was using at least 5GB of ram. What a strange side effect!

But like I said Twitch downloads are working good now so again, thanks.
Reply With Quote
  #9  
Old 16.05.2018, 17:34
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 57,425
Default

Thanks for the feedback! What exactly are collections? I've opened the link and I can't see a difference to normal video? You just want to add the video that shows when you open that link, don't you?

There are no changes within JDownloader itself except the required changes for updated ffmpeg.
JDownloader parses the output of ffmpeg and updates the progress/gui according to the output of ffmpeg.
So yes, very likely that there have been changes to ffmpeg that cause this 256kb steps.
I can do further checking if this important to you.

So all okay now?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #10  
Old 16.05.2018, 18:15
plip plip is offline
Vacuum Cleaner
 
Join Date: Jun 2016
Posts: 16
Default

As the progress updating isn't bugged it isn't that important, so don't worry about it unless you're curious about it yourself.

Twitch collections are like Youtube playlists, where there's multiple videos per collection.

You can see what I mean by going to someone's collections page.
**External links are only visible to Supporters****External links are only visible to Supporters**

Collections themselves are cumbersome on Twitch. Opening one up goes to the first video. While that's playing, if you mouse over the left side of the video then a list of all the other videos in the collections comes up.

You can't right click and copy to get the vod addresses though. You have to click and go to each individual vod for some reason to get each individual vod's address from the browser's address bar. So it's really clunky.

On the other hand, sometimes a streamer's most recent collection will show on top of their videos page, in which case you can easily right click and copy each vod address. That only happens with the most recent collection though.

Example:
**External links are only visible to Supporters****External links are only visible to Supporters**

It's pretty strange how they handle those so differently depending on where the collection is shown. But there's no simple way to get all the videos from a collection, unlike Youtube playlists, unless the collection happens to show up at the top of the streamer's videos page.

I don't know if there would be a way to automate it or not so that JD2 could grab everything in a collection just by checking the collection address.

Last edited by plip; 16.05.2018 at 18:20.
Reply With Quote
  #11  
Old 16.05.2018, 18:23
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 57,425
Default


Thanks for the explanation!
__________________
JD-Dev & Server-Admin
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 07:57.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2018, Jelsoft Enterprises Ltd.