JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 18.11.2022, 02:28
rdlnwdj rdlnwdj is offline
Junior Loader
 
Join Date: Nov 2022
Posts: 13
Post Video resolution and audio codec missing in filename (generic HLS)

I use Ctrl+V to batch analyze generic HLS streams, the filenames used to be like this "index (2160p_ac3 375kbits).mp4" or this "index (1080p_aac).mp4", and thus would be sorted into different packages.

This used to work when I tried it back on 23 September, but no longer works as of the newest update, now the filename is only "index.mp4".

I already updated ffmpeg and ffprobe before September and I did it again this time, so I guess it's irrelevant.

Last edited by rdlnwdj; 18.11.2022 at 02:30.
Reply With Quote
  #2  
Old 18.11.2022, 11:45
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,044
Default

@rdlnwdj: can you please provide example links?
We've optimized the crawler plugin to be faster and thus the plugin only fetches those infos on actual download or linkcheck by default.
You can change it in Settings->Plugins->m3u8, set to slow
__________________
JD-Dev & Server-Admin
Reply With Quote
  #3  
Old 18.11.2022, 13:17
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 64,713
Default

@rdlnwdj
Sorry for the problems our changes may have caused.
While working on this thread we found multiple ways to improve the performance when adding m3u8 links.
We've tried to set reasonable default options for the newly available settings but we might have failed with this and/or there are still some bugs left.
Please provide example URLs so we can look into this.
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #4  
Old 18.11.2022, 15:33
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,044
Default

@rdlnwdj: Change Settings->Plugins->m3u8 with next plugin to AUTOMATIC_SUPERFAST

But would be great if you could provide example links for further testing
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 18.11.2022 at 15:37.
Reply With Quote
  #5  
Old 18.11.2022, 19:27
rdlnwdj rdlnwdj is offline
Junior Loader
 
Join Date: Nov 2022
Posts: 13
Default

Update

With the newest rev.47130 core update, some links are still not getting recognized(missing audio codec) as they did back in september, more example links are updated below.

-------------------------------------------------------------------------------

Guys, sorry for the late reply, issue is almost resolved with the rev.47086 core update rolled out today, thanks!

Just as a side note, I examined the output of "ffprobe.exe -loglevel 48 -show_format -show_streams -analyzeduration 15000000 -of json -i URL", and noticed that the output contains duration info, so I guess it would be really awesome if duration can be sorted on the LinkGrabber tab just like what we can do with file size.

Please see below for example links and further relevent info.

**External links are only visible to Support Staff****External links are only visible to Support Staff**

Last edited by rdlnwdj; 25.11.2022 at 02:46. Reason: update current situation
Reply With Quote
  #6  
Old 18.11.2022, 20:07
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,044
Default

@rdlnwdj: nothing to be sorry for. I wil take look at your pastebin as soon as I find time. You can also send mail to support@jdownloader.org so we can talk about it,but no need to!
__________________
JD-Dev & Server-Admin
Reply With Quote
  #7  
Old 22.11.2022, 15:00
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 64,713
Default

Quote:
Originally Posted by rdlnwdj View Post
it would be really awesome if duration can be sorted on the LinkGrabber tab just like what we can do with file size.
Internally we do already obtain the duration but there is no- and will be no "duration" column.
If you want to sort items by duration, you can do this using a custom EventScripter script.
You can access the duration via plugin property "duration_estimated_millis".
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #8  
Old 23.11.2022, 07:51
rdlnwdj rdlnwdj is offline
Junior Loader
 
Join Date: Nov 2022
Posts: 13
Default

if (name == "Sort Package(s) by 'Duration'") {
var Duration[] = lgSelection.duration_estimated_millis;

Wish I could do it as you suggested, but this is as far as I can get in terms of EventScripter, I added a buttom named "Sort Package(s) by 'Duration'" to the LinkGrabber tab, but I got stuck on EventScripter part, help is appreciated.
Reply With Quote
  #9  
Old 23.11.2022, 13:48
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 64,713
Default

Please post EventScripter related questions in our EventScripter thread.
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #10  
Old 23.11.2022, 17:21
rdlnwdj rdlnwdj is offline
Junior Loader
 
Join Date: Nov 2022
Posts: 13
Default

Hey, I just realized that EventScripter might not be suitable for this usecase. Displaying duration on the LinkGrabber tab is not achievable through EventScripter, even if one manage to sort items by duration using EventScripter, he would not know exactly how long a video is without manually checking it again.
Reply With Quote
  #11  
Old 23.11.2022, 17:30
rdlnwdj rdlnwdj is offline
Junior Loader
 
Join Date: Nov 2022
Posts: 13
Default

And I do understand your decision to not include duration on the LinkGrabber tab when it comes to general videos, but for HLS streams, the situation is a little different, extimated filesizes sometimes deviate too much from the actual ones(mostly because many HLS streams nowadays use scene based encoding, opening titles with complicated effects and fast moving/switching scenes tend to give ffprobe wrong estimation on average bitrates), but duration values are most likely accurate, so instead of displaying only estimated filesizes calculated from duration and estimated bitrates, so please reconsider the necessity of "duration" column for HLS streams.
Reply With Quote
  #12  
Old 23.11.2022, 21:14
rdlnwdj rdlnwdj is offline
Junior Loader
 
Join Date: Nov 2022
Posts: 13
Default

Update:
@mgpai helped me find a easier way to achive this, namely, by adding duration values to comment, I used a simple packagizer rule and it worked, duration is sorted like this "1656000 175000 1769000 193000 1962000", so I guess problem is not solved here:crying:

@pspzockerscene I checked your posts in this thread, and tried several m3u8 link properties in packagizer, "height, width, hlsBandwidth, duration_estimated_millis" did return values, while "hlsBandwidthAverage, bitrate, framerate, ffmpeg_codec_type, ffmpeg_codec_name, m3u8_codecs, m3u8_name" did not return any value.

Last edited by rdlnwdj; 23.11.2022 at 22:47. Reason: not fixed
Reply With Quote
  #13  
Old 23.11.2022, 22:29
rdlnwdj rdlnwdj is offline
Junior Loader
 
Join Date: Nov 2022
Posts: 13
Default

Update

With the newest rev.47128 core update, some links are still not getting recognized(missing audio codec) as they did back in september.

More example links are added in this post.
Reply With Quote
  #14  
Old 25.11.2022, 14:02
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 64,713
Default

Quote:
Originally Posted by rdlnwdj View Post
And I do understand your decision to not include duration on the LinkGrabber tab when it comes to general videos
That wasn't just a "no" in my answer.
I understand that this would be super nice and indeed it would probably be the best to be able to add custom columns and tell JD "use plugin property <duration>, put it in that column and maybe format it to data type <timeHumanReadable>" but at this moment that's just out of reach.

Quote:
Originally Posted by rdlnwdj View Post
but for HLS streams, the situation is a little different, extimated filesizes sometimes deviate too much from the actual ones(mostly because many HLS streams nowadays use scene based encoding, opening titles with complicated effects and fast moving/switching scenes tend to give ffprobe wrong estimation on average bitrates)
Any examples for that?
Are the estimated filesizes mostly too high or too low?
Are they much better when doing another linkcheck afterwards or adding links with "Crawl mode: Slow"?


Quote:
Originally Posted by rdlnwdj View Post
duration is sorted like this "1656000 175000 1769000 193000 1962000", so I guess problem is not solved here
Those internally stored duration values are given in milliseconds.

Quote:
Originally Posted by rdlnwdj View Post
I checked your posts in this thread, and tried several m3u8 link properties in packagizer, "height, width, hlsBandwidth, duration_estimated_millis" did return values, while "hlsBandwidthAverage, bitrate, framerate, ffmpeg_codec_type, ffmpeg_codec_name, m3u8_codecs, m3u8_name" did not return any value.
Well that again depends on the "Crawl Mode" setting (to keep it simple).
When single-checking links, JD will obtain much more information by doing the FFprobe step.
In "Fast" or "Superfast crawl mode(s)", JDownloader will only be able to obtain the information given inside "master.m3u8" (hls list containing multiple streams) let's call it the "master playlist" and that information can vary.

The problem is that JD will only go through the Packagizer rules one time so when fast-adding links some information might be missing at that point as single online-check will be skipped. That information will be available later on downloadstart or when manually single-linkchecking such files.
Here are your options:
If you want to use the Packagizer and don't care about the speed:
- Set m3u8 Crawl Mode to "Slow" and that will work just fine
If you do care about speed + filenames:
- Use an EventScripter script instead of a Packagizer rule (much more complex!) so your filenames can be auto-altered at a later stage when all of those properties become available

Quote:
Originally Posted by rdlnwdj View Post
With the newest rev.47128 core update, some links are still not getting recognized(missing audio codec) as they did back in september.
Jiaz will look into this once he finds the time.
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #15  
Old 26.11.2022, 07:05
rdlnwdj rdlnwdj is offline
Junior Loader
 
Join Date: Nov 2022
Posts: 13
Default

Quote:
Originally Posted by pspzockerscene View Post
Any examples for that?
Yes, the example below is actually 17.5GiB in size, while detected as 21.26GiB.
**External links are only visible to Support Staff**ffprobe -loglevel 48 -show_format -show_streams -analyzeduration 15000000 -of json -i "**External links are only visible to Support Staff** logs as well as mediainfo of downloaded file

Quote:
Originally Posted by pspzockerscene View Post
Are the estimated filesizes mostly too high or too low?
With HLS streams, extimated filesizes are mostly higher, and depending on complexity of beginning scenes, slightly or much higher.

Quote:
Originally Posted by pspzockerscene View Post
Are they much better when doing another linkcheck afterwards or adding links with "Crawl mode: Slow"?
No, my observations are based on slow mode, so my thought is, there is currently not much to do to improve this, increasing analyzeduration may help but is not needed, the increased time costing outrun the benifit.

Quote:
Originally Posted by pspzockerscene View Post
The problem is that JD will only go through the Packagizer rules one time so when fast-adding links some information might be missing at that point as single online-check will be skipped. That information will be available later on downloadstart or when manually single-linkchecking such files.
Here are your options:
If you want to use the Packagizer and don't care about the speed:
- Set m3u8 Crawl Mode to "Slow" and that will work just fine
Unfortunately, "Slow" mode does not work here, difference between super fast, fast and slow mode is that super fast does not return resolution, while fast and slow mode are basically the same behavior.
Quote:
Originally Posted by rdlnwdj View Post
"height, width, hlsBandwidth, duration_estimated_millis" did return values, while "hlsBandwidthAverage, bitrate, framerate, ffmpeg_codec_type, ffmpeg_codec_name, m3u8_codecs, m3u8_name" did not return any value.
Quote:
Originally Posted by pspzockerscene View Post
If you do care about speed + filenames:
- Use an EventScripter script instead of a Packagizer rule (much more complex!) so your filenames can be auto-altered at a later stage when all of those properties become available
Writing java script is beyond my grasp, so I'm still waiting for script master mgpai's reponse., see post below, duration sorting problem is solved.

Quote:
Originally Posted by pspzockerscene View Post
Jiaz will look into this once he finds the time.
Got it. Meanwhile I'll keep testing on new releases to see if problem gets solved and would update here asap.

Last edited by rdlnwdj; 30.11.2022 at 10:42.
Reply With Quote
  #16  
Old 30.11.2022, 10:39
rdlnwdj rdlnwdj is offline
Junior Loader
 
Join Date: Nov 2022
Posts: 13
Default

For anyone who would like to sort duration of HLS streams in linkgrabber tab, , first add a packagizer rule to add <jd:prop:duration_estimated_millis> to comments, then create your custom 'sort by duration' buttom and try the following EventScripter code by our script master mgpai.

Code:
/*
    sort by duration
    trigger: Toolbar button pressed
*/


if (name == "sort by duration") {
    getAllCrawledPackages().forEach(function(package) {
        package.downloadLinks.filter(function(link) {
            return link.comment.match(/^[\d]+$/);
        }).sort(function(a, b) {
            return a.comment - b.comment;
        }).reverse().forEach(function(link) {
            callAPI("linkgrabberv2", "moveLinks", [link.UUID], -1, package.UUID);
        })
    })
}
Reply With Quote
  #17  
Old 05.12.2022, 21:32
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 64,713
Default

Quote:
Originally Posted by rdlnwdj View Post
Yes, the example below is actually 17.5GiB in size, while detected as 21.26GiB.
What's your suggestion to improve this?

Quote:
Originally Posted by rdlnwdj View Post
ffprobe -loglevel 48...
That hyperlink to a pastebin appears to be offline.

Quote:
Originally Posted by rdlnwdj View Post
With HLS streams, extimated filesizes are mostly higher, and depending on complexity of beginning scenes, slightly or much higher.
Higher is better than lower if you ask me because JD will then still stop if you're running low on free space.

Quote:
Originally Posted by rdlnwdj View Post
No, my observations are based on slow mode, so my thought is, there is currently not much to do to improve this, increasing analyzeduration may help but is not needed, the increased time costing outrun the benifit.
Yap most likely not.
Maybe we could add some "factor" setting to let users have an influence on this?
But I still wonder why such a setting should be useful for anyone...

Quote:
Originally Posted by rdlnwdj View Post
Unfortunately, "Slow" mode does not work here, difference between super fast, fast and slow mode is that super fast does not return resolution, while fast and slow mode are basically the same behavior.
* it does not return the resolution right away.
The resolution will be available once such items get "fully checked" once in linkgrabber/on-downloadstart

Quote:
Originally Posted by rdlnwdj View Post
For anyone who would like to sort duration of HLS streams in linkgrabber tab...
Nice one!
__________________
JD Supporter, Plugin Dev. & Community Manager
JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
That's true James
Quote:
Originally Posted by James
Die Leute verstehen einfach nicht dass nur weil man mit einer Waffe auch auf Menschen schießen kann dass ein Schützenver​ein kein Ort für Amoklaufide​en ist
Reply With Quote
  #18  
Old 06.12.2022, 20:10
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,044
Default

Quote:
Originally Posted by pspzockerscene View Post
Higher is better than lower if you ask me because JD will then still stop if you're running low on free space.
I agree with this. Better a higher estimation and enough disk space than lower estimation and user wonders why it takes longer and maybe not enough disk space available
__________________
JD-Dev & Server-Admin
Reply With Quote
  #19  
Old 06.12.2022, 20:18
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,044
Default

Quote:
Originally Posted by rdlnwdj View Post
Unfortunately, "Slow" mode does not work here, difference between super fast, fast and slow mode is that super fast does not return resolution, while fast and slow mode are basically the same behavior.
That's why I added two new AUTO Modes, see https://board.jdownloader.org/showpo...53&postcount=4. Those will check superfast/fast only if resolution is available.
SuperFast = do not trust estimated file size
Fast = if (average)bandwidth is available, estimate file size
Slow = use ffprobe to obtain information
Auto-SuperFast -> SuperFast only if video + resolution or no video
Auto-Fast -> Fast only if video + resolution or no video

Quote:
Originally Posted by rdlnwdj View Post
so I guess it would be really awesome if duration can be sorted on the LinkGrabber tab just like what we can do with file size.
I'm sorry but I don't see us adding native column with such meta information. At least not in near future, because this information is not available in many cases. Your *workaround* via Eventscripter and accessing the properties is quiet nice!
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 06.12.2022 at 20:24.
Reply With Quote
  #20  
Old 10.12.2022, 02:38
rdlnwdj rdlnwdj is offline
Junior Loader
 
Join Date: Nov 2022
Posts: 13
Default

Quote:
Originally Posted by pspzockerscene View Post
Higher is better than lower if you ask me because JD will then still stop if you're running low on free space.
Well, this does make point:P, I was focusing on one side of the coin and ignored the other side.

Quote:
Originally Posted by pspzockerscene View Post
it does not return the resolution right away.
The resolution will be available once such items get "fully checked" once in linkgrabber/on-downloadstart
Actually, resolution detection is already working with slow, fast and auto-fast mode starting with rev.47086 core update, which according to Jiaz is the expected behavior;)

Quote:
Originally Posted by Jiaz View Post
SuperFast = do not trust estimated file size
Fast = if (average)bandwidth is available, estimate file size
Slow = use ffprobe to obtain information
Auto-SuperFast -> SuperFast only if video + resolution or no video
Auto-Fast -> Fast only if video + resolution or no video
The current situation is, fast and super-fast mode work as they should, they have done quite the job on speeding up the linkcrawing process, but for some links slow mode is missing info(no audio codec in filename) compared to the old code back in september, I posted some example links to help troubleshooting, but at least according to mgpai, these links were not accessible, if that's also the case for you, would me posting ffprobe logs of certain links help?

Last edited by rdlnwdj; 10.12.2022 at 02:41.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 18:18.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2023, Jelsoft Enterprises Ltd.