[rdlnwdj]

I use Ctrl+V to batch analyze generic HLS streams, the filenames used to be like this "index (2160p_ac3 375kbits).mp4" or this "index (1080p_aac).mp4", and thus would be sorted into different packages.

This used to work when I tried it back on 23 September, but no longer works as of the newest update, now the filename is only "index.mp4".

I already updated ffmpeg and ffprobe before September and I did it again this time, so I guess it's irrelevant.

[Jiaz]

@rdlnwdj: can you please provide example links?
We've optimized the crawler plugin to be faster and thus the plugin only fetches those infos on actual download or linkcheck by default.
You can change it in Settings->Plugins->m3u8, set to slow

[pspzockerscene]

@rdlnwdj
Sorry for the problems our changes may have caused.
While working on this thread we found multiple ways to improve the performance when adding m3u8 links.
We've tried to set reasonable default options for the newly available settings but we might have failed with this and/or there are still some bugs left.
Please provide example URLs so we can look into this.

[Jiaz]

@rdlnwdj: Change Settings->Plugins->m3u8 with next plugin to AUTOMATIC_SUPERFAST

But would be great if you could provide example links for further testing

[rdlnwdj]

Update

With the newest rev.47130 core update, some links are still not getting recognized(missing audio codec) as they did back in september, more example links are updated below.

-------------------------------------------------------------------------------

Guys, sorry for the late reply, issue is almost resolved with the rev.47086 core update rolled out today, thanks!

Just as a side note, I examined the output of "ffprobe.exe -loglevel 48 -show_format -show_streams -analyzeduration 15000000 -of json -i URL", and noticed that the output contains duration info, so I guess it would be really awesome if duration can be sorted on the LinkGrabber tab just like what we can do with file size.

Please see below for example links and further relevent info.

**External links are only visible to Support Staff****External links are only visible to Support Staff**

[Jiaz]

@rdlnwdj: nothing to be sorry for. I wil take look at your pastebin as soon as I find time. You can also send mail to support@jdownloader.org so we can talk about it,but no need to!

[pspzockerscene]

Originally Posted by rdlnwdj

it would be really awesome if duration can be sorted on the LinkGrabber tab just like what we can do with file size.

Internally we do already obtain the duration but there is no- and will be no "duration" column.
If you want to sort items by duration, you can do this using a custom EventScripter script.
You can access the duration via plugin property "duration_estimated_millis".

[rdlnwdj]

if (name == "Sort Package(s) by 'Duration'") {
var Duration[] = lgSelection.duration_estimated_millis;

Wish I could do it as you suggested, but this is as far as I can get in terms of EventScripter, I added a buttom named "Sort Package(s) by 'Duration'" to the LinkGrabber tab, but I got stuck on EventScripter part, help is appreciated.

[pspzockerscene]

Please post EventScripter related questions in our EventScripter thread.

[rdlnwdj]

Hey, I just realized that EventScripter might not be suitable for this usecase. Displaying duration on the LinkGrabber tab is not achievable through EventScripter, even if one manage to sort items by duration using EventScripter, he would not know exactly how long a video is without manually checking it again.

[rdlnwdj]

And I do understand your decision to not include duration on the LinkGrabber tab when it comes to general videos, but for HLS streams, the situation is a little different, extimated filesizes sometimes deviate too much from the actual ones(mostly because many HLS streams nowadays use scene based encoding, opening titles with complicated effects and fast moving/switching scenes tend to give ffprobe wrong estimation on average bitrates), but duration values are most likely accurate, so instead of displaying only estimated filesizes calculated from duration and estimated bitrates, so please reconsider the necessity of "duration" column for HLS streams.

[rdlnwdj]

Update:
@mgpai helped me find a easier way to achive this, namely, by adding duration values to comment, I used a simple packagizer rule and it worked, duration is sorted like this "1656000 175000 1769000 193000 1962000", so I guess problem is not solved here:crying:

@pspzockerscene I checked your posts in this thread, and tried several m3u8 link properties in packagizer, "height, width, hlsBandwidth, duration_estimated_millis" did return values, while "hlsBandwidthAverage, bitrate, framerate, ffmpeg_codec_type, ffmpeg_codec_name, m3u8_codecs, m3u8_name" did not return any value.

[rdlnwdj]

Update

With the newest rev.47128 core update, some links are still not getting recognized(missing audio codec) as they did back in september.

More example links are added in this post.

[pspzockerscene]

Originally Posted by rdlnwdj

And I do understand your decision to not include duration on the LinkGrabber tab when it comes to general videos

That wasn't just a "no" in my answer.
I understand that this would be super nice and indeed it would probably be the best to be able to add custom columns and tell JD "use plugin property <duration>, put it in that column and maybe format it to data type <timeHumanReadable>" but at this moment that's just out of reach.

Originally Posted by rdlnwdj

but for HLS streams, the situation is a little different, extimated filesizes sometimes deviate too much from the actual ones(mostly because many HLS streams nowadays use scene based encoding, opening titles with complicated effects and fast moving/switching scenes tend to give ffprobe wrong estimation on average bitrates)

Any examples for that?
Are the estimated filesizes mostly too high or too low?
Are they much better when doing another linkcheck afterwards or adding links with "Crawl mode: Slow"?

Originally Posted by rdlnwdj

duration is sorted like this "1656000 175000 1769000 193000 1962000", so I guess problem is not solved here

Those internally stored duration values are given in milliseconds.

Originally Posted by rdlnwdj

I checked your posts in this thread, and tried several m3u8 link properties in packagizer, "height, width, hlsBandwidth, duration_estimated_millis" did return values, while "hlsBandwidthAverage, bitrate, framerate, ffmpeg_codec_type, ffmpeg_codec_name, m3u8_codecs, m3u8_name" did not return any value.

Well that again depends on the "Crawl Mode" setting (to keep it simple).
When single-checking links, JD will obtain much more information by doing the FFprobe step.
In "Fast" or "Superfast crawl mode(s)", JDownloader will only be able to obtain the information given inside "master.m3u8" (hls list containing multiple streams) let's call it the "master playlist" and that information can vary.

The problem is that JD will only go through the Packagizer rules one time so when fast-adding links some information might be missing at that point as single online-check will be skipped. That information will be available later on downloadstart or when manually single-linkchecking such files.
Here are your options:
If you want to use the Packagizer and don't care about the speed:
- Set m3u8 Crawl Mode to "Slow" and that will work just fine
If you do care about speed + filenames:
- Use an EventScripter script instead of a Packagizer rule (much more complex!) so your filenames can be auto-altered at a later stage when all of those properties become available

Originally Posted by rdlnwdj

With the newest rev.47128 core update, some links are still not getting recognized(missing audio codec) as they did back in september.

Jiaz will look into this once he finds the time.

[rdlnwdj]

Originally Posted by pspzockerscene

Any examples for that?

Yes, the example below is actually 17.5GiB in size, while detected as 21.26GiB.
**External links are only visible to Support Staff**ffprobe -loglevel 48 -show_format -show_streams -analyzeduration 15000000 -of json -i "**External links are only visible to Support Staff** logs as well as mediainfo of downloaded file

Originally Posted by pspzockerscene

Are the estimated filesizes mostly too high or too low?

With HLS streams, extimated filesizes are mostly higher, and depending on complexity of beginning scenes, slightly or much higher.

Originally Posted by pspzockerscene

Are they much better when doing another linkcheck afterwards or adding links with "Crawl mode: Slow"?

No, my observations are based on slow mode, so my thought is, there is currently not much to do to improve this, increasing analyzeduration may help but is not needed, the increased time costing outrun the benifit.

Originally Posted by pspzockerscene

The problem is that JD will only go through the Packagizer rules one time so when fast-adding links some information might be missing at that point as single online-check will be skipped. That information will be available later on downloadstart or when manually single-linkchecking such files.
Here are your options:
If you want to use the Packagizer and don't care about the speed:
- Set m3u8 Crawl Mode to "Slow" and that will work just fine

Unfortunately, "Slow" mode does not work here, difference between super fast, fast and slow mode is that super fast does not return resolution, while fast and slow mode are basically the same behavior.

Originally Posted by rdlnwdj

"height, width, hlsBandwidth, duration_estimated_millis" did return values, while "hlsBandwidthAverage, bitrate, framerate, ffmpeg_codec_type, ffmpeg_codec_name, m3u8_codecs, m3u8_name" did not return any value.

Originally Posted by pspzockerscene

If you do care about speed + filenames:
- Use an EventScripter script instead of a Packagizer rule (much more complex!) so your filenames can be auto-altered at a later stage when all of those properties become available

Writing java script is beyond my grasp, so I'm still waiting for script master mgpai's reponse., see post below, duration sorting problem is solved.

Originally Posted by pspzockerscene

Jiaz will look into this once he finds the time.

Got it. Meanwhile I'll keep testing on new releases to see if problem gets solved and would update here asap.

[rdlnwdj]

For anyone who would like to sort duration of HLS streams in linkgrabber tab, , first add a packagizer rule to add <jd:prop:duration_estimated_millis> to comments, then create your custom 'sort by duration' buttom and try the following EventScripter code by our script master mgpai.

/*
    sort by duration
    trigger: Toolbar button pressed
*/


if (name == "sort by duration") {
    getAllCrawledPackages().forEach(function(package) {
        package.downloadLinks.filter(function(link) {
            return link.comment.match(/^[\d]+$/);
        }).sort(function(a, b) {
            return a.comment - b.comment;
        }).reverse().forEach(function(link) {
            callAPI("linkgrabberv2", "moveLinks", [link.UUID], -1, package.UUID);
        })
    })
}

[pspzockerscene]

Originally Posted by rdlnwdj

Yes, the example below is actually 17.5GiB in size, while detected as 21.26GiB.

What's your suggestion to improve this?

Originally Posted by rdlnwdj

ffprobe -loglevel 48...

That hyperlink to a pastebin appears to be offline.

Originally Posted by rdlnwdj

With HLS streams, extimated filesizes are mostly higher, and depending on complexity of beginning scenes, slightly or much higher.

Higher is better than lower if you ask me because JD will then still stop if you're running low on free space.

Originally Posted by rdlnwdj

No, my observations are based on slow mode, so my thought is, there is currently not much to do to improve this, increasing analyzeduration may help but is not needed, the increased time costing outrun the benifit.

Yap most likely not.
Maybe we could add some "factor" setting to let users have an influence on this?
But I still wonder why such a setting should be useful for anyone...

Originally Posted by rdlnwdj

Unfortunately, "Slow" mode does not work here, difference between super fast, fast and slow mode is that super fast does not return resolution, while fast and slow mode are basically the same behavior.

* it does not return the resolution right away.
The resolution will be available once such items get "fully checked" once in linkgrabber/on-downloadstart

Originally Posted by rdlnwdj

For anyone who would like to sort duration of HLS streams in linkgrabber tab...

Nice one!

[Jiaz]

Originally Posted by pspzockerscene

Higher is better than lower if you ask me because JD will then still stop if you're running low on free space.

I agree with this. Better a higher estimation and enough disk space than lower estimation and user wonders why it takes longer and maybe not enough disk space available

[Jiaz]

Originally Posted by rdlnwdj

Unfortunately, "Slow" mode does not work here, difference between super fast, fast and slow mode is that super fast does not return resolution, while fast and slow mode are basically the same behavior.

That's why I added two new AUTO Modes, see https://board.jdownloader.org/showpo...53&postcount=4. Those will check superfast/fast only if resolution is available.
SuperFast = do not trust estimated file size
Fast = if (average)bandwidth is available, estimate file size
Slow = use ffprobe to obtain information
Auto-SuperFast -> SuperFast only if video + resolution or no video
Auto-Fast -> Fast only if video + resolution or no video

Originally Posted by rdlnwdj

so I guess it would be really awesome if duration can be sorted on the LinkGrabber tab just like what we can do with file size.

I'm sorry but I don't see us adding native column with such meta information. At least not in near future, because this information is not available in many cases. Your *workaround* via Eventscripter and accessing the properties is quiet nice!

[rdlnwdj]

Originally Posted by pspzockerscene

Higher is better than lower if you ask me because JD will then still stop if you're running low on free space.

Well, this does make point:P, I was focusing on one side of the coin and ignored the other side.

Originally Posted by pspzockerscene

it does not return the resolution right away.
The resolution will be available once such items get "fully checked" once in linkgrabber/on-downloadstart

Actually, resolution detection is already working with slow, fast and auto-fast mode starting with rev.47086 core update, which according to Jiaz is the expected behavior;)

Originally Posted by Jiaz

SuperFast = do not trust estimated file size
Fast = if (average)bandwidth is available, estimate file size
Slow = use ffprobe to obtain information
Auto-SuperFast -> SuperFast only if video + resolution or no video
Auto-Fast -> Fast only if video + resolution or no video

The current situation is, fast and super-fast mode work as they should, they have done quite the job on speeding up the linkcrawing process, but for some links slow mode is missing info(no audio codec in filename) compared to the old code back in september, I posted some example links to help troubleshooting, but at least according to mgpai, these links were not accessible, if that's also the case for you, would me posting ffprobe logs of certain links help?

[Jiaz]

Originally Posted by rdlnwdj

but for some links slow mode is missing info(no audio codec in filename) compared to the old code back in september, I posted some example links to help troubleshooting

@rdlnwdj: the pastebin has expired. maybe you can send them to support@jdownloader.org again?

[rdlnwdj]

Originally Posted by Jiaz

@rdlnwdj: the pastebin has expired. maybe you can send them to support@jdownloader.org again?

**External links are only visible to Support Staff****External links are only visible to Support Staff**
;)

[Jiaz]

@rdlnwdj: possible to provide non IPv4 URL or only available as IPv6?
Does it always fail or just for IPv6 URLs?
Maybe the IPv6 somehow causes the ffprobe in JDownloader to fail. I will have to check and get hands on IPv6 connectivity here

[rdlnwdj]

Originally Posted by Jiaz

@rdlnwdj: possible to provide non IPv4 URL or only available as IPv6?

IPv4 URL is available but only accessible through my local ISP's connection, namely it's geo-locked(but much stricker, requires same ISP in the same area), the IPv6 URL is working global wide as long as the testing end has IPv6 connectivity. So yes, it is only available as IPv6.

Originally Posted by Jiaz

Does it always fail or just for IPv6 URLs?
Maybe the IPv6 somehow causes the ffprobe in JDownloader to fail. I will have to check and get hands on IPv6 connectivity here

Yes, (for the current core version)it always fails (to detect some audio codec), regards of whether it's IPv4 or IPv6, but resolution and duration is correctly detected most of the time.

Also, forgot to mention, I updated the ffmpeg and ffprobe binary originally downloaded by jdownloader, and the log of ffprobe I provided was generated using that ffprobe binary(pastbin updated). **External links are only visible to Support Staff****External links are only visible to Support Staff**

The build used:
ffmpeg-master-latest-win64-gpl-shared.zip from **External links are only visible to Support Staff****External links are only visible to Support Staff**

Screenshot of crawled link from rev.47198 core and m3u8 plugin set to slow mode:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

[Jiaz]

@rdlnwdj: took a look at this again and found the cause. the stream uses mpeg2(mp2) for audio codec and this one was not yet support/detected internally

"codec_name": "mp2",
"codec_long_name": "MP2 (MPEG audio layer 2)",

[Jiaz]

@rdlnwdj:mp2 audio codec will be correctly recognized with next core update. In case this happens again for other codecs in future, please either provide output of ffprobe or example link

[rdlnwdj]

Originally Posted by Jiaz

@rdlnwdj:mp2 audio codec will be correctly recognized with next core update. In case this happens again for other codecs in future, please either provide output of ffprobe or example link

Okay, thanks Jiaz

[Jiaz]

@rdlnwdj: Thank me once the update is live and it's working for you

[naancy584]

Originally Posted by rdlnwdj

I use Ctrl+V to batch analyze generic HLS streams, the filenames used to be like this "index (2160p_ac3 375kbits).mp4" or this "index (1080p_aac).mp4", and thus would be sorted into different packages.

This used to work when I tried it back on 23 September, but no longer works as of the newest update, now the filename is only "index.mp4".

I already updated ffmpeg and ffprobe before September and I did it again this time, so I guess it's irrelevant.

Why do my batch-analyzed HLS video filename results now not display resolution and audio codec after a recent update, when they were previously arranged neatly like "index (2160p_ac3 375kbits)" or "index (1080p_aac).mp4" for easy organization? The issue persists despite updating ffmpeg and ffprobe.