JDownloader Community - Appwork GmbH
 

Notices

Reply
 
Thread Tools Display Modes
  #1  
Old 29.06.2022, 06:34
UriF UriF is offline
Linkgrabbing Monster
 
Join Date: Oct 2013
Posts: 81
Default missav.com - download HLS URLs with referer

I got a link to stream from Video DownloadHelper but jDownloader doesn't recognize it. For example
**External links are only visible to Support Staff****External links are only visible to Support Staff**

**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #2  
Old 29.06.2022, 14:32
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

Working on it
__________________
JD-Dev & Server-Admin
Reply With Quote
  #3  
Old 29.06.2022, 17:16
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

@UriF: you will be able to add the m3u8 with next core update, but you also have to add
Quote:
#forced_referer=68747470733A2F2F6D69737361762E636F6D2F
at the end of the link, as site checks for valid referer. hexencoded website, see convertstring.com/EncodeDecode/HexEncode
__________________
JD-Dev & Server-Admin
Reply With Quote
  #4  
Old 29.06.2022, 21:25
UriF UriF is offline
Linkgrabbing Monster
 
Join Date: Oct 2013
Posts: 81
Default

Sorry, what would I have to encode or decode? bcdn_token? token_path? Could write a original string and converted string, for example... Thank you in advance

P.S. I downloaded using Ant Downloader but I got a strange effect. The length of resulted video were 2-4 min greater than original (for 4 hrs video - 4 min, for 2 hrs video - 2 min)

Last edited by Jiaz; 30.06.2022 at 10:02.
Reply With Quote
  #5  
Old 30.06.2022, 10:02
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

@UriF: You have to wait for next core update (not yet released) first, else the m3u8 link won't work at all. But as this site also checks for correct referer, you have to add https://board.jdownloader.org/showpo...91&postcount=3 at the end of the url before copying to JDownloader to tell JDownloader what referer to use.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #6  
Old 30.06.2022, 10:03
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

Quote:
Originally Posted by UriF View Post
Sorry, what would I have to encode or decode? bcdn_token? token_path? Could write a original string and converted string, for example... Thank you in advance
example
Quote:
.....link..../master.m3u8#forced_referer=68747470733A2F2F6D69737361762E636F6D2F
__________________
JD-Dev & Server-Admin
Reply With Quote
  #7  
Old 30.06.2022, 16:43
UriF UriF is offline
Linkgrabbing Monster
 
Join Date: Oct 2013
Posts: 81
Default

I tried this referrer for a video
**External links are only visible to Support Staff****External links are only visible to Support Staff**

but it doesn't work. How did you get a referrer for any video at this site? Thanks. I just want to try to get it for myself.
Reply With Quote
  #8  
Old 30.06.2022, 16:48
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

@UriF: Those m3u8 DO NOT work right now. You will have to wait for next core update.
AFTER next core update, just append https://board.jdownloader.org/showpo...91&postcount=3 at the end of the url and copy to clipboard
__________________
JD-Dev & Server-Admin
Reply With Quote
  #9  
Old 30.06.2022, 20:41
UriF UriF is offline
Linkgrabbing Monster
 
Join Date: Oct 2013
Posts: 81
Default

Thanks

P.S. I got a solution:

ffmpeg -headers "Referer: **External links are only visible to Support Staff** -i "**External links are only visible to Support Staff** -c copy -bsf:a aac_adtstoasc test.mp4

Last edited by UriF; 01.07.2022 at 04:14.
Reply With Quote
  #10  
Old 01.07.2022, 10:05
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

@UriF: yes, that's what JDownloader would do as well but the & within the URL is triggering a bug and therefore you'll have to wait for core update. the Referer is set via forcedreferer and hex encoded
__________________
JD-Dev & Server-Admin
Reply With Quote
  #11  
Old 01.07.2022, 14:47
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,918
Default

Bitte auf das nächste CORE-Update warten!

Please wait for the next CORE-Update!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!


-psp-
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #12  
Old 02.07.2022, 02:03
UriF UriF is offline
Linkgrabbing Monster
 
Join Date: Oct 2013
Posts: 81
Default

Thank you anyway. I downloaded everything that I wanted from this site using this ffmpeg script with referrer
Reply With Quote
  #13  
Old 02.07.2022, 20:30
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

Quote:
Originally Posted by UriF View Post
Thank you anyway.
Nothing to thank us for! You helped to find/fix a bug in JDownloader
__________________
JD-Dev & Server-Admin
Reply With Quote
  #14  
Old 03.07.2022, 22:11
UriF UriF is offline
Linkgrabbing Monster
 
Join Date: Oct 2013
Posts: 81
Default

Thank you to give me a hint (referrer) where to search. There are some web sites which follow the same rules. I will use this method FOR now until Ant Downloader would fix their bug which I mentioned above (if jDownloader of course would be able to implement for all of sites which have the same restrictions).
Reply With Quote
  #15  
Old 07.07.2022, 17:33
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,918
Default

Please first wait for the update and then provide more example URLs if you find more similar websites which JD fails to download streams from.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #16  
Old 22.07.2022, 01:32
UriF UriF is offline
Linkgrabbing Monster
 
Join Date: Oct 2013
Posts: 81
Default

1. **External links are only visible to Support Staff****External links are only visible to Support Staff**
I can get a link to stream but I get only first 3mb (ts files misnumbered???) I tried using Google development tools but I was able to see a lot ts file with the same name
sample:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

2. **External links are only visible to Support Staff****External links are only visible to Support Staff**
Video are not recognized neither VDH no Ant Downloader
sample:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #17  
Old 22.07.2022, 12:03
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

Quote:
Originally Posted by UriF View Post
1. **External links are only visible to Support Staff****External links are only visible to Support Staff**
I can get a link to stream but I get only first 3mb (ts files misnumbered???) I tried using Google development tools but I was able to see a lot ts file with the same name
Because the server only serves max 3mb chunks and then the next 3mb and so on. This will be supported with next core/plugin update. Please wait and when update is released, try the stream url with JDownloader
__________________
JD-Dev & Server-Admin
Reply With Quote
  #18  
Old 24.07.2022, 21:32
UriF UriF is offline
Linkgrabbing Monster
 
Join Date: Oct 2013
Posts: 81
Default

Re. 2. It interesting that some browsers let you a download button. When I clicked on it I got

**External links are only visible to Support Staff****External links are only visible to Support Staff**

I clicked on it and it asked me to register. After I registered they asked me to pay something to download. I tried Firefox Development tool but I has been kick off from this web page. By the way I found out how to download from hellabyte but I can get only 1080p resolution file


P.S. There were this button on Dissenter and PaleMoon. Now this button disappeared on Dissenter. Palemoon stopped to play this file at all (hourglass forever)

P.P.s. I believe that jable.tv and javrip.net can be used with the same technology like missav.com. If you are interested with yt-dlp (like ffmpeg) I can post here a sample.

Last edited by UriF; 24.07.2022 at 22:26.
Reply With Quote
  #19  
Old 25.07.2022, 00:35
UriF UriF is offline
Linkgrabbing Monster
 
Join Date: Oct 2013
Posts: 81
Default

By the way I wanted to update ffmpeg version (or check if it is current). I opened a directory
C:\Users\MyUserName\AppData\Local\JDownloader 2.0\tools\Windows\ffmpeg\x64

I found out that files are TOO old. Moreover you use a download link to
**External links are only visible to Support Staff****External links are only visible to Support Staff**
This site is dead. Therefore you are not able to update old files and user is not able to get all of bug fixes
Reply With Quote
  #20  
Old 25.07.2022, 17:14
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,918
Default

Quote:
Originally Posted by UriF View Post
I found out that files are TOO old. Moreover you use a download link to
ffmpeg.zeranoe.com/builds/
This site is dead. Therefore you are not able to update old files and user is not able to get all of bug fixes
This is just the link to the website where we got the build from.

You can use any FFmpeg build you like from any source you like as long as it's not super outdated nor some kind of special fork of FFmpeg.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #21  
Old 25.07.2022, 19:00
UriF UriF is offline
Linkgrabbing Monster
 
Join Date: Oct 2013
Posts: 81
Default

I apologize but the latest ffmpeg build from this site - 4.3. The current version of ffmpeg - 5.1
Reply With Quote
  #22  
Old 26.07.2022, 13:29
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,918
Default

I'm sorry but in this thread Jiaz has explained why there is no need to constantly update FFmpeg.
As explained, if you want to use the latest FFmpeg version or any other version apart from the one we deliver with JD, you are free to do so.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #23  
Old 27.07.2022, 01:44
UriF UriF is offline
Linkgrabbing Monster
 
Join Date: Oct 2013
Posts: 81
Default

Sorry, l don't know how to access pro settings. I see a standard (???) GUI with following icons: General, Reconnect, Connection Manager and so on. Now I don't see a lot of jDownloader options which were available in previous versions.

P.S. Google translate translated from German "Pro Settings" instead of "Advanced Settings". By the way which version of ffmpeg version to be used - ffmpeg-release-full-shared or ffmpeg-release-full?

I didn't create a path to ffmpeg in Environmental Variables. Could I create adirectore somewhere where I will copy the newest version of ffmpeg and just replace a path in jDownloader Advanced Settings with mine or I will have to create a new Environmental variable?

Last edited by UriF; 27.07.2022 at 02:20.
Reply With Quote
  #24  
Old 27.07.2022, 13:44
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,918
Default

Quote:
Originally Posted by UriF View Post
Sorry, l don't know how to access pro settings. I see a standard (???) GUI with following icons: General, Reconnect, Connection Manager and so on.
Scroll down and you will see "Advanced Settings".

Quote:
Originally Posted by UriF View Post
By the way which version of ffmpeg version to be used - ffmpeg-release-full-shared or ffmpeg-release-full?
Use "ffmpeg-release-full".

Quote:
Originally Posted by UriF View Post
I didn't create a path to ffmpeg in Environmental Variables.
You don't need to do that.

Quote:
Originally Posted by UriF View Post
Could I create adirectore somewhere where I will copy the newest version of ffmpeg and just replace a path in jDownloader Advanced Settings
Exactly that.
Either place FFmpeg in the JDownloader folder where the current version also sits (overwrite it) or manually change the location in the advanced settings (not recommended), see the following advanced settings:
FFmpegSetup.binarypathprobe
and:
FFmpegSetup.binarypath
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #25  
Old 27.07.2022, 20:34
UriF UriF is offline
Linkgrabbing Monster
 
Join Date: Oct 2013
Posts: 81
Default

Thanks
Reply With Quote
  #26  
Old 10.08.2022, 20:25
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

@UriF:
I will update bundled ffmpeg version as soon as I find some time for it
__________________
JD-Dev & Server-Admin
Reply With Quote
  #27  
Old 10.08.2022, 20:26
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

Quote:
Originally Posted by UriF View Post
P.S. Google translate translated from German "Pro Settings" instead of "Advanced Settings". By the way which version of ffmpeg version to be used - ffmpeg-release-full-shared or ffmpeg-release-full?
Doesn't really matter. Shared just requires less memory because both binaries share the same library files while the full ones are larger but *normally* don't depend on additional library files
__________________
JD-Dev & Server-Admin
Reply With Quote
  #28  
Old 02.09.2022, 21:20
UriF UriF is offline
Linkgrabbing Monster
 
Join Date: Oct 2013
Posts: 81
Default

Sorry. I tried but it doesn't work. I'm able to download using ffmpeg directly using referrer. I don't know how to add referrer
Reply With Quote
  #29  
Old 02.09.2022, 21:27
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

Quote:
Originally Posted by UriF View Post
I don't know how to add referrer
You mean you're adding m3u8 and want to specify the referer? Or do you mean something else?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #30  
Old 03.09.2022, 00:20
UriF UriF is offline
Linkgrabbing Monster
 
Join Date: Oct 2013
Posts: 81
Default

Yes, you are right. I tried missav by adding link has been taken from Video Download Helper or The Stream Detector but I got in jDownloader link grabber something like "file off-line" t.e. typical j Downloader message. I tried the same link with referrer at ffmpeg and single threaded (one connection) yt-dlp and downloaded video without any problems. Therefore the problem in jDownloader that I didn't pass a referrer. The Stream Detector is a more simple tool than VDH but it generate a script for ffmpeg and yt-dlp.

Last edited by UriF; 04.09.2022 at 19:55.
Reply With Quote
  #31  
Old 04.09.2022, 20:05
UriF UriF is offline
Linkgrabbing Monster
 
Join Date: Oct 2013
Posts: 81
Default

Quote:
You mean you're adding m3u8 and want to specify the referer?
Not just m3u8. There are sites which have links to mp4 files but to download files it req'd referrer (sorry, I don't remember them right now)
Reply With Quote
  #32  
Old 05.09.2022, 11:16
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

@UriF: thanks for clarification. can you please provide example links and referer to support@jdownloader.org then I will look into it and add a way to set custom referer
__________________
JD-Dev & Server-Admin
Reply With Quote
  #33  
Old 05.09.2022, 15:44
UriF UriF is offline
Linkgrabbing Monster
 
Join Date: Oct 2013
Posts: 81
Default

That's samples for ffmpeg (for yt-dlp the similar code):
ffmpeg -headers "Referer: **External links are only visible to Support Staff** -i "**External links are only visible to Support Staff** -c copy -bsf:a aac_adtstoasc test.mp4

ffmpeg -headers "Referer: **External links are only visible to Support Staff** -i "**External links are only visible to Support Staff** -c copy -bsf:a aac_adtstoasc test.mp4

yt-dlp --add-header "User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36" --add-header "Referer:**External links are only visible to Support Staff** "**External links are only visible to Support Staff**

As a rule I never put a reference to browser user-agent but maybe it is necessary when it req'd browser's cookies (get session variables)

P.S. The Stream Detector generate link with referrer and user-agent for ffmpeg and yt-dlp.

I believe that the easiest way to implement it in GUI just add a text box into Analyze and Add Screen in Link Grabber

Last edited by UriF; 05.09.2022 at 15:52.
Reply With Quote
  #34  
Old 05.09.2022, 15:52
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,918
Default

Quote:
Originally Posted by UriF View Post
P.S. The Stream Detector generate link with referrer and user-agent for ffmpeg and yt-dlp.
My guess is that that is not a link but a command for FFmpeg that you can easily copy & paste.

Anyways depending on how you are copying URLs JD can also already auto find the referer e.g. if you copy a URL on a page.

I guess when copying it using browser addons that reference won't be there so exactly such cases are hard/impossible to auto handle in JD atm.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #35  
Old 05.09.2022, 23:02
UriF UriF is offline
Linkgrabbing Monster
 
Join Date: Oct 2013
Posts: 81
Default

Yes, you are right but if you get a link from Video Download Helper it contains just a link to stream/file. If I copy/paste it in jDownloader link grabber it will not contain a referrer. As a rule a referrer is a link to video hoster site or link to a web page where a video is located. That's why I gave you sample links are generated by Stream Detector. You have 2 choices:

1. Link grabber will accept all of link is generated by Stream Detector including referrer or (if it is necessary) user-agent.

2. Accept direct link generated by Visual Download Helper and add a text box for referrer.

3. There is an opportunity to create jDownloader plug-in which automatically recognize a link to stream/file and copy this link with a link web page to video as a referrer but I am afraid it is too much job for jDownloader. This application don't intercept a stream but use like ffmpeg or yt-dlp because it is possibly to recognize a video when user start to watch a video.

P.S. By the way I got an interesting response from Ant Downloader developers r.e. javhdworld.com | soe-681

Quote:
Unfortunately the video from the website from your ticket doesn't provide any information about playlist, streaming manifest or video segments.

Such cases is usually related to the protected RTMP well known real time messaging protocol.
Reply With Quote
  #36  
Old 06.09.2022, 10:38
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

@UriF: Can you provide complete examples of 1 and 2 ? Maybe we can add support via simple parser
Quote:
Originally Posted by UriF View Post
P.S. By the way I got an interesting response from Ant Downloader developers r.e. javhdworld.com | soe-681
It's a wrong answer, the site uses normal hls streaming but the urls are missing known file extension and most likely therefore their detection fails to find it. Same reason you need to look more accurate in developer mode of browser to find the stream
__________________
JD-Dev & Server-Admin
Reply With Quote
  #37  
Old 06.09.2022, 15:48
pspzockerscene's Avatar
pspzockerscene pspzockerscene is offline
Community Manager
 
Join Date: Mar 2009
Location: Deutschland
Posts: 70,918
Default

@Jiaz
Depending on the case he could also use a LinkCrawler REWRITE rule to make that link get picked up by JD.
__________________
JD Supporter, Plugin Dev. & Community Manager

Erste Schritte & Tutorials || JDownloader 2 Setup Download
Spoiler:

A users' JD crashes and the first thing to ask is:
Quote:
Originally Posted by Jiaz View Post
Do you have Nero installed?
Reply With Quote
  #38  
Old 06.09.2022, 17:58
UriF UriF is offline
Linkgrabbing Monster
 
Join Date: Oct 2013
Posts: 81
Default

1. **External links are only visible to Support Staff****External links are only visible to Support Staff**

or **External links are only visible to Support Staff****External links are only visible to Support Staff**

2. Under a large text box add a standard text box for referrer. You can also add a text box for an user-agent

Last edited by UriF; 06.09.2022 at 18:03.
Reply With Quote
  #39  
Old 06.09.2022, 18:08
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 79,289
Default

@UriF: the first url is missing the text referer? did you remove it or why first one is missing it while second one does have it?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #40  
Old 06.09.2022, 19:34
UriF UriF is offline
Linkgrabbing Monster
 
Join Date: Oct 2013
Posts: 81
Default

No. It is the same link. I just joined the link to stream/file and referral through character "|" as a separator

https//"link to stream"|https"link referral" or https//"link to stream"|Referral:https"link referral"

or https//"link to stream"|https"link referral" or https//"link to stream"|Referral:https"link referral"|user-agent:"user agent"

Whatever you will chose a format it is up to you. If you don't want to change GUI and add a text box(s) the "|" as a separator would be the best chice
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 13:37.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.