[ElCho]

Hi, could you please add direct support to DW's streaming website?

Main streaming website (ES):

**External links are only visible to Support Staff****External links are only visible to Support Staff**

Example links:

**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

I don't know if you can make JD2 replicates what browsers do, regarding the RTMP streams' request, but that's not the most important issue about my request. Reading each source pages from the aforementioned example links, there is always a direct "HTTP" link for the higher quality version from the videos, which for some weird reason they hide.

Since downloading through direct links is a lot better than doing it via RTMP ones, it would be good if JD2 just grabs them directly after copying such links to the clipboard, and uses the websites' titles as filenames.

Thanks.

[Jiaz]

[pspzockerscene]

Hi
No need for a plugin.
Simply go to Settings --> Advanced Settings --> Search for "link crawler rules" --> Enter this:

[
  {
    "enabled": true,
    "maxDecryptDepth": 1,
    "id": 1422443765154,
    "name": "dw.com TEST",
    "pattern": "https?://(www\\.)?dw\\.com/.+av\\-\\d+",
    "rule": "DEEPDECRYPT",
    "packageNamePattern": "<title>(.*?)</title>",
    "formPattern": null,
    "deepPattern": null,
    "rewriteReplaceWith": null
  }
]

Best regards,
pspzockerscene

[Jiaz]

@pspzocker: you should also learn to specify the packageNamePattern and deepPattern

[pspzockerscene]

I'm usually making minimalistic crawler rules - if a user wants it to do more, simply ask

GreeZ psp

[ElCho]

Thanks, I just tried it and it works, but I would like it grabs just the video files and leaves out all the rest of links. For example, for this link:

**External links are only visible to Support Staff**

JD2 will grab 159 links, from which 158 are useless, since just the video related one is needed.

[pspzockerscene]

Updated - try this:

[
  {
    "enabled": true,
    "maxDecryptDepth": 1,
    "id": 1422443765154,
    "name": "dw.com TEST",
    "pattern": "https?://(www\\.)?dw\\.com/.+av\\-\\d+",
    "rule": "DEEPDECRYPT",
    "packageNamePattern": "<title>(.*?)</title>",
    "formPattern": null,
    "deepPattern": "\"(http[^\"]+\\.mp4)\"",
    "rewriteReplaceWith": null
  }
]

GreeZ pspzockerscene

[ElCho]

It works completely fine now, thanks.

[Jiaz]

Danke fürs Feedback!

[ElCho]

False alarm. Delete please.

[Jiaz]

Thanks for the feedback!

[ElCho]

Hi, since some weeks ago, I just noted they dropped their habit of hiding the HD MP4 direct download link of the videos in the source pages; now, they leave just a low quality, FLV vp6 direct link over there. Despite that, at least two other qualities of the videos can still be downloaded once they are reproduced by the embedded player. So, it would be good if you can make JD2 grab those ones from now on.

Example links:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

**External links are only visible to Support Staff****External links are only visible to Support Staff**

The first links have two qualities (as far as I can see):

pri20170928_9673mikrobenHD_sd_sor.mp4 (121,54 MB)

pri20170928_9673mikrobenHD_sd_avc.mp4 (808.87 MB)

The second one has two as well:

pri20171010_9684SterneNahHD_sd_sor.mp4 (184,20 MB)

pri20171010_9684SterneNahHD_sd_avc.mp4 (805,18 MB)

Thanks.

[Jiaz]

[Jiaz]

we will check and add proper plugin support

[ElCho]

I just saw this issue got a %100 progress status and has been already closed, but it still has the "bugtracker" tag in its title and downloads are still not working.

[Jiaz]

Forgot to reopen the ticket, sorry

[ElCho]

It's OK.

[ElCho]

Hello, is there some hope left about adding support for this website?

Thanks.

[pspzockerscene]

Added basic crawler plugin.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[ElCho]

Very nice, it's getting the streams correctly so far I tested it.

There is a missing detail about the saved streams' filenames, which maybe can be automatized as well. Basically, naming files to be saved according to the web page they are on, like this:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

Original name: "pri200205_10618redchildren_sd_avc.mp4"

Renamed: "Niños rojos - La infancia bajo el gobierno de Mao.mp4"

**External links are only visible to Support Staff****External links are only visible to Support Staff**

Original name: "pri20200212_10613dresden_sd_avc.mp4"

Renamed: "El mito de Dresde - La sombra de una noche de bombardeos.mp4"

Thanks.

[pspzockerscene]

Done.

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[ElCho]

It is complete now, thanks once again.

[pspzockerscene]

Thanks for your feedback.

-psp-

[ElCho]

@pspzockerscene

Hello, it's been sometime since I downloaded something form DW, but I just noticed the already fixed issue I reported here back then:

https://board.jdownloader.org/showpost.php?p=457626&postcount=20

came back, but in a different way. It seems the latest changes you introduced for some sites, regarding playlists' processing and the like, altered the naming convention for this site as well. So, now, with a link like this:

**External links are only visible to Support Staff**

JD2 still grabs the videos titles correctly, as that from the website's title, but since it doesn't apply it to the filenames themselves, but to a container, which lists the stream's different qualities, and additional manual renaming will be required each time.

For example, the above video is processed this way:

Primer plano - Qatar - Polémica Copa del Mundo _ DW _ 21 11 2022 (container)

index-f1-v1-a1 (1080p_aac)_bw_9251776.mp4
index-f2-v1-a1 (720p_aac)_bw_7045912.mp4
index-f3-v1-a1 (540p_aac)_bw_5157216.mp4
index-f4-v1-a1 (360p_aac)_bw_2026488.mp4
index-f5-v1-a1 (288p_aac)_bw_1064720.mp4
index-f6-v1-a1 (270p_aac)_bw_543824.mp4

Would it be possible, to make JD2 adds just the website's title to each stream, like it was before? Or make it puts it at the beginning of
each filename, so just the unneeded parts get deleted later. Something like this:

Primer plano - Qatar - Polémica Copa del Mundo _ DW _ 21 11 2022_index-f1-v1-a1 (1080p_aac)_bw_9251776.mp4

Thanks.

P.D. Just for your information, the board's bug about introduced links getting missing after previewing a post, or submitting it, is still kicking. So, the "CODE" tag should be mandatorily used each time to avoid that glitch.

[pspzockerscene]

Originally Posted by ElCho

Hello, it's been sometime since I downloaded something form DW, but I just noticed the already fixed issue I reported here back then:...came back, but in a different way.

Nope it didn't really come back.
The website dw.com seems to have removed all of thei direct http URLs and is now only providing HLS/.m3u8 links for streaming.
Filename handling for those was never added.

Originally Posted by ElCho

Would it be possible, to make JD2 adds just the website's title to each stream, like it was before? Or make it puts it at the beginning of

For the next update:
Added title property so m3u8 generic handling will put that title into the filenames.

Originally Posted by ElCho

Just for your information, the board's bug about introduced links getting missing after previewing a post, or submitting it, is still kicking. So, the "CODE" tag should be mandatorily used each time to avoid that glitch.

I need to say in recent times I did not encounter that bug anymore at all...
Sorry for the trouble.

Bitte auf das nächste CORE-Update warten!

Please wait for the next CORE-Update!

Wartest du auf einen angekündigten Bugfix oder ein neues Feature?
Updates werden nicht immer sofort bereitgestellt!
Bitte lies unser Update FAQ! | Please read our Update FAQ!
---
Are you waiting for recently announced changes to get released?
Updates to not necessarily get released immediately!
Bitte lies unser Update FAQ! | Please read our Update FAQ!

-psp-

[pspzockerscene]

CORE-Updates have been released!
All announced bugfixes and features are live!
Please update your JDownloader and report any issues you find asap.
If this thread gets marked as "[Solved]" by our forum staff you can still post in it and we will read- and reply to it!

CORE-Updates wurden released!
Alle angekündigten Bugfixes/Features sind nun verfügbar!
Bitte JDownloader updaten und eventuelle Bugs schnellstmöglich an uns melden.
Falls dieser Thread vom Team als "[Erledigt]" markiert wird, kannst du weiterhin darin antworten und wir lesen/beantworten auch solche Threads!

-psp-

[ElCho]

Originally Posted by pspzockerscene

Nope it didn't really come back...

Yes, that's correct. I forgot they were using direct, http based downloads before.

Originally Posted by pspzockerscene

For the next update:
Added title property so m3u8 generic handling will put that title into the filenames.

I just tested it, and it's working alright.

By the way, there is a minor issue I found by chance, regarding the handling of unknown
characters. For example, with the next links:

**External links are only visible to Support Staff**

Originally Posted by pspzockerscene

I need to say in recent times I did not encounter that bug anymore at all...
Sorry for the trouble.

It is really easy to reproduce it. I leave you a short video about it:

**External links are only visible to Support Staff****External links are only visible to Support Staff**

By the way, would be possible to filter the download links, so JD2 shows only those with a certain word in their titles?
For example, "1080p". Or the files with the higher sizes only? I already tried with the "Linkgrabber Filter",
but to no avail. Thanks.

[Jiaz]

Originally Posted by ElCho

By the way, there is a minor issue I found by chance, regarding the handling of unknown
characters. For example, with the next links:

check again with next update. I guess you were talking about for example

"

? in case you mean something different, maybe provide screenshot

[Jiaz]

Originally Posted by ElCho

For example, "1080p". .... I already tried with the "Linkgrabber Filter",
but to no avail. Thanks.

Linkgrabber Filter should work just fine. Can you maybe share screenshot of the rule you tried?

Originally Posted by ElCho

Or the files with the higher sizes only?

Filters cannot rely on other links as condition as there are not guarantees about processing order and this can cause circular dependencies. Such filters/selections must be coded/supported directly within the plugin as only this has a whole *view* on the link and its results.

[Jiaz]

Originally Posted by ElCho

It is really easy to reproduce it. I leave you a short video about it:
**External links are only visible to Support Staff****External links are only visible to Support Staff**

The censor module/plugin blacklists/filters URLs for all non staff members. Else
it would be easy to workaround it by just quoting other ppl posts and go to edit mode.

[ElCho]

Originally Posted by Jiaz

check again with next update. I guess you were talking about for example ? in case you mean something different, maybe provide screenshot

It is partially correct now; the next character:

**External links are only visible to Support Staff**

is missing from the next three titles:

**External links are only visible to Support Staff**
**External links are only visible to Support Staff**
**External links are only visible to Support Staff**

[ElCho]

Originally Posted by Jiaz

Linkgrabber Filter should work just fine. Can you maybe share screenshot of the rule you tried?

Filters cannot rely on other links as condition as there are not guarantees about processing order and this can cause circular dependencies. Such filters/selections must be coded/supported directly within the plugin as only this has a whole *view* on the link and its results.

I tried with the next rule, and although it works, I doubt it is completely right:

[ElCho]

Originally Posted by Jiaz

The censor module/plugin blacklists/filters URLs for all non staff members. Else
it would be easy to workaround it by just quoting other ppl posts and go to edit mode.

I'm sorry, but I didn't understand you. I know the links posted by any "non staff" member will be hidden to all other regular members, but the problem shown in that video, is that any introduced link will be deleted, after pressing the "Go Advanced" button; and the only workaround I know to this issue is, using the "code" tags before pressing the button.

[ElCho]

@Jiaz

Some news about this?

Thanks.

[Jiaz]

Originally Posted by ElCho

It is partially correct now; the next character:
software.hixie.ch/utilities/cgi/unicode-decoder/character-identifier?characters=%22

This character will be auto replaced because it's not allowed on all OS, see
learn.microsoft.com/en-us/windows/win32/fileio/naming-a-file
JDownloader auto replaces all known forbidden chars independent of the underlying OS so you will always be able to move around your downloaded files and use JDownloader portable across different OS/filesystem.

[Jiaz]

Originally Posted by ElCho

I tried with the next rule, and although it works, I doubt it is completely right:

You should limit the 1080p rule to match on dw.com links only
sourceURL - contains - dw.com

[Jiaz]

Originally Posted by ElCho

I'm sorry, but I didn't understand you. I know the links posted by any "non staff" member will be hidden to all other regular members, but the problem shown in that video, is that any introduced link will be deleted, after pressing the "Go Advanced" button; and the only workaround I know to this issue is, using the "code" tags before pressing the button.

The link censor module auto censors the URLs when not part of BBCode tag. I know that this is disadvantageous but right now I don't find time to debug the module in order to check if this can be fixed/changed. sorry for the inconvenience

[ElCho]

@Jiaz

Alright, thanks for your explanations.

[Jiaz]

@ElCho: You're welcome!