[Mistery]

Here information for my last search:
The links for Rai1 don't work: eg: **External links are only visible to Support Staff****External links are only visible to Support Staff**

Thank you :-)

[Mistery]

Hello

Now these are the links of rai2 and 3 don't work ;-)

Thank you

[pspzockerscene]

Regarding your posts:

#119:
That means if you add a url with a date but there are no videos for this date, JDownloader will not add anything - before, it added videos of all/other days then.

#120:
Fixed wrong date issue.
Then about the 1800, I still don't understand the problem.
Test URL:

**External links are only visible to Support Staff**

-->
HLS url:

**External links are only visible to Support Staff**

This is what JD finds and results in this filename:

2017-01-06_raitv_Un Selfie con il Papa - Un selfie con il Papa - Le storie - video - RaiPlay_928x522_avc1.77.30, mp4a.40.2.mp4

Of course I can change the filename - just tell me how you prefer it to look like

#121:
What do you mean by this?
When I add this URL, I get a couple of urls:

**External links are only visible to Support Staff**
**External links are only visible to Support Staff**
**External links are only visible to Support Staff**
**External links are only visible to Support Staff**
**External links are only visible to Support Staff**
**External links are only visible to Support Staff**
**External links are only visible to Support Staff**
**External links are only visible to Support Staff**
**External links are only visible to Support Staff**
**External links are only visible to Support Staff**
**External links are only visible to Support Staff**
**External links are only visible to Support Staff**
**External links are only visible to Support Staff**
**External links are only visible to Support Staff**
**External links are only visible to Support Staff**
**External links are only visible to Support Staff**
**External links are only visible to Support Staff**
**External links are only visible to Support Staff**
**External links are only visible to Support Staff**
**External links are only visible to Support Staff**
**External links are only visible to Support Staff**
**External links are only visible to Support Staff**
**External links are only visible to Support Staff**
**External links are only visible to Support Staff**

Screenshot:

(By the way I did fix the 'null' in the filenames (missing date) too.)

#122:
As always - we need a detailed problem description and example URLs no matter which issues you have
---

[Please read]=>How/when do I get updates?
Read it carefully BEFORE complaining that something would still be not working!
Update cycles depend on your installed JD version!!

---
[Bitte lesen]=>Wie/wann bekomme ich Updates?
Bitte vollständig lesen bevor ihr euch beschwert, dass etwas immernoch nicht funktionieren würde!
Die Updatezyklen variieren je nach installierter JD Version!!

GreeZ pspzockerscene

[Mistery]

119: Ok i understand

120:
- The problem of dates persists via links "days":
ex:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

- For "800": The problem is that sometimes it does not appear in the linkgrabber (while it exists)

- For "files names": no worries, i created rules ! ;-)

121:
- I got few links, even nothing, that's why i reported it. But it seems that it works now... to follow...

- And your print shows that many videos of that day are not detected

122:
- I gave no examples of links because these same as usual: ex: **External links are only visible to Support Staff****External links are only visible to Support Staff** or ch=2 or ch=3

[pspzockerscene]

120:
?
Do you mean the wrong date inside the filename?
Please post specific example urls and I do not mean the direct-urls - I mean such URLs:

**External links are only visible to Support Staff**

Also I need to know exactly where which quality is missing - without this information, I cannot help you.

121:
Thanks for the feedback.
Regarding missing content:
Again, without knowing what is missing, I cannot help you.
A LOT of the content appears to be offline/geoblocked here ...

122:
See above what I mean with "example urls" --> URLs of single videos - NOT the daily overview/playlist urls!

GreeZ pspzockerscene

[Mistery]

120:
- Yes "dates" in the files names obtained. Examples there are many. Just analyze any day via a "day" link and you will get all sorts of different dates in the files names that the day analyzed
- For qualities: sometimes it is the "800" missing and sometimes the "1800". I give you another example when i find it...

121:
What is missing in the results, i know it by deduction, when for example there is a small number of videos found or some missing as for example: during these analyzes there was very little or no at all "tg1"
For all this, i speak naturally of files not geoblocked (besides that there were also very few results)

122:
I was talking about "day" links that gave no results, so sorry i can not give you links for single videos because there were no videos.
But now it seems that it works. So to follow ...

Thanks

[pspzockerscene]

Thanks for your feedback.
I've checked the date-thing and indeed you're right - the day-crawler will also crawl videos of other days but I cannot see any error on our side.
Example:

**External links are only visible to Support Staff**

--> json:

**External links are only visible to Support Staff**

--> Example of a video with "wrong" date:

**External links are only visible to Support Staff**

-->

2017-01-14_raitv_Tg3 - TG3 Agenda del mondo - video - RaiPlay_928x522_avc1.77.30, mp4a.40.2.mp4

json source of this item:

 "ID" : "ContentItem-fcd06184-ea41-435c-b009-e0971b8d74db", "pathID" : "/raiplay/video/2017/01/TG3-Agenda-del-mondo-fcd06184-ea41-435c-b009-e0971b8d74db.html?json", "name" : "TG3 Agenda del mondo", "caption" : "", "description" : "TG3 Agenda del mondo", "type" : "RaiTv Media Video Item", "videoQuality" : "", "webLink" : "/raiplay/video/2017/01/TG3-Agenda-del-mondo-fcd06184-ea41-435c-b009-e0971b8d74db.html", "rating" : "0" } ,{ "datePublished":"22/01/2017", "timePublished":"01:35", "duration":"00:15:00", "hasVideo": true, "hasHighlights": false, "hasSubtitle": false, "isPartOf": { "name" : "TG3 Chi \u00E8 di scena", "ID" : "ContentItem-fee0d709-84ed-4baf-bb23-bfcf93273c54", "tipologia"

As you can see it says:

"datePublished":"22/01/2017"

JDownloader has to rely on this information ...

GreeZ pspzockerscene

[Mistery]

"
As you can see it says:
Code:
"datePublished":"22/01/2017"
JDownloader has to rely on this information ...
"
So that i understand well, it is the modification that you made in the plugin to get the right date ?
(But the problem of dates seems to occur only for the "day" links)

Here two example for date and quality problem:
- **External links are only visible to Support Staff****External links are only visible to Support Staff**
- **External links are only visible to Support Staff****External links are only visible to Support Staff**
: for quality, the problem is due to the links found:
=> **External links are only visible to Support Staff****External links are only visible to Support Staff**
=> **External links are only visible to Support Staff****External links are only visible to Support Staff**
But the "1800" is available too...

Thank you :-)

[pspzockerscene]

1. No you misunderstood me - this is not the place I get the date from - this is the place where I make sure that the videos I add are only from the date you wanted e.g.:

**External links are only visible to Support Staff**

--> date: 2017-01-22
All videos where "datePublished" is NOT "22/01/2017" will not get added.

--> But yeah I just fixed the "wrong date" issue.
Please watch this and tell me if it happens again.
Please always provide example links the same type as you just posted

2. Regarding the quality:
How do you know that 1800 is available?
Where can I see that?
As said, I could simply always add 1800 but then you'll get a lot of dead urls too ...
---

[Please read]=>How/when do I get updates?
Read it carefully BEFORE complaining that something would still be not working!
Update cycles depend on your installed JD version!!

---
[Bitte lesen]=>Wie/wann bekomme ich Updates?
Bitte vollständig lesen bevor ihr euch beschwert, dass etwas immernoch nicht funktionieren würde!
Die Updatezyklen variieren je nach installierter JD Version!!

GreeZ pspzockerscene

[Mistery]

1.
Correct me if i'm wrong but it seems to me that this way it means that if the date is incorrect in the file then it will not be considered then that perhaps it should so if in fact it is part of the date requested... ?
(But again, what is bizarre is that it happens with the links "days" and (it seems to me) not with normal links)

2.
Because i understood how to transform the links to get what i want. In the case as in the example links given above, the site proposes the version "800" but the script that i proposed at the beginning of the subject or if i transform manually the links, we always get a "1800" valid.
(Do not worry about the dead urls because there is always an "1800")

Thank you

[Mistery]

Here are other links problem:
I tested all the links of this show and it does not work
**External links are only visible to Support Staff****External links are only visible to Support Staff**

[pspzockerscene]

1. Sorry, I failed to understand you.
What I wanted to say is if you add date XY and the server's json-response says video "blabla" has date "XY", JDownloader will trust this.
The real date will be shown later when the video gets added to the linkgrabber.

2. Which script do you mean?
Sorry, it must have been a while ago that you posted that^^

3. The url you posted leads to a video overview page:

**External links are only visible to Support Staff**

(Some urls are in there twice or even more often.)

GreeZ pspzockerscene

[Mistery]

1. Ok i understand and i trust you too ;-)

2. This script finds the direct link of the video file in one block (therefore not segmented file) and to the max of available quality on each page...
I gave it to you in case it would help you... : **External links are only visible to Support Staff****External links are only visible to Support Staff**

3. Yes but you only scanned a single page (12 different links)
but there are other sub pages on the page of the show-page that i sent you, there are 4 (so we should get about 40 different links) and so we should get about 40 videos ;-)

[Mistery]

...And days links don't work fine (especially for Rai 2 & 3) :-s :
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

[pspzockerscene]

Regarding the 1800 URLs:
The script uses an external website/unofficial API to get the HQ http urls - this is not good:

**External links are only visible to Support Staff**

However, I've implemented it directly in JD without this "video.lazza.dk" website so that should be fine.

To the date URLs:
Indeed there is a problem - the website returns bad json - I'm not in ...
EDIT
Fixed!
---

[Please read]=>How/when do I get updates?
Read it carefully BEFORE complaining that something would still be not working!
Update cycles depend on your installed JD version!!

---
[Bitte lesen]=>Wie/wann bekomme ich Updates?
Bitte vollständig lesen bevor ihr euch beschwert, dass etwas immernoch nicht funktionieren würde!
Die Updatezyklen variieren je nach installierter JD Version!!

GreeZ pspzockerscene

[Mistery]

Thank you !

Ha ok i did not know for the script work...

The day links work better but:
- most links are "**External links are only visible to Support Staff** and not "**External links are only visible to Support Staff**
Is this normal despite the script you have incorporated ?
- And the list of programs-shows videos links is still incomplete because the video link is not found for some shows:
random example: **External links are only visible to Support Staff****External links are only visible to Support Staff**
(you can see all programs here : **External links are only visible to Support Staff****External links are only visible to Support Staff** and select the day you want...)

[pspzockerscene]

Hi
1. I've changed some code to only add http urls if possible - this will also make the filenames of all found qualities look more similar to each other.

2. Your example works fine here - sorry I failed to understand what you mean here:

**External links are only visible to Support Staff**

---

[Please read]=>How/when do I get updates?
Read it carefully BEFORE complaining that something would still be not working!
Update cycles depend on your installed JD version!!

---
[Bitte lesen]=>Wie/wann bekomme ich Updates?
Bitte vollständig lesen bevor ihr euch beschwert, dass etwas immernoch nicht funktionieren würde!
Die Updatezyklen variieren je nach installierter JD Version!!

GreeZ pspzockerscene

[Mistery]

Thanks

2. Yesterday, it was not working and was not found via the search "day". It appeared as a file "document" but no "video"
I don't know why this difference between research "day" and research "direct link"
You have to tell me ;-)

[Mistery]

- Now the new problem is the mixture of obtained links:
"**External links are only visible to Support Staff** and "http: // Creativemedia3.rai .it /..._ 1800.mp4 "

- The case "**External links are only visible to Support Staff** => "1800" is not resolve.

It's much more complicated to manage and it also creates duplicates ...
The best is the direct links to the file: "creative ..."
But whatever it is it would be better if there is only one type of links please :-s

[pspzockerscene]

Example urls please.
The different qualitry/protocol types should NOT be mixed (anymore)!

GreeZ pspzockerscene

[Mistery]

The links i gave you above, it is always the same ...:
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**
=> but now the pages expired but i suppose that the same problem with others days...

When i scanned it, i obtained the same video but with two different links, example for RAI - 05.02.2017:
- Linea verde - Sicilia - Linea Verde Domenica del - 05.02.2017.mp4
: **External links are only visible to Support Staff****External links are only visible to Support Staff**
and
- Linea verde - Sicilia - Linea Verde Domenica del - 05.02.2017.mp4
: **External links are only visible to Support Staff****External links are only visible to Support Staff**

[pspzockerscene]

I'm sorry, I cannot reproduce your issues.
For all filename issues, I still need single video urls (NOT direct urls!).

GreeZ psp

[Mistery]

1. Sorry : **External links are only visible to Support Staff****External links are only visible to Support Staff**

2. But this creates a lot of duplicates and for management it is (very much)better that it is then only one type link (therefore as before), please:
or type: **External links are only visible to Support Staff****External links are only visible to Support Staff**
or type: **External links are only visible to Support Staff****External links are only visible to Support Staff**

Thanks

[pspzockerscene]

@Mistery
For me this is the case - I get 2 urls of 1 type:

2017-02-05_raitv_Linea verde - Sicilia - Linea Verde Domenica del 05⁄02⁄2017 - video - RaiPlay_800..mp4

2017-02-05_raitv_Linea verde - Sicilia - Linea Verde Domenica del 05⁄02⁄2017 - video - RaiPlay_1800..mp4

GreeZ pspzockerscene

[Mistery]

Ok but see the directs links obtained...
In my case, i obtained there (as shown above) two types directs links (but it was via "day" scan)
It seems that the type of direct links obtained is different depending on whether you scan a link to the page or via the links "day"

Another important tip for this site is that when you get the direct link to the video, in most cases it works even if it geoblocked ;-)

Thanks

[pspzockerscene]

Everything should be implemented as far as possible.
I was not able to find any remaining issues.

GreeZ psp

[Mistery]

Ok no problem, thanks
I will therefore continue manually for the geoblocked vidéos

[pspzockerscene]

Originally Posted by Mistery

I will therefore continue manually for the geoblocked vidéos

How do you unblock them?

If the Greasemonkey addon does this:
Sorry, I will not implement their API as it uses a third party website - in terms of security this is not a good idea.

GreeZ psp

[Mistery]

Yes i understand, no problem ;-)

But a new problem with "day" links:
Rai 2 don't work
**External links are only visible to Support Staff****External links are only visible to Support Staff**
**External links are only visible to Support Staff****External links are only visible to Support Staff**

[pspzockerscene]

Fixed!
---

[Please read]=>How/when do I get updates?
Read it carefully BEFORE complaining that something would still be not working!
Update cycles depend on your installed JD version!!

---
[Bitte lesen]=>Wie/wann bekomme ich Updates?
Bitte vollständig lesen bevor ihr euch beschwert, dass etwas immernoch nicht funktionieren würde!
Die Updatezyklen variieren je nach installierter JD Version!!

GreeZ pspzockerscene

[Mistery]

Thanks

Just by curiosity: why, while it is the same pages, sometimes there is one of the channels that does not pass ?

[Mistery]

sorry error message but i don't delete it

Originally Posted by pspzockerscene

Sorry, I will not implement their API as it uses a third party website - in terms of security this is not a good idea.

The most important point about not using my API in Jdownloader is not about security, it's the fact that the API usage is monitored and can be blocked any time if abuses are detected, like third-party software using it.

[pspzockerscene]

Originally Posted by Lazza

The most important point about not using my API in Jdownloader is not about security, it's the fact that the API usage is monitored and can be blocked any time if abuses are detected, like third-party software using it.

I do not want to argue here but seriously - how would you want to detect- and block us ... and why?

Anways, reasons agains using "your"(are you the admin, can you prove that??) API:
1. Security - our plugin is about downloading from rai.it - if we now use a third party API just to find the video directurls and it get's hacked --> Boom now it could easily send infected .exe files to our users - why should we take that risc?

2. Why should we use that API if we can do exactly what you do serverside in our plugins?
I don't even exactly understand why the Greasemonkey script uses the API.

3. Finally, if you want you can help us to improve our RAI plugin - we will certainly not use/abuse your API

GreeZ psp

Nothing to argue about, your points make sense.

Originally Posted by pspzockerscene

why?

Because, as you pointed out, Jdownloader can be programmed to work without using external helpers. Also, it is not kind to use another's developer server, owned by a person who pays for the hosting and doesn't get a cent for it, if your program can happily work without it.

Originally Posted by pspzockerscene

are you the admin, can you prove that?

Maybe you should check which email address I used to register my account here. Also, do I really need to prove that I am the developer of my script?

Originally Posted by pspzockerscene

Why should we use that API if we can do exactly what you do serverside in our plugins?

Yes, this is the important point. And, by the way, I am not arguing in favor of using my API. I actually strongly support your choice to avoid doing that!

Originally Posted by pspzockerscene

I don't even exactly understand why the Greasemonkey script uses the API.

Basically, convenience. It's hard to do some stuff in JS while supporting several browser add-ons, e.g. multiple synchronous HTTP requests or UA spoofing. This is stuff that is very easy to do in a Jdownloader plug-in, for instance.

Also, there is more than meets the eye. The server component has been developed and bug-fixed for more than 2 years and it's about 500 lines of Python:

1. Special cases are handled specially (e.g. the Italian regional news programmes have a different subdomain structure than all other videos, but it's not the only exception).
2. Links and programmes lists are cached in a DB featuring roughly 157000 pre-translated URLs.
3. It handles geo-unblocking of videos who can actually be watched even if geo-restricted (but not all of them can, so it tests it as well).
4. It "upscales" the video quality up to 2400 even if the URL resolves to say 800 (but again, only if possible).
5. M3U8 URLs are "converted" to MP4 direct links whenever possible (99% of the time) because saving M3U8 with a browser is complicated (that would not be an issue with Jdownloader).

Originally Posted by pspzockerscene

Finally, if you want you can help us to improve our RAI plugin

I'm not experienced enough in the plug-in architecture of Jdownloader to be a plug-in developer and I honestly don't have much time either, not to mention I used the program in the past and quite rarely.

I joined because I saw someone was suggesting to implement HTTP requests to my server in a Jdownloader plug-in and given how many people use it, that was kinda scary.:rolleyes:

[Mistery]

@Lazza: Help us please and thanks a lot for your script ! :-) :-) :-)

the mediaset links don't work:
eg.: **External links are only visible to Support Staff****External links are only visible to Support Staff**

Interestingly, I saw the URL in my email notification but I cannot see it here.:blink:

Anyways... that works correctly with my script. I guess the Jdownloader plug-in might be using the old CDN selector URL also on new videos...? Honestly, I don't know.

But maybe this snippet of code can give some hint to the plug-in developers:

$url = "**External links are only visible to Support Staff** . $id;
if ($new)
        $url = "**External links are only visible to Support Staff** . $id;

Enjoy.

[pspzockerscene]

@Lazza
First, thanks for your kind response - I agree with all of your points
Also my suggestions to help us was not ment to be with Java code - simply a description of how some RAI things work is fine - you seem to be the one who knows it best

@Mistery
Thanks for reporting - fixed!

@Lazza
1. Yap, our forum link censor system is quite buggy ...
2. Yap, you're right, it works via "cdnsel01.mediaset.net".
Can you tell us how to find out which URL is required for the request?
In your code snippet I can see the boolean value "new" --> How do you decide whether an URL is to be considered "new" or not?

Thanks in advance

GreeZ pspzockerscene

Originally Posted by pspzockerscene

simply a description of how some RAI things work is fine

Yes, if you want to know something just ask. If I know the answer and have time to write a post, I will reply.

Originally Posted by pspzockerscene

Can you tell us how to find out which URL is required for the request?
In your code snippet I can see the boolean value "new" --> How do you decide whether an URL is to be considered "new" or not?

Honestly, I haven't checked if there is a simple rule. It is the default CDN for new URLs. Regarding what "new" means, I don't know. There was a day in the first weeks of the year when they changed it for new uploads.

Probably it's possible to find the exact video ID where the new CDN started to be used using a binary search approach.

I do it the lazy way: I check the Javascript variable called CDN_SELECTOR_URL when the page loads. If it contains mediaset.net I add new to the query string.

[pspzockerscene]

Thanks, I've implemented it the same way you did
---

[Please read]=>How/when do I get updates?
Read it carefully BEFORE complaining that something would still be not working!
Update cycles depend on your installed JD version!!

---
[Bitte lesen]=>Wie/wann bekomme ich Updates?
Bitte vollständig lesen bevor ihr euch beschwert, dass etwas immernoch nicht funktionieren würde!
Die Updatezyklen variieren je nach installierter JD Version!!

GreeZ pspzockerscene