JDownloader Community - Appwork GmbH
 

Reply
 
Thread Tools Display Modes
  #1  
Old 01.10.2019, 23:38
yurikus yurikus is offline
JD Alpha
 
Join Date: Jun 2015
Posts: 21
Default Twitch segments with compressed response headers fail to download

URL in question:
Code:
**External links are only visible to Support Staff**
<SEGMENT_ID> sample range: 2900 - 2915

Example:
Code:
**External links are only visible to Support Staff**
JD2 downloader get 403 from CloudFront:
Code:
URL: https://vod-metro.twitch.tv/809481d0548b9f09c374_sips__35835997232_1308432620/chunked/2900.ts
SSLCipher: BC:TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
ConnectIP: vod-metro.twitch.tv/13.249.87.81:443
Connection-Timeout: 20000ms
Read-Timeout: 60000ms
----------------Request-------------------------
GET /809481d0548b9f09c374_sips__35835997232_1308432620/chunked/2900.ts HTTP/1.1
Host: vod-metro.twitch.tv
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:67.0) Gecko/20100101 Firefox/67.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-gb, en;q=0.8
Accept-Encoding: identity
Cache-Control: no-cache
Referer: https://vod-metro.twitch.tv/809481d0548b9f09c374_sips__35835997232_1308432620/chunked/2900.ts

----------------Response Information------------
Connection-Time: keep-Alive
Request-Time: 191ms
----------------Response------------------------
HTTP/1.1 403 Forbidden
Content-Type: text/html
Content-Length: 17
Connection: keep-alive
Last-Modified: Mon, 28 Mar 2016 20:26:38 GMT
ETag: "f2ba37ab539401d32d53e77d7527788e"
x-amz-error-code: AccessDenied
x-amz-error-message: Access Denied
Date: Tue, 01 Oct 2019 20:16:15 GMT
Server: AmazonS3
X-Cache: Error from cloudfront
Via: 1.1 7c79be60fcabd36655f1da4928e2adea.cloudfront.net (CloudFront)
X-Amz-Cf-Pop: ORD52-C1
X-Amz-Cf-Id: 52qPn65SrSCdQgaPoBCM1GKzckVWsuGK6cfttWZMGjJVOhrRZKebCw==
------------------------------------------------

This curl fails:
Code:
curl --output 2900.ts **External links are only visible to Support Staff**

This curl suceeds:
Code:
curl --compressed --output 2900.ts **External links are only visible to Support Staff**
Reply With Quote
  #2  
Old 02.10.2019, 15:13
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 66,134
Default

I'm sorry but the urls are expired (403 with Object not found).
In case you still have the issue and can reproduce it with curl, can you please provide request/response headers from curl? curl -v ....

Waiting for new infos/logs, Thanks!

Edit: Seems like the webserver must explicitly support serving compressed content with 'identity' aka uncompress server-side. I just need example logs/headers to verify this
__________________
JD-Dev & Server-Admin

Last edited by Jiaz; 02.10.2019 at 15:17.
Reply With Quote
  #3  
Old 02.10.2019, 17:29
yurikus yurikus is offline
JD Alpha
 
Join Date: Jun 2015
Posts: 21
Default

Segments are from this video **External links are only visible to Support Staff****External links are only visible to Support Staff**
Reply With Quote
  #4  
Old 02.10.2019, 17:33
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 66,134
Default

Would be great if you could create the debug infos/reproduce the issue. Sorry but I don't find much time to wait to reproduce the issue myself
__________________
JD-Dev & Server-Admin
Reply With Quote
  #5  
Old 02.10.2019, 19:40
yurikus yurikus is offline
JD Alpha
 
Join Date: Jun 2015
Posts: 21
Default

Quote:
Originally Posted by Jiaz View Post
Would be great if you could create the debug infos/reproduce the issue. Sorry but I don't find much time to wait to reproduce the issue myself
No worries, if it happens again, I'll update.
LMK what exactly do you need.
Reply With Quote
  #6  
Old 02.10.2019, 19:42
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 66,134
Default

just try again those curl commands with
curl -v ...
so the output will show the request/response headers, so I can check those
__________________
JD-Dev & Server-Admin
Reply With Quote
  #7  
Old 02.10.2019, 21:57
yurikus yurikus is offline
JD Alpha
 
Join Date: Jun 2015
Posts: 21
Default

I think Twitch either implemented a nasty protection against downloading streams or their browser app deals with absent segments gracefully. Case in point: new stream that ended an hour ago:
Code:
**External links are only visible to Support Staff**
This segment is absent, it's somewhere in the middle of the stream. Browser app doesn't mind, JD2 stops with "Segment not found", mp4 is missing moov and unplayable.

Code:
**External links are only visible to Support Staff**
.
Reply With Quote
  #8  
Old 02.10.2019, 22:01
yurikus yurikus is offline
JD Alpha
 
Join Date: Jun 2015
Posts: 21
Default compressed headers sample

SUCCESS
Code:
curl -Lv --compressed --output 2124.ts https://vod-secure.twitch.tv/e1d16b09f72b8b913cdf_sips__35846269392_1309074795/chunked/6.ts
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 13.249.87.47...
* TCP_NODELAY set
* Connected to vod-secure.twitch.tv (13.249.87.47) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [218 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [70 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [4818 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.twitch.tv
*  start date: Feb  8 00:00:00 2019 GMT
*  expire date: Mar  8 12:00:00 2020 GMT
*  subjectAltName: host "vod-secure.twitch.tv" matched cert's "*.twitch.tv"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x7fffc491ab70)
} [5 bytes data]
> GET /e1d16b09f72b8b913cdf_sips__35846269392_1309074795/chunked/6.ts HTTP/2
> Host: vod-secure.twitch.tv
> User-Agent: curl/7.58.0
> Accept: */*
> Accept-Encoding: deflate, gzip
>
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
} [5 bytes data]
< HTTP/2 200
< content-type: video/MP2T
< content-length: 8018200
< date: Wed, 02 Oct 2019 17:19:17 GMT
< last-modified: Wed, 02 Oct 2019 10:00:59 GMT
< etag: "fd1c476b2f1bbc9f97cac3e6ed906ab7"
< server: AmazonS3
< x-cache: Hit from cloudfront
< via: 1.1 3b3cb210495e4d1678591271d0921206.cloudfront.net (CloudFront)
< x-amz-cf-pop: ORD52-C1
< x-amz-cf-id: aP4tah1qzGZWMwt_neup55dRXFZICTgnoGMUJkxMt-u88UJxQgOtCQ==
< age: 6040
<
{ [16071 bytes data]
100 7830k  100 7830k    0     0  25.7M      0 --:--:-- --:--:-- --:--:-- 25.7M
* Connection #0 to host vod-secure.twitch.tv left intact
FAILURE
Code:
curl -Lv --output 2124.ts https://vod-secure.twitch.tv/e1d16b09f72b8b913cdf_sips__35846269392_1309074795/chunked/6.ts
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 13.249.87.47...
* TCP_NODELAY set
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to vod-secure.twitch.tv (13.249.87.47) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [218 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [70 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [4818 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.twitch.tv
*  start date: Feb  8 00:00:00 2019 GMT
*  expire date: Mar  8 12:00:00 2020 GMT
*  subjectAltName: host "vod-secure.twitch.tv" matched cert's "*.twitch.tv"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x7ffff76efb70)
} [5 bytes data]
> GET /e1d16b09f72b8b913cdf_sips__35846269392_1309074795/chunked/6.ts HTTP/2
> Host: vod-secure.twitch.tv
> User-Agent: curl/7.58.0
> Accept: */*
>
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
} [5 bytes data]
< HTTP/2 403
< content-type: text/html
< content-length: 17
< last-modified: Mon, 28 Mar 2016 20:26:38 GMT
< etag: "f2ba37ab539401d32d53e77d7527788e"
< x-amz-error-code: AccessDenied
< x-amz-error-message: Access Denied
< date: Wed, 02 Oct 2019 18:59:48 GMT
< server: AmazonS3
< x-cache: Error from cloudfront
< via: 1.1 b5d0d50a85cb855c8d83851f71d078e3.cloudfront.net (CloudFront)
< x-amz-cf-pop: ORD52-C1
< x-amz-cf-id: a2gytfnFDFw2YlBlqNNBqKtLz6wvT-Q6C2V75kvh3RFGCsAVZzRXnQ==
<
{ [17 bytes data]
100    17  100    17    0     0     40      0 --:--:-- --:--:-- --:--:--    39
* Connection #0 to host vod-secure.twitch.tv left intact
Reply With Quote
  #9  
Old 02.10.2019, 22:09
yurikus yurikus is offline
JD Alpha
 
Join Date: Jun 2015
Posts: 21
Default

I have a hunch that it is related to the muted copyrighted music sections. Twitch own apps just mute sound (as instructed by JSON). Maybe missing some special-sauce headers?
Reply With Quote
  #10  
Old 02.10.2019, 22:30
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 66,134
Default

Yes, that might be one possible cause. There is also an open ticket for this. Maybe it's possible to *fake* a dummy segment in such a case or hop to next one and ignore the missing one.
Another solution (not yet implemented) is to avoid to use ffmpeg for merging the stream and just download the segments and merge them.
Maybe you can find more streams that have this and then I can look deeper in this.
Also would be great if you can check the *live/chunked* version vs the recoded version. Maybe the recoded works while chunked/live fails?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #11  
Old 02.10.2019, 22:34
yurikus yurikus is offline
JD Alpha
 
Join Date: Jun 2015
Posts: 21
Default

Link (**External links are only visible to Support Staff****External links are only visible to Support Staff**) is to a recorded video, it fails. While stream was going (I caught it half-way) was downloading fine. When stream ended and JD2 was near the end of the download, it stopped with a missing segment. Might be related to the muting post-processing done by twitch and muted segments are at different location. As far as more streams go, last 3 videos from here (**External links are only visible to Support Staff****External links are only visible to Support Staff**) will have this.
Reply With Quote
  #12  
Old 03.10.2019, 10:47
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 66,134
Default

Thanks! in case you stumple upon any shorter stream, would be great
What quality did you try? chunked or recoded?
__________________
JD-Dev & Server-Admin
Reply With Quote
  #13  
Old 03.10.2019, 13:44
yurikus yurikus is offline
JD Alpha
 
Join Date: Jun 2015
Posts: 21
Default

> shorter stream
haha, will do

I didn't realize I can change it, so always defaulted to chunked

UPD: Checked: there's only "Extract Audio Stream" and "Original Video" options available

UPD: I now start to think that Twitch/CloudFront/S3 were having availability issues. Latest stream downloaded just fine.

Last edited by yurikus; 03.10.2019 at 13:49.
Reply With Quote
  #14  
Old 03.10.2019, 21:22
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 66,134
Default

@yurikus: by quality I mean, check filename, when it contains chunked, then it's the live(upload) version of the stream while the streams without chunked are the recorded streams.
__________________
JD-Dev & Server-Admin
Reply With Quote
  #15  
Old 03.10.2019, 21:45
yurikus yurikus is offline
JD Alpha
 
Join Date: Jun 2015
Posts: 21
Default

It is chunked
Reply With Quote
  #16  
Old 03.10.2019, 22:07
Jiaz's Avatar
Jiaz Jiaz is offline
JD Manager
 
Join Date: Mar 2009
Location: Germany
Posts: 66,134
Default

When it fails with chunked, would be great to test with different recoded version and check if it happens as well
__________________
JD-Dev & Server-Admin
Reply With Quote
  #17  
Old 04.10.2019, 00:25
yurikus yurikus is offline
JD Alpha
 
Join Date: Jun 2015
Posts: 21
Default

Not sure if transcoded version exists. This video failed again today. **External links are only visible to Support Staff****External links are only visible to Support Staff**

Interesting effect. I started download when the stream was still running. JD2 downloaded 100% up to the time code when I started it. Stream ended, I restarted the download. Now it failed at 89% with "segment 4 not found". Segment 4 is at 30-40s mark. strange.

I wonder if it's possible to continue downloading past the time code at which JD2 requested list of segments? In other words - we need JD2 to keep asking for more segments until stream is over. This way I can start it (pretty much) at any point and since download speed is greater than streaming speed it will eventually catch up and will slowly follow the new segments when they arrive.

Last edited by yurikus; 04.10.2019 at 00:28.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +2. The time now is 10:34.
Provided By AppWork GmbH | Privacy | Imprint
Parts of the Design are used from Kirsch designed by Andrew & Austin
Powered by vBulletin® Version 3.8.10 Beta 1
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.