|
[Declined] Twitch segments with compressed response headers fail to download |
|
Thread Tools | Display Modes |
#1
|
|||
|
|||
Twitch segments with compressed response headers fail to download
URL in question:
Code:
**External links are only visible to Support Staff** Example: Code:
**External links are only visible to Support Staff** Code:
URL: **External links are only visible to Support Staff** SSLCipher: BC:TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 ConnectIP: vod-metro.twitch.tv/13.249.87.81:443 Connection-Timeout: 20000ms Read-Timeout: 60000ms ----------------Request------------------------- GET /809481d0548b9f09c374_sips__35835997232_1308432620/chunked/2900.ts HTTP/1.1 Host: vod-metro.twitch.tv User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:67.0) Gecko/20100101 Firefox/67.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-gb, en;q=0.8 Accept-Encoding: identity Cache-Control: no-cache Referer: **External links are only visible to Support Staff** ----------------Response Information------------ Connection-Time: keep-Alive Request-Time: 191ms ----------------Response------------------------ HTTP/1.1 403 Forbidden Content-Type: text/html Content-Length: 17 Connection: keep-alive Last-Modified: Mon, 28 Mar 2016 20:26:38 GMT ETag: "f2ba37ab539401d32d53e77d7527788e" x-amz-error-code: AccessDenied x-amz-error-message: Access Denied Date: Tue, 01 Oct 2019 20:16:15 GMT Server: AmazonS3 X-Cache: Error from cloudfront Via: 1.1 7c79be60fcabd36655f1da4928e2adea.cloudfront.net (CloudFront) X-Amz-Cf-Pop: ORD52-C1 X-Amz-Cf-Id: 52qPn65SrSCdQgaPoBCM1GKzckVWsuGK6cfttWZMGjJVOhrRZKebCw== ------------------------------------------------ This curl fails: Code:
curl --output 2900.ts **External links are only visible to Support Staff** This curl suceeds: Code:
curl --compressed --output 2900.ts **External links are only visible to Support Staff** |
#2
|
||||
|
||||
I'm sorry but the urls are expired (403 with Object not found).
In case you still have the issue and can reproduce it with curl, can you please provide request/response headers from curl? curl -v .... Waiting for new infos/logs, Thanks! Edit: Seems like the webserver must explicitly support serving compressed content with 'identity' aka uncompress server-side. I just need example logs/headers to verify this
__________________
JD-Dev & Server-Admin Last edited by Jiaz; 02.10.2019 at 14:17. |
#3
|
|||
|
|||
Segments are from this video **External links are only visible to Support Staff****External links are only visible to Support Staff**
|
#4
|
||||
|
||||
Would be great if you could create the debug infos/reproduce the issue. Sorry but I don't find much time to wait to reproduce the issue myself
__________________
JD-Dev & Server-Admin |
#5
|
|||
|
|||
Quote:
LMK what exactly do you need. |
#6
|
||||
|
||||
just try again those curl commands with
curl -v ... so the output will show the request/response headers, so I can check those
__________________
JD-Dev & Server-Admin |
#7
|
|||
|
|||
I think Twitch either implemented a nasty protection against downloading streams or their browser app deals with absent segments gracefully. Case in point: new stream that ended an hour ago:
Code:
**External links are only visible to Support Staff** Code:
**External links are only visible to Support Staff** |
#8
|
|||
|
|||
compressed headers sample
SUCCESS
Code:
curl -Lv --compressed --output 2124.ts **External links are only visible to Support Staff** % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 13.249.87.47... * TCP_NODELAY set * Connected to vod-secure.twitch.tv (13.249.87.47) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/certs/ca-certificates.crt CApath: /etc/ssl/certs } [5 bytes data] * TLSv1.2 (OUT), TLS handshake, Client hello (1): } [218 bytes data] * TLSv1.2 (IN), TLS handshake, Server hello (2): { [70 bytes data] * TLSv1.2 (IN), TLS handshake, Certificate (11): { [4818 bytes data] * TLSv1.2 (IN), TLS handshake, Server key exchange (12): { [333 bytes data] * TLSv1.2 (IN), TLS handshake, Server finished (14): { [4 bytes data] * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): } [70 bytes data] * TLSv1.2 (OUT), TLS change cipher, Client hello (1): } [1 bytes data] * TLSv1.2 (OUT), TLS handshake, Finished (20): } [16 bytes data] * TLSv1.2 (IN), TLS handshake, Finished (20): { [16 bytes data] * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 * ALPN, server accepted to use h2 * Server certificate: * subject: CN=*.twitch.tv * start date: Feb 8 00:00:00 2019 GMT * expire date: Mar 8 12:00:00 2020 GMT * subjectAltName: host "vod-secure.twitch.tv" matched cert's "*.twitch.tv" * issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon * SSL certificate verify ok. * Using HTTP2, server supports multi-use * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 } [5 bytes data] * Using Stream ID: 1 (easy handle 0x7fffc491ab70) } [5 bytes data] > GET /e1d16b09f72b8b913cdf_sips__35846269392_1309074795/chunked/6.ts HTTP/2 > Host: vod-secure.twitch.tv > User-Agent: curl/7.58.0 > Accept: */* > Accept-Encoding: deflate, gzip > { [5 bytes data] * Connection state changed (MAX_CONCURRENT_STREAMS updated)! } [5 bytes data] < HTTP/2 200 < content-type: video/MP2T < content-length: 8018200 < date: Wed, 02 Oct 2019 17:19:17 GMT < last-modified: Wed, 02 Oct 2019 10:00:59 GMT < etag: "fd1c476b2f1bbc9f97cac3e6ed906ab7" < server: AmazonS3 < x-cache: Hit from cloudfront < via: 1.1 3b3cb210495e4d1678591271d0921206.cloudfront.net (CloudFront) < x-amz-cf-pop: ORD52-C1 < x-amz-cf-id: aP4tah1qzGZWMwt_neup55dRXFZICTgnoGMUJkxMt-u88UJxQgOtCQ== < age: 6040 < { [16071 bytes data] 100 7830k 100 7830k 0 0 25.7M 0 --:--:-- --:--:-- --:--:-- 25.7M * Connection #0 to host vod-secure.twitch.tv left intact Code:
curl -Lv --output 2124.ts **External links are only visible to Support Staff** % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 13.249.87.47... * TCP_NODELAY set 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to vod-secure.twitch.tv (13.249.87.47) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/certs/ca-certificates.crt CApath: /etc/ssl/certs } [5 bytes data] * TLSv1.2 (OUT), TLS handshake, Client hello (1): } [218 bytes data] * TLSv1.2 (IN), TLS handshake, Server hello (2): { [70 bytes data] * TLSv1.2 (IN), TLS handshake, Certificate (11): { [4818 bytes data] * TLSv1.2 (IN), TLS handshake, Server key exchange (12): { [333 bytes data] * TLSv1.2 (IN), TLS handshake, Server finished (14): { [4 bytes data] * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): } [70 bytes data] * TLSv1.2 (OUT), TLS change cipher, Client hello (1): } [1 bytes data] * TLSv1.2 (OUT), TLS handshake, Finished (20): } [16 bytes data] * TLSv1.2 (IN), TLS handshake, Finished (20): { [16 bytes data] * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 * ALPN, server accepted to use h2 * Server certificate: * subject: CN=*.twitch.tv * start date: Feb 8 00:00:00 2019 GMT * expire date: Mar 8 12:00:00 2020 GMT * subjectAltName: host "vod-secure.twitch.tv" matched cert's "*.twitch.tv" * issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon * SSL certificate verify ok. * Using HTTP2, server supports multi-use * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 } [5 bytes data] * Using Stream ID: 1 (easy handle 0x7ffff76efb70) } [5 bytes data] > GET /e1d16b09f72b8b913cdf_sips__35846269392_1309074795/chunked/6.ts HTTP/2 > Host: vod-secure.twitch.tv > User-Agent: curl/7.58.0 > Accept: */* > { [5 bytes data] * Connection state changed (MAX_CONCURRENT_STREAMS updated)! } [5 bytes data] < HTTP/2 403 < content-type: text/html < content-length: 17 < last-modified: Mon, 28 Mar 2016 20:26:38 GMT < etag: "f2ba37ab539401d32d53e77d7527788e" < x-amz-error-code: AccessDenied < x-amz-error-message: Access Denied < date: Wed, 02 Oct 2019 18:59:48 GMT < server: AmazonS3 < x-cache: Error from cloudfront < via: 1.1 b5d0d50a85cb855c8d83851f71d078e3.cloudfront.net (CloudFront) < x-amz-cf-pop: ORD52-C1 < x-amz-cf-id: a2gytfnFDFw2YlBlqNNBqKtLz6wvT-Q6C2V75kvh3RFGCsAVZzRXnQ== < { [17 bytes data] 100 17 100 17 0 0 40 0 --:--:-- --:--:-- --:--:-- 39 * Connection #0 to host vod-secure.twitch.tv left intact |
#9
|
|||
|
|||
I have a hunch that it is related to the muted copyrighted music sections. Twitch own apps just mute sound (as instructed by JSON). Maybe missing some special-sauce headers?
|
#10
|
||||
|
||||
Yes, that might be one possible cause. There is also an open ticket for this. Maybe it's possible to *fake* a dummy segment in such a case or hop to next one and ignore the missing one.
Another solution (not yet implemented) is to avoid to use ffmpeg for merging the stream and just download the segments and merge them. Maybe you can find more streams that have this and then I can look deeper in this. Also would be great if you can check the *live/chunked* version vs the recoded version. Maybe the recoded works while chunked/live fails?
__________________
JD-Dev & Server-Admin |
#11
|
|||
|
|||
Link (**External links are only visible to Support Staff****External links are only visible to Support Staff**) is to a recorded video, it fails. While stream was going (I caught it half-way) was downloading fine. When stream ended and JD2 was near the end of the download, it stopped with a missing segment. Might be related to the muting post-processing done by twitch and muted segments are at different location. As far as more streams go, last 3 videos from here (**External links are only visible to Support Staff****External links are only visible to Support Staff**) will have this.
|
#12
|
||||
|
||||
Thanks! in case you stumple upon any shorter stream, would be great
What quality did you try? chunked or recoded?
__________________
JD-Dev & Server-Admin |
#13
|
|||
|
|||
> shorter stream
haha, will do I didn't realize I can change it, so always defaulted to chunked UPD: Checked: there's only "Extract Audio Stream" and "Original Video" options available UPD: I now start to think that Twitch/CloudFront/S3 were having availability issues. Latest stream downloaded just fine. Last edited by yurikus; 03.10.2019 at 12:49. |
#14
|
||||
|
||||
@yurikus: by quality I mean, check filename, when it contains chunked, then it's the live(upload) version of the stream while the streams without chunked are the recorded streams.
__________________
JD-Dev & Server-Admin |
#15
|
|||
|
|||
It is chunked
|
#16
|
||||
|
||||
When it fails with chunked, would be great to test with different recoded version and check if it happens as well
__________________
JD-Dev & Server-Admin |
#17
|
|||
|
|||
Not sure if transcoded version exists. This video failed again today. **External links are only visible to Support Staff****External links are only visible to Support Staff**
Interesting effect. I started download when the stream was still running. JD2 downloaded 100% up to the time code when I started it. Stream ended, I restarted the download. Now it failed at 89% with "segment 4 not found". Segment 4 is at 30-40s mark. strange. I wonder if it's possible to continue downloading past the time code at which JD2 requested list of segments? In other words - we need JD2 to keep asking for more segments until stream is over. This way I can start it (pretty much) at any point and since download speed is greater than streaming speed it will eventually catch up and will slowly follow the new segments when they arrive. Last edited by yurikus; 03.10.2019 at 23:28. |
#18
|
||||
|
||||
Hi,
no. JD is not [yet] (officially) able to download live streams. If that has happened as described, JD did not "know" that this was a running livestream otherwise our plugin would have never attempted the download. Doe this mean, JD would in general try to download twitch livestreams atm.? That would not be our desired behavior ... -psp-
__________________
JD Supporter, Plugin Dev. & Community Manager
Erste Schritte & Tutorials || JDownloader 2 Setup Download |
|
|