Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

YouTube feed discovery broken #1468

Open
jtojnar opened this issue Nov 12, 2023 · 1 comment
Open

YouTube feed discovery broken #1468

jtojnar opened this issue Nov 12, 2023 · 1 comment
Labels
misconfigured site spouts

Comments

@jtojnar
Copy link
Member

jtojnar commented Nov 12, 2023

Using YouTube spout with https://www.youtube.com/c/BreakingTaps URL now fails with the following:

2023-11-12 11:02:24error loading feed content: https://www.youtube.com/c/BreakingTaps is invalid XML, likely due to invalid characters. XML error: > required at line 1, column 117

It is caused by YouTube deciding to start redirecting to a consent page:

Logs 
Nov 12 01:25:38 azazel php[1147758]: [2023-11-12T00:25:38.948869+00:00] selfoss.DEBUG: start fetching source "Breaking Taps (id: 463)
Nov 12 01:25:38 azazel php[1147758]: [2023-11-12T00:25:38.948883+00:00] selfoss.DEBUG: spout successfully loaded: spouts\youtube\youtube
Nov 12 01:25:38 azazel php[1147758]: [2023-11-12T00:25:38.948892+00:00] selfoss.DEBUG: fetch content
Nov 12 01:25:39 azazel php[1147758]: [2023-11-12T00:25:39.007424+00:00] selfoss.DEBUG: >>>>>>>>
Nov 12 01:25:39 azazel php[1147758]: GET /c/BreakingTaps HTTP/1.1
Nov 12 01:25:39 azazel php[1147758]: Host: www.youtube.com
Nov 12 01:25:39 azazel php[1147758]: User-Agent: Selfoss/2.20-cdc7f3e (+https://selfoss.aditu.de)
Nov 12 01:25:39 azazel php[1147758]: Referer: https://www.youtube.com/c/BreakingTaps
Nov 12 01:25:39 azazel php[1147758]: Accept: application/atom+xml, application/rss+xml, application/rdf+xml;q=0.9, application/xml;q=0.8, text/xml;q=0.8, text/html;q=0.7, unknown/unknown;q=0.1, application/unknown;q=0.1, */*;q=0.1
Nov 12 01:25:39 azazel php[1147758]: <<<<<<<<
Nov 12 01:25:39 azazel php[1147758]: HTTP/1.1 302 Found
Nov 12 01:25:39 azazel php[1147758]: Content-Type: application/binary
Nov 12 01:25:39 azazel php[1147758]: X-Content-Type-Options: nosniff
Nov 12 01:25:39 azazel php[1147758]: Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Nov 12 01:25:39 azazel php[1147758]: Pragma: no-cache
Nov 12 01:25:39 azazel php[1147758]: Expires: Mon, 01 Jan 1990 00:00:00 GMT
Nov 12 01:25:39 azazel php[1147758]: Date: Sun, 12 Nov 2023 00:25:38 GMT
Nov 12 01:25:39 azazel php[1147758]: Location: https://consent.youtube.com/m?continue=https%3A%2F%2Fwww.youtube.com%2Fc%2FBreakingTaps%3Fcbrd%3D1&gl=CZ&m=0&pc=yt&cm=2&hl=cs&src=1
Nov 12 01:25:39 azazel php[1147758]: X-Frame-Options: SAMEORIGIN
Nov 12 01:25:39 azazel php[1147758]: Strict-Transport-Security: max-age=31536000
Nov 12 01:25:39 azazel php[1147758]: Report-To: {"group":"youtube_main","max_age":2592000,"endpoints":[{"url":"https://csp.withgoogle.com/csp/report-to/youtube_main"}]}
Nov 12 01:25:39 azazel php[1147758]: Permissions-Policy: ch-ua-arch=*, ch-ua-bitness=*, ch-ua-full-version=*, ch-ua-full-version-list=*, ch-ua-model=*, ch-ua-wow64=*, ch-ua-form-factor=*, ch-ua-platform=*, ch-ua-platform-version=*
Nov 12 01:25:39 azazel php[1147758]: Cross-Origin-Opener-Policy: same-origin-allow-popups; report-to="youtube_main"
Nov 12 01:25:39 azazel php[1147758]: Origin-Trial: AvC9UlR6RDk2crliDsFl66RWLnTbHrDbp+DiY6AYz/PNQ4G4tdUTjrHYr2sghbkhGQAVxb7jaPTHpEVBz0uzQwkAAAB4eyJvcmlnaW4iOiJodHRwczovL3lvdXR1YmUuY29tOjQ0MyIsImZlYXR1cmUiOiJXZWJWaWV3WFJlcXVlc3RlZFdpdGhEZXByZWNhdGlvbiIsImV4cGlyeSI6MTcxOTUzMjc5OSwiaXNTdWJkb21haW4iOnRydWV9
Nov 12 01:25:39 azazel php[1147758]: P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=cs for more info."
Nov 12 01:25:39 azazel php[1147758]: Server: ESF
Nov 12 01:25:39 azazel php[1147758]: Content-Length: 0
Nov 12 01:25:39 azazel php[1147758]: X-XSS-Protection: 0
Nov 12 01:25:39 azazel php[1147758]: Set-Cookie: SOCS=CAAaBgiAq8CqBg; Domain=.youtube.com; Expires=Wed, 11-Dec-2024 00:25:38 GMT; Path=/; Secure; SameSite=lax, YSC=k-IrTaIrtsc; Domain=.youtube.com; Path=/; Secure; HttpOnly; SameSite=none, __Secure-YEC=CgtuTjV2S2diREVOVSiCt8CqBjIICgJDWhICEgA%3D; Domain=.youtube.com; Expires=Wed, 11-Dec-2024 00:25:37 GMT; Path=/; Secure; HttpOnly; SameSite=lax, VISITOR_PRIVACY_METADATA=CgJDWhICEgA%3D; Domain=.youtube.com; Expires=Wed, 11-Dec-2024 00:25:38 GMT; Path=/; Secure; HttpOnly; SameSite=lax, VISITOR_INFO1_LIVE=; Domain=.youtube.com; Expires=Mon, 15-Feb-2021 00:25:38 GMT; Path=/; Secure; HttpOnly; SameSite=none, CONSENT=PENDING+472; expires=Tue, 11-Nov-2025 00:25:38 GMT; path=/; domain=.youtube.com; Secure
Nov 12 01:25:39 azazel php[1147758]: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Nov 12 01:25:39 azazel php[1147758]: --------
Nov 12 01:25:39 azazel php[1147758]: NULL
Nov 12 01:25:39 azazel php[1147758]: [2023-11-12T00:25:39.058897+00:00] selfoss.DEBUG: >>>>>>>>
Nov 12 01:25:39 azazel php[1147758]: GET /m?continue=https%3A%2F%2Fwww.youtube.com%2Fc%2FBreakingTaps%3Fcbrd%3D1&gl=CZ&m=0&pc=yt&cm=2&hl=cs&src=1 HTTP/1.1
Nov 12 01:25:39 azazel php[1147758]: Host: consent.youtube.com
Nov 12 01:25:39 azazel php[1147758]: User-Agent: Selfoss/2.20-cdc7f3e (+https://selfoss.aditu.de)
Nov 12 01:25:39 azazel php[1147758]: Accept: application/atom+xml, application/rss+xml, application/rdf+xml;q=0.9, application/xml;q=0.8, text/xml;q=0.8, text/html;q=0.7, unknown/unknown;q=0.1, application/unknown;q=0.1, */*;q=0.1
Nov 12 01:25:39 azazel php[1147758]: <<<<<<<<
Nov 12 01:25:39 azazel php[1147758]: HTTP/1.1 303 See Other
Nov 12 01:25:39 azazel php[1147758]: Content-Type: application/binary
Nov 12 01:25:39 azazel php[1147758]: Vary: Sec-Fetch-Dest, Sec-Fetch-Mode, Sec-Fetch-Site
Nov 12 01:25:39 azazel php[1147758]: Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Nov 12 01:25:39 azazel php[1147758]: Pragma: no-cache
Nov 12 01:25:39 azazel php[1147758]: Expires: Mon, 01 Jan 1990 00:00:00 GMT
Nov 12 01:25:39 azazel php[1147758]: Date: Sun, 12 Nov 2023 00:25:39 GMT
Nov 12 01:25:39 azazel php[1147758]: Location: https://www.youtube.com/c/BreakingTaps?cbrd=1&ucbcb=1
Nov 12 01:25:39 azazel php[1147758]: Content-Security-Policy: require-trusted-types-for 'script';report-uri /_/ConsentUi/cspreport, script-src 'report-sample' 'nonce-HhJ_V8ae9aH6QoLE0b614Q' 'unsafe-inline';object-src 'none';base-uri 'self';report-uri /_/ConsentUi/cspreport;worker-src 'self'
Nov 12 01:25:39 azazel php[1147758]: Cross-Origin-Opener-Policy: unsafe-none
Nov 12 01:25:39 azazel php[1147758]: Accept-CH: Sec-CH-UA-Arch, Sec-CH-UA-Bitness, Sec-CH-UA-Full-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Model, Sec-CH-UA-WoW64, Sec-CH-UA-Form-Factor, Sec-CH-UA-Platform, Sec-CH-UA-Platform-Version
Nov 12 01:25:39 azazel php[1147758]: Permissions-Policy: ch-ua-arch=*, ch-ua-bitness=*, ch-ua-full-version=*, ch-ua-full-version-list=*, ch-ua-model=*, ch-ua-wow64=*, ch-ua-form-factor=*, ch-ua-platform=*, ch-ua-platform-version=*
Nov 12 01:25:39 azazel php[1147758]: Cross-Origin-Resource-Policy: same-site
Nov 12 01:25:39 azazel php[1147758]: Server: ESF
Nov 12 01:25:39 azazel php[1147758]: Content-Length: 0
Nov 12 01:25:39 azazel php[1147758]: X-XSS-Protection: 0
Nov 12 01:25:39 azazel php[1147758]: X-Frame-Options: SAMEORIGIN
Nov 12 01:25:39 azazel php[1147758]: X-Content-Type-Options: nosniff
Nov 12 01:25:39 azazel php[1147758]: Set-Cookie: CONSENT=PENDING+507; expires=Tue, 11-Nov-2025 00:25:39 GMT; path=/; domain=.youtube.com; Secure
Nov 12 01:25:39 azazel php[1147758]: P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
Nov 12 01:25:39 azazel php[1147758]: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Nov 12 01:25:39 azazel php[1147758]: --------
Nov 12 01:25:39 azazel php[1147758]: NULL
Nov 12 01:25:39 azazel php[1147758]: [2023-11-12T00:25:39.244316+00:00] selfoss.DEBUG: >>>>>>>>
Nov 12 01:25:39 azazel php[1147758]: GET /c/BreakingTaps?cbrd=1&ucbcb=1 HTTP/1.1
Nov 12 01:25:39 azazel php[1147758]: Host: www.youtube.com
Nov 12 01:25:39 azazel php[1147758]: User-Agent: Selfoss/2.20-cdc7f3e (+https://selfoss.aditu.de)
Nov 12 01:25:39 azazel php[1147758]: Accept: application/atom+xml, application/rss+xml, application/rdf+xml;q=0.9, application/xml;q=0.8, text/xml;q=0.8, text/html;q=0.7, unknown/unknown;q=0.1, application/unknown;q=0.1, */*;q=0.1
Nov 12 01:25:39 azazel php[1147758]: <<<<<<<<
Nov 12 01:25:39 azazel php[1147758]: HTTP/1.1 200 OK
Nov 12 01:25:39 azazel php[1147758]: Content-Type: text/html; charset=utf-8
Nov 12 01:25:39 azazel php[1147758]: X-Content-Type-Options: nosniff
Nov 12 01:25:39 azazel php[1147758]: Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Nov 12 01:25:39 azazel php[1147758]: Pragma: no-cache
Nov 12 01:25:39 azazel php[1147758]: Expires: Mon, 01 Jan 1990 00:00:00 GMT
Nov 12 01:25:39 azazel php[1147758]: Date: Sun, 12 Nov 2023 00:25:39 GMT
Nov 12 01:25:39 azazel php[1147758]: X-Frame-Options: SAMEORIGIN
Nov 12 01:25:39 azazel php[1147758]: Strict-Transport-Security: max-age=31536000
Nov 12 01:25:39 azazel php[1147758]: Report-To: {"group":"youtube_main","max_age":2592000,"endpoints":[{"url":"https://csp.withgoogle.com/csp/report-to/youtube_main"}]}
Nov 12 01:25:39 azazel php[1147758]: Permissions-Policy: ch-ua-arch=*, ch-ua-bitness=*, ch-ua-full-version=*, ch-ua-full-version-list=*, ch-ua-model=*, ch-ua-wow64=*, ch-ua-form-factor=*, ch-ua-platform=*, ch-ua-platform-version=*
Nov 12 01:25:39 azazel php[1147758]: Origin-Trial: AvC9UlR6RDk2crliDsFl66RWLnTbHrDbp+DiY6AYz/PNQ4G4tdUTjrHYr2sghbkhGQAVxb7jaPTHpEVBz0uzQwkAAAB4eyJvcmlnaW4iOiJodHRwczovL3lvdXR1YmUuY29tOjQ0MyIsImZlYXR1cmUiOiJXZWJWaWV3WFJlcXVlc3RlZFdpdGhEZXByZWNhdGlvbiIsImV4cGlyeSI6MTcxOTUzMjc5OSwiaXNTdWJkb21haW4iOnRydWV9
Nov 12 01:25:39 azazel php[1147758]: Cross-Origin-Opener-Policy: same-origin-allow-popups; report-to="youtube_main"
Nov 12 01:25:39 azazel php[1147758]: P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=cs for more info."
Nov 12 01:25:39 azazel php[1147758]: Server: ESF
Nov 12 01:25:39 azazel php[1147758]: X-XSS-Protection: 0
Nov 12 01:25:39 azazel php[1147758]: Set-Cookie: YSC=f2hckoP0eKc; Domain=.youtube.com; Path=/; Secure; HttpOnly; SameSite=none, __Secure-YEC=CgsyNHAtTTJIUUQ0USiDt8CqBjIICgJDWhICEgA%3D; Domain=.youtube.com; Expires=Wed, 11-Dec-2024 00:25:38 GMT; Path=/; Secure; HttpOnly; SameSite=lax, VISITOR_PRIVACY_METADATA=CgJDWhICEgA%3D; Domain=.youtube.com; Expires=Wed, 11-Dec-2024 00:25:39 GMT; Path=/; Secure; HttpOnly; SameSite=lax, VISITOR_INFO1_LIVE=; Domain=.youtube.com; Expires=Mon, 15-Feb-2021 00:25:39 GMT; Path=/; Secure; HttpOnly; SameSite=none, CONSENT=PENDING+796; expires=Tue, 11-Nov-2025 00:25:39 GMT; path=/; domain=.youtube.com; Secure
Nov 12 01:25:39 azazel php[1147758]: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Nov 12 01:25:39 azazel php[1147758]: Accept-Ranges: none
Nov 12 01:25:39 azazel php[1147758]: Vary: Accept-Encoding
Nov 12 01:25:39 azazel php[1147758]: Transfer-Encoding: chunked
Nov 12 01:25:39 azazel php[1147758]: --------
Nov 12 01:25:39 azazel php[1147758]: NULL
Nov 12 01:25:39 azazel php[1147758]: [2023-11-12T00:25:39.304459+00:00] selfoss.DEBUG: >>>>>>>>
Nov 12 01:25:39 azazel php[1147758]: GET /c/BreakingTaps HTTP/1.1
Nov 12 01:25:39 azazel php[1147758]: Host: www.youtube.com
Nov 12 01:25:39 azazel php[1147758]: User-Agent: Selfoss/2.20-cdc7f3e (+https://selfoss.aditu.de)
Nov 12 01:25:39 azazel php[1147758]: Referer: https://www.youtube.com/c/BreakingTaps
Nov 12 01:25:39 azazel php[1147758]: Accept: application/atom+xml, application/rss+xml, application/rdf+xml;q=0.9, application/xml;q=0.8, text/xml;q=0.8, text/html;q=0.7, unknown/unknown;q=0.1, application/unknown;q=0.1, */*;q=0.1
Nov 12 01:25:39 azazel php[1147758]: <<<<<<<<
Nov 12 01:25:39 azazel php[1147758]: HTTP/1.1 302 Found
Nov 12 01:25:39 azazel php[1147758]: Content-Type: application/binary
Nov 12 01:25:39 azazel php[1147758]: X-Content-Type-Options: nosniff
Nov 12 01:25:39 azazel php[1147758]: Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Nov 12 01:25:39 azazel php[1147758]: Pragma: no-cache
Nov 12 01:25:39 azazel php[1147758]: Expires: Mon, 01 Jan 1990 00:00:00 GMT
Nov 12 01:25:39 azazel php[1147758]: Date: Sun, 12 Nov 2023 00:25:39 GMT
Nov 12 01:25:39 azazel php[1147758]: Location: https://consent.youtube.com/m?continue=https%3A%2F%2Fwww.youtube.com%2Fc%2FBreakingTaps%3Fcbrd%3D1&gl=CZ&m=0&pc=yt&cm=2&hl=cs&src=1
Nov 12 01:25:39 azazel php[1147758]: Strict-Transport-Security: max-age=31536000
Nov 12 01:25:39 azazel php[1147758]: X-Frame-Options: SAMEORIGIN
Nov 12 01:25:39 azazel php[1147758]: Permissions-Policy: ch-ua-arch=*, ch-ua-bitness=*, ch-ua-full-version=*, ch-ua-full-version-list=*, ch-ua-model=*, ch-ua-wow64=*, ch-ua-form-factor=*, ch-ua-platform=*, ch-ua-platform-version=*
Nov 12 01:25:39 azazel php[1147758]: Cross-Origin-Opener-Policy: same-origin-allow-popups; report-to="youtube_main"
Nov 12 01:25:39 azazel php[1147758]: Origin-Trial: AvC9UlR6RDk2crliDsFl66RWLnTbHrDbp+DiY6AYz/PNQ4G4tdUTjrHYr2sghbkhGQAVxb7jaPTHpEVBz0uzQwkAAAB4eyJvcmlnaW4iOiJodHRwczovL3lvdXR1YmUuY29tOjQ0MyIsImZlYXR1cmUiOiJXZWJWaWV3WFJlcXVlc3RlZFdpdGhEZXByZWNhdGlvbiIsImV4cGlyeSI6MTcxOTUzMjc5OSwiaXNTdWJkb21haW4iOnRydWV9
Nov 12 01:25:39 azazel php[1147758]: Report-To: {"group":"youtube_main","max_age":2592000,"endpoints":[{"url":"https://csp.withgoogle.com/csp/report-to/youtube_main"}]}
Nov 12 01:25:39 azazel php[1147758]: P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=cs for more info."
Nov 12 01:25:39 azazel php[1147758]: Server: ESF
Nov 12 01:25:39 azazel php[1147758]: Content-Length: 0
Nov 12 01:25:39 azazel php[1147758]: X-XSS-Protection: 0
Nov 12 01:25:39 azazel php[1147758]: Set-Cookie: SOCS=CAAaBgiAq8CqBg; Domain=.youtube.com; Expires=Wed, 11-Dec-2024 00:25:39 GMT; Path=/; Secure; SameSite=lax, YSC=DkWZnYEFvcs; Domain=.youtube.com; Path=/; Secure; HttpOnly; SameSite=none, __Secure-YEC=CgtOVnZKTEgtZHFRUSiDt8CqBjIICgJDWhICEgA%3D; Domain=.youtube.com; Expires=Wed, 11-Dec-2024 00:25:38 GMT; Path=/; Secure; HttpOnly; SameSite=lax, VISITOR_PRIVACY_METADATA=CgJDWhICEgA%3D; Domain=.youtube.com; Expires=Wed, 11-Dec-2024 00:25:39 GMT; Path=/; Secure; HttpOnly; SameSite=lax, VISITOR_INFO1_LIVE=; Domain=.youtube.com; Expires=Mon, 15-Feb-2021 00:25:39 GMT; Path=/; Secure; HttpOnly; SameSite=none, CONSENT=PENDING+335; expires=Tue, 11-Nov-2025 00:25:39 GMT; path=/; domain=.youtube.com; Secure
Nov 12 01:25:39 azazel php[1147758]: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Nov 12 01:25:39 azazel php[1147758]: --------
Nov 12 01:25:39 azazel php[1147758]: NULL
Nov 12 01:25:39 azazel php[1147758]: [2023-11-12T00:25:39.331475+00:00] selfoss.DEBUG: >>>>>>>>
Nov 12 01:25:39 azazel php[1147758]: GET /m?continue=https%3A%2F%2Fwww.youtube.com%2Fc%2FBreakingTaps%3Fcbrd%3D1&gl=CZ&m=0&pc=yt&cm=2&hl=cs&src=1 HTTP/1.1
Nov 12 01:25:39 azazel php[1147758]: Host: consent.youtube.com
Nov 12 01:25:39 azazel php[1147758]: User-Agent: Selfoss/2.20-cdc7f3e (+https://selfoss.aditu.de)
Nov 12 01:25:39 azazel php[1147758]: Accept: application/atom+xml, application/rss+xml, application/rdf+xml;q=0.9, application/xml;q=0.8, text/xml;q=0.8, text/html;q=0.7, unknown/unknown;q=0.1, application/unknown;q=0.1, */*;q=0.1
Nov 12 01:25:39 azazel php[1147758]: <<<<<<<<
Nov 12 01:25:39 azazel php[1147758]: HTTP/1.1 303 See Other
Nov 12 01:25:39 azazel php[1147758]: Content-Type: application/binary
Nov 12 01:25:39 azazel php[1147758]: Vary: Sec-Fetch-Dest, Sec-Fetch-Mode, Sec-Fetch-Site
Nov 12 01:25:39 azazel php[1147758]: Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Nov 12 01:25:39 azazel php[1147758]: Pragma: no-cache
Nov 12 01:25:39 azazel php[1147758]: Expires: Mon, 01 Jan 1990 00:00:00 GMT
Nov 12 01:25:39 azazel php[1147758]: Date: Sun, 12 Nov 2023 00:25:39 GMT
Nov 12 01:25:39 azazel php[1147758]: Location: https://www.youtube.com/c/BreakingTaps?cbrd=1&ucbcb=1
Nov 12 01:25:39 azazel php[1147758]: Content-Security-Policy: script-src 'report-sample' 'nonce-Pn2ZNms9ulOluE0dBy6lcQ' 'unsafe-inline';object-src 'none';base-uri 'self';report-uri /_/ConsentUi/cspreport;worker-src 'self', require-trusted-types-for 'script';report-uri /_/ConsentUi/cspreport
Nov 12 01:25:39 azazel php[1147758]: Cross-Origin-Resource-Policy: same-site
Nov 12 01:25:39 azazel php[1147758]: Cross-Origin-Opener-Policy: unsafe-none
Nov 12 01:25:39 azazel php[1147758]: Permissions-Policy: ch-ua-arch=*, ch-ua-bitness=*, ch-ua-full-version=*, ch-ua-full-version-list=*, ch-ua-model=*, ch-ua-wow64=*, ch-ua-form-factor=*, ch-ua-platform=*, ch-ua-platform-version=*
Nov 12 01:25:39 azazel php[1147758]: Accept-CH: Sec-CH-UA-Arch, Sec-CH-UA-Bitness, Sec-CH-UA-Full-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Model, Sec-CH-UA-WoW64, Sec-CH-UA-Form-Factor, Sec-CH-UA-Platform, Sec-CH-UA-Platform-Version
Nov 12 01:25:39 azazel php[1147758]: Server: ESF
Nov 12 01:25:39 azazel php[1147758]: Content-Length: 0
Nov 12 01:25:39 azazel php[1147758]: X-XSS-Protection: 0
Nov 12 01:25:39 azazel php[1147758]: X-Frame-Options: SAMEORIGIN
Nov 12 01:25:39 azazel php[1147758]: X-Content-Type-Options: nosniff
Nov 12 01:25:39 azazel php[1147758]: Set-Cookie: CONSENT=PENDING+657; expires=Tue, 11-Nov-2025 00:25:39 GMT; path=/; domain=.youtube.com; Secure
Nov 12 01:25:39 azazel php[1147758]: P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
Nov 12 01:25:39 azazel php[1147758]: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Nov 12 01:25:39 azazel php[1147758]: --------
Nov 12 01:25:39 azazel php[1147758]: NULL
Nov 12 01:25:39 azazel php[1147758]: [2023-11-12T00:25:39.479455+00:00] selfoss.DEBUG: >>>>>>>>
Nov 12 01:25:39 azazel php[1147758]: GET /c/BreakingTaps?cbrd=1&ucbcb=1 HTTP/1.1
Nov 12 01:25:39 azazel php[1147758]: Host: www.youtube.com
Nov 12 01:25:39 azazel php[1147758]: User-Agent: Selfoss/2.20-cdc7f3e (+https://selfoss.aditu.de)
Nov 12 01:25:39 azazel php[1147758]: Accept: application/atom+xml, application/rss+xml, application/rdf+xml;q=0.9, application/xml;q=0.8, text/xml;q=0.8, text/html;q=0.7, unknown/unknown;q=0.1, application/unknown;q=0.1, */*;q=0.1
Nov 12 01:25:39 azazel php[1147758]: <<<<<<<<
Nov 12 01:25:39 azazel php[1147758]: HTTP/1.1 200 OK
Nov 12 01:25:39 azazel php[1147758]: Content-Type: text/html; charset=utf-8
Nov 12 01:25:39 azazel php[1147758]: X-Content-Type-Options: nosniff
Nov 12 01:25:39 azazel php[1147758]: Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Nov 12 01:25:39 azazel php[1147758]: Pragma: no-cache
Nov 12 01:25:39 azazel php[1147758]: Expires: Mon, 01 Jan 1990 00:00:00 GMT
Nov 12 01:25:39 azazel php[1147758]: Date: Sun, 12 Nov 2023 00:25:39 GMT
Nov 12 01:25:39 azazel php[1147758]: X-Frame-Options: SAMEORIGIN
Nov 12 01:25:39 azazel php[1147758]: Strict-Transport-Security: max-age=31536000
Nov 12 01:25:39 azazel php[1147758]: Permissions-Policy: ch-ua-arch=*, ch-ua-bitness=*, ch-ua-full-version=*, ch-ua-full-version-list=*, ch-ua-model=*, ch-ua-wow64=*, ch-ua-form-factor=*, ch-ua-platform=*, ch-ua-platform-version=*
Nov 12 01:25:39 azazel php[1147758]: Cross-Origin-Opener-Policy: same-origin-allow-popups; report-to="youtube_main"
Nov 12 01:25:39 azazel php[1147758]: Origin-Trial: AvC9UlR6RDk2crliDsFl66RWLnTbHrDbp+DiY6AYz/PNQ4G4tdUTjrHYr2sghbkhGQAVxb7jaPTHpEVBz0uzQwkAAAB4eyJvcmlnaW4iOiJodHRwczovL3lvdXR1YmUuY29tOjQ0MyIsImZlYXR1cmUiOiJXZWJWaWV3WFJlcXVlc3RlZFdpdGhEZXByZWNhdGlvbiIsImV4cGlyeSI6MTcxOTUzMjc5OSwiaXNTdWJkb21haW4iOnRydWV9
Nov 12 01:25:39 azazel php[1147758]: Report-To: {"group":"youtube_main","max_age":2592000,"endpoints":[{"url":"https://csp.withgoogle.com/csp/report-to/youtube_main"}]}
Nov 12 01:25:39 azazel php[1147758]: P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=cs for more info."
Nov 12 01:25:39 azazel php[1147758]: Server: ESF
Nov 12 01:25:39 azazel php[1147758]: X-XSS-Protection: 0
Nov 12 01:25:39 azazel php[1147758]: Set-Cookie: YSC=B9f4hlGdRP0; Domain=.youtube.com; Path=/; Secure; HttpOnly; SameSite=none, __Secure-YEC=CgtKcUdscEtJc2pkdyiDt8CqBjIICgJDWhICEgA%3D; Domain=.youtube.com; Expires=Wed, 11-Dec-2024 00:25:38 GMT; Path=/; Secure; HttpOnly; SameSite=lax, VISITOR_PRIVACY_METADATA=CgJDWhICEgA%3D; Domain=.youtube.com; Expires=Wed, 11-Dec-2024 00:25:39 GMT; Path=/; Secure; HttpOnly; SameSite=lax, VISITOR_INFO1_LIVE=; Domain=.youtube.com; Expires=Mon, 15-Feb-2021 00:25:39 GMT; Path=/; Secure; HttpOnly; SameSite=none, CONSENT=PENDING+051; expires=Tue, 11-Nov-2025 00:25:39 GMT; path=/; domain=.youtube.com; Secure
Nov 12 01:25:39 azazel php[1147758]: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Nov 12 01:25:39 azazel php[1147758]: Accept-Ranges: none
Nov 12 01:25:39 azazel php[1147758]: Vary: Accept-Encoding
Nov 12 01:25:39 azazel php[1147758]: Transfer-Encoding: chunked
Nov 12 01:25:39 azazel php[1147758]: --------
Nov 12 01:25:39 azazel php[1147758]: NULL
Nov 12 01:25:39 azazel php[1147758]: [2023-11-12T00:25:39.484557+00:00] selfoss.ERROR: error loading feed content for Breaking Taps {"exception":"[object] (Exception(code: 0): https://www.youtube.com/c/BreakingTaps is invalid XML, likely due to invalid characters. XML error: > required at line 1, column 117 at /nix/store/zwvd6rdhahqkxgn8nqd9d36zyr79c6pg-selfoss-2.20-cdc7f3e/src/helpers/FeedReader.php:61)
Nov 12 01:25:39 azazel php[1147758]: [stacktrace]
Nov 12 01:25:39 azazel php[1147758]: #0 /nix/store/zwvd6rdhahqkxgn8nqd9d36zyr79c6pg-selfoss-2.20-cdc7f3e/src/spouts/rss/feed.php(63): helpers\\FeedReader->load('https://www.you...')
Nov 12 01:25:39 azazel php[1147758]: #1 /nix/store/zwvd6rdhahqkxgn8nqd9d36zyr79c6pg-selfoss-2.20-cdc7f3e/src/spouts/youtube/youtube.php(50): spouts\\rss\\feed->load(Array)
Nov 12 01:25:39 azazel php[1147758]: #2 /nix/store/zwvd6rdhahqkxgn8nqd9d36zyr79c6pg-selfoss-2.20-cdc7f3e/src/helpers/ContentLoader.php(117): spouts\\youtube\\youtube->load(Array)
Nov 12 01:25:39 azazel php[1147758]: #3 /nix/store/zwvd6rdhahqkxgn8nqd9d36zyr79c6pg-selfoss-2.20-cdc7f3e/src/helpers/ContentLoader.php(56): helpers\\ContentLoader->fetch(Array)
Nov 12 01:25:39 azazel php[1147758]: #4 /nix/store/zwvd6rdhahqkxgn8nqd9d36zyr79c6pg-selfoss-2.20-cdc7f3e/cliupdate.php(23): helpers\\ContentLoader->update(Object(helpers\\UpdateVisitor@anonymous))
Nov 12 01:25:39 azazel php[1147758]: #5 {main}
Nov 12 01:25:39 azazel php[1147758]: "}
@jtojnar
Copy link
Member Author

jtojnar commented Nov 12, 2023

Looks like it is user-agent dependent

$ curl https://www.youtube.com/c/BreakingTaps -H 'User-Agent: Selfoss/2.20-cdc7f3e (+https://selfoss.aditu.de)' 
$ curl --silent https://www.youtube.com/c/BreakingTaps -H 'User-Agent: Selfoss/2.20-cdc7f3e (Googlebot; +https://selfoss.aditu.de)' | rg --only-matching 'rel="alternate".+?href=".+?"'
rel="alternate" media="handheld" href="https://m.youtube.com/c/BreakingTaps"
rel="alternate" media="only screen and (max-width: 640px)" href="https://m.youtube.com/c/BreakingTaps"
rel="alternate" type="application/rss+xml" title="RSS" href="https://www.youtube.com/feeds/videos.xml?channel_id=UC06HVrkOL33D5lLnCPjr6NQ"
rel="alternate" href="android-app://com.google.android.youtube/http/www.youtube.com/channel/UC06HVrkOL33D5lLnCPjr6NQ"
rel="alternate" href="ios-app://544007664/vnd.youtube/www.youtube.com/channel/UC06HVrkOL33D5lLnCPjr6NQ"

Do-Not-Track and Global Privacy Control headers do not appear to help:

$ curl https://www.youtube.com/c/BreakingTaps -H 'User-Agent: Selfoss/2.20-cdc7f3e (+https://selfoss.aditu.de)' -H 'Sec-GPC: 1'
$ curl https://www.youtube.com/c/BreakingTaps -H 'User-Agent: Selfoss/2.20-cdc7f3e (+https://selfoss.aditu.de)' -H 'Do-Not-Track: 1'

But it is possible to set a SOCS=CAI cookie to prevent the redirect:

$ curl --silent https://www.youtube.com/c/BreakingTaps -H 'User-Agent: Selfoss/2.20-cdc7f3e (+https://selfoss.aditu.de)' --cookie 'SOCS=CAI' | rg --only-matching 'rel="alternate".+?href=".+?"'
rel="alternate" media="handheld" href="https://m.youtube.com/c/BreakingTaps"
rel="alternate" media="only screen and (max-width: 640px)" href="https://m.youtube.com/c/BreakingTaps"
rel="alternate" type="application/rss+xml" title="RSS" href="https://www.youtube.com/feeds/videos.xml?channel_id=UC06HVrkOL33D5lLnCPjr6NQ"
rel="alternate" href="android-app://com.google.android.youtube/http/www.youtube.com/channel/UC06HVrkOL33D5lLnCPjr6NQ"
rel="alternate" href="ios-app://544007664/vnd.youtube/www.youtube.com/channel/UC06HVrkOL33D5lLnCPjr6NQ"

Thanks to gpodder/gpodder#1549 for the info.

@jtojnar jtojnar mentioned this issue Jul 3, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
misconfigured site spouts
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jtojnar