[YouTube] Workaround HTTP 403s on streaming URLs of WEB client (after some time or instantly for some JavaScript players), update clients info#1197
Merged
Stypox merged 7 commits intoTeamNewPipe:devfrom Jul 24, 2024
Conversation
Member
Can't you just use This is the response for an invalid id: I don't have a deleted video to test. |
Stypox
reviewed
Jul 21, 2024
f22ed6d to
e80515c
Compare
…time These changes work around an anti-bot token, for which its requirement is A/B tested on the WEB client. In this test, streaming URLs of this client return HTTP errors 403 if the token is not provided after some time. It also allows to not fetch the JavaScript player for non-age restricted videos, reducing data usage. The TVHTML5 embed client is now only fetched in the case of age-restricted videos. The methods forceFetchAndroidClient and forceFetchIosClient of YoutubeStreamExtractor have been removed, as they are now not needed anymore. These changes also break the extraction of appropriate error states for private and deleted videos and invalid video IDs.
The "blue whale" search query does not return a crisis resource panel anymore, so it was changed to a different word, "suicide".
- Fix typo in folder name of DescriptionTestPewdiepie test; - Fix constant usage of DownloaderTestImpl as download implementation for UnlistedTest and CCLicensed tests.
This commits fixes extraction of the function name decoding the n parameter for HTML5 clients' streaming URLs for YouTube base JavaScript player 3400486c. Two new regexes have been added to the existing ones. All regexes and what they extract has been documented.
This param used to throttle bandwidth of streaming URLs which have this parameter when the correct value is not provided but it is not the case anymore, as the streaming URLs return now an HTTP response code 403 in this case.
e80515c to
d73de6b
Compare
Stypox
approved these changes
Jul 24, 2024
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR works around an anti-bot token, for which its requirement is A/B tested on the WEB client. In this test, streaming URLs of this client return HTTP errors 403 if the token is not provided after some time.
It also allows to not fetch the JavaScript player for non-age restricted videos, reducing data usage. The TVHTML5 embed client is now also only fetched in the case of age-restricted videos.
forceFetchAndroidClientandforceFetchIosClientofYoutubeStreamExtractorhave been removed, as they are now not needed anymore. This is a breaking change for users of these methods.The PR changes also break the extraction of appropriate error states for private and deleted videos and invalid video IDs (an invalid player response message is thrown instead of a content not available one). This should fixed before this PR is merged, but I can't find a solution, so I will need help for that.
Clients info have been updated, this means that mocks have to be updated and several tests unrelated to my changes are failing. I fixed one in this PR (at least for the connection I used to ran the tests),
YoutubeSearchExtractorTest.CrisisResources.Fixes TeamNewPipe/NewPipe#11191.