Skip to content

[YouTube] Fixes for n param deobfuscation function #1253

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Stypox merged 11 commits into from
Jan 26, 2025
Merged

[YouTube] Fixes for n param deobfuscation function #1253

Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
56595bd
[YouTube] Fix extraction of n param deobfuscation function name and f…
gechoto Dec 29, 2024
68465b2
fix formatting
gechoto Dec 29, 2024
1393644
formatting again
gechoto Dec 29, 2024
e0d39b6
correct comment
gechoto Dec 31, 2024
4181625
add comment to throttling parameter deobfuscation function fixup
gechoto Dec 31, 2024
d007e36
don't commit testing code kids
gechoto Dec 31, 2024
14f7ef1
YoutubeThrottlingParameterUtils: add back second regex and swap it wi…
gechoto Jan 5, 2025
c7d5988
YoutubeThrottlingParameterUtils: swap regex positions again
gechoto Jan 6, 2025
ca9f193
YoutubeThrottlingParameterUtils: make third regex longer to ensure it…
gechoto Jan 6, 2025
2b70b5f
Improve new regex to account for more context
Stypox Jan 26, 2025
b74a8d0
Remove manually writing cardinals of regexes
Stypox Jan 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 65 additions & 18 deletions ...n/java/org/schabi/newpipe/extractor/services/youtube/YoutubeThrottlingParameterUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,33 +30,45 @@ final class YoutubeThrottlingParameterUtils {
private static final Pattern[] DEOBFUSCATION_FUNCTION_NAME_REGEXES = {

/*
* The first regex matches the following text, where we want Wma and the array index
* accessed:
* Matches the following text, where we want SDa and the array index accessed:
*
* a.D&&(b="nn"[+a.D],WL(a),c=a.j[b]||null)&&(c=SDa[0](c),a.set(b,c),SDa.length||Wma("")
*/
Pattern.compile(SINGLE_CHAR_VARIABLE_REGEX + "=\"nn\"\\[\\+" + MULTIPLE_CHARS_REGEX
+ "\\." + MULTIPLE_CHARS_REGEX + "]," + MULTIPLE_CHARS_REGEX + "\\("
+ MULTIPLE_CHARS_REGEX + "\\)," + MULTIPLE_CHARS_REGEX + "="
+ MULTIPLE_CHARS_REGEX + "\\." + MULTIPLE_CHARS_REGEX + "\\["
+ MULTIPLE_CHARS_REGEX + "]\\|\\|null\\).+\\|\\|(" + MULTIPLE_CHARS_REGEX
+ ")\\(\"\"\\)"),
+ MULTIPLE_CHARS_REGEX + "]\\|\\|null\\)&&\\(" + MULTIPLE_CHARS_REGEX + "=("
+ MULTIPLE_CHARS_REGEX + ")" + ARRAY_ACCESS_REGEX),

/*
* The second regex matches the following text, where we want SDa and the array index
* accessed:
* Matches the following text, where we want Wma:
*
* a.D&&(b="nn"[+a.D],WL(a),c=a.j[b]||null)&&(c=SDa[0](c),a.set(b,c),SDa.length||Wma("")
*/
Pattern.compile(SINGLE_CHAR_VARIABLE_REGEX + "=\"nn\"\\[\\+" + MULTIPLE_CHARS_REGEX
+ "\\." + MULTIPLE_CHARS_REGEX + "]," + MULTIPLE_CHARS_REGEX + "\\("
+ MULTIPLE_CHARS_REGEX + "\\)," + MULTIPLE_CHARS_REGEX + "="
+ MULTIPLE_CHARS_REGEX + "\\." + MULTIPLE_CHARS_REGEX + "\\["
+ MULTIPLE_CHARS_REGEX + "]\\|\\|null\\)&&\\(" + MULTIPLE_CHARS_REGEX + "=("
+ MULTIPLE_CHARS_REGEX + ")" + ARRAY_ACCESS_REGEX),
+ MULTIPLE_CHARS_REGEX + "]\\|\\|null\\).+\\|\\|(" + MULTIPLE_CHARS_REGEX
+ ")\\(\"\"\\)"),

/*
* Matches the following text, where we want cvb and the array index accessed:
*
* ,Vb(m),W=m.j[c]||null)&&(W=cvb[0](W),m.set(c,W)
*/
Pattern.compile("," + MULTIPLE_CHARS_REGEX + "\\("
+ MULTIPLE_CHARS_REGEX + "\\)," + MULTIPLE_CHARS_REGEX + "="
+ MULTIPLE_CHARS_REGEX + "\\." + MULTIPLE_CHARS_REGEX + "\\["
+ MULTIPLE_CHARS_REGEX + "]\\|\\|null\\)&&\\(\\b" + MULTIPLE_CHARS_REGEX + "=("
+ MULTIPLE_CHARS_REGEX + ")" + ARRAY_ACCESS_REGEX + "\\("
+ SINGLE_CHAR_VARIABLE_REGEX + "\\)," + MULTIPLE_CHARS_REGEX
+ "\\.set\\((?:\"n+\"|" + MULTIPLE_CHARS_REGEX + ")," + MULTIPLE_CHARS_REGEX
+ "\\)"),

/*
* The third regex matches the following text, where we want rma:
* Matches the following text, where we want rma:
*
* a.D&&(b="nn"[+a.D],c=a.get(b))&&(c=rDa[0](c),a.set(b,c),rDa.length||rma("")
*/
Expand All @@ -66,8 +78,7 @@ final class YoutubeThrottlingParameterUtils {
+ MULTIPLE_CHARS_REGEX + ")\\(\"\"\\)"),

/*
* The fourth regex matches the following text, where we want rDa and the array index
* accessed:
* Matches the following text, where we want rDa and the array index accessed:
*
* a.D&&(b="nn"[+a.D],c=a.get(b))&&(c=rDa[0](c),a.set(b,c),rDa.length||rma("")
*/
Expand All @@ -77,8 +88,7 @@ final class YoutubeThrottlingParameterUtils {
+ MULTIPLE_CHARS_REGEX + "=(" + MULTIPLE_CHARS_REGEX + ")\\[(\\d+)]"),

/*
* The fifth regex matches the following text, where we want BDa and the array index
* accessed:
* Matches the following text, where we want BDa and the array index accessed:
*
* (b=String.fromCharCode(110),c=a.get(b))&&(c=BDa[0](c)
*/
Expand All @@ -89,8 +99,7 @@ final class YoutubeThrottlingParameterUtils {
+ SINGLE_CHAR_VARIABLE_REGEX + "\\)"),

/*
* The sixth regex matches the following text, where we want Yva and the array index
* accessed:
* Matches the following text, where we want Yva and the array index accessed:
*
* .get("n"))&&(b=Yva[0](b)
*/
Expand All @@ -112,6 +121,13 @@ final class YoutubeThrottlingParameterUtils {
private static final String FUNCTION_NAMES_IN_DEOBFUSCATION_ARRAY_REGEX =
"\\s*=\\s*\\[(.+?)][;,]";

private static final String FUNCTION_ARGUMENTS_REGEX =
"=\\s*function\\s*\\(\\s*([^)]*)\\s*\\)";

private static final String EARLY_RETURN_REGEX =
";\\s*if\\s*\\(\\s*typeof\\s+" + MULTIPLE_CHARS_REGEX
+ "+\\s*===?\\s*([\"'])undefined\\1\\s*\\)\\s*return\\s+";

private YoutubeThrottlingParameterUtils() {
}

Expand Down Expand Up @@ -154,19 +170,21 @@ static String getDeobfuscationFunctionName(@Nonnull final String javaScriptPlaye
* Get the throttling parameter deobfuscation code of YouTube's base JavaScript file.
*
* @param javaScriptPlayerCode the complete JavaScript base player code
* @return the throttling parameter deobfuscation function name
* @return the throttling parameter deobfuscation function code
* @throws ParsingException if the throttling parameter deobfuscation code couldn't be
* extracted
*/
@Nonnull
static String getDeobfuscationFunction(@Nonnull final String javaScriptPlayerCode,
@Nonnull final String functionName)
throws ParsingException {
String function;
try {
return parseFunctionWithLexer(javaScriptPlayerCode, functionName);
function = parseFunctionWithLexer(javaScriptPlayerCode, functionName);
} catch (final Exception e) {
return parseFunctionWithRegex(javaScriptPlayerCode, functionName);
function = parseFunctionWithRegex(javaScriptPlayerCode, functionName);
}
return fixupFunction(function);
}

/**
Expand Down Expand Up @@ -214,4 +232,33 @@ private static String validateFunction(@Nonnull final String function) {
JavaScript.compileOrThrow(function);
return function;
}

/**
* Removes an early return statement from the code of the throttling parameter deobfuscation function.
*
* <p>In newer version of the player code the function contains a check for something defined outside of the function.
* If that was not found it will return early.
*
* <p>The check can look like this (JS):<br>
* if(typeof RUQ==="undefined")return p;
*
* <p>In this example RUQ will always be undefined when running the function as standalone.
* If the check is kept it would just return p which is the input parameter and would be wrong.
* For that reason this check and return statement needs to be removed.
*
* @param function the original throttling parameter deobfuscation function code
* @return the throttling parameter deobfuscation function code with the early return statement removed
*/
@Nonnull
private static String fixupFunction(@Nonnull final String function)
Comment thread
AudricV marked this conversation as resolved.
throws Parser.RegexException {
final String firstArgName = Parser
.matchGroup1(FUNCTION_ARGUMENTS_REGEX, function)
.split(",")[0].trim();
final Pattern earlyReturnPattern = Pattern.compile(
EARLY_RETURN_REGEX + firstArgName + ";",
Pattern.DOTALL);
final Matcher earlyReturnCodeMatcher = earlyReturnPattern.matcher(function);
return earlyReturnCodeMatcher.replaceFirst(";");
}
}