Selenium request returns error 400 - Bad request - selenium

Question

I'm trying to get pull data from http://43.248.49.97/indexEn
Normally on the browser there is a first request to the url above returns error 412, the second one is to a JS file and the third one is also to the url above and returns ok (200).
When using selenium the third request returns error 400 - Bad request instead.
I'm using Python. Any ideas on why this is happening?]
Thanks

Answer 1

I had the same problem with you but I found a solution which perfectly solves my problem.
Maybe your program fails because you are detected as a robot using selenium.
So here is the method to solve it or hide your identity(window.navigator.webdriver) by using JavaScript:
With CDP(Chrome Devtools-Protocol), you run the code before the frame is loaded by the JS file(detector). Therefore, use these codes to remove the "webdriver True" property:
Object.defineProperty(navigator, 'webdriver', {
get: () => undefined
})
key codes:
from selenium.webdriver import Chrome
driver = Chrome('D://chromedriver.exe')
driver.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {
"source": """
Object.defineProperty(navigator, 'webdriver', {
get: () => undefined
})
"""
})
driver.get('http://pythonlearner.com')
However, if you upgrade your Chrome to 88 right now. The method mentioned above will be useless. Fortunately, we still have a solution (add this code)
chrome_options.add_argument("--disable-blink-features=AutomationControlled")
These codes should help you pass the JS file so that you can obtain the data. At least I successfully run my program.

Answer 2

I am using ChromeDriver 104.0.5112.79.0 and the solution with chrome_options.add_argument("--disable-blink-features=AutomationControlled") works for me!