encountered a suspected TLS fingerprint risk control when using the python crawler to send Amazon to modify the zip code interface - ssl

Question

When I use python to carry a cookie to send a simulated request to
https://www.amazon.com/gp/delivery/ajax/address-change.html
a response of \n\n will be returned. But when I use Charles as the middleman agent, the same http message responds normally.
Similarly, when you use nodejs to send a simulation request, you can also get a normal response. I tried three different network request libraries in python, requests, httpx, and aiohttp got the same result.
For the response of \n\n, I locked the problem on the tls handshake package requested by python. After modifying urllib3.util.ssl_.DEFAULT_CIPHERS, it still returns a response of \n\n.
After comparison with Wireshark captures, it is found that in addition to the CIPHERS part, the Signature Algorithm part is also fixed, and the Signature Algorithm parts of the three request libraries are the same, with the way of curl, the way of nodejs, the way of Charles, and the way of Chrome to obtain the TSL Client Hello package. It's not the same.
I want to simulate the Signature Algorithm part of the Python TSL Client Hello package into Chrome. After I DEBUG the source code of the python request library, I found that the SSL Signature Algorithm part control seems to exist in the openssl so file.
This problem is It has troubled me for a long time, hope it can be resolved, thank you very much
import requests
from aiohttp import ClientSession
import httpx
cookies = {
'csm-hit': 'tb:s-B8ZK0QTPQCGWKHY3QDT5|1620287052879&t:1620287054928&adb:adblk_no',
'i18n-prefs': 'USD',
'lc-main': 'en_US',
'session-id': '143-0501748-3847056',
'session-id-time': '2082787201l',
'session-token': 'NxLWWkB7RnpUvmQEl7OcUzk44D9PnlSt/swrqvnSwBvry9WAPSeQt5U2hVCa7IeEEDwj+qzLHwrNhCnA+7pN8H7HELP5WYZuPjtTJ1d8jrTxLueLIQB+wh+3e+1c1vRrfYDa4FTsdm6jN2QR55zq0ybhNJt0jrXCTdlaktZ+e0tHPIjQnCsu1lidMvyOksR+',
'skin': 'noskin',
'sp-cdn': 'L5Z9:CN',
'ubid-main': '134-5202210-0613519',
}
headers = {
'Host': 'www.amazon.com',
'user-agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 UBrowser/6.2.4094.1',
'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
'accept-language': 'zh-CN,zh;q=0.9',
'anti-csrftoken-a2z': 'gBtJDelwICZ60r+pGBgwbzjAf4Wr+LTRIoyWRyMAAAAMAAAAAGC1xeJyYXcAAAAA',
'content-type': 'application/x-www-form-urlencoded;charset=utf-8',
}
data = 'locationType=LOCATION_INPUT&zipCode=90001&storeContext=generic&deviceType=web&pageType=Gateway&actionSource=glow&almBrandId=undefined'
url = 'https://www.amazon.com/gp/delivery/ajax/address-change.html'
# url = 'https://www.python-spider.com/nginx'
your_proxy_url = 'http://127.0.0.1:8888'
# your_proxy_url = ''
#
with httpx.Client(
# http2=True,
# proxies=your_proxy_url,
verify=False) as client:
# This HTTP request will be tunneled instead of forwarded.
response = client.post(url=url, headers=headers, cookies=cookies, data=data)
print(response.status_code)
print(response.text)
# cert='/Users/yangyanhui/lbs/spider/amazon/amazon_cookie_pool/charles-ssl-proxying-certificate.pem'
response = requests.post(url, headers=headers, cookies=cookies, data=data)
print(response.status_code)
print(response.text)
import aiohttp, asyncio
# asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy()) # 加上这一行
async def main(): # aiohttp必须放在异步函数中使用
async with ClientSession(cookies=cookies, headers=headers) as session:
async with session.post(url, data=data,
# proxy=your_proxy_url,
verify_ssl=False) as resp:
print(await resp.text())
print(resp.status)
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
curl -H 'Host: www.amazon.com' -H 'Cookie: csm-hit=tb:s-B8ZK0QTPQCGWKHY3QDT5|1620287052879&t:1620287054928&adb:adblk_no; i18n-prefs=USD; lc-main=en_US; session-id=143-0501748-3847056; session-id-time=2082787201l; session-token=NxLWWkB7RnpUvmQEl7OcUzk44D9PnlSt/swrqvnSwBvry9WAPSeQt5U2hVCa7IeEEDwj+qzLHwrNhCnA+7pN8H7HELP5WYZuPjtTJ1d8jrTxLueLIQB+wh+3e+1c1vRrfYDa4FTsdm6jN2QR55zq0ybhNJt0jrXCTdlaktZ+e0tHPIjQnCsu1lidMvyOksR+; skin=noskin; sp-cdn=L5Z9:CN; ubid-main=134-5202210-0613519' -H 'user-agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 UBrowser/6.2.4094.1' -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' -H 'accept-language: zh-CN,zh;q=0.9' -H 'anti-csrftoken-a2z: gBtJDelwICZ60r+pGBgwbzjAf4Wr+LTRIoyWRyMAAAAMAAAAAGC1xeJyYXcAAAAA' -H 'content-type: application/x-www-form-urlencoded;charset=utf-8' --data-binary "locationType=LOCATION_INPUT&zipCode=90001&storeContext=generic&deviceType=web&pageType=Gateway&actionSource=glow&almBrandId=undefined" --compressed 'https://www.amazon.com/gp/delivery/ajax/address-change.html'

Answer 1

maybe consider using golang which has a package can modify your tls fingerprint