I have the code below to initialise a setup of Selenium + Chrome driver + Browsermob Proxy to login to a certain website.
If I just use the setup above, the login fails with HTTP 403. However, if I had another Proxy (Selenium + Chrome driver + Browsermob Proxy + e.g. Fiddler Everywhere, see lines after "to pass requests from proxy server to Fiddler Everywhere") and force the requests from Browsermob Proxy to be proxied by Fiddler Everywhere, then the login works well.
Can anyone provide me a hint on how to debug and/or clear this problem?
I have investigated the headers between all legs:
"Via: 1.1 browsermobproxy" is being added by Browsermob but I replayed a login request after removing "Via" header from Browsermob, and did not have any effect.
BrowserMob is removing "Connection:keep-alive". Could this be the cause (or hint at the cause) ?
I also checked:
The body of the request is not changed across the full chain.
HTTP protocol version does not change across the full chain (HTTP/1.1).
Here is a detailed log of the request without Fiddler:
POST /api/sts/v2/oauth/auth?client_request_id=34634696-b122-40de-b144-ab812a4b3784 HTTP/1.1
Host: www.example.com
Connection: keep-alive
Content-Length: 147
AccountType: Real
X-STS-ClientTime: 2020-08-27T18:10:51
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.135 Safari/537.36
X-STS-AppDomain: https://www.example.com
Content-Type: application/json;charset=UTF-8
Accept: application/json, text/plain, */*
X-STS-GatewayAppId: 90631448-9A01-4860-9FA5-B4EBCDE5EA1D
ApplicationIdentifier: Rexample
ApplicationVersion: 268.0.1
X-DEVICE-ID: 77f281a4-7914-4417-b144-4811998c3e5b
Origin: https://www.example.com
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: cors
Sec-Fetch-Dest: empty
Referer: https://www.example.com/login
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
Cookie: __cfduid=d722b1913f887e5df9b0c740ccef4744a1598548223; exampleLocale=en-gb; TS01047baf=01f1b32d7e1354b9f541c1bab1a81960be72277a638bce5a4865b5b1d5b2e0a9cfddddbcba0d21185b68a7d599391e972d5f065031; __cflb=02DiuEAg8LPSYevHEYiyx9rnANkoz3ETfpNySVtv8fQVr; TMIS2=9a74f2a90237486aab4dcb9d44dec14a0520aaf521b9e87e81f271264e25038c2919d29ee219162aa2ad6bdb68fbe78a86ce282ef510aa77456a3876e4e790d3fdbb0ca31af4d206c67f7b9b5fda93e11b80782ed9095c6a7d8766b4642a69785e7ba9d0a86cd98f3b42c2172a8821bbe6906b6fafc8178f0d742271591193; __cf_bm=50d03de4ce1301af67fa03c59387373fba4d8b3e-1598548241-1800-AZ39AzpjNXxDQ+qFhgfI0OYbMauZ3vXOA1MUejwqJsfE3kOEmEVmKXtsstKg5vZZ5ie/d3yHbgzM8RzzLqqSQEVnd5Vy+l06FUm5Z90kFf54ffUqgq+ywM4EOcGhdP22aetxTezQvFieWgfScPvH0SzqDmFpvrmZJmRuLvnywgcLX14Ys9uA2tMCLWSlDesiQA==; mp_dbbd7bd9566da85f012f7ca5d8c6c944_mixpanel=%7B%22distinct_id%22%3A%20%2217430e66d1679d-015c94d6c30eda-31647304-fa000-17430e66d17591%22%2C%22%24device_id%22%3A%20%2217430e66d1679d-015c94d6c30eda-31647304-fa000-17430e66d17591%22%2C%22%24initial_referrer%22%3A%20%22%24direct%22%2C%22%24initial_referring_domain%22%3A%20%22%24direct%22%7D; _gcl_au=1.1.706805090.1598548242; G_ENABLED_IDPS=google; _uetsid=5c779e7394432399f739848297b9a350; _uetvid=d421e48c4acc1f88b81573014b4fa429; _fbp=fb.1.1598548244752.1489311425; __adal_ses=*; __adal_id=c4b052ee-96b4-4643-b30e-af4af0f7460d.1598548246.1.1598548246.1598548246.2a5a85bd-5415-4275-942c-e02a8048acc2; __adal_ca=so%3Ddirect%26me%3Dnone%26ca%3Ddirect%26co%3D%28not%2520set%29%26ke%3D%28not%2520set%29; __adal_cw=1598548245533; _ga=GA1.2.436171435.1598548249; _gid=GA1.2.1493657885.1598548249; _gat=1; _gat_UA-2056847-65=1
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ClientToProxyConnection.java:183) - (AWAITING_INITIAL) [id: 0x367d8cd0, L:/[0:0:0:0:0:0:0:1]:64108 - R:/[0:0:0:0:0:0:0:1]:64132]: Received raw request: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
POST /api/sts/v2/oauth/auth?client_request_id=34634696-b122-40de-b144-ab812a4b3784 HTTP/1.1
Host: www.example.com
Connection: keep-alive
Content-Length: 147
AccountType: Real
X-STS-ClientTime: 2020-08-27T18:10:51
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.135 Safari/537.36
X-STS-AppDomain: https://www.example.com
Content-Type: application/json;charset=UTF-8
Accept: application/json, text/plain, */*
X-STS-GatewayAppId: 90631448-9A01-4860-9FA5-B4EBCDE5EA1D
ApplicationIdentifier: Rexample
ApplicationVersion: 268.0.1
X-DEVICE-ID: 77f281a4-7914-4417-b144-4811998c3e5b
Origin: https://www.example.com
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: cors
Sec-Fetch-Dest: empty
Referer: https://www.example.com/login
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
Cookie: __cfduid=d722b1913f887e5df9b0c740ccef4744a1598548223; exampleLocale=en-gb; TS01047baf=01f1b32d7e1354b9f541c1bab1a81960be72277a638bce5a4865b5b1d5b2e0a9cfddddbcba0d21185b68a7d599391e972d5f065031; __cflb=02DiuEAg8LPSYevHEYiyx9rnANkoz3ETfpNySVtv8fQVr; TMIS2=9a74f2a90237486aab4dcb9d44dec14a0520aaf521b9e87e81f271264e25038c2919d29ee219162aa2ad6bdb68fbe78a86ce282ef510aa77456a3876e4e790d3fdbb0ca31af4d206c67f7b9b5fda93e11b80782ed9095c6a7d8766b4642a69785e7ba9d0a86cd98f3b42c2172a8821bbe6906b6fafc8178f0d742271591193; __cf_bm=50d03de4ce1301af67fa03c59387373fba4d8b3e-1598548241-1800-AZ39AzpjNXxDQ+qFhgfI0OYbMauZ3vXOA1MUejwqJsfE3kOEmEVmKXtsstKg5vZZ5ie/d3yHbgzM8RzzLqqSQEVnd5Vy+l06FUm5Z90kFf54ffUqgq+ywM4EOcGhdP22aetxTezQvFieWgfScPvH0SzqDmFpvrmZJmRuLvnywgcLX14Ys9uA2tMCLWSlDesiQA==; mp_dbbd7bd9566da85f012f7ca5d8c6c944_mixpanel=%7B%22distinct_id%22%3A%20%2217430e66d1679d-015c94d6c30eda-31647304-fa000-17430e66d17591%22%2C%22%24device_id%22%3A%20%2217430e66d1679d-015c94d6c30eda-31647304-fa000-17430e66d17591%22%2C%22%24initial_referrer%22%3A%20%22%24direct%22%2C%22%24initial_referring_domain%22%3A%20%22%24direct%22%7D; _gcl_au=1.1.706805090.1598548242; G_ENABLED_IDPS=google; _uetsid=5c779e7394432399f739848297b9a350; _uetvid=d421e48c4acc1f88b81573014b4fa429; _fbp=fb.1.1598548244752.1489311425; __adal_ses=*; __adal_id=c4b052ee-96b4-4643-b30e-af4af0f7460d.1598548246.1.1598548246.1598548246.2a5a85bd-5415-4275-942c-e02a8048acc2; __adal_ca=so%3Ddirect%26me%3Dnone%26ca%3Ddirect%26co%3D%28not%2520set%29%26ke%3D%28not%2520set%29; __adal_cw=1598548245533; _ga=GA1.2.436171435.1598548249; _gid=GA1.2.1493657885.1598548249; _gat=1; _gat_UA-2056847-65=1
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ClientToProxyConnection.java:269) - (AWAITING_INITIAL) [id: 0x367d8cd0, L:/[0:0:0:0:0:0:0:1]:64108 - R:/[0:0:0:0:0:0:0:1]:64132]: Ensuring that hostAndPort are available in /api/sts/v2/oauth/auth?client_request_id=34634696-b122-40de-b144-ab812a4b3784
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ClientToProxyConnection.java:281) - (AWAITING_INITIAL) [id: 0x367d8cd0, L:/[0:0:0:0:0:0:0:1]:64108 - R:/[0:0:0:0:0:0:0:1]:64132]: Finding ProxyToServerConnection for: www.example.com
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ClientToProxyConnection.java:331) - (AWAITING_INITIAL) [id: 0x367d8cd0, L:/[0:0:0:0:0:0:0:1]:64108 - R:/[0:0:0:0:0:0:0:1]:64132]: Reusing existing server connection: org.littleshoot.proxy.impl.ProxyToServerConnection#7c392fd2
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ClientToProxyConnection.java:1084) - (AWAITING_INITIAL) [id: 0x367d8cd0, L:/[0:0:0:0:0:0:0:1]:64108 - R:/[0:0:0:0:0:0:0:1]:64132]: Modifying request for proxy chaining
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ClientToProxyConnection.java:1088) - (AWAITING_INITIAL) [id: 0x367d8cd0, L:/[0:0:0:0:0:0:0:1]:64108 - R:/[0:0:0:0:0:0:0:1]:64132]: Stripped host from uri: /api/sts/v2/oauth/auth?client_request_id=34634696-b122-40de-b144-ab812a4b3784 yielding: /api/sts/v2/oauth/auth?client_request_id=34634696-b122-40de-b144-ab812a4b3784
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ClientToProxyConnection.java:1093) - (AWAITING_INITIAL) [id: 0x367d8cd0, L:/[0:0:0:0:0:0:0:1]:64108 - R:/[0:0:0:0:0:0:0:1]:64132]: Modifying request headers for proxying
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ClientToProxyConnection.java:350) - (AWAITING_INITIAL) [id: 0x367d8cd0, L:/[0:0:0:0:0:0:0:1]:64108 - R:/[0:0:0:0:0:0:0:1]:64132]: Writing request to ProxyToServerConnection
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ProxyToServerConnection.java:316) - (AWAITING_INITIAL) [id: 0xba528f83, L:/172.20.10.2:64134 - R:www.example.com/104.18.19.67:443]: Requested write of DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
POST /api/sts/v2/oauth/auth?client_request_id=34634696-b122-40de-b144-ab812a4b3784 HTTP/1.1
Host: www.example.com
Content-Length: 147
AccountType: Real
X-STS-ClientTime: 2020-08-27T18:10:51
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.135 Safari/537.36
X-STS-AppDomain: https://www.example.com
Content-Type: application/json;charset=UTF-8
Accept: application/json, text/plain, */*
X-STS-GatewayAppId: 90631448-9A01-4860-9FA5-B4EBCDE5EA1D
ApplicationIdentifier: Rexample
ApplicationVersion: 268.0.1
X-DEVICE-ID: 77f281a4-7914-4417-b144-4811998c3e5b
Origin: https://www.example.com
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: cors
Sec-Fetch-Dest: empty
Referer: https://www.example.com/login
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
Cookie: __cfduid=d722b1913f887e5df9b0c740ccef4744a1598548223; exampleLocale=en-gb; TS01047baf=01f1b32d7e1354b9f541c1bab1a81960be72277a638bce5a4865b5b1d5b2e0a9cfddddbcba0d21185b68a7d599391e972d5f065031; __cflb=02DiuEAg8LPSYevHEYiyx9rnANkoz3ETfpNySVtv8fQVr; TMIS2=9a74f2a90237486aab4dcb9d44dec14a0520aaf521b9e87e81f271264e25038c2919d29ee219162aa2ad6bdb68fbe78a86ce282ef510aa77456a3876e4e790d3fdbb0ca31af4d206c67f7b9b5fda93e11b80782ed9095c6a7d8766b4642a69785e7ba9d0a86cd98f3b42c2172a8821bbe6906b6fafc8178f0d742271591193; __cf_bm=50d03de4ce1301af67fa03c59387373fba4d8b3e-1598548241-1800-AZ39AzpjNXxDQ+qFhgfI0OYbMauZ3vXOA1MUejwqJsfE3kOEmEVmKXtsstKg5vZZ5ie/d3yHbgzM8RzzLqqSQEVnd5Vy+l06FUm5Z90kFf54ffUqgq+ywM4EOcGhdP22aetxTezQvFieWgfScPvH0SzqDmFpvrmZJmRuLvnywgcLX14Ys9uA2tMCLWSlDesiQA==; mp_dbbd7bd9566da85f012f7ca5d8c6c944_mixpanel=%7B%22distinct_id%22%3A%20%2217430e66d1679d-015c94d6c30eda-31647304-fa000-17430e66d17591%22%2C%22%24device_id%22%3A%20%2217430e66d1679d-015c94d6c30eda-31647304-fa000-17430e66d17591%22%2C%22%24initial_referrer%22%3A%20%22%24direct%22%2C%22%24initial_referring_domain%22%3A%20%22%24direct%22%7D; _gcl_au=1.1.706805090.1598548242; G_ENABLED_IDPS=google; _uetsid=5c779e7394432399f739848297b9a350; _uetvid=d421e48c4acc1f88b81573014b4fa429; _fbp=fb.1.1598548244752.1489311425; __adal_ses=*; __adal_id=c4b052ee-96b4-4643-b30e-af4af0f7460d.1598548246.1.1598548246.1598548246.2a5a85bd-5415-4275-942c-e02a8048acc2; __adal_ca=so%3Ddirect%26me%3Dnone%26ca%3Ddirect%26co%3D%28not%2520set%29%26ke%3D%28not%2520set%29; __adal_cw=1598548245533; _ga=GA1.2.436171435.1598548249; _gid=GA1.2.1493657885.1598548249; _gat=1; _gat_UA-2056847-65=1
Accept-Encoding: gzip, deflate, br
Via: 1.1 browsermobproxy
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ProxyToServerConnection.java:348) - (AWAITING_INITIAL) [id: 0xba528f83, L:/172.20.10.2:64134 - R:www.example.com/104.18.19.67:443]: Using existing connection to: www.example.com/104.18.19.67:443
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ProxyConnection.java:228) - (AWAITING_INITIAL) [id: 0xba528f83, L:/172.20.10.2:64134 - R:www.example.com/104.18.19.67:443]: Writing: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
POST /api/sts/v2/oauth/auth?client_request_id=34634696-b122-40de-b144-ab812a4b3784 HTTP/1.1
Host: www.example.com
Content-Length: 147
AccountType: Real
X-STS-ClientTime: 2020-08-27T18:10:51
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.135 Safari/537.36
X-STS-AppDomain: https://www.example.com
Content-Type: application/json;charset=UTF-8
Accept: application/json, text/plain, */*
X-STS-GatewayAppId: 90631448-9A01-4860-9FA5-B4EBCDE5EA1D
ApplicationIdentifier: Rexample
ApplicationVersion: 268.0.1
X-DEVICE-ID: 77f281a4-7914-4417-b144-4811998c3e5b
Origin: https://www.example.com
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: cors
Sec-Fetch-Dest: empty
Referer: https://www.example.com/login
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
Cookie: __cfduid=d722b1913f887e5df9b0c740ccef4744a1598548223; exampleLocale=en-gb; TS01047baf=01f1b32d7e1354b9f541c1bab1a81960be72277a638bce5a4865b5b1d5b2e0a9cfddddbcba0d21185b68a7d599391e972d5f065031; __cflb=02DiuEAg8LPSYevHEYiyx9rnANkoz3ETfpNySVtv8fQVr; TMIS2=9a74f2a90237486aab4dcb9d44dec14a0520aaf521b9e87e81f271264e25038c2919d29ee219162aa2ad6bdb68fbe78a86ce282ef510aa77456a3876e4e790d3fdbb0ca31af4d206c67f7b9b5fda93e11b80782ed9095c6a7d8766b4642a69785e7ba9d0a86cd98f3b42c2172a8821bbe6906b6fafc8178f0d742271591193; __cf_bm=50d03de4ce1301af67fa03c59387373fba4d8b3e-1598548241-1800-AZ39AzpjNXxDQ+qFhgfI0OYbMauZ3vXOA1MUejwqJsfE3kOEmEVmKXtsstKg5vZZ5ie/d3yHbgzM8RzzLqqSQEVnd5Vy+l06FUm5Z90kFf54ffUqgq+ywM4EOcGhdP22aetxTezQvFieWgfScPvH0SzqDmFpvrmZJmRuLvnywgcLX14Ys9uA2tMCLWSlDesiQA==; mp_dbbd7bd9566da85f012f7ca5d8c6c944_mixpanel=%7B%22distinct_id%22%3A%20%2217430e66d1679d-015c94d6c30eda-31647304-fa000-17430e66d17591%22%2C%22%24device_id%22%3A%20%2217430e66d1679d-015c94d6c30eda-31647304-fa000-17430e66d17591%22%2C%22%24initial_referrer%22%3A%20%22%24direct%22%2C%22%24initial_referring_domain%22%3A%20%22%24direct%22%7D; _gcl_au=1.1.706805090.1598548242; G_ENABLED_IDPS=google; _uetsid=5c779e7394432399f739848297b9a350; _uetvid=d421e48c4acc1f88b81573014b4fa429; _fbp=fb.1.1598548244752.1489311425; __adal_ses=*; __adal_id=c4b052ee-96b4-4643-b30e-af4af0f7460d.1598548246.1.1598548246.1598548246.2a5a85bd-5415-4275-942c-e02a8048acc2; __adal_ca=so%3Ddirect%26me%3Dnone%26ca%3Ddirect%26co%3D%28not%2520set%29%26ke%3D%28not%2520set%29; __adal_cw=1598548245533; _ga=GA1.2.436171435.1598548249; _gid=GA1.2.1493657885.1598548249; _gat=1; _gat_UA-2056847-65=1
Accept-Encoding: gzip, deflate, br
Via: 1.1 browsermobproxy
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ProxyConnection.java:237) - (AWAITING_INITIAL) [id: 0xba528f83, L:/172.20.10.2:64134 - R:www.example.com/104.18.19.67:443]: Wrote: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
POST /api/sts/v2/oauth/auth?client_request_id=34634696-b122-40de-b144-ab812a4b3784 HTTP/1.1
Host: www.example.com
Content-Length: 147
AccountType: Real
X-STS-ClientTime: 2020-08-27T18:10:51
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.135 Safari/537.36
X-STS-AppDomain: https://www.example.com
Content-Type: application/json;charset=UTF-8
Accept: application/json, text/plain, */*
X-STS-GatewayAppId: 90631448-9A01-4860-9FA5-B4EBCDE5EA1D
ApplicationIdentifier: Rexample
ApplicationVersion: 268.0.1
X-DEVICE-ID: 77f281a4-7914-4417-b144-4811998c3e5b
Origin: https://www.example.com
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: cors
Sec-Fetch-Dest: empty
Referer: https://www.example.com/login
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
Cookie: __cfduid=d722b1913f887e5df9b0c740ccef4744a1598548223; exampleLocale=en-gb; TS01047baf=01f1b32d7e1354b9f541c1bab1a81960be72277a638bce5a4865b5b1d5b2e0a9cfddddbcba0d21185b68a7d599391e972d5f065031; __cflb=02DiuEAg8LPSYevHEYiyx9rnANkoz3ETfpNySVtv8fQVr; TMIS2=9a74f2a90237486aab4dcb9d44dec14a0520aaf521b9e87e81f271264e25038c2919d29ee219162aa2ad6bdb68fbe78a86ce282ef510aa77456a3876e4e790d3fdbb0ca31af4d206c67f7b9b5fda93e11b80782ed9095c6a7d8766b4642a69785e7ba9d0a86cd98f3b42c2172a8821bbe6906b6fafc8178f0d742271591193; __cf_bm=50d03de4ce1301af67fa03c59387373fba4d8b3e-1598548241-1800-AZ39AzpjNXxDQ+qFhgfI0OYbMauZ3vXOA1MUejwqJsfE3kOEmEVmKXtsstKg5vZZ5ie/d3yHbgzM8RzzLqqSQEVnd5Vy+l06FUm5Z90kFf54ffUqgq+ywM4EOcGhdP22aetxTezQvFieWgfScPvH0SzqDmFpvrmZJmRuLvnywgcLX14Ys9uA2tMCLWSlDesiQA==; mp_dbbd7bd9566da85f012f7ca5d8c6c944_mixpanel=%7B%22distinct_id%22%3A%20%2217430e66d1679d-015c94d6c30eda-31647304-fa000-17430e66d17591%22%2C%22%24device_id%22%3A%20%2217430e66d1679d-015c94d6c30eda-31647304-fa000-17430e66d17591%22%2C%22%24initial_referrer%22%3A%20%22%24direct%22%2C%22%24initial_referring_domain%22%3A%20%22%24direct%22%7D; _gcl_au=1.1.706805090.1598548242; G_ENABLED_IDPS=google; _uetsid=5c779e7394432399f739848297b9a350; _uetvid=d421e48c4acc1f88b81573014b4fa429; _fbp=fb.1.1598548244752.1489311425; __adal_ses=*; __adal_id=c4b052ee-96b4-4643-b30e-af4af0f7460d.1598548246.1.1598548246.1598548246.2a5a85bd-5415-4275-942c-e02a8048acc2; __adal_ca=so%3Ddirect%26me%3Dnone%26ca%3Ddirect%26co%3D%28not%2520set%29%26ke%3D%28not%2520set%29; __adal_cw=1598548245533; _ga=GA1.2.436171435.1598548249; _gid=GA1.2.1493657885.1598548249; _gat=1; _gat_UA-2056847-65=1
Accept-Encoding: gzip, deflate, br
Via: 1.1 browsermobproxy
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ProxyConnection.java:111) - (AWAITING_CHUNK) [id: 0x367d8cd0, L:/[0:0:0:0:0:0:0:1]:64108 - R:/[0:0:0:0:0:0:0:1]:64132]: Reading: DefaultLastHttpContent(data: SlicedAbstractByteBuf(ridx: 0, widx: 147, cap: 147/147, unwrapped: UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf(ridx: 147, widx: 147, cap: 176)), decoderResult: success)
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ProxyToServerConnection.java:316) - (AWAITING_INITIAL) [id: 0xba528f83, L:/172.20.10.2:64134 - R:www.example.com/104.18.19.67:443]: Requested write of DefaultLastHttpContent(data: SlicedAbstractByteBuf(ridx: 0, widx: 147, cap: 147/147, unwrapped: UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf(ridx: 147, widx: 147, cap: 176)), decoderResult: success)
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ProxyToServerConnection.java:319) - (AWAITING_INITIAL) [id: 0xba528f83, L:/172.20.10.2:64134 - R:www.example.com/104.18.19.67:443]: Retaining reference counted message
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ProxyToServerConnection.java:348) - (AWAITING_INITIAL) [id: 0xba528f83, L:/172.20.10.2:64134 - R:www.example.com/104.18.19.67:443]: Using existing connection to: www.example.com/104.18.19.67:443
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ProxyConnection.java:228) - (AWAITING_INITIAL) [id: 0xba528f83, L:/172.20.10.2:64134 - R:www.example.com/104.18.19.67:443]: Writing: DefaultLastHttpContent(data: SlicedAbstractByteBuf(ridx: 0, widx: 147, cap: 147/147, unwrapped: UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf(ridx: 147, widx: 147, cap: 176)), decoderResult: success)
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ProxyConnection.java:249) - (AWAITING_INITIAL) [id: 0xba528f83, L:/172.20.10.2:64134 - R:www.example.com/104.18.19.67:443]: Writing an empty buffer to signal the end of our chunked transfer
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ProxyConnection.java:237) - (AWAITING_INITIAL) [id: 0xba528f83, L:/172.20.10.2:64134 - R:www.example.com/104.18.19.67:443]: Wrote: DefaultLastHttpContent(data: SlicedAbstractByteBuf(ridx: 0, widx: 147, cap: 147/147, unwrapped: UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf(ridx: 147, widx: 147, cap: 176)), decoderResult: success)
DEBUG [LittleProxy-0-ProxyToServerWorker-0] (ProxyConnection.java:111) - (AWAITING_INITIAL) [id: 0xba528f83, L:/172.20.10.2:64134 - R:www.example.com/104.18.19.67:443]: Reading: DefaultHttpResponse(decodeResult: success, version: HTTP/1.1)
HTTP/1.1 403 Forbidden
Date: Thu, 27 Aug 2020 17:10:51 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
X-Frame-Options: SAMEORIGIN
Cache-Control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires: Thu, 01 Jan 1970 00:00:01 GMT
cf-request-id: 04d27fab5700001bfece21f200000001
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Vary: Accept-Encoding
Strict-Transport-Security: max-age=15552000; includeSubDomains; preload
X-Content-Type-Options: nosniff
Server: cloudflare
CF-RAY: 5c97688bbb9f1bfe-OSL
Content-Encoding: br
alt-svc: h3-27=":443"; ma=86400, h3-28=":443"; ma=86400, h3-29=":443"; ma=86400
DEBUG [LittleProxy-0-ProxyToServerWorker-0] (ProxyToServerConnection.java:220) - (AWAITING_INITIAL) [id: 0xba528f83, L:/172.20.10.2:64134 - R:www.example.com/104.18.19.67:443]: Received raw response: DefaultHttpResponse(decodeResult: success, version: HTTP/1.1)
Thank you!
System.setProperty("webdriver.chrome.driver", "/usr/local/bin/chromedriver");
System.setProperty("webdriver.chrome.silentOutput", "true");
System.setProperty("webdriver.chrome.verboseLogging", "false");
ChromeOptions options = new ChromeOptions();
// route Chrome requests to proxy server
proxy = new BrowserMobProxyServer();
// to pass requests from proxy server to Fiddler Everywhere
// InetSocketAddress inetsockaddr = new InetSocketAddress('localhost', 8866);
// proxy.setTrustAllServers(true);
// proxy.setChainedProxy(inetsockaddr);
// start proxy server
proxy.start(0);
Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
seleniumProxy.setHttpProxy("localhost:"+ proxy.getPort());
seleniumProxy.setSslProxy("localhost:"+ proxy.getPort());
options.setCapability(CapabilityType.PROXY, seleniumProxy);
options.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
options.setCapability (CapabilityType.ACCEPT_INSECURE_CERTS, true);
//options.addArguments("headless");
//options.addArguments("window-size=1200x600");
options.setExperimentalOption("excludeSwitches", Collections.singletonList("enable-automation"));
options.setExperimentalOption("useAutomationExtension", false);
options.addArguments("--disable-blink-features=AutomationControlled");
driver = new ChromeDriver(options);`
Solved this by replacing the JDK, from OpenJDK to Oracle
Im trying to download a simple web page as text using Weblcient but all time i get a problem,
i think the problem in the user-agent but when i set one for the weblclient i get the same problem
the page httpheader Capture :
GET /wp-json/binlist/v1/441442/?_wpnonce=335f68c9e2 HTTP/1.1
Host: binlist.org:443
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Cookie: _ga=GA1.2.1639241798.1540059335
User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.0.1617 Safari/537.36
HTTP/1.1 200
access-control-allow-headers: Authorization, Content-Type
access-control-expose-headers: X-WP-Total, X-WP-TotalPages
allow: GET
alt-svc: quic=":443"; ma=86400; v="43,39"
cache-control: max-age=0
content-encoding: gzip
content-length: 221
content-type: application/json; charset=UTF-8
date: Sat, 22 Jun 2019 10:02:14 GMT
expires: Sat, 22 Jun 2019 10:02:13 GMT
host-header: 192fc2e7e50945beb8231a492d6a8024
link: <https://binlist.org/wp-json></https:>; rel="https://api.w.org/"
server: nginx
set-cookie: wpSGCacheBypass=0; expires=Sat, 22-Jun-2019 09:02:13 GMT; Max- Age=0; path=/
status: 200
vary: Accept-Encoding
x-cache-enabled: True
x-content-type-options: nosniff
x-proxy-cache: MISS
x-robots-tag: noindex
x-wp-nonce: 335f68c9e2
my code :
Private Sub Button4_Click(sender As Object, e As EventArgs) Handles Button4.Click
Dim webClient As New System.Net.WebClient
webClient.Headers("User-Agent") = "Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko Firefox/11.0 (via ggpht.com GoogleImageProxy)"
Dim result As String = WebClient.DownloadString("https://binlist.org/wp-json/binlist/v1/441442/?_wpnonce=a7ddc554d3")
RichTextBox3.Text = result
End Sub
I'm sending a POST request via POSTMAN, but my server (WebServer) is receiving it as a GET request - can anyone think why this can happen? Here is some info from postman (part of the URL/hostnames annonymised):
POST /blaa/DescriptionEntities/description HTTP/1.1
Host: blaaa
Content-Type: application/x-www-form-urlencoded
Cache-Control: no-cache
Postman-Token: 3901995b-78a0-598f-40d2-83b09e09d12e
id=20&description=hi
The code:
#POST
#Path("description")
#Consumes(MediaType.APPLICATION_FORM_URLENCODED)
#Produces({MediaType.APPLICATION_JSON,MediaType.APPLICATION_XML})
public DescriptionEntity createDescriptionEntity(MultivaluedMap<String,String>map){
System.out.println(map.getFirst("id"));
System.out.println(map.getFirst("description"));
return null;
}
The Server (WebSphere):
No operation matching request path "/blaa/DescriptionEntities/description" is found, Relative Path: /description, HTTP Method: GET, ContentType: */*, Accept: */*,. Please enable FINE/TRACE log level for more details-is anyone able to kindly help here.
Not sure how to set the log level here either(I'm new to this).The relative path at the class level is "DescriptionEntities" Thanks so much.
EDIT- this is the whole class (minus import statements-it's completely experimental and ignore non related errors):
#Path("DescriptionEntities")
public class DescriptionEntityResource {
private DescriptionEntityRepository descriptionEntityRepository=new DescriptionEntityRepository();
#Path("description")
#POST
#Consumes(MediaType.APPLICATION_FORM_URLENCODED)
#Produces({MediaType.APPLICATION_JSON,MediaType.APPLICATION_XML})
public DescriptionEntity createDescriptionEntity(MultivaluedMap<String,String>map){
System.out.println(map.getFirst("id"));
System.out.println(map.getFirst("description"));
return null;
}
}
And the URL in Postman: http://blaa/DescriptionEntities/Description
EDIT: Server.xml
<server description="new server">
<!-- Enable features -->
<featureManager>
<feature>jndi-1.0</feature>
<feature>json-1.0</feature>
<feature>localConnector-1.0</feature>
<feature>appSecurity-2.0</feature>
<feature>ssl-1.0</feature>
<feature>servlet-3.1</feature>
<feature>jaxrs-2.0</feature>
<feature>jpa-2.1</feature>
<feature>jsf-2.2</feature>
<feature>jsp-2.3</feature>
</featureManager>
<httpEndpoint host="*" httpPort="80" httpsPort="443" id="defaultHttpEndpoint" sslOptionsRef="endpointSSLOptions"/>
<sslOptions id="endpointSSLOptions" sslRef="endpointSSL"/>
<ssl enabledCiphers="TLS_RSA_WITH_AES_128_CBC_SHA TLS_RSA_WITH_AES_128_CBC_SHA256 " id="endpointSSL" keyStoreRef="customKeyStore"/>
<keyStore id="defaultKeyStore" password="annonymised"/>
<basicRegistry/>
<mimeTypes>
<type>svg=image/svg+xml</type>
</mimeTypes>
<httpSession useContextRootAsCookiePath="true" cookieSecure="true" />
<logging consoleLogLevel="INFO"/>
<applicationMonitor updateTrigger="mbean"/>
<webApplication id="annonymised" location="annonymised" name="annonymised"/>
EDIT: Network traffic (using fiddler):
POST /blaa/DescriptionEntities/description HTTP/1.1
Host: blaa
Connection: keep-alive
Content-Length: 20
Cache-Control: no-cache
Origin: chrome-extension://fhbjgbiflinjbdggehcddcbncdddomop
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8
EDIT: The GET request:
GET /blaa/DescriptionEntities/description HTTP/1.1
Host: blaa
Connection: keep-alive
Cache-Control: no-cache
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
Accept: */*
Accept-Encoding: gzip, deflate, sdch, br
Accept-Language: en-US,en;q=0.8
Cookie: IBMISP=b645ef125eca11e6aeed1341c031a83e- b645ef125eca11e6aeed1341c031a83e-0801740c1b0bc161a63c23e998bba6ec; CoreM_State=83~-1~-1~-1~-1~3~3~5~3~3~7~7~|~~|~~|~~|~||||||~|~~|~~|~~|~~|~~|~~|~~|~; CoreM_State_Content=6~|~~|~|; optimizelyEndUserId=oeu1470813611355r0.3187708418093793; ajs_user_id=null; ajs_group_id=null; amplitude_idibm.com=eyJkZXZpY2VJZCI6Ijk0ZGZlZWE0LTg4NjEtNDMzNi04Yzk5LWMyZWMwNjMxOGZmNFIiLCJ1c2VySWQiOm51bGwsIm9wdE91dCI6ZmFsc2UsInNlc3Npb25JZCI6MTQ3MDgxMzY4MDQ4MCwibGFzdEV2ZW50VGltZSI6MTQ3MDgxMzY4MDQ5MCwiZXZlbnRJZCI6MSwiaWRlbnRpZnlJZCI6MSwic2VxdWVuY2VOdW1iZXIiOjJ9; ajs_anonymous_id=%22b645ef125eca11e6aeed1341c031a83e%22; intercom-id=19922b40-fb41-4744-9dfb-89d6875c0294; UnicaNIODID=enSkfJ02Gh7-Z3dNjvn; optimizelySegments=%7B%222753990332%22%3A%22gc%22%2C%222757810742%22%3A%22search%22%2C%222762380326%22%3A%22false%22%2C%223081321317%22%3A%22none%22%2C%223855873690%22%3A%22referral%22%2C%223859505928%22%3A%22none%22%2C%223860673668%22%3A%22gc%22%2C%223861195110%22%3A%22false%22%7D; optimizelyBuckets=%7B%7D; cvo_sid1=EEHN9V3GC6UB; cvo_tid1=BF-jgeF2_Y4|1470813613|1471260957|0; mmcore.tst=0.794; mmapi.store.p.0=%7B%22mmparams.d%22%3A%7B%7D%2C%22mmparams.p%22%3A%7B%22mmid%22%3A%221502797170504%7C%5C%22716298243%7CCgAAAApStaCTtQ0AAA%3D%3D%5C%22%22%2C%22pd%22%3A%221502797170506%7C%5C%22-1893569528%7CCgAAAAoBQlK1oJO1DfyoSkgDAOpF09EAxdNIDwAAABoXwicJwdNIAAAAAP%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FAAZEaXJlY3QBtQ0DAAAAAAAAAAAAAP%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwAAAAAAAUU%3D%5C%22%22%2C%22srv%22%3A%221502797170510%7C%5C%22ldnvwcgus01%5C%22%22%7D%7D; __auc=f59597cf1567352a7c8a1f92cde; _ga=GA1.2.1447575942.1471939647; CoreID6=67988125510914708136127&ci=50200000|Bluemix_51040000|IBMTESTW3_51040000|HR_50200000|IBM_GlobalMarketing_50200000|devwrkscon_50200000|ESTKCS_50200000|DWNEXT; utag_main=v_id:01567352a47a000fc70d49709b1a04072002b06a00718$_sn:4$_ss:0$_st:1471941446924$dc_visit:4$_pn:7%3Bexp-session$ses_id:1471937020076%3Bexp-session$dc_event:19%3Bexp-session$dc_region:eu-central-1%3Bexp-session$ttd_uuid:f57e787f-a10b-48bc-833c-add2cf0e063d%3Bexp-session; 50200000_clogin=v=1&l=1471937020&e=1471941448777
EDIT: I have purged this little test project for now as I can't figure out what's wrong. I may try another time a little further down the line. Still curious as to what went wrong though so let me know if you spot anything obvious. Thanks everyone
I am trying to add Cache-Control: no-cache, no-store to HTTP response header. I put the following to Web.config:
<configuration>
<system.webServer>
<staticContent>
<clientCache cacheControlMode="DisableCache" />
</staticContent>
</system.webServer>
</configuration>
It yields Pragma: no-cache directive in the header.
CONNECT ...:443 HTTP/1.0
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko
Connection: Keep-Alive
Content-Length: 0
Host: ...
Pragma: no-cache
Is this equivalent to Cache-Control: no-cache? How do change Web.Config to produce Cache-Control: no-cache, no-store in the HTTP header on IIS 8.5? Thanks in advance.
HTTP/1.1 header field values can be
folded onto multiple lines if the
continuation line begins with a space
or horizontal tab. All linear white
space, including folding, has the same
semantics as SP. A recipient MAY
replace any linear white space with a
single SP before interpreting the
field value or forwarding the message
downstream.(quoted from here)
Here's my server side script,which just dumps the cookie content:
var_dump($_COOKIE);exit;
Here comes my test,please pay attention to the cookie part:
GET /logtest.php HTTP/1.1
Host: localhost
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; zh-CN; rv:1.9.2.17) Gecko/20110420 AlexaToolbar/alxf-2.11 Firefox/3.6.17
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: zh-cn,zh;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: GB2312,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Cookie: A=t;
artDate=t
Cache-Control: max-age=0
HTTP/1.1 200 OK
Server: iis/8.0
Date: Mon, 23 May 2011 12:38:00 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive
Keep-Alive: timeout=20
X-Powered-By: PHP/5.3.2
Set-Cookie: ZDEDebuggerPresent=php,phtml,php3; path=/
27
array(1) {
["A"]=>
string(1) "t"
}
0
GET /logtest.php HTTP/1.1
Host: localhost
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; zh-CN; rv:1.9.2.17) Gecko/20110420 AlexaToolbar/alxf-2.11 Firefox/3.6.17
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: zh-cn,zh;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: GB2312,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Cookie: A=t;
artDate=t
Cache-Control: max-age=0
HTTP/1.1 200 OK
Server: iis/8.0
Date: Mon, 23 May 2011 12:38:11 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive
Keep-Alive: timeout=20
X-Powered-By: PHP/5.3.2
Set-Cookie: ZDEDebuggerPresent=php,phtml,php3; path=/
27
array(1) {
["A"]=>
string(1) "t"
}
0
GET /logtest.php HTTP/1.1
Host: localhost
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; zh-CN; rv:1.9.2.17) Gecko/20110420 AlexaToolbar/alxf-2.11 Firefox/3.6.17
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: zh-cn,zh;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: GB2312,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Cookie: A=t;artDate=t
Cache-Control: max-age=0
HTTP/1.1 200 OK
Server: iis/8.0
Date: Mon, 23 May 2011 12:38:55 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive
Keep-Alive: timeout=20
X-Powered-By: PHP/5.3.2
Set-Cookie: ZDEDebuggerPresent=php,phtml,php3; path=/
47
array(2) {
["A"]=>
string(1) "t"
["artDate"]=>
string(1) "t"
}
0
It's a known issue that doesn't have a high priority.