I have a Javascript object that i am trying to post to the server with XMLHttpRequest() using JSON.stringify(). my code works fine in all major browsers except for Safari (5.1.2). My analysis shows that Safari, is in fact sending the data. i can see the message in the Safari Developer Tools and i see the bytes received in the IIS logs and it seems accurate (48kb) but the WCF function doesn't get the object data. In looking into the wcf logs i see that the content-length is 0 for safari and has a value for chrome. Does anyone have any insight to this issue?
SAFARI:
<httprequest>
<Method>POST</Method>
<QueryString></QueryString>
<WebHeaders>
<Connection>keep-alive</Connection>
<Content-Length>0</Content-Length>
<Content-Type>application/json</Content-Type>
<Accept>*/*</Accept>
<Accept-Encoding>gzip, deflate</Accept-Encoding>
<Accept-Language>en-US</Accept-Language>
<Authorization>Negotiate TlRMTVNTUA</Authorization>
<Cookie>ASP.NET_SessionId=2ynxibj2jovjo345nckpsskm</Cookie>
<Host>localhost</Host>
<Referer>http://localhost/xRMS.Net/PROFILE/Employee.htm?winid=flENz4mLt1TRBTvL&theme=ThemeDevelopment.css</Referer>
<User-Agent>Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.52.7 (KHTML, like Gecko) Version/5.1.2 Safari/534.52.7</User-Agent>
<Origin>http://localhost</Origin>
</WebHeaders>
</httprequest>
CHROME:
<httprequest>
<Method>POST</Method>
<QueryString></QueryString>
<WebHeaders>
<Connection>keep-alive</Connection>
<Content-Length>48822</Content-Length>
<Content-Type>application/json</Content-Type>
<Accept>*/*</Accept>
<Accept-Charset>ISO-8859-1,utf-8;q=0.7,*;q=0.3</Accept-Charset>
<Accept-Encoding>gzip,deflate,sdch</Accept-Encoding>
<Accept-Language>en-US,en;q=0.8</Accept-Language>
<Cookie>ASP.NET_SessionId=gapksa2mmuh3wcrntz32mipw</Cookie>
<Host>localhost</Host>
<Referer>http://localhost/xRMS.Net/PROFILE/Employee.htm?winid=FTWLXL4b8aTWaaaM&theme=ThemeDevelopment.css</Referer>
<User-Agent>Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2</User-Agent>
<Origin>http://localhost</Origin>
</WebHeaders>
</httprequest>
Related
whatsweb headless using chromedriver only works correctly when used user agent:
chrome_options.add_argument("user-agent=User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36")
then it works, but its asking the qr code even when I already used
options.add_argument(r"user-data-dir
when I go without headless then chromedriver recognize the user data dir, but in the headless its not working, what's the solution ?
I am trying to scrape one web page from amazon with the help of Scrapy 2.4.1 over shell. Without any prior scraping amazon instantly askes for captcha entries.
I am setting another user agent as only prevention but have never before scraped the page:
scrapy shell -s USER_AGENT="Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36"
Get one page:
>>> fetch('https://www.amazon.de/Eastpak-Provider-Rucksack-Noir-Black/dp/B0815FZ3C6/')
>>> view(response)
Results in a captcha question.
I also tried it with headers:
headers = {"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0", "Accept-Encoding":"gzip, deflate", "Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "DNT":"1","Connection":"close", "Upgrade-Insecure-Requests":"1"}
>>> req = Request("https://www.amazon.de/Eastpak-Provider-Rucksack-Noir-Black/dp/B0815FZ3C6/", headers=headers)
>>> fetch(req)
This also results in a captcha question, while the main page can be scraped in this case.
How does amazon detect that this is a bot and how to prevent that?
what is the difference between the following user agent strings
Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko
Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; .NET4.0E; .NET4.0C; rv:11.0) like Gecko
I am currently trying to debug a JS issue specific to IE and while debugging noticed that there are these 2 varying user agent strings, for windows 10 and IE 11.
Also, additionally, the user agent some times has Gecko/20100101 like in the string Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0, does presence of Gecko/20100101 indicate that its a Desktop always?
Thanks.
Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko
By default, Internet Explorer 11 on Windows 10 sends the above User-Agent string:
Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; .NET4.0E; .NET4.0C;
rv:11.0) like Gecko
By default, Internet Explorer 11 on Windows 8.1 sends the following User-Agent string:
Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko
Internet Explorer 11 continues the IE9 tradition of exposing extensible tokens in the navigator.userAgent property but not sending those tokens in the request header. For instance, by default this property returns the following on IE11/Win8.1:
Mozilla/5.0 (Windows NT 6.3; Trident/7.0; .NET4.0E; .NET4.0C; rv:11.0) like Gecko
The .NET tokens here were pulled from the registry and allow JavaScript to detect that the .NET Framework is installed on the computer. (They’re a bit misleading because Windows 8.1 includes the 4.5 version of the Framework.)
More detail information, please check the User Agent and Internet Explorer 11’s Many User-Agent String.
The UA string of Firefox itself is broken down into four components:
Mozilla/5.0 (platform; rv:geckoversion) Gecko/geckotrail Firefox/firefoxversion
Gecko/geckotrail indicates that the browser is based on Gecko.
On Desktop, geckotrail is the fixed string "20100101".
More detail information about Firefox user agent string, please check the Firefox user agent string reference.
Edit:
[Note] In Windows, Macintosh, Linux, Firefox for Maemo (Nokia N900), Camino on Mac, SeaMonkey on Windows, SeaMonkey on Mac and SeaMonkey on Linux, the Firefox user agent string contains "20100101".
In testing headless chrome I have noticed that it does not transmit the Accept-Language header entry. I have confirmed that it does get sent when there is a visible browser window.
Is there a reason for this and does chrome have an option to require/force it to send these normal values?
To see this, you can fire up Fiddler and type this at the command line:
chrome --headless --incognito --window-size=1920,1080 --disable-gpu --no-sandbox http://www.daringfireball.net/
In the Fiddler inspector (raw view) you'll see this:
GET https://daringfireball.net/css/ie_sucks HTTP/1.1
Host: daringfireball.net
Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/65.0.3325.181 Safari/537.36
Accept: text/css,*/*;q=0.1
Referer: https://daringfireball.net/
Accept-Encoding: gzip, deflate
Whereas running the same command without --headless gives you this:
GET https://daringfireball.net/css/ie_sucks.php HTTP/1.1
Host: daringfireball.net
Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36
Accept: text/css,*/*;q=0.1
Referer: https://daringfireball.net/
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
When running headless mode, set the following as an option:
'--lang=en-GB'
It would appear that Accept-Language is a user-profile header sent only when the browser has a confirmed user specifying a language and as headless has no user it can/does not send that header. This was raised in these posts and looks to be a lacking feature for the foreseeable future:
https://github.com/SeleniumHQ/selenium/issues/4437
https://bugs.chromium.org/p/chromedriver/issues/detail?id=1925
I'm designing some end to end testing for my job, and I've got it up and running using nightwatch.js through chromedriver. However, we're looking to have this run on our servers, and so I wanted to be able to run it using phantomjs. Although the test performs without incident using chromedriver, Phantomjs yields the following error "phantomjs unable to locate item using css selector"
Any ideas? I've scoured the internet for a solution, to no avail.
First, check decates' comment here: https://github.com/nightwatchjs/nightwatch/issues/243#issuecomment-94287511
See how depending on the user-agent info passed from your browser to the site, the site returns different XHTML data? So if you want to use phantomjs, but are okay with it spoofing as a different browser via the user agent, you can configure phantomjs' user-agent capabilities, like this (spoofing Mac Chrome):
"desiredCapabilities": {
"browserName": "phantomjs",
"phantomjs.cli.args" : ["--ignore-ssl-errors=true"],
"phantomjs.page.settings.userAgent" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36"
}
Then your tests should act the same as your other browser. Using any browser you like, you can check the user-agent string that it sends here: http://www.httpuseragent.org/. Here are some other examples:
// Mac Chrome 46
"phantomjs.page.settings.userAgent" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36"
// Windows Chrome 46
"phantomjs.page.settings.userAgent" : "Mozilla/5.0 (Windows NT 6.3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36"
// Mac Firefox 42.0
"phantomjs.page.settings.userAgent" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:42.0) Gecko/20100101 Firefox/42.0"
// Windows Firefox 42.0
"phantomjs.page.settings.userAgent" : "Mozilla/5.0 (Windows NT 6.3; rv:42.0) Gecko/20100101 Firefox/42.0"
// PhantomJS 2.0
"phantomjs.page.settings.userAgent" : "Mozilla/5.0 (Macintosh; Intel Mac OS X) AppleWebKit/538.1 (KHTML, like Gecko) PhantomJS/2.0.0 Safari/538.1"
I sometimes have this effect in the difference browsers, not only phantoms. The reason seemed to be that elements are not loaded at the time of evaluating for one browser (and are loaded for another). You can debug it with checking screenshots at the point of failure.
The solution for me was using waitForElementPresent/Visible.