Can anybody please help with the grok pattern for below example of logs?
85.85.85.85 webmail.company.com "CN=First Last/O=Company/C=CZ" [14/Dec/2020:05:58:18 +0100] "GET /mail/User.nsf/iNotes/Proxy/?OpenDocument&Form=s_ReadViewEntries&PresetFields=DBQuotaInfo;1,FolderName;($Inbox),UnreadCountInfo;1,SearchSort;DateD,s_UsingHttps;1,noPI;1&TZType=UTC&Start=1&Count=23&resortdescending=6 HTTP/1.1" 200 2054 "https://webmail.company.com/mail/User.nsf/iNotes/Proxy/?OpenDocument&Form=l_ScriptFrame&l=en&gz&CR&MX&TSF=20170318T181650,92Z&TSX=20180206T185427,18Z&EFF=%2FiNotes%2FForms9_x&charset=UTF-8&charset=UTF-8&KIC&ua=safari&pt" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36" 125 INOTES_LOGIN_ID=First%20Last; Shimmer=SI_TLM:20210209T072811%2C40Z&ST_Counter:3&LAO:mail&SAB:1&CS_TLM:20210209T072831%2C15Z&V_TLM:20210210T080147%2C82Z&DMS:5&ui:X&MOTLM:20210129T113159%2C00Z&DBQS:1503571%2C%207168000%2C%206963200%2C%200%2C%201503571&SPRKL:1&KOSCZ:GTB&FISD:1; INOTES_LOGIN_ID=First%20Last; DWAShared=0; DWAMode=0; INOTES_LOGIN_ID=First%20Last; DWAShared=0; DWAMode=0; LtpaToken2=FpoGJJz33bYLI+CtWy6OlIgoTJouNGEiduvxvQbcN8HRI7K6LThCsb1Dl8CzN72Zi05RGOUmQRMiOQcTk1norKHi6SbkEGI6GlXzjSIweBRSc8c+XPyAwA44PKPbu3WzrPfR0+uoC0sgTPvochvQ/VfPL/sSaqUFoRswRwyI+UeaOwTs/DvKiWLCpiKrVkFk3SmDjrxPBHb/WiL5nDkpp8Dsjjxnlo4vpx7BdOoVNai1jybvHkW28KXxkb21o8SSpmU7ZFdHyZFjDWCYuuCVOx7asV/q4a3lWdxlPfWdPcUguHML+xDmsrMPm6fTUSKeKIKdQEPr6VDmitBi7Z5URIlkRrUyslkTcc28y6fQir3Y20Hc9TmOvwaBlG/ehnpv; LtpaToken=0x4JJ4oWKojdqoz08Ng+MRUkkJq2vYGLGN9lp8HL8FxbD+xnivE7qzCzf92Q6x5OAPOBFRNgxd3Qg225zLwnJFWO0lGeIweH8VDgyWOMImNe6E9z9HBnQAN43vQ2uwtpv3X5E5DN0oLIPKLxAkqsHUDJqJ0SE6NZ6UnfLoR82JyjZVC/s6QEov5DNdpAY/o2Gxh0vWmE+wuQGuCh4mVCIP9KU/dbX4F0Ld9JEExzIpkdzKELibU2Akov0Krv0eWADSV++m/5ECLpaf6N6/VzkZEkt5XoOoL6OD/6ni4zojvo3O+X9Bn7Mdk2MnsQ1AccIohj5eN8Oi81QbD0a9b7jw==; ShimmerS=ET:20210210T114045%2c00Z&R:0&AT:M" "D:/Lotus/Domino/Data/mail/User.nsf"
What I would need is Client IP (85.85.85.85), VirtualHostname (webmail / webmail.company.com) , User (part after CN=, First Last), Time (14/Dec/2020:05:58:18), URL (GET /mail/User.nsf/iNotes/Proxy/?OpenDocument&Form=s_ReadViewEntries&PresetFields=DBQuotaInfo;1,FolderName; ... ) and the Device Info ( "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36" )
I know it should start with below, however I can't get anyhow the User name to proceed with [%{HTTPDATE:timestamp}] and possible next would be "(?:%{WORD:verb} %{NOTSPACE:request} and not sure how to get the Device info.
Any help would be appreciated!
%{IPORHOST:clientip} %{WORD:VirtualHost} ???
Since you have customized your log format, you have to build your own grok to match the log. You can use https://grokdebug.herokuapp.com/ to debug the pattern you're going to use and you can copy some patterns from https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns
Solved:
%{IPORHOST:clientip} %{IPORHOST:destination.domain} "CN=%{DATA:username}" [%{HTTPDATE:apache.access.time}] "(?:%{WORD:http.request.method} %{DATA:url.original} HTTP/%{NUMBER:http.version}|-)?" %{NUMBER:http.response.status_code:long} (?:%{NUMBER:http.response.body.bytes:long}|-) ("%{DATA:http.request.referrer}") ("%{DATA:user_agent.original}")
Related
whatsweb headless using chromedriver only works correctly when used user agent:
chrome_options.add_argument("user-agent=User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36")
then it works, but its asking the qr code even when I already used
options.add_argument(r"user-data-dir
when I go without headless then chromedriver recognize the user data dir, but in the headless its not working, what's the solution ?
I have a simple query. Looking for a word in string field like the following:
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.90 Safari/537.36 2345Explorer/9.2.1.17116
I EXPECT "CHROME" ROWS ONLY AS AN OUTPUT
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Safari/56.0.2924.90 Chrome/537.36 2345Explorer/9.2.1.17116
I EXPECT "SAFARI" ROWS ONLY AS AN OUTPUT
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Explorer/56.0.2924.90 Safari/537.36 2345Chrome/9.2.1.17116
I EXPECT "EXPLORER" ROWS ONLY AS AN OUTPUT
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Explorer/56.0.2924.90 Safari/537.36 2345Chrome/9.2.1.17116
I do not want "CHROME" rows as an output when I query for rows with "CHROME"
Think about Chrome only and chrome = 1
Then if I have 1-2-3, I want to output
If 2-1-3 i don't want to output
If 2-3-1 don't want to output
I want it only when it comes first.
I want to be able to display only field where Chrome appear first, then with another query only field where Safari is first (not this case).
DO you have an idea please? beginning with the following code
SELECT *
FROM user_logins
WHERE user_agent NOT LIKE '%iPhone%'
AND user_agent NOT LIKE '%Linux; Android%'
order BY id DESC
LIMIT 1000
I would regex it, smth like:
so=# with c(s) as (values
('blahChrome/56.0.2924.90 Safari/537.36 2345')
,('blahSafariblahChromeblah')
)
select s ~ 'Chrome.*Safari', s ~ 'Safari.*Chrome',s from c;
?column? | ?column? | s
----------+----------+--------------------------------------------
t | f | blahChrome/56.0.2924.90 Safari/537.36 2345
f | t | blahSafariblahChromeblah
(2 rows)
You can use substring() with a pattern argument to extract the element of the form /#.#.#.#. Then compare it to 'Chrome':
select substring(x from '[^ ]*/[0-9]+[.][0-9]+[.][0-9]+[.][0-9]'),
v.*
from (values ('Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.90 Safari/537.36 2345Explorer/9.2.1.17116'),
('Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Safari/56.0.2924.90 Chrome/537.36 2345Explorer/9.2.1.17116')
) v(x)
where substring(x from '[^ ]*/[0-9]+[.][0-9]+[.][0-9]+[.][0-9]') like 'Chrome/%';
I'm designing some end to end testing for my job, and I've got it up and running using nightwatch.js through chromedriver. However, we're looking to have this run on our servers, and so I wanted to be able to run it using phantomjs. Although the test performs without incident using chromedriver, Phantomjs yields the following error "phantomjs unable to locate item using css selector"
Any ideas? I've scoured the internet for a solution, to no avail.
First, check decates' comment here: https://github.com/nightwatchjs/nightwatch/issues/243#issuecomment-94287511
See how depending on the user-agent info passed from your browser to the site, the site returns different XHTML data? So if you want to use phantomjs, but are okay with it spoofing as a different browser via the user agent, you can configure phantomjs' user-agent capabilities, like this (spoofing Mac Chrome):
"desiredCapabilities": {
"browserName": "phantomjs",
"phantomjs.cli.args" : ["--ignore-ssl-errors=true"],
"phantomjs.page.settings.userAgent" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36"
}
Then your tests should act the same as your other browser. Using any browser you like, you can check the user-agent string that it sends here: http://www.httpuseragent.org/. Here are some other examples:
// Mac Chrome 46
"phantomjs.page.settings.userAgent" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36"
// Windows Chrome 46
"phantomjs.page.settings.userAgent" : "Mozilla/5.0 (Windows NT 6.3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36"
// Mac Firefox 42.0
"phantomjs.page.settings.userAgent" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:42.0) Gecko/20100101 Firefox/42.0"
// Windows Firefox 42.0
"phantomjs.page.settings.userAgent" : "Mozilla/5.0 (Windows NT 6.3; rv:42.0) Gecko/20100101 Firefox/42.0"
// PhantomJS 2.0
"phantomjs.page.settings.userAgent" : "Mozilla/5.0 (Macintosh; Intel Mac OS X) AppleWebKit/538.1 (KHTML, like Gecko) PhantomJS/2.0.0 Safari/538.1"
I sometimes have this effect in the difference browsers, not only phantoms. The reason seemed to be that elements are not loaded at the time of evaluating for one browser (and are loaded for another). You can debug it with checking screenshots at the point of failure.
The solution for me was using waitForElementPresent/Visible.
I'm trying to determine the user agent string for surface RT for testing purposes.
Just do some Google'ing and you will find your answer.
Internet Explorer 10 User-agent string
Internet Explorer 10 on Windows RT:
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; ARM; Trident/6.0)
Update after comment below
The link above also states:
Identifying touch-enabled systemsInternet Explorer 10 introduces the "Touch" UA string token. If this token is present at the end of the UA string, the computer has touch capability, and is running Windows 8 (or later). This UA string will be transmitted on a touch-enabled system running Windows 8. Note Internet Explorer 10 on Windows 7 will never report a UA string with the "Touch" token.
Internet Explorer 10 on Windows RT with Touch enabled:
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; ARM; Trident/6.0; Touch)
Update for Internet Explorer 11
User-agent string changes
Here is what's reported for Internet Explorer 11 on Windows 8.1:
Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko
Here is the string for Internet Explorer 11 on Windows 7:
Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko
If you compare these values to those reported by earlier versions of Internet Explorer, you'll find the following changes:
The compatible ("compatible") and browser ("MSIE") tokens have been removed.
The "like Gecko" token has been added (for consistency with other browsers).
The version of the browser is now reported by a new revision ("rv") token.
I went to a Microsoft retail location yesterday (November 13, 2012) and used IE to browse to http://whatsmyuseragent.com/ in both Metro and Desktop modes.
Here is the user agent given in both cases:
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; ARM; Trident/6.0; Touch)
For those interested. Here is the User Agent string for a Surface Pro (128Gb):
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Win64; x64; Trident/6.0; Touch)
Actually, everyone is wrong. The actual user agent that comes up in metro mode is
Mozilla/4.0 (Compatible; msie 7.0; windows nt 6.2; arm; trident/6.0;
touch; .net4.0e; .net4.0c; tablet PC 2.0; Version).
This is what I get when I visit the whatsmyuseragent site:
Mozilla/5.0 (Windows NT 6.3; Win64; x64; Trident/7.0; Touch; rv:11.0) like Gecko
Hope this helps.
surface RT will run only IE 10.
The User Agent string is
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; ARM; Trident/6.0)
(source: this MSDN blog entry)
This type of information is typically published well in advance of the delivery of the underlying browsers / machines, because of the interest Web Browser manufacturers have in seeing the new browsers well supported by most Web Sites.
UserAgent for devices -
IE desktop - "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727; .NET CLR 3.0.30729; .NET CLR 3.5.30729; InfoPath.3; rv:11.0) like Gecko"
IE Surface Pro - "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; Touch; .NET4.0C; .NET4.0E; Tablet PC 2.0; rv 11.0) like Gecko"
Edge desktop - "Mozilla/5.0 (Windows NT 10.0; Win64; x64; ServiceUI 9) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Edge/15.15063"
Edge surface - "Mozilla/5.0 (Windows NT 10.0; Win64; x64; ServiceUI 13) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/17.17134"
By looking at the above user agents we don't have any clear distinguish between desktop and surface pro for Edge(IE is having Tablet PC check available ). So here to detect the window device first(surface pro is window tablet) and then verify if the device is touch device.
window + touch: true - surface pro
window + touch: false - desktop
isSurface: function () {
// Window device Check
if(!!navigator.userAgent.match(/Win/)) {
// Check if the device is touch
return !!navigator.userAgent.match(/Tablet PC/i) || "ontouchstart" in document.documentElement;
}
}
I have a Javascript object that i am trying to post to the server with XMLHttpRequest() using JSON.stringify(). my code works fine in all major browsers except for Safari (5.1.2). My analysis shows that Safari, is in fact sending the data. i can see the message in the Safari Developer Tools and i see the bytes received in the IIS logs and it seems accurate (48kb) but the WCF function doesn't get the object data. In looking into the wcf logs i see that the content-length is 0 for safari and has a value for chrome. Does anyone have any insight to this issue?
SAFARI:
<httprequest>
<Method>POST</Method>
<QueryString></QueryString>
<WebHeaders>
<Connection>keep-alive</Connection>
<Content-Length>0</Content-Length>
<Content-Type>application/json</Content-Type>
<Accept>*/*</Accept>
<Accept-Encoding>gzip, deflate</Accept-Encoding>
<Accept-Language>en-US</Accept-Language>
<Authorization>Negotiate TlRMTVNTUA</Authorization>
<Cookie>ASP.NET_SessionId=2ynxibj2jovjo345nckpsskm</Cookie>
<Host>localhost</Host>
<Referer>http://localhost/xRMS.Net/PROFILE/Employee.htm?winid=flENz4mLt1TRBTvL&theme=ThemeDevelopment.css</Referer>
<User-Agent>Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.52.7 (KHTML, like Gecko) Version/5.1.2 Safari/534.52.7</User-Agent>
<Origin>http://localhost</Origin>
</WebHeaders>
</httprequest>
CHROME:
<httprequest>
<Method>POST</Method>
<QueryString></QueryString>
<WebHeaders>
<Connection>keep-alive</Connection>
<Content-Length>48822</Content-Length>
<Content-Type>application/json</Content-Type>
<Accept>*/*</Accept>
<Accept-Charset>ISO-8859-1,utf-8;q=0.7,*;q=0.3</Accept-Charset>
<Accept-Encoding>gzip,deflate,sdch</Accept-Encoding>
<Accept-Language>en-US,en;q=0.8</Accept-Language>
<Cookie>ASP.NET_SessionId=gapksa2mmuh3wcrntz32mipw</Cookie>
<Host>localhost</Host>
<Referer>http://localhost/xRMS.Net/PROFILE/Employee.htm?winid=FTWLXL4b8aTWaaaM&theme=ThemeDevelopment.css</Referer>
<User-Agent>Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2</User-Agent>
<Origin>http://localhost</Origin>
</WebHeaders>
</httprequest>