I am trying to extract few fields from an event log using rex command and display the fields in a tabular format.
This is my log:
LOG_LEVEL="INFO" MESSAGE="Type_of_Call = Sample Call LOB = F Date/Time_Stamp = 2022-10-10T21:10:53.900129 Policy_Number = 12-AB-1234-5 Requester_Id = A1231301 Last_Name = SAMPLE State = IL City = Chicago Zip 12345" APPLICATION_VERSION="appVersion_IS_UNDEFINED"
Fields that I want to extract are: Type_of_Call, LOB, Date/Time_Stamp, Policy_Number, Requester_Id, Last_Name, State, City, Zip
This is my splunk rex command:
rex field=_raw "Type_of_Call\s*=\s*(?<Type_Of_Call>\w+)\s+Call\s+LOB\s*=\s*(?<LOB>\w+)\s+Date/Time_Stamp\s*=\s*(?<Date_Time_Stamp>[0-9TZ.:-]+)\s+Policy_Number\s*=\s*(?<Policy_Number>[\w-]+)\s+Requestor_Id\s*=\s*(?<Requestor_Id>\w+)\s+Last_Name\s*=\s*(\w+)\s+State\s*=\s*(?<State>\w+)"
| table msg "Type of Call" "LOB" "Date/Time Stamp" "Policy Number" "Requester Id" "LastName" "State"
The issue that I am having is that Only LOB field and State field come back with values, State field for some reason is adding an escape character and pulling the last "
This is what the results look like:
Can someone please help
If changing the logs itself could be a fix then i can do that as well
In addition to what #Mads Hansen offered, the slash in "Date/Time_Stamp" must be escaped. Try this regex:
Type_of_Call\s*=\s*(?<Type_Of_Call>\w+)\s+Call\s+LOB\s*=\s*(?<LOB>\w+)\s+Date\/Time_Stamp\s*=\s*(?<Date_Time_Stamp>[0-9TZ.:-]+)\s+Policy_Number\s*=\s*(?<Policy_Number>[\w-]+)\s+Requester_Id\s*=\s*(?<Requestor_Id>\w+)\s+Last_Name\s*=\s*(\w+)\s+State\s*=\s*(?<State>\w+)
Related
I have the following query I'm working with which results 1 per row but I know there is more than one email address stored within the field separated by semicolon
SELECT UID, EmailToField,
EmailToField REGEXP '[a-zA-Z0-9+._-]+#[a-zA-Z0-9._-]+\.[a-zA-Z0-9_-]+' AS valid_emailTo
FROM table
For example my DB have
UID
EmailTo
EmailCC
EmailBCC
001
emailTo_1#domain.com; emailTo_2#domain.com
emailCC_1#domain.com
EmailBcc1#domain.com
Expecting results to show
UID
validEmailToCcBcc_count
001
4
Used AWK instead of SQL to obtain results, the following worked!
awk '{print NR " " gsub(/[a-zA-Z0-9+._-]+#[a-zA-Z0-9._-]+\.[a-zA-Z0-9_-]+/, "")}' test.csv > results.csv
I have the following query which provides me with all the data I need exported but I would like text '' removed from my final query. How would I achieve this?
| where type == "microsoft.security/assessments"
| project id = tostring(id),
Vulnerabilities = properties.metadata.description,
Severity = properties.metadata.severity,
Remediations = properties.metadata.remediationDescription
| parse kind=regex id with '/virtualMachines/' Name '/providers/'
| where isnotempty(Name)
| project Name, Severity, Vulnerabilities, Remediations ```
You could use replace_string() (https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/replace-string-function) to replace any substring with an empty string
Could you help me extract file name in table format.
Here the below field just before file name is always constant. "Put File /test/abc/test/test/test to /test/test/test/test/test/test/test/test/test/test destFolderPath: /test/test/test/test/test/test/test/abc/def/hij"
This is an event from splunk
2021-04-08T01:03:40.155069+00:00 somedata||someotherdata||..|||Put File /test/abc/test/test/test to /test/test/test/test/test/test/test/test/test/test destFolderPath: /test/test/test/test/test/test/test/abc/def/hij/CHARGEBACK_20210407_060334_customer.csv
Result should be in table format: (font / format doesnt matter)
File Name
CHARGEBACK_20210407_060334_customer.csv
Assuming the original event/field ends with the file name, you should use this regular expression:
(?<file_name>[^\/]+)$
This will extract the text between the last "/" and the end of the event/field ("$").
You can test it here: https://regex101.com/r/J6bU3m/1
Now you can use Splunk's rex command to extract fields at search-time:
| makeresults
| eval _raw="2021-04-08T01:03:40.155069+00:00 somedata||someotherdata||..|||Put File /test/abc/test/test/test to /test/test/test/test/test/test/test/test/test/test destFolderPath: /test/test/test/test/test/test/test/abc/def/hij/CHARGEBACK_20210407_060334_customer.csv"
| fields - _time
| rex field=_raw "(?<file_name>[^\/]+)$"
Alternatively, you could also use this regular expression since you mentioned that the file path is always the same:
| rex field=_raw "abc\/def\/hij\/(?<file_name>.+)"
The splunk query outputs a table where one of the column has these kind of json
the part of the query that gives this output is details.ANALYSIS
{"stepSuccess":false,"SR":false,"propertyMap":{"Url":"https://example.com","ErrCode":"401","transactionId":"7caf34342524-3d232133da","status":"API failing with error code 401"}}
I want to edit my splunk query so that instead of this json, I get only Url in this same column.
Here is my splunk query I was using
|dbxquery connection="AT" query="select service.req_id, service.out,details.ANALYSIS from servicerequest service,SERVICEREQUEST_D details where service.out like 'XYZ is%' and service.row_created > sysdate-1 and service.SERVICEREQUEST_ID = details.SERVICEREQUEST_ID and details.ANALYSIS_CLASS_NAME = 'GetProduction' " shortnames=0 maxrows=100000001
I tried using details.ANALYSIS.propertyMap.Url but it throws error.
You can probably use spathto extract the fields from details.ANALYSIS
Try the following to extract all fields
| spath field="details.ANALYSIS"
Or this just for the url field you are after
| spath field="details.ANALYSIS" path="propertyMap.Url"
scenarioi:-
1) User enters an address into a search field. Address maybe zip,state or country
2) Now i have to take that address which may be in any order or which may just contain the country, state or zip and check the database if the the results exists or not.
Any advice on this topic would be awesome
How about this one?
'SELECT * FROM table WHERE address LIKE \''. addslashes($search) .'\'
OR zip = '. (int)$search .'
OR country LIKE \''. addslashes($search) .'\''
I would do something like :
split address on space
build where clause as
WHERE zip||state||country like '%address_part_one%'
OR zip||state||country like '%address_part_two%'
and so on ...
If you have fulltext enabled for the three fields :
WHERE MATCH (zip,state,country) AGAINST ('address_part_one')
OR MATCH (zip,state,country) AGAINST ('address_part_two')
...