tranform field url_address by rempving \/ (backslash) from value - splunk

I would like to remove \ (backslash) from my field url_address that is extracted using this regex :
Address: : (?<URL>.*?)\\r
my actually output is : blablahost:80\/api\/cars\/bmw\/g\/v1
Èxpected output is : blablahost:80/api/cars/bmw/g/v1
Is it possible with Splunk transformation to remove that backslash from my url_address ?
Many Thanks


You can use the rex command in sed mode to fix that at search time. | rex field=URL mode=sed "s/\//g".
Or do it automatically in props.conf. This applies to all fields so it's a little different so as not to strip too many backslashes.
[mysourcetype]
SEDCMD-fixurl = s:\\/:/:g

Related

Awk - How to escape the | in sub?

I'd like to substitue a string, which contains a |
My STDIN :
13|Test|123|6232
14|Move|126|6692
15|Test|123|6152
I'd like to obtain :
13|Essai|666|6232
14|Move|126|6692
15|Essai|666|6152
I tried like this
{sub("|Test|123","|Essai|666") ;} {print;}
But I think the | is bothers me.... I really need to replace the complete string WITH the |.
How should I do to get this result ?
Many thanks for you precious help

You can use
awk '{sub(/\|Test\|123\|/,"|Essai|666|")}1' file
See the online demo.
Note:
/\|Test\|123\|/ is a regex that matches |Test|123| substring
sub(/\|Test\|123\|/,"|Essai|666|") - replaces the first occurrence of the regex pattern in the whole record (since the input is omitted, $0 is assumed)
1 triggers the default print action, no need to explicitly call print here.

Getting Error as "Regex: syntax error in subpattern name (missing terminator)." in SPLUNK

I have been extracting fields in Splunk and this looks to be working fine for all headers but for the header l-s-m, I am getting the error as "syntax error in subpattern name (missing terminator)."
I have done similar for other headers and all works but this is the only header with "hypen" sign that is giving this error, I have tried multiple times but this is not helping.
Headers:
Content-Type: application/json
Accept: application/json,application/problem json
l-m-n: txxxmnoltr
Accept-Encoinding:gzip
Regex I am trying is "rex field=u "l-m-n: (?<l-m-n>.*)" in SPLUNK. Could you please guide me here?

rex cannot extract into a field name with hyphens. However, you can solve this with rename
| rex field=u "l-m-n: (?<lmn>.*)" | rename lmn AS "l-m-n"
In general, I would avoid the use of hyphens in a field name, as it can be mistaken for a minus. If you want to use the field l-m-n, you will need to quote it everywhere, like 'l-m-n' . I would strongly suggest you stick with using the field name lmn.
Try running the following to see what I mean
| makeresults | eval l-m-n=10 | eval l=1 | eval m=1 | eval n=1 | eval result_noquote=l-m-n | eval result_quoted='l-m-n'

Splunk Rex: Extracting fields of a string to a value

I'm a newbie to SPlunk trying to do some dashboards and need help in extracting fields of a particular variable
Here in my case i want to extract only KB_List":"KB000119050,KB000119026,KB000119036" values to a column
Expected output:
KB_List
KB000119050,KB000119026,KB000119036
i have tried:
| rex field=_raw "\*"KB_List":(?<KB_List>\d+)\*"
highlighted the part below in the log
svc_log_ERROR","Impact":4.0,"CategoryId":"94296c474f356a0009019ffd0210c738","hasKBList":"true","lastNumOfAlerts":1,"splunkURL":false,"impactedInstances":"","highestSeverity":"Minor","Source":"hsym-plyfss01","reqEmail":"true","AlertGroup":"TIBCOP","reqPage":"","KB_List":"KB000119050,KB000119026,KB000119036","reqTicket":"true","autoTicket":true,"SupportGroup":"TESTPP","Environment":"UAT","Urgency":4.0,"AssetId":"AST000000000159689","LiveSupportGroup":"TESTPP","sentPageTo":"TESTPP"},"Notification":{"":{"requestId":"532938335"}},"":

rex field=_raw "KB_List\":\"(?<KB_List>[^\"])\""
This regular expression will look for anything that begins with KB_List":", the capture everything except a ".
In your example, you are only capturing digits (\d+), whereas the contents in the KB_List field also contain characters ("KB" and ",")

Alas:
I figured out by looking into so many articles:
| rex "KB_List\":\"(?<KB_Listed>[^\"]+)" | table KB_Listed

How to extract data from the String in splunk?

I was given a log from splunk and I want to get a particular data in the middle of the string and use it for the dashboard. For example:
msg="somestring1 somestring2 500 somestring3 ..."
How do I get the value 500?
Sorry, I am not expert in splunk. Thanks in advance

I think what you're looking for is the rex command.
Example usage:
... | rex field=message "\S+ \S+ (?<extracted_field>\d+) \S+" | stats count by extracted_field

Data between quotes and field separator

In the example given below, the last line is not uploaded. I get an error:
Data between close double quote (") and field separator:
This looks like a bug since all the data between pipe symbol should be treated as a single field.
Schema: one:string,two:string,three:string,four:string
Upload file:
This | is | test only | to check quotes
second | line | "with quotes" | no text
third line | with | "start quote" and | a word after quotes
The first and second line above is processed. But not the third.
Update:
Can some please explain why does the following work except the third line?
This | is | test only | to check quotes
second | line | "with quotes" | no text
third line | with | "start quote" and | a word after quotes
forth line | enclosed | {"GPRS","MCC_DETECTED":false,"MNC_DETECTED":false} | how does this work?
fifth line | with | {"start quote"} and | a word after quotes
There can be some fancy explanation to this. From the end user perspective this is absurd.

From the CSV RFC4180 page: "If double-quotes are used to enclose fields, then a double-quote appearing inside a field must be escaped by preceding it with another double quote."
You probably want to do this:
This | is | test only | to check quotes
second | line | "with quotes" | no text
third line | with | " ""start quote"" and " | a word after quotes
More about our CSV input format here.

Using --quote worked perfectly.
bq load
--source_format CSV --quote ""
--field_delimiter \t
--max_bad_records 10
-E UTF-8
destination table
Source files

API V2
https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.load.quote
bq command
--quote: Quote character to use to enclose records. Default is ". To indicate no quote character at all, use an empty string.

Try this as an alternative:
Load the MySQL backup files into a Cloud SQL instance.
Read the data in BigQuery straight out of MySQL.
Longer how-to:
https://medium.com/google-cloud/loading-mysql-backup-files-into-bigquery-straight-from-cloud-sql-d40a98281229

You can use the other flags also while uploading the data. I used the bq tool with following flags
bq load -F , --source_format CSV --skip_leading_rows 1 --max_bad_records 1 --format csv -E UTF-8 yourdatset gs://datalocation.

Try loading every time with bq shell.
I had to load 1100 columns. While trying with the console with all the error options, it threw lot many errors. Ignoring the errors in the console means loosing records.
Hence tried with the shell and succeeded loading all the records.
Try the following:
bq load --source_format CSV --quote "" --field_delimiter \t --allow_jagged_rows --ignore_unknown_values --allow_quoted_newlines --max_bad_records 10 -E UTF-8 {dataset_name}.{table_name} gs://{google_cloud_storage_location}/* {col_1}:{data_type1},{col_2}:{data_type2}, ....
References:
https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-csv#bigquery_load_table_gcs_csv-cli
https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-csv#csv-options