How to return distinct multi-word strings from blocks of usage log text with POSIX? - sql

First time poster and still learning the ropes, so I apologize if the description below is overly verbose.
I have a database of usage logs I'm pulling data from via various pre-parsed fields. This query is intended to return the count of how many times a distinct error signature was logged over a given period of time. Each logged error is assigned a signature_id, errors of the same type are all assigned the same signature_id. One of the fields I'm returning in my query, message, returns the entire message stack trace/block of usage log text.
I want my query to group by signature_id, which is a pre-parsed field in the table I'm selecting from. I'm struggling to make it work because, while similar error types are assigned the same signature_id', every usage log differs slightly due to the timestamp of when the message was logged. So my query is grouping bymessageinstead ofsignature`.
EX: Of what my query returns if I return the entire usage log message
signature_id
b2dea422
message
2019-01-17 18:01:52,130 ip-BLANK [WARN ][159] [request_id=00e74d7c] Type=Blank.Multiverse.Web.Mvc.Attributes.Usage+UsageLoggingException Message=UsageLogContext not present in HttpContext.Current.Items Data: Signature=b2dea422 Stack Trace: at Blank.Blank.Web.Mvc.Attributes.Usage.GetUsageLogContext() at Blank.Blank.Web.Mvc.Attributes.Usage.AddData(Object data)
count
1
signature_id
b2dea422
message
2019-01-17 16:21:36,681,130 ip-BLANK [WARN ][38] [request_id=c140f8ea] Type=Blank.Multiverse.Web.Mvc.Attributes.Usage+UsageLoggingException Message=UsageLogContext not present in HttpContext.Current.Items Data: Signature=b2dea422 Stack Trace: at Blank.Blank.Web.Mvc.Attributes.Usage.GetUsageLogContext() at Blank.Blank.Web.Mvc.Attributes.Usage.AddData(Object data)
count
1
I mentioned above that every usage log differs due to the timestamp of when a given message was logged, but similar error types are assigned the same signature_id. Similar error types also share the same Exception Message=...
EX: Every time a message is logged with signature_id=ab7d890pq, it will also have Exception Message=Cannot read property 'get' of undefined in the message block.
Since the table I'm selecting from doesn't have a pre-parsed exception_message field, I want to parse out the Exception Message= string so my GROUP BY will return the count of distinct logged signature_id's and a column with the exception message is for each distinct signature.
My current query shown below begins to parse out the exceptionmessgage string, but I can't get it to return the entire string:
SELECT CASE
WHEN sourcecategory = 'source_hello_world_category' THEN 'hwCategory'
END AS Service,
signature,
NULLIF(SUBSTRING(REGEXP_SUBSTR(message, 'Message=\\w+[[:space:]]+'), 9), '') AS exceptionmessage,
count(*)
FROM user_usage_logs
WHERE (signature IS NOT NULL
AND signature NOT IN ('ccce9e73',
'787dd1b5',
'17fc66bc',
'ca384d1f',
'20121ecb',
'ccce9e73'))
AND sourcecategory IN ('source_hello_world_category')
AND messagetime > (getdate() - 1)
GROUP BY signature,
sourcecategory,
exceptionmessage
ORDER BY COUNT DESC
LIMIT 10;
The code shown above returns:
signature_id exceptionmessage count
b1det422 Cannot 31,321
330ope77 Unauthorized 1,207
53m6m466 Reference 311
This is an example of I want returned:
signature_id exceptionmessage count
b1det422 Cannot read property 'get' of undefined Stack 31,321
330ope77 Unauthorized access response for many users 1,207
53m6m466 Reference cannot be set to an empty.object.3 311

Related

"The maximum total size of all input parameters is 5242880 bytes." Error message when using BigQuery remote function

When calling a remote function on BigQuery, I get the following error The maximum total size of all input parameters is 5242880 bytes.. This error is undocumented (or my Googling ability is lacking). It is even more unexpected as the query that pop this error is as simple as it gets:
SELECT
`remote_function`( payload ) AS results
FROM
`source_table`
The error is definitely on BQ's side, as no error is observed on the cloud-function called.
How to proceed on such issue, I'm guessing I hitting some kind of limit related to remote functions, but which one?

How do I solve "-307" error on ZKTeco SDK?

Hello everyone and thanks for reading this problem.
I have a solution in C# using the zkemkeeper dll to get the records from some access control devices. When I "ping" them, there isn't any problem, but when I try to connect to them (Using my solution or the standalone demo to get attendance) I get the "-307" error with the "Unable to connect message". That's not very clear and I would really aprecciate if someone can explain what this error is (please!!!!). I would really like to understand these errors myself, so, where can I find all the definition of these errors?
In short:
1.- What is the problem regarding the "-307" error?
2.- Is there any place where all these errors are documented?
Thanks in advance!!
enter image description here
maybe you should check your device, you can ping them
Attention
The dwErrorCode parameter specifies the error code. The values are described as follows:
During connection, the following error codes may be returned:
0 Connected successfully
-1 Failed to invoke the interface
-2 Failed to initialize
-3 Failed to initialize parameters
-5 Data mode read error
-6 Wrong password
-7 Reply error
-8 Receive timeout
-307 Connection timeout
In invoking other interfaces, the following error codes may be returned:
-201 Device is busy
-199 New Mode
-103 device send back error of face version error
-102 face template version error, like 8.0 face template send to 7.0 device
-101 malloc memory failed
-100 Not supported or the data does not exist
-10 The length of transmitted data is incorrect
-5 Data already exists
-4 Insufficient space
-3 Wrong size
-2 File read/write error
-1 The SDK is not initialized and needs to be reconnected
0 Data not found or duplicate data
1 Correct operation
4 Parameter error
101 Buffer allocation error
102 repeat invoking
Underlying error codes:
-12001 Socket creation timeout (connection timeout)
-12002 Insufficient memory
-12003 Wrong Socket version
-12004 Not TCP protocol
-12005 Waiting timeout
-12006 Data transmission timeout
-12007 Data reading timeout
-12008 Failed to read Socket
-13009 Waiting event error
-13010 Exceeded retry attempts
-13011 Wrong reply ID
-13012 Checksum error
-13013 Waiting event timeout
-13014 DIRTY_DATA
-13015 Buffer size too small
-13016 Wrong data length
-13017 Invalid data read1
-13018 Invalid data read2
-13019 Invalid data read3
-13020 Data loss
-13021 Memory initialization error
-15001 Invoking return value of status key issued by SetShortkey interface repeatedly
-15002 Invoking return value of description issued by SetShortkey interface repeatedly
-15003 The two level menu is not opened in the device, and the data need not be issued
getdevicedata and setdevicedata invocation error codes
-15100 Error occurs in obtaining table structure
-15101 The condition field does not exist in the table structure
-15102 Inconsistency in the total number of fields
-15103 Inconsistency in sorting fields
-15104 Memory allocation error
-15105 Data parsing error
-15106 Data overflow as the transmitted data exceeds 4M
-15108 Invalid options
-15113 Data parsing error: table ID not found
-15114 A data exception is returned as the number of fields is smaller than or equal to 0
-15115 A data exception is returned as the total number of table fields is inconsistent with the
total number of fields of the data
Firmware error codes:
2000 Return OK to execute
-2001 Return Fail to execute command
-2002 Return Data
-2003 Regstered event occorred
-2004 Return REPEAT Command
-2005 Return UNAUTH Command
0xffff Return Unknown Command
-4999 Device parameter read error
-4998 Device parameter write error
-4997 The length of the data sent by the software to the device is incorrect
228
229
-4996 A parameter error exists in the data sent by the software to the device
-4995 Failed to add data to the database
-4994 Failed to update the database
-4993 Failed to read data from the database
-4992 Failed to delete data in the database
-4991 Data not found in the database
-4990 The data amount in the database reaches the limit
-4989 Failed to allocate memory to a session
-4988 Insufficient space in the memory allocated to a session
-4987 The memory allocated to a session overflows
-4986 File does not exist
-4985 File read failure
-4984 File write failure
-4983 Failed to calculate the hash value
-4982 Failed to allocate memory
Note
This interface is applicable to the new architecture firmware.

ERROR: Limit set by ERRORS= option reached. Further errors for this INPUT function will not be printed

While using the code able I'm getting an error message as below and also I'm getting the output table.
input(put(serv_to_DT_KEY,8.),yymmdd8.)
between datepart(D.throughdate)
and datepart(intnx('day',d.throughdate,31))
Error: INPUT function reported 'ERROR: Invalid date value' while processing WHERE clause.
ERROR: Limit set by ERRORS= option reached. Further errors for this INPUT function will not be printed.
Could you please help
It's not a true error, exactly; it's a data error, which SAS will let by. What it's saying is the value in serv_to_DT_KEY is not a valid yymmdd8 for some records. (The rest of the message, about "limit set by ...", is just SAS saying it's only going to tell you about 20 or so individual data errors instead of showing you every single one, so your log isn't hopeless.)
To fix this you have several options:
If there is a data issue [ie, all rows should have a valid date], then fix that.
If it's okay that some rows don't have a valid date (for example, they might have a missing), you can use the ?? modifier in the informat in input to tell it to ignore these.
Like this:
input(put(serv_to_DT_KEY,8.),?yymmdd8.)
between datepart(D.throughdate)
and datepart(intnx('day',d.throughdate,31))

Can you combine multivalue fields to form a consolidated Splunk alert?

I have a Splunk search which returns several logs of the same exception, one for each ID number (from a batch process). I have no problem extracting the field from the log with reg-ex and can build a single alert for each ID number easily.
Slack Message: "Reference number $result.extractedField$ has failed processing."
Since the error happens in batches, sending out an alert for every reference ID that failed would clutter up my Slack channel very quickly. Is it possible to collect all of the extracted fields and set the alert to send only one message? Like this...
Slack Message: "Reference numbers $result.listOfExtractedFields$ have failed to process."
To have a consolidated alert you need consolidated search results. Do that like this:
index=the_index_youre_searching "the class where the error occurs" "the exception you're looking for"
| stats values(*) as * by referenceID
Be sure to select the "Once" Trigger Condition in the alert setup.

Pentaho - Condition to go to next block

I have a transformation where i call a REST client to post to an API. The API is expected to return a Reference number, which i use to log and use it for other functionalities.
An exception occurred and i received a status code 200 but the response was "Object reference not set to an instance of an object." which is not a number. The next step after Rest client expects a number but since the response is a text fails. (Rest client 2 to Modified Javascript 2 in the image)
In this scenario is it possible to have an intermediate step which checks if the response is a number else should not allow to go to next step?
Also, a related question. this transformation is run for each record from previous transformation. If the if condition fails, then it should continue with the next record.
There are multiple options.
One of the simplest ones is to insert a Select Values step to convert the field to a number and then add a Error handling hop connected to a Dummy step.
Rows that fail the data type conversion cause errors and are then sent through the error handling hop to the dummy step and will not be sent to the javascript step.