Multiple conditional branches in Serilog expression - conditional-statements

We've started using Serilog to generate structure logs for our application. Using Serilog Expressions we've been able to generate the following JSON, which is fairly cool as far as logging goes.
{
"#t": "2022-02-11T15:57:57.6087361+01:00",
"#m": "GET Request received on API endpoint /foo",
"#l": "Info",
"SourceContext": "Company.API.FooController",
"ActionId": "bd248542-138b-4053-9d50-b9e62e0ab1fd",
"ActionName": "Company.API.FooController.GetFoo (Company.API)",
"RequestId": "0HMFDDRDSMUUI:00000001",
"RequestPath": "/foo",
"SpanId": "|85c9c2db-489a260ce2c6915e.",
"TraceId": "85c9c2db-489a260ce2c6915e",
"ParentId": "",
"ConnectionId": "0HMFDDRDSMUUI",
"ThreadId": 7,
"ContextFieldA": "ValueA",
"ContextFieldB": "ValueA"
}
These log events are generated using the following expression, which is read from the environment variables:
"{ {#t, #m, #r, #l: if #l = 'Information' then 'Info' else #l, #x, ..#p} }\n"
However we'd like to map the other log levels to short-hands as well. Such as Warning to Warn etc. How would we go about adding multiple if-else branches to this expression?

The answer is surprisingly simple actually. Multiple branches can be done exactly like one might expect:
"{ {#t, #m, #r, #l: if #l = 'Information' then 'Info'
else if #l = 'Warning' then 'Warn' else #l, #x, ..#p} }\n"

Related

Karate API : * def from response is throwing syntax error

I am setting up an E2E test and chaining my request/responses. I am defining variables from each response and using them in the next call.
Its working up to a point, and then a problem surfaces when defining off the 2nd response.
If I def operationId, operationSubject, or operationStatus (e.g response.operationId), it works.
If I store anything from the results (e.g response.results.0.personId) it throws this error
Expected ; but found .0
response.results.0.personId
My response:
{
"operationId": "922459ecxxxxx",
"operationSubject": "BATCH_ENROLLMENT",
"operationStatus": "PROCESSED",
"results": {
"0": {
"personId": "367a73b5xxxx",
"status": "PRE_AUTH",
"email": "mquinter+TEST.69387488#email.com",
"loanNumber": null
},
"1": {
"personId": "56f060fd-e34xxxxxx",
"status": "PRE_AUTH",
"email": "mquintxxxx#email.com",
"loanNumber": null
}
}
}
That's not how to access data in JSON. See this similar question: https://stackoverflow.com/a/71847841/143475
Maybe you meant to do this:
* def foo = response.results[0].personId
https://stackoverflow.com/users/143475/peter-thomas
I see the issue - It wasn't finding the response because I wasn't giving it enough time before the next call.
I put a sleep in there and its working as expected.
Thanks

Extracting particular nested properties with a $ prefix in Amazon Redshift or Quicksight

I am using PostHog for product analytics and have exported some event data to Amazon Redshift as well as S3 to be used in Quicksight.
Under the personal properties part of the JSON, each individual property is nested but begins with a $
I am quite new to SQL queries as well as getting specific details from JSON. in Quicksight using parseJson
Here is an example of the JSON from PostHog
"properties": {
"$active_feature_flags": [],
"$browser": "Chrome",
"$browser_version": 98,
"$ce_version": 1,
"$device_type": "Desktop",
"$environment": "test",
"$event_type": "click",
"$lib": "web",
"$lib_version": "1.17.8",
"$os": "Mac OS X",
"$pathname": "/events",
"$plugins_deferred": [],
"$plugins_failed": [],
"$plugins_succeeded": [
"First Event Today (4914)",
"GeoIP (5539)"
],
I have sought help from a few sources who have mentioned it isn't as simple because of the $ symbol at the beginning.
So my question would be,
How would I query this in Redshift to successfully extract $device_type and $os for example?
How would I pull the same properties using parseJson in Amazon Quicksight?
I can answer #1.
The json provided looks to be a snippet and invalid as is. So I removed the trailing ',' and used SQL to provide the surrounding '{}'. Once it is valid json this runs fine:
create table test as select '"properties": {
"$active_feature_flags": [],
"$browser": "Chrome",
"$browser_version": 98,
"$ce_version": 1,
"$device_type": "Desktop",
"$environment": "test",
"$event_type": "click",
"$lib": "web",
"$lib_version": "1.17.8",
"$os": "Mac OS X",
"$pathname": "/events",
"$plugins_deferred": [],
"$plugins_failed": [],
"$plugins_succeeded": [
"First Event Today (4914)",
"GeoIP (5539)"
]
}' as json_text;
select json_extract_path_text('{' || json_text ||'}', 'properties' ,'$device_type') as device_type,
json_extract_path_text('{' || json_text ||'}', 'properties' ,'$os') as os
from test;

karate scenario outline - Create a dynamic example table based on JSON array index size

Here I would like to clarify about creating dynamic example table for a dynamic JSON index size
My JSON looks like
Env - Dev - 2 servers
"response": {
"abc": [{
"status": "pass"
.
.
},
{
"status": "pass"
.
.
}
]
}
Env - Uat - 3 servers
{
"response": {
"abc": [{
"status": "pass"
},
{
"status": "pass"
},
{
"status": "pass"
}
]
}
}
My scenario outline looks like
Scenario Outline: validating .....
When def result = callonce read('featurefilename#tagname')
Then print result
And print <status>
And print ...
And match ....
Examples:
|result.response.abc|
Errors for the above:
1) * dynamic expression evaluation failed:result.response.abc
2) com.intuit.karate.karateExpresion: ---- javascript evaluation failed result.response.abc, ReferenceError:"result" is not defined in at line number 1
Note - If I move step 'When def result = callonce read('featurefilename#tagname') to background it's working as expected but I can't use background in my feature file due to other factors.
Thanks in advance
Instead of providing index in a table you can leverage Dynamic Scenario Outline feature in karate.
In this case you you can pass the variable as a input to Examples. If the JSON provided above is from variable result then,
Examples:
| result.response.abc |
Refer the docs for more insights.

TableData.insertAll with templateSuffix - frequent 503 errors

We are using TableData.insertAll with a templateSuffix and are experiencing frequent 503 errors with our usage pattern.
We set the templateSuffix based on two pieces of information - the name of the event being inserted and the data of the event being inserted. E.g. 'NewPlayer20160712'. The table ID is set to 'events'.
In most cases this works as expected, but relatively often it will fail and return an error. Approximately 1 in every 200 inserts will fail, which seems way too often for expected behaviour.
The core of our event ingestion service looks like this:
//Handle all rows in rowsBySuffix
async.mapLimit(Object.keys(rowsBySuffix), 5, function(suffix) {
//Construct request for suffix
var request = {
projectId: "tactile-analytics",
datasetId: "discoducksdev",
tableId: "events",
resource: {
"kind": "bigquery#tableDataInsertAllRequest",
"skipInvalidRows": true,
"ignoreUnknownValues": true,
"templateSuffix": suffix, // E.g. NewPlayer20160712
"rows": rowsBySuffix[suffix]
},
auth: jwt // valid google.auth.JWT instance
};
//Insert all rows into BigQuery
var cb = arguments[arguments.length-1];
bigquery.tabledata.insertAll(request, function(err, result) {
if(err) {
console.log("Error insertAll. err=" + JSON.stringify(err) + ", request.resource=" + JSON.stringify(request.resource));
}
cb(err, result);
});
}, arguments[arguments.length-1]);
A typical error would look like this:
{
   "code": 503,
   "errors": [
      {
         "domain": "global",
         "reason": "backendError",
         "message": "Error encountered during execution. Retrying may solve the problem."
      }
   ]
}
The resource part for the insertAll that fails looks like this:
{
   "kind": "bigquery#tableDataInsertAllRequest",
   "skipInvalidRows": true,
   "ignoreUnknownValues": true,
   "templateSuffix": "GameStarted20160618",
   "rows": [
      {
         "insertId": "1f4786eaccd1c16d7ce865fea4c7af89",
         "json": {
            "eventName": "gameStarted",
            "eventSchemaHash": "unique-schema-hash-value",
            "eventTimestamp": 1466264556,
            "userId": "f769dc78-3210-4fd5-a2b0-ca4c48447578",
            "sessionId": "821f8f40-ed08-49ff-b6ac-9a1b8194286b",
            "platform": "WEBPLAYER",
            "versionName": "1.0.0",
            "versionCode": 12345,
            "ts_param1": "2016-06-04 00:00",
            "ts_param2": "2014-01-01 00:00",
            "i_param0": 598,
            "i_param1": 491,
            "i_param2": 206,
            "i_param3": 412,
            "i_param4": 590,
            "i_param5": 842,
            "f_param0": 5945.442,
            "f_param1": 1623.4111,
            "f_param2": 147.04747,
            "f_param3": 6448.521,
            "b_param0": true,
            "b_param1": false,
            "b_param2": true,
            "b_param3": true,
            "s_param0": "Im guesior ti asorne usse siorst apedir eamighte rel kin.",
            "s_param1": "Whe autiorne awayst pon, lecurt mun.",
            "eventHash": "1f4786eaccd1c16d7ce865fea4c7af89",
            "collectTimestamp": "1468346812",
            "eventDate": "2016-06-18"
         }
      }
   ]
}
We have noticed that, if we avoid including the name of the event in the suffix (e.g. the NewPlayer part) and instead just have the date as the suffix, then we never experience these errors.
Is there any way that this can be made to work reliably?
Backend errors happen, we usually see 5 from 10000 requests. We simply retry, and we have more constant rate, and we can provide a reconstructable use case we put a ticket on the Bigquery issue tracker. This way if there is something wrong with our project it can be investigated.
https://code.google.com/p/google-bigquery/

how to filter Defect search by active projects in Rally using Web API

I built a custom search tool to allow searching Rally via the Web API from other applications and I've run into an issue. Right now I am allowing defects to be searched but I noticed that defects are coming back in the search results that are related to a project that is closed. I need to filter these out. I am wondering if there is a way to access attributes on a referenced object when querying another object, for example, if I have a query to search for defects where the name contains some text, such as https://rally1.rallydev.com/slm/webservice/v2.0/defect?query=(Name contains "keyword"), can I include something in that query to say that I only want defects for open projects by using the Project attribute on Defect, such as Project.State equals "Open". Basically I'm wondering if there is a way to do it in one query in an OData-ish format. Or as an alternative, if I separately query for a list of all open projects, could I add conditions to the query to say something like (Name contains "keyword") AND (ProjectId = ... OR ProjectId OR ...)? Any thoughts or suggestions are much appreciated.
A query for defects (or any other work item types) is not expected to return items from closed projects. WS API queries do not search closed projects.
Created a defect in a project. It happens to have FormattedID DE529
Tested (FormattedID = DE529) in WS API.
This json was returned:
{
QueryResult: {
_rallyAPIMajor: "2",
_rallyAPIMinor: "0",
Errors: [ ],
Warnings: [ ],
TotalResultCount: 1,
StartIndex: 1,
PageSize: 20,
Results: [
{
_rallyAPIMajor: "2",
_rallyAPIMinor: "0",
_ref: "https://rally1.rallydev.com/slm/webservice/v2.0/defect/36182496495",
_refObjectUUID: "aa35839a-5e49-44c6-8be7-2fb17bbd91bf",
_refObjectName: "bad defect",
_type: "Defect"
}
]
}
}
Closed the project. Ran the same query:
No result:
{
QueryResult: {
_rallyAPIMajor: "2",
_rallyAPIMinor: "0",
Errors: [ ],
Warnings: [ ],
TotalResultCount: 0,
StartIndex: 1,
PageSize: 20,
Results: [ ]
}
}
Also, it is not possible to query Projects by State. This query will return 0 results even when there are closed projects in the workspace 1234:
https://rally1.rallydev.com/slm/webservice/v2.0/project?workspace=https://rally1.rallydev.com/slm/webservice/v2.0/workspace/12345&query=(State = Closed)
Project names in Rally do not have to be unique. Identifying a project by Name may produce a misleading result in a corner case when you have two projects with the same name(one is Open, the other is Closed).