How to escape new lines in rendered message using Serilog and Console sink? - asp.net-core

For the reasons that I will cover in the background at the end of this post, I need to escape new lines in both logged messages and exceptions printed to console using Serilog (i.e. convert \r and \n to \\r and \\n respectively). I created two enrichers - one for exceptions, another for messages - in which I perform the substitutions and register new values as EscapedMessage and EscapedException. The exception enricher works fine, but the problem with the message enricher is that the LogEvent object does not contain the formatted message, it only holds the message template. To generate the message, I call RenderMessage:
public class MessageEnricher: ILogEventEnricher
{
public void Enrich(LogEvent logEvent, ILogEventPropertyFactory propertyFactory)
{
if (logEvent.MessageTemplate == null)
return;
string message = logEvent.RenderMessage();
var logEventProperty = propertyFactory.CreateProperty(
"EscapedMessage",
message
.Replace("\r", "\\r")
.Replace("\n", "\\n"));
logEvent.AddPropertyIfAbsent(logEventProperty);
}
}
The RenderMessage call gives me the formatted message, but there is one problem: it always wraps string value placeholder substitutions in quotes, so I end up with text like:
Something happened to '"this"' or '"that"'.
Assuming that the logging code was like:
_logger.LogDebug($"Something happened to '{0}' or '{1}'.", "this", "that");
Is there a better way to escape new lines in messages or apply the lj modifiers to the RenderMessage call when formatting the message? I saw some posts (e.g. this one) suggesting that just applying the j modifier to a log message takes care of the new lines, but it does not seem to be the case (I suspect it would take care of the new lines in the embedded object, but when I tried to apply it to a message containing a new line, it did nothing).
Background: The reason why I need new lines removed is because when logs are processed by Azure, each new line gets converted to a separate log entry, so we lose all context (timestamp, correlation ID, etc. defined in the log entry template header) in the subsequent log entries. And the reason why I want to escape the new lines (instead of replacing them with, say, white space) is because later, we send Azure logs to Splunk and Splunk can revert the escaped new lines back to real new lines.

Related

Using a local image with EmbedBuilder

According to the Discord.NET documentation page for the EmbedBuilder class, the syntax (converted to VB) to add a local image to an EmbedBuilder object should look something like this:
Dim fileName = "image.png"
Dim embed = New EmbedBuilder() With {
.ImageUrl = $"attachment://{fileName}"
}.Build()
I'm trying to use something like this to add a dynamically created image to the EmbedBuilder, but I can't seem to get it to work properly. Here's basically what I've got:
Dim TweetBuilder As New Discord.EmbedBuilder
Dim DynamicImagePath As String = CreateDynamicImage()
Dim AttachURI As String = $"attachment:///" & DynamicImagePath.Replace("\", "/").Replace(" ", "%20")
With Builder
.Description = "SAMPLE DESCRIPTION"
.ImageUrl = AttachURI
End With
MyClient.GetGuild(ServerID).GetTextChannel(PostChannelID).SendMessageAsync("THIS IS A TEST", False, Builder.Build)
My CreateDynamicImage method returns the full path to the locally created image (e.g., C:\Folder\Another Folder\image.png). I've done a fair amount of "fighting"/testing with this to get past the Url must be a well-formed URI exception I was initially getting because of the [SPACE] in the path.
MyClient is a Discord.WebSocket.SocketClient object set elsewhere.
The SendMessageAsync method does send the Embed to Discord on the correct channel, but without the embedded image.
If I instead send the image using the SendFileAsync method (like so):
MyClient.GetGuild(ServerID).GetTextChannel(PostChannelID).SendFileAsync(DynamicImagePath, "THIS IS A TEST", False, Builder.Build)
the image is sent, but as a part of the message, rather than included as a part of the Embed (this is expected behavior - I only bring it up b/c it was a part of my testing to ensure that there wasn't a problem with actually sending the image to Discord).
I've tried using the file:/// scheme instead of the attachment:/// scheme, but that results in the entire post never making it to Discord at all.
Additionally, I've tried setting the ImageUrl property to a Web resource (e.g., https://www.somesite.com/someimage.png) and the Embed looks exactly as expected with the image and everything when it successfully posts to Discord.
So, I'm just wondering at this point if I'm just missing something, or if I'm just doing it completely wrong?
I cross-posted this to issue #1609 in the Discord.Net GitHub project to get a better idea of what options are available for this and received a good explanation of the issue:
The Embed (and EmbedImage) objects don't do anything with files. They simply pass the URI as configured straight into Discord. Discord then expects a URI in the form attachment://filename.ext if you want to refer to an attached image.
What you need to do is use SendFileAsync with the embed. You have two options here:
Use SendFileAsync with the Stream stream, string filename overload. I think this makes it clear what you need to do: you provide a file stream (via File.OpenRead or similar) and a filename. The provided filename does not have to match any file on disk. > So, for example:
var embed = new EmbedBuilder()
.WithImageUrl("attachment://myimage.png")
.Build();
await channel.SendFileAsync(stream, "myimage.png", embed: embed);
Alternatively, you can use SendFileAsync with the string filePath overload. Internally, this gets a stream of the file at the path, and sets filename (as sent to Discord) to the last part of the path. So it's equivalent to:
using var stream = File.OpenRead(filePath);
var filename = Path.GetFileName(filePath);
await channel.SendFileAsync(stream, filename);
From here, you can see that if you want to use the string filePath overload, you need to set embed image URI to something like $"attachment://{Path.GetFileName(filePath)}", because the attachment filename must match the one sent to Discord.
I almost had it with my code above, but I misunderstood the intention and usage of the method and property. I guess I thought the .ImageUrl property somehow "automatically" initiated a Stream in the background. Additionally, I missed one very important piece:
As it's an async method, you must await (or whatever the VB.NET equivalent is) on SendFileAsync.
So, after making my calling method into an async method, my code now looks like this:
Private Async Sub TestMessageToDiscord()
Dim Builder As New Discord.EmbedBuilder
Dim AttachmentPath As String = CreateDynamicImage() '<-- Returns the full, local path to the created file
With Builder
.Description = "SAMPLE DESCRIPTION"
.ImageUrl = $"attachment://{IO.Path.GetFileName(AttachmentPath)}"
End With
Using AttachmentStream As IO.Stream = IO.File.OpenRead(AttachmentPath)
Await MyClient.GetGuild(ServerID).GetTextChannel(PostChannelID).SendFileAsync(AttachmentStream, IO.Path.GetFileName(AttachmentPath), "THIS IS A TEST", False, Builder.Build)
End Using
End Sub
Now, everything works exactly as expected and I didn't have to resort to uploading the image to a hosting site and using the new URL (I actually had that working before I got the response on GitHub. I'm sure that code won't go to waste).
EDIT
Okay, so I still ended up going back to my separately hosted image option for one reason: I have a separate event method that modifies the original Embed object during which I want to remove the image and replace the text. However, when that event fired, while the text was replaced, the image was "moved" to the body of the Discord message. While I may have been able to figure out how to get rid of the image entirely, I decided to "drop back and punt" since I had already worked out the hosted image solution.
I've tried everyting I could, but I got stuck at the same point at where you are now.
My guesses are that Discord doesn't like the embedded images from https://cdn.discordapp.com/attachments, and only accepts the new files from https://media.discordapp.net. I might be wrong though, this is the way it worked for me.
I believe it's only a visual glitch, as I found if you send a link for an image from cdn.discordapp.com/attchments in your regular Discord client, it bugs out and shows an empty embed for some reason.
That would make sense since the default link used in an embedded image actually starts with https://cdn.discordapp.com/attachments/...
You could solve this issue by using https://media.discordapp.net, but it seems like Discord.net is configured to use the old domain.

Debugging u-sql Jobs

I would like to know if there are any tips and tricks to find error in data lake analytics jobs. The error message seems most of the time to be not very detailed.
When trying to extract from CSV file I often get error like this
Vertex failure triggered quick job abort. Vertex failed: SV1_Extract[0] with >error: Vertex user code error.
Vertex failed with a fail-fast error
It seems that these error occur when trying to convert the columns to specified types.
The technique I found is to extract all columns to string and then do a SELECT that will try to convert the columns to the expected type. Doing that columns by columns can help find the specific column in error.
#data =
EXTRACT ClientID string,
SendID string,
FromName string,
FROM "wasb://..."
USING Extractors.Csv();
//convert some columns to INT, condition to skip header
#clean =
SELECT Int32.Parse(ClientID) AS ClientID,
Int32.Parse(SendID) AS SendID,
FromName,
FROM #data
WHERE !ClientID.StartsWith("ClientID");
Is it also possible to use something like a TryParse to return null or default values in case of a parsing error, instead of the whole job failing?
Thanks
Here is a solution without having to use code behind (although Codebehind will make your code a bit more readable):
SELECT ((Func<string, Int32?>)(v => { Int32 res; return Int32.TryParse(v, out res)? (Int32?) res : (Int32?) null; }))(ClientID) AS ClientID
Also, the problem you see regarding error message being cryptic has to do with a bug that should be fixed soon in returning so called inner error messages. The work around today is to do the following:
In the ADL Tools for VisualStudio, open the Job View of the failed job.
In the lower left corner, click on “resources” link in the job detail area.
Once the job resources are loaded, click on “Profile”.
Search for the string “jobError” at the beginning of the line. Copy the entire line of text and paste in notepad (or other text editor) to read the actual error.
That should give you the exact error message.
Yes, you can use TryParse using U-SQL user defined functions. You can do this like:
In code behind:
namespace TestNS
{
public class TestClass
{
public static int TryConvertToInt(string s)
{
int i = 0;
if (Int32.TryParse(s, out i))
return i;
return 0;
}
}
}
In U-SQL Script:
TestNS.TestClass.TryConvertToInt(ClientID) AS clientID
It looks like you have some other issues, as I always get appropriate error in case of conversion problem, something like:
"E_RUNTIME_USER_EXTRACT_COLUMN_CONVERSION_INVALID_ERROR","message":"Invalid character when attempting to convert column data."

Trying to get node-webkit console output formatted on terminal

Fairly new to node-webkit, so I'm still figuring out how everything works...
I have some logging in my app:
console.log("Name: %s", this.name);
It outputs to the browser console as expected:
Name: Foo
But in the invoking terminal, instead I get some fairly ugly output:
[7781:1115/085317:INFO:CONSOLE(43)] ""Name: %s" "Foo"", source: /file/path/to/test.js (43)
The numerical output within the brackets might be useful, but I don't know how to interpret it. The source info is fine. But I'd really like the printed string to be printf-style formatted, rather than shown as individual arguments.
So, is there a way to get stdout to be formatted either differently, or to call a custom output function of some sort so I can output the information I want?
I eventually gave up, and wrapped console.log() with:
log = function() {
console.log(util.format.apply(this, arguments));
}
The actual terminal console output is done via RenderFrameHostImpl::OnAddMessageToConsole in chromium, with the prefix info being generated via LogMessage::Init() in the format:
[pid:MMDD/HHMMSS:severity:filename(line)]
The javascript console.log is implemented in console.cc, via the Log() function. The printf style formatting is being done at a higher level, so that by the time the Log() function (or similar) are called, they are only passed a single string.
It's not a satisfying answer, but a tolerable workaround.
I was looking to create a command line interface to go along side my UI and had a similar problem. Along with not logging the values as I wanted I also wanted to get rid of the [pid:MMDD/HHMMSS:severity:filename(line)] output prefix so I added the following:
console.log = function (d) {
process.stdout.write(d + '\n');
};
so that console logging was set back to stdout without extra details. Unfortunately also a work around.

wcf rest appending escape characters to HTML tags

I am trying to convert an existing .Net shop site to an Android app.
(This is in VB)
One of the main objects holds product data.
Within that object there is, for instance:
<H2>Product Title</H2>
<P>A description</P>
I have already built a WCF Rest service which returns the data I expect:...
Having said that, I have trialled Newtonsoft.Json and DataContractJsonSerializer which produce the same, but different, outputs.
When running the WCF service in debug using Newtonsoft.Json it returns those items as I would expect:
Newtonsoft:
<H2>Product Title<\/H2><P>A description<\/P>
DataContractJsonSerializer:
<H2>Product Title<\\\/H2><P>A description<\\\/P>
However, when I run the Android app through Eclipse I get the error of "Invalid escape sequence (valid ones are \b \t \n \f \r \" \' \ )"
So, in short; how to stop Newtonsoft or DataContractJsonSerializer from inserting these escape sequences?
Thanks
Dave
UPDATE:
I have tracked this down to something (?) that WCF is doing. Here is the final bit of my code which returns the JSON string:
retVal = CacheManager.JSONFullProduct("P" & ProductID)
At this point 'retVal' is storing closing HTML tags with just '/'
retVal = retVal.Replace("\/", "/")
At this point 'retVal' is still storing closing HTML tags with just '/'
Return retVal
At this point 'retVal' itself is still storing closing HTML tags with just '/', but when it actually returns (either to Notepad if I'm running the Service direct, or to Android) '/' suddenly becomes '{backslash}/'
I've tried to do a string replace in the android app:
result.replace("\/", "/");
But that returns the same error of "Invalid escape sequence...", and anyway, I don't really want to be doing this kind of work on the phone.
So, what is happening at Return retVal to suddenly insert all of these escape characters??
I have searched high and wide for an answer to this, and have finally found it.
I will share for anyone else experiencing the same issue:
It is the WCF Rest service.
Learning WCF and Android at the same time led me to believe that the response from WCF should be a String serialized in the Json format.
To do this, a .Net object, array or whatever would go through DataContractJsonSerializer before being returned as a String to Android for further parsing.
Something like this:
Dim stream1 As MemoryStream = New MemoryStream
Dim ser As DataContractJsonSerializer = New DataContractJsonSerializer(GetType(myType))
ser.WriteObject(stream1, myThing)
Dim _json As String = Encoding.UTF8.GetString(stream1.ToArray())
stream1.Close()
return _json
Wrong.
Keep your object, array or whatever and return that instead; WCF will take care of the proper escaping for you.
For example (this is VB);
IService:
<OperationContract()> _
<WebGet(BodyStyle:=WebMessageBodyStyle.WrappedRequest, RequestFormat:=WebMessageFormat.Json, ResponseFormat:=WebMessageFormat.Json, UriTemplate:="/MyKit/{AccountID}")> _
Function GetKit(ByVal AccountID As String) As MyKit
Service:
Public Function GetKit(ByVal AccountID As String) As MyKit Implements IService1.GetKit
Dim allKit As New MyKit() //Your object
objDal.CommandText = 'run some sql here - or whatever
Using dr As SqlDataReader = "blah"
//populate your object
End Using
Return allKit //return the object, not the string representation of it
End Function
Using DataContractJsonSerializer for sending as Json to Android from WCF effectively 'pre-escapes' the data. When it gets to Android, the Json parser is unable to handle it, because it also escapes the data.

How can I send different SMS' to multiple recipients in a loop

I'm using Symbian C++ to create my code, I'm using S60 5th Ed SDK
I want to know how to send different messages - Their body text not the same - to multiple recipients in a for-loop ?
I've tried the example below, but when I try to use it in a loop it crashes due to ActiveObjects properties, as I should wait to AO to finish before calling it again.
Sending_SMS_in_S60_3rd_Edition_MTM
Below is example of what I need to do:
SendSMSL(); // **I call this function once to start the process**
// **iRecepients is a CDesCArray contains phone numbers**
// ** iSMSBody is a CDesCArray contains each contact SMS body text**
void CSMS::SendSMSL()
{
if(iRecepients->Count() >= 1)
{
TInt x = iRecepients->Count()-1;
TInt y = iSMSBody->Count()-1;
// **If the sms validating and scheduling succeeded then delete last item from both arrays**
if(iSMSHandler->SendL((*iRecepients)[x],(*iSMSBody)[y])
{
iRecepients->Delete(x);
iSMSBody->Delete(y);
}
}
}
Now, in the code above I call iSMSHandler->SendL() which send sms using AO, and in iSMSHandler object RunL() function, I call back the function above CSMS::SendSMSL() , which in turn checks if there is still anymore iRecepients elements and then call again iSMSHandler->SendL() AO , and keeps this way till no more iRecepients.
Looking forward to hear your feedback on the modification above.
Many thanks in advance.
The link you posted doesn't work for me so I can't see the rest of the code.
Assuming that iSmsHandler is a class that uses active objects to send SMS messages,
I see several issues with your loop.
1) You need to wait for the first asynchronous SendL to complete before you can issue the next SendL
2) The buf variable can not go out of scope until the SendL completes. (This may be the reason for your crash)
I suggest that you keep the textbuffer somewhere else, like together with iSmsHandler, and then code the active object that is called when SendL completes to issue the next SendL.
All of this is guesses since I have no idea what class iSmsHandler is....