ASP.NET Core 2.1 API POST body is null when called using HttpWebRequest, seems it can't be parsed as JSON - asp.net-core

Question

I'm facing a strange bug, where .NET Core 2.1 API seems to ignore a JSON body on certain cases.
I advised many other questions (e.g this one, which itself references others), but couldn't resolve my problem.
I have something like the following API method:
[Route("api/v1/accounting")]
public class AccountingController
{ sometimes it's null
||
[HttpPost("invoice/{invoiceId}/send")] ||
public async Task<int?> SendInvoice( \/
[FromRoute] int invoiceId, [FromBody] JObject body
)
{
// ...
}
}
And the relevant configuration is:
public IServiceProvider ConfigureServices(IServiceCollection services)
{
services
.AddMvcCore()
.AddJsonOptions(options =>
{
options.SerializerSettings.Converters.Add(new TestJsonConverter());
})
.AddJsonFormatters()
.AddApiExplorer();
// ...
}
Where TestJsonConverter is a simple converter I created for testing why things doesn't work as they should, and it's simple as that:
public class TestJsonConverter : JsonConverter
{
public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
{
var token = JToken.Load(reader);
return token;
}
public override bool CanRead
{
get { return true; }
}
public override bool CanConvert(Type objectType)
{
return true;
}
public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
{
throw new NotImplementedException("Unnecessary (would be neccesary if used for serialization)");
}
}
Calling the api method using Postman works, meaning it goes through the JSON converter's CanConvert, CanRead, ReadJson, and then routed to SendInvoice with body containing the parsed json.
However, calling the api method using HttpWebRequest (From a .NET Framework 4, if that matters) only goes through CanConvert, then routes to SendInvoice with body being null.
The request body is just a simple json, something like:
{
"customerId": 1234,
"externalId": 5678
}
When I read the body directly, I get the expected value on both cases:
using (var reader = new StreamReader(context.Request.Body))
{
var requestBody = await reader.ReadToEndAsync(); // works
var parsed = JObject.Parse(requestBody);
}
I don't see any meaningful difference between the two kinds of requests - to the left is Postman's request, to the right is the HttpWebRequest:
To be sure, the Content-Type header is set to application/json. Also, FWIW, the HttpWebRequest body is set as follows:
using(var requestStream = httpWebRequest.GetRequestStream())
{
JsonSerializer.Serialize(payload, requestStream);
}
And called with:
var response = (HttpWebResponse)request.GetResponse();
Question
Why does body is null when used with HttpWebRequest? Why does the JSON converter read methods are skipped in such cases?

Answer 1

The problem was in the underlying code of the serialization. So this line:
JsonSerializer.Serialize(payload, requestStream);
Was implemented using the default UTF8 property:
public void Serialize<T>(T instance, Stream stream)
{
using(var streamWriter = new StreamWriter(stream, Encoding.UTF8) // <-- Adds a BOM
using(var jsonWriter = new JsonTextWriter(streamWriter))
{
jsonSerializer.Serialize(jsonWriter, instance); // Newtonsoft.Json's JsonSerializer
}
}
The default UTF8 property adds a BOM character, as noted in the documentation:
It returns a UTF8Encoding object that provides a Unicode byte order
mark (BOM). To instantiate a UTF8 encoding that doesn't provide a BOM,
call any overload of the UTF8Encoding constructor.
It turns out that passing the BOM in a json is not allowed per the spec:
Implementations MUST NOT add a byte order mark (U+FEFF) to the
beginning of a networked-transmitted JSON text.
Hence .NET Core [FromBody] internal deserialization failed.
Lastly, as for why the following did work (see demo here):
using (var reader = new StreamReader(context.Request.Body))
{
var requestBody = await reader.ReadToEndAsync(); // works
var parsed = JObject.Parse(requestBody);
}
I'm not very sure. Certainly, StreamReader also uses UTF8 property by default (see remarks here), so it shouldn't remove the BOM, and indeed it doesn't. Per a test I did (see it here), it seems that ReadToEnd is responsible for removing the BOM.
For elaboration:
StreamWriter and UTF-8 Byte Order Marks
The Curious Case of the JSON BOM