NodaTime database json serialization - serialization

I'm saving list of schedules in my jsonb column in postgres database and I'm having problems with LocalDate NodaTime type.
Here is my object that is being serialized (actually List<Schedule> is serialized and stored in database)
public class Schedule
{
public LocalTime? Start { get; set; }
public LocalTime? End { get; set; }
}
This is what's get stored in the database
[
{
"End": {
"Hour": 10,
"Minute": 0,
"Second": 0,
"TickOfDay": 360000000000,
"Millisecond": 0,
"TickOfSecond": 0,
"NanosecondOfDay": 36000000000000,
"ClockHourOfHalfDay": 10,
"NanosecondOfSecond": 0
},
"Start": {
"Hour": 8,
"Minute": 0,
"Second": 0,
"TickOfDay": 288000000000,
"Millisecond": 0,
"TickOfSecond": 0,
"NanosecondOfDay": 28800000000000,
"ClockHourOfHalfDay": 8,
"NanosecondOfSecond": 0
}
}
]
Which seems pretty fine to me, but the problem is when I'm fetching the data from database my LocalDate's are '00:00:00'.
Serialization/Deserialization is done using
var converter = new ValueConverter<T, string>
(
v => JsonConvert.SerializeObject(v),
v => JsonConvert.DeserializeObject<T>(v) ?? null
);

I've added the NodaTime.Serialization.JsonNet as suggested in the comments and changed the converter for Entity Framework to:
var converter = new ValueConverter<T, string>
(
v => JsonConvert.SerializeObject(v, NodaConverters.LocalTimeConverter),
v => JsonConvert.DeserializeObject<T>(v, NodaConverters.LocalTimeConverter) ?? null
);
This now correctly serializes to string and successfully deserializes.
[{"End": "10:00:00", "Start": "09:00:00"}]

Related

ASPNET Core Options Binding does not populate dictionary key/value pair

I have the following JSON configuration
"Configurations": {
"KeyA": {
"Ids": []
},
"KeyB": {
"Ids": [1, 2, 3]
},
"KeyC": {
"Ids": [1, 2, 3],
"OptionalData": "asdf"
}
}
This is then read into the following object
public class AppConfiguration
{
public Dictionary<ConfigType, ConfigurationData> Configurations {get; set;} = new Dictionary<ConfigType, ConfigurationData>();
}
public class ConfigurationData
{
public HashSet<int> Ids {get;set;} = new HashSet<int>();
public string OptionalData = "";
}
public Enum ConfigType
{
KeyA = 1,
KeyB = 2,
KeyC = 3
}
I then bind this in ConfigureServices(IServiceCollection services) method using
services.Configure<AppConfiguration>(this.Configuration);
However, I notied that the configuration binding produces my AppConfiguration's dictionary with only the KeyB and KeyC keys, skipping KeyA, because its Ids array is empty. I read up on the behaviour of the configuration binding online, but as far as I saw it should bind the Ids to null, but here it just does not generate a key value pair in the dictionary altogether.
I tried removing the "Ids" property, leaving my config like "KeyA": {}", but this still did not work. The only way I can get it to parse is if I put in some numbers in the array, but this obviously not what I want.
I would like to know if there is anyway I can bind such a key-value pair, where I don't have any Ids in my array. This seems like it should be somehow supported out of the box, but I'm not sure why it's not working and how could I resolve it, without implementing some hacky custom configuration loader/binder.
For anyone who stumbles upon this in the future, I managed to solve this by setting my array to null, instead of an empty array like so.
"Configurations": {
"KeyA": {
"Ids": null
}
}

How to change JSON returned by query using Helidon 2.0.0-M-2

I'm using Helidon 2.0.0-M2.
When I run the query below I get back a list of JSON objects.
dbClient.execute(exec -> exec.createNamedQuery("select-dsitem-by-id")
.addParam("userId", dataItemId)
.execute())
.thenAccept(response::send)
.exceptionally(throwable -> sendError(throwable, response));
Returned list
[
{
"data": "qwerty",
"user_id": "12345"
},
{
"data": "qwerty123",
"user_id": "22345"
}
]
The attribute names seem to be taken directly from the database column name. e.g. one attribute name returned is "user_id". However, I want it to be "userId". I also want to create a parent wrapper for this list like:
{
"userList": [
{
"data": "qwerty",
"user_id": "12345"
},
{
"data": "qwerty123",
"user_id": "22345"
}
]
}
What is the best way to do this with the dbclient?
Thanks
Simple approach:
Change your SQL statement to return the correct name, such as:
SELECT data, user_id as userId FROM mytable
Complicated approach:
We are working on a better support to map to a JSON stream.
Currently there is only one (a bit complicated) way to achieve this:
You can create a custom mapper from a DbRow to JsonObject. This mapper needs to be a general one (it must work for any DbRow of any query).
The built-in mapper uses metadata provided on the columns. I have prepared a simple example (that just expects to have a single type of statements):
class DbRecordMapperProvider implements DbMapperProvider {
private static final DbMapper<JsonObject> MAPPER = new DbRecordMapper();
#SuppressWarnings("unchecked")
#Override
public <T> Optional<DbMapper<T>> mapper(Class<T> aClass) {
if (JsonObject.class.equals(aClass)) {
return Optional.of((DbMapper<T>)MAPPER);
}
return Optional.empty();
}
}
class DbRecordMapper implements DbMapper<JsonObject> {
#Override
public JsonObject read(DbRow dbRow) {
return Json.createObjectBuilder()
.add("name", dbRow.column("FIRSTPART").as(String.class))
.add("message", dbRow.column("SECONDPART").as(String.class))
.build();
}
#Override
public Map<String, ?> toNamedParameters(JsonObject dbRecord) {
return dbRecord;
}
#Override
public List<?> toIndexedParameters(JsonObject dbRecord) {
throw new IllegalStateException("Cannot convert json object to indexed parameters");
}
}
The important method is public JsonObject read(DbRow dbRow).
Once you have such a DbMapperProvider, you register it with the DbClient:
dbClient = DbClient.builder()
.config(config.get("db"))
.mapperProvider(new DbRecordMapperProvider())
.build();

RestSharp returns object with null values

I'm using RestSharp 106.6.10. I expect the response to return a simple object with four string properties. I've defined the object's class in C# with the correct property types, spelling, and capitalization.
I'm posting a request and getting a response with no errors. The response's Content appears to be clean JSON, with a Data object that includes good values for my "TransactionAcknowledgement" object. But here's my problem: the Data object returned by RestSharp has null values for the deserialized properties.
I've tried several suggestions from SO and other sites: Adding "DeserializeAs" hints on the class properties; using OnBeforeDeserialization to set ContentType = "application/json"; and a few other ideas others have suggested. Nothing seems to help.
Here is the class definition:
public class TransactionAcknowledgement
{
[DeserializeAs(Name = "BusinessEntityID")] // These hints don't seem to help.
public string BusinessEntityID { get; set; }
[DeserializeAs(Name = "BusinessEntityMedicaidIdentifier")]
public string BusinessEntityMedicaidIdentifier { get; set; }/
[DeserializeAs(Name = "TransactionID")]
public string TransactionID { get; set; }
[DeserializeAs(Name = "Reason")]
public string Reason { get; set; }
}
And a C# code snippet:
RestClient restClient;
RestRequest restRequest;
IRestResponse<TransactionAcknowledgement> restResponse;
restClient = new RestClient(MyBaseUrl);
restClient.Authenticator = new HttpBasicAuthenticator(evvCompanyAggregator.EffectiveUserID, evvCompanyAggregator.EffectiveUserPassword);
restRequest = new RestRequest("MyResource", Method.POST, DataFormat.Json);
restRequest.AddJsonBody(myData);
restRequest.OnBeforeDeserialization = r => { r.ContentType = "application/json"; }; // This doesn't seem to help.
restResponse = restClient.Execute<TransactionAcknowledgement>(restRequest);
The restResponse.Content looks good, including good values in the "Data" object within the raw restResponse.Content string.
The restResponse.Content.Data object is also the correct class, TransactionAcknowledgement.
However, the property values within restResponse.Content.Data are all null.
Here is the raw restResponse.Content string:
"{\n \"id\": \"1de51d1a-8086-4ba7-8aa6-2d1431986a99\",\n \"status\": null,\n \"token\": null,\n \"messageSummary\": \"Transaction Received.\",\n \"messageDetail\": null,\n \"errorMessage\": null,\n \"failedCount\": 0,\n \"succeededCount\": 0,\n \"cached\": false,\n \"cachedDate\": null,\n \"totalRows\": 0,\n \"page\": 0,\n \"pageSize\": 0,\n \"orderByColumn\": null,\n \"orderByDirection\": null,\n \"data\": {\n \"BusinessEntityID\": \"200248\",\n \"BusinessEntityMedicaidIdentifier\": \"8471209\",\n \"TransactionID\": \"1de51d1a-8086-4ba7-8aa6-2d1431986a99\",\n \"Reason\": \"Transaction Received.\"\n }\n}"
And here is the same restResponse.Content nicely formatted for readability:
{
"id": "bf0606a3-f21d-4d51-980c-bee407adc561",
"status": null,
"token": null,
"messageSummary": "Transaction Received.",
"messageDetail": null,
"errorMessage": null,
"failedCount": 0,
"succeededCount": 0,
"cached": false,
"cachedDate": null,
"totalRows": 0,
"page": 0,
"pageSize": 0,
"orderByColumn": null,
"orderByDirection": null,
"data": {
"BusinessEntityID": "200248",
"BusinessEntityMedicaidIdentifier": "8471209",
"TransactionID": "bf0606a3-f21d-4d51-980c-bee407adc561",
"Reason": "Transaction Received."
}
}
And here is the object in the debugger, showing null property values:
I've spent a couple of days on this. I'm sure this works for most people, so I'm probably just missing something simple. Any suggestions?
Update:
To prove that the data coming in with the response is ok and matches my class definition, I deserialized the response content directly. I created a new wrapper class:
public class HttpRestRootObject
{
public TransactionAcknowledgement data { get; set; }
}
Then, I deserialized the the restResponse.Content string without RestSharp:
HttpRestRootObject testRoot = javaScriptSerializer.Deserialize<HttpRestRootObject>(restResponse.Content);
TransactionAcknowledgement transactionAcknowledgement = testRoot.data;
So, everything works except RestSharp's deserialization. Am I using RestSharp wrong somehow? Any suggestions?
I have the exact same problem. The only fix I've found is to revert back to RestSharp 106.10.1. All 106.11.x have the same problem. It's also possible to declare your own custom serializer, which could be simply NewtonSoft, however, depending on the complexity of what you are serializing/deserializing this could be more headache than it's worth.

RestSharp deserialization containing both a value and a dictionary under the root element

I am trying to construct POCO classes so that RestSharp can deserialize a particular JSON result.
The sticking point seems to be that the root object contains both a nb_results key, and a dictionary of other keys ("0:", "1:", etc. each with a complex value).
I have tried both Dictionary(Of Integer, mediaGalleryData) and Dictionary(Of String, mediaGalleryData). Neither works. nb_results always serializes, but the dictionary never does.
{
"nb_results": 2,
"0": {
"id": 51976254,
"title": "Bumblebee species Bombus terrestris",
"media_type_id": 1,
"creator_name": "paulrommer",
"creator_id": 201446851,
"thumbnail_url": "http:\/\/t1.ftcdn.net\/jpg\/00\/51\/97\/62\/110_F_51976254_dVCbgGVey5xEuLkvk1e4vhnmPqxI4J0X.jpg",
"thumbnail_html_tag": "<img src=\"http:\/\/t1.ftcdn.net\/jpg\/00\/51\/97\/62\/110_F_51976254_dVCbgGVey5xEuLkvk1e4vhnmPqxI4J0X.jpg\" alt=\"Bumblebee species Bombus terrestris\" title=\"Photo: Bumblebee species Bombus terrestris\" width=\"110\" height=\"73\" \/>",
"thumbnail_width": 110,
"thumbnail_height": 73,
"affiliation_link": "http:\/\/api.fotolia.com\/id\/51976254",
"thumbnail_30_url": "http:\/\/t1.ftcdn.net\/jpg\/00\/51\/97\/62\/30_F_51976254_dVCbgGVey5xEuLkvk1e4vhnmPqxI4J0X.jpg",
"thumbnail_30_width": 30,
"thumbnail_30_height": 20,
"thumbnail_110_url": "http:\/\/t1.ftcdn.net\/jpg\/00\/51\/97\/62\/110_F_51976254_dVCbgGVey5xEuLkvk1e4vhnmPqxI4J0X.jpg",
"thumbnail_110_width": 110,
"thumbnail_110_height": 73,
"thumbnail_400_url": "http:\/\/t1.ftcdn.net\/jpg\/00\/51\/97\/62\/400_F_51976254_dVCbgGVey5xEuLkvk1e4vhnmPqxI4J0X.jpg",
"thumbnail_400_width": 400,
"thumbnail_400_height": 267,
"licenses": [
{
"name": "XS",
"price": 1
},
{
"name": "S",
"price": 3
},
{
"name": "M",
"price": 5
},
{
"name": "L",
"price": 7
},
{
"name": "XL",
"price": 8
},
{
"name": "XXL",
"price": 10
},
{
"name": "X",
"price": 100
}
]
},
"1": {
"id": 44488015,
"title": "Vintage Style Birds, Bees and Banners Vector Set",
"media_type_id": 3,
"creator_name": "artnerdluxe",
"creator_id": 203491263,
"thumbnail_url": "http:\/\/t1.ftcdn.net\/jpg\/00\/44\/48\/80\/110_F_44488015_hvEpYPw8yILDsnAi6BChYWXtzmiH6jWd.jpg",
"thumbnail_html_tag": "<img src=\"http:\/\/t1.ftcdn.net\/jpg\/00\/44\/48\/80\/110_F_44488015_hvEpYPw8yILDsnAi6BChYWXtzmiH6jWd.jpg\" alt=\"Vintage Style Birds, Bees and Banners Vector Set\" title=\"Vector: Vintage Style Birds, Bees and Banners Vector Set\" width=\"105\" height=\"110\" \/>",
"thumbnail_width": 105,
"thumbnail_height": 110,
"affiliation_link": "http:\/\/api.fotolia.com\/id\/44488015",
"thumbnail_30_url": "http:\/\/t1.ftcdn.net\/jpg\/00\/44\/48\/80\/30_F_44488015_hvEpYPw8yILDsnAi6BChYWXtzmiH6jWd.jpg",
"thumbnail_30_width": 29,
"thumbnail_30_height": 30,
"thumbnail_110_url": "http:\/\/t1.ftcdn.net\/jpg\/00\/44\/48\/80\/110_F_44488015_hvEpYPw8yILDsnAi6BChYWXtzmiH6jWd.jpg",
"thumbnail_110_width": 105,
"thumbnail_110_height": 110,
"thumbnail_400_url": "http:\/\/t1.ftcdn.net\/jpg\/00\/44\/48\/80\/400_F_44488015_hvEpYPw8yILDsnAi6BChYWXtzmiH6jWd.jpg",
"thumbnail_400_width": 380,
"thumbnail_400_height": 400,
"licenses": [
{
"name": "XS",
"price": 1
},
{
"name": "S",
"price": 3
},
{
"name": "M",
"price": 5
},
{
"name": "L",
"price": 7
},
{
"name": "XL",
"price": 8
},
{
"name": "XXL",
"price": 10
},
{
"name": "V",
"price": 5
},
{
"name": "XV",
"price": 40
}
]
}
}
Here are the classes I have so far:
Public Class mediaGallery
Public Property nb_results As Integer
Public Property results As Dictionary(Of String, mediaGalleryData)
End Class
Public Class mediaGalleryData
Public Property id As Integer
Public Property title As String
Public Property creator_name As String
Public Property creator_id As Integer
Public Property media_type_id As Integer
Public Property thumbnail_url As String
Public Property thumbnail_width As Integer
Public Property thumbnail_height As Integer
Public Property thumbnail_html_tag As String
Public Property thumbnail_30_url As String
Public Property thumbnail_30_width As Integer
Public Property thumbnail_30_height As Integer
Public Property thumbnail_110_url As String
Public Property thumbnail_110_width As Integer
Public Property thumbnail_110_height As Integer
Public Property thumbnail_400_url As String
Public Property thumbnail_400_width As Integer
Public Property thumbnail_400_height As Integer
Public Property licenses As List(Of licensedata)
End Class
Public Class licensedata
Public Property name As String
Public Property price As Integer
End Class
My RestSharp-consuming code:
Public Function getUserGalleryMedias(page As Integer?, nb_per_page As Integer?, thumbnail_size As Integer?, id As String, detail_level As Integer?) As mediaGallery
Dim request = New RestRequest("user/getUserGalleryMedias", Method.GET)
If page.HasValue Then request.AddParameter("page", page.Value)
If nb_per_page.HasValue Then request.AddParameter("nb_per_page", nb_per_page.Value)
If thumbnail_size.HasValue Then request.AddParameter("thumbnail_size", thumbnail_size.Value)
If Not String.IsNullOrEmpty(id) Then request.AddParameter("id", id)
If detail_level.HasValue Then request.AddParameter("detail_level", detail_level.Value)
Return Execute(Of mediaGallery)(request)
End Function
I've accomplished what I needed to do, but I had to go a bit of the long way around.
As suggested (but not very well explained) in the RestSharp documentation on the wiki, I implemented IDeserializer to handle this class specially. I had to add SimpleJson to the website separately, since it is marked Friend inside RestSharp. And instead of registering my deserializer as a handler with the RestClient, which would have required my deserializer to handle ALL classes, I just invoked it inside my API class's Execute(Of T) method, for just the single case where T is the troublesome class.
The Execute(Of T) method:
Public Function Execute(Of T As New)(request As RestRequest) As T
Dim client = New RestClient()
' set up client with authenticator
client.BaseUrl = BaseURL
If String.IsNullOrEmpty(_sessionID) Then
client.Authenticator = New HttpBasicAuthenticator(_apikey, "")
Else
client.Authenticator = New HttpBasicAuthenticator(_apikey, _sessionID)
End If
' used on every request
Dim response = client.Execute(Of T)(request)
' Check for exceptions or bad status code
If response.ErrorException IsNot Nothing Then
Throw response.ErrorException
ElseIf response.StatusCode > 299 Then
If response.Content.StartsWith("{""error""") Then
Throw New Exception(String.Format("Status {0} {1} : {2}", response.StatusCode, response.StatusDescription, response.Content))
Else
Throw New HttpException(response.StatusCode, response.StatusDescription)
End If
End If
' Handle troublesome class
If GetType(T).Equals(GetType(mediaGallery)) Then
Dim fjs As New fotoliaJSONDeSerializer
response.Data = fjs.Deserialize(Of T)(response)
End If
Return response.Data
End Function
The deserializer class:
''' <summary>
''' specific to the FotoliaAPI.mediaGallery we actually want to handle
''' </summary>
''' <param name="response"></param>
''' <returns></returns>
''' <remarks></remarks>
Private Function DeserializeMediaLibrary(response As IRestResponse) As FotoliaAPI.mediaGallery
Dim result As New FotoliaAPI.mediaGallery
' turn to an iDictionary so we can readily enumerate keys
Dim j As IDictionary(Of String, Object) = CType(SimpleJSON.SimpleJson.DeserializeObject(response.Content), IDictionary(Of String, Object))
For Each key In j.Keys
' this top level key is a direct property
If key = "nb_results" Then
result.nb_results = j(key)
Else
' all other keys become members of the list.
' Turn the value back to a string, then feed it back into SimpleJson to deserialize to the strongly typed subclass.
result.media.Add( _
SimpleJSON.SimpleJson.DeserializeObject(Of FotoliaAPI.mediaGalleryData)( _
j(key).ToString))
End If
Next
Return result
End Function
' required but unimportant properties of IDeserializer
Public Property DateFormat As String Implements Deserializers.IDeserializer.DateFormat
Public Property [Namespace] As String Implements Deserializers.IDeserializer.Namespace
Public Property RootElement As String Implements Deserializers.IDeserializer.RootElement
End Class

RavenDB poor select performance

I'm testing RavenDB for my future projects. Database performance is a must requirement for me, that's why I want to be able to tune RavenDB to be at least in SQL Server's performance range, but my tests shows that raven db is approximately 10x-20x slower in select queries than SQL Server, even when RavenDB is indexed and SQL Server doesn't have any indexes.
I populated database with 150k documents. Each document has a collection of child elements. Db size is approx. 1GB and so is index size too. Raven/Esent/CacheSizeMax is set to 2048 and Raven/Esent/MaxVerPages is set to 128.
Here's how the documents looks like:
{
"Date": "2028-09-29T01:27:13.7981628",
"Items": [
{
{
"ProductId": "products/673",
"Quantity": 26,
"Price": {
"Amount": 2443.0,
"Currency": "USD"
}
},
{
"ProductId": "products/649",
"Quantity": 10,
"Price": {
"Amount": 1642.0,
"Currency": "USD"
}
}
],
"CustomerId": "customers/10"
}
public class Order
{
public DateTime Date { get; set; }
public IList<OrderItem> Items { get; set; }
public string CustomerId { get; set; }
}
public class OrderItem
{
public string ProductId { get; set; }
public int Quantity { get; set; }
public Price Price { get; set; }
}
public class Price
{
public decimal Amount { get; set; }
public string Currency { get; set; }
}
Here's the defined index:
from doc in docs.Orders
from docItemsItem in ((IEnumerable<dynamic>)doc.Items).DefaultIfEmpty()
select new { Items_Price_Amount = docItemsItem.Price.Amount, Items_Quantity = docItemsItem.Quantity, Date = doc.Date }
I defined the index using Management studio, not from code BTW (don't know if it has any negative/positive effect on perfromance).
This query takes from 500ms to 1500ms to complete (Note that this is the time that is needed to execute the query, directly shown from ravendb's console. So it doesn't contain http request time and deserialization overhead. Just query execution time).
session.Query<Order>("OrdersIndex").Where(o =>
o.Items.Any(oi => oi.Price.Amount > 0 && oi.Quantity < 100)).Take(128).ToList();
I'm running the query on quad core i5 cpu running at 4.2 GHz and the db is located on a SSD.
Now when I populated same amount of data on sql server express, with same schema and same amount of associated objects. without index, sql server executes the same query which includes joins in 35ms. With index it takes 0ms :|.
All tests were performed when db servers were warmed up.
Though, I'm still very satisfied with RavenDB's performance, I'm curious if I am missing something or RavenDB is slower than a relational database?
Sorry for my poor english.
Thanks
UPDATE
Ayande, I tried what you suggested, but when I try to define the index you sent me, I get the following error:
public Index_OrdersIndex()
{
this.ViewText = #"from doc in docs.Orders
select new { Items_Price_Amount = doc.Items(s=>s.Price.Amount), Items_Quantity = doc.Items(s=>s.Quantity), Date = doc.Date }
";
this.ForEntityNames.Add("Orders");
this.AddMapDefinition(docs => from doc in docs
where doc["#metadata"]["Raven-Entity-Name"] == "Orders"
select new { Items_Price_Amount = doc.Items(s => s.Price.Amount), Items_Quantity = doc.Items.(s => s.Quantity), Date = doc.Date, __document_id = doc.__document_id });
this.AddField("Items_Price_Amount");
this.AddField("Items_Quantity");
this.AddField("Date");
this.AddField("__document_id");
this.AddQueryParameterForMap("Date");
this.AddQueryParameterForMap("__document_id");
this.AddQueryParameterForReduce("Date");
this.AddQueryParameterForReduce("__document_id");
}
}
error CS1977: Cannot use a lambda expression as an argument to a dynamically dispatched operation without first casting it to a delegate or expression tree type
Davita,
The following index generate ~8 million index entries:
from doc in docs.Orders
from docItemsItem in ((IEnumerable<dynamic>)doc.Items).DefaultIfEmpty()
select new { Items_Price_Amount = docItemsItem.Price.Amount, Items_Quantity = docItemsItem.Quantity, Date = doc.Date }
This one generates far less:
from doc in docs.Orders
select new { Items_Price_Amount = doc.Items(s=>s.Price.Amount), Items_Quantity = doc.Items.(s=>s.Quantity), Date = doc.Date }
And can be queried with the same results, but on our tests showed up to be about twice as fast.
The major problem is that you are making several range queries, which are expensive with a large number of potential values, and then you have a large number of actual matches for the query.
Doing an exact match is significantly faster, by the way.
We are still working on ways to try to speed things up.