Another failure at deserializing data with discriminated unions, in F# - serialization

Following a question where the answer provided a working solution to serialize / deserialize discriminated unions (IgnoreMissingMember setting doesn't seem to work with FSharpLu.Json deserializer)
I have now a practical case where this fails (although it works in simpler cases).
here is the test code:
open System.Collections.Generic
open Microsoft.FSharpLu.Json
open Newtonsoft.Json
open Newtonsoft.Json.Serialization
// set up the serialization / deserialization based on answer from:
// https://stackoverflow.com/questions/62364229/ignoremissingmember-setting-doesnt-seem-to-work-with-fsharplu-json-deserializer/62364913#62364913
let settings =
JsonSerializerSettings(
NullValueHandling = NullValueHandling.Ignore,
Converters = [| CompactUnionJsonConverter(true, true) |]
)
let serialize object =
JsonConvert.SerializeObject(object, settings)
let deserialize<'a> object =
JsonConvert.DeserializeObject<'a>(object, settings)
// define the type used
type BookSide =
| Bid
| Ask
type BookEntry =
{
S : float
P : float
}
type BookSideData =
Dictionary<int, BookEntry>
type BookData =
{
Data: Dictionary<BookSide, BookSideData>
}
static member empty =
{
Data = Dictionary<BookSide, BookSideData>(dict [ (BookSide.Bid, BookSideData()); (BookSide.Ask, BookSideData()) ])
}
// make some sample data
let bookEntry = { S=3.; P=5. }
let bookData = BookData.empty
bookData.Data.[BookSide.Bid].Add(1, bookEntry)
// serialize. This part works
let s = serialize bookData
// deserialize. This part fails
deserialize<BookData> s
the serialized test data will look like this:
{"Data":{"Bid":{"1":{"S":3.0,"P":5.0}},"Ask":{}}}
but deserializing will crash like this:
Could not convert string 'Bid' to dictionary key type 'FSI_0023+BookSide'. Create a TypeConverter to convert from the string to the key type object.
although the serialization / deserialization of the DU through FSharpLu which has a DU converter.
The reason I am trying to find some automated solution, vs writing a custom TypeConverter (besides the fact I've never done it) is that I have a lot of types I do not control to go through.
Here is a fiddle:
https://dotnetfiddle.net/Sx0k4x

Your basic problem is that you are using BookSide as a dictionary key -- but this is an f# union which makes it a complex key -- one not immediately convertible to and from a string. Unfortunately Json.NET does not support complex dictionary keys out of the box as is stated in its Serialization Guide:
When serializing a dictionary, the keys of the dictionary are converted to strings and used as the JSON object property names. The string written for a key can be customized by either overriding ToString() for the key type or by implementing a TypeConverter. A TypeConverter will also support converting a custom string back again when deserializing a dictionary.
There are two basic approaches to handling this issue:
Implement a TypeConverter as is shown in, e.g., Not ableTo Serialize Dictionary with Complex key using Json.net.
Serialize the dictionary as an array of key/value pair objects e.g. as is shown in Serialize dictionary as array (of key value pairs).
Since your data model includes dictionaries with a variety of keys (DU, strings and ints) the second solution would appear to be the only possibility. The following DictionaryConverter should have the necessary logic:
let inline isNull (x:^T when ^T : not struct) = obj.ReferenceEquals (x, null)
type Type with
member t.BaseTypesAndSelf() =
t |> Seq.unfold (fun state -> if isNull state then None else Some(state, state.BaseType))
member t.DictionaryKeyValueTypes() =
t.BaseTypesAndSelf()
|> Seq.filter (fun i -> i.IsGenericType && i.GetGenericTypeDefinition() = typedefof<Dictionary<_,_>>)
|> Seq.map (fun i -> i.GetGenericArguments())
type JsonReader with
member r.ReadAndAssert() =
if not (r.Read()) then raise (JsonReaderException("Unexpected end of JSON stream."))
r
member r.MoveToContentAndAssert() =
if r.TokenType = JsonToken.None then r.ReadAndAssert() |> ignore
while r.TokenType = JsonToken.Comment do r.ReadAndAssert() |> ignore
r
type internal DictionaryReadOnlySurrogate<'TKey, 'TValue>(i : IDictionary<'TKey, 'TValue>) =
interface IReadOnlyDictionary<'TKey, 'TValue> with
member this.ContainsKey(key) = i.ContainsKey(key)
member this.TryGetValue(key, value) = i.TryGetValue(key, &value)
member this.Item with get(index) = i.[index]
member this.Keys = i.Keys :> IEnumerable<'TKey>
member this.Values = i.Values :> IEnumerable<'TValue>
member this.Count = i.Count
member this.GetEnumerator() = i.GetEnumerator()
member this.GetEnumerator() = i.GetEnumerator() :> IEnumerator
type DictionaryConverter () =
// ReadJson adapted from this answer https://stackoverflow.com/a/28633769/3744182
// To https://stackoverflow.com/questions/28451990/newtonsoft-json-deserialize-dictionary-as-key-value-list-from-datacontractjsonse
// By https://stackoverflow.com/users/3744182/dbc
inherit JsonConverter()
override this.CanConvert(t) = (t.DictionaryKeyValueTypes().Count() = 1) // If ever implemented for IReadOnlyDictionary<'TKey, 'TValue> then reject DictionaryReadOnlySurrogate<'TKey, 'TValue>
member private this.ReadJsonGeneric<'TKey, 'TValue> (reader : JsonReader, t : Type, existingValue : obj, serializer : JsonSerializer) : obj =
let contract = serializer.ContractResolver.ResolveContract(t)
let dict = if (existingValue :? IDictionary<'TKey, 'TValue>) then existingValue :?> IDictionary<'TKey, 'TValue> else contract.DefaultCreator.Invoke() :?> IDictionary<'TKey, 'TValue>
match reader.MoveToContentAndAssert().TokenType with
| JsonToken.StartArray ->
let l = serializer.Deserialize<List<KeyValuePair<'TKey, 'TValue>>>(reader)
for p in l do dict.Add(p)
dict :> obj
| JsonToken.StartObject ->
serializer.Populate(reader, dict)
dict :> obj
| JsonToken.Null -> null // Or throw an exception if you prefer
| _ -> raise (JsonSerializationException(String.Format("Unexpected token {0}", reader.TokenType)))
override this.ReadJson(reader, t, existingValue, serializer) =
let keyValueTypes = t.DictionaryKeyValueTypes().Single(); // Throws an exception if not exactly one.
let m = typeof<DictionaryConverter>.GetMethod("ReadJsonGeneric", BindingFlags.NonPublic ||| BindingFlags.Instance ||| BindingFlags.Public);
m.MakeGenericMethod(keyValueTypes).Invoke(this, [| reader; t; existingValue; serializer |])
member private this.WriteJsonGeneric<'TKey, 'TValue> (writer : JsonWriter, value : obj, serializer : JsonSerializer) =
let dict = value :?> IDictionary<'TKey, 'TValue>
let keyContract = serializer.ContractResolver.ResolveContract(typeof<'Key>)
// Wrap the value in an enumerator or read-only surrogate to prevent infinite recursion.
match keyContract with
| :? JsonPrimitiveContract -> serializer.Serialize(writer, new DictionaryReadOnlySurrogate<'TKey, 'TValue>(dict))
| _ -> serializer.Serialize(writer, seq { yield! dict })
()
override this.WriteJson(writer, value, serializer) =
let keyValueTypes = value.GetType().DictionaryKeyValueTypes().Single(); // Throws an exception if not exactly one.
let m = typeof<DictionaryConverter>.GetMethod("WriteJsonGeneric", BindingFlags.NonPublic ||| BindingFlags.Instance ||| BindingFlags.Public);
m.MakeGenericMethod(keyValueTypes).Invoke(this, [| writer; value; serializer |])
()
Which you would add to settings as follows:
let settings =
JsonSerializerSettings(
NullValueHandling = NullValueHandling.Ignore,
Converters = [| CompactUnionJsonConverter(true, true); DictionaryConverter() |]
)
And generates the following JSON for your bookData:
{
"Data": [
{
"Key": "Bid",
"Value": [
{
"Key": 1,
"Value": {
"S": 3.0,
"P": 5.0
}
}
]
},
{
"Key": "Ask",
"Value": []
}
]
}
Notes:
The converter works for all Dictionary<TKey, TValue> types (and subtypes).
The converter detects whether the dictionary keys will be serialized using a primitive contract, and if so, serializes the dictionary compactly as a JSON object. If not the dictionary is serialized as an array. You can observe this in the JSON shown above: the Dictionary<BookSide, BookSideData> dictionary is serialized as a JSON array, and the Dictionary<int, BookEntry> dictionary is serialized as a JSON object.
During deserialization the converter detects whether the incoming JSON value is an array or object, and adapts as required.
The converter is only implemented for the mutable .Net Dictionary<TKey, TValue> type. The logic would require some slight modification to deserialize the immutable Map<'Key,'Value> type.
Demo fiddle here.

Related

List of array object iteration in java 8

I am setting the type for sql query result set as List in java. I am trying to convert it into
a dto.
When I see the List<Object[]> structure from query. It shows
resultList-ArrayList<E>
[0....9999]
[0..99]
[0]=Object[3]
[0]="jjj"
[1]="8787"
[2]="7686"
So is this expected. How can I access the object values here(jjj,8787...) by setting it to dto.
I tried something like this
List<Dto> dtoList = resultList.stream().map(obj->{
Dto dt = new Dto()
dt.setName(obj[0]);
).collect(Collectors.toList())
This is not correct as I am not able to access the object
Should I do another level of iteration in order to reach that object or is my generic type for result set is right
Thanks
Try this:
Object[] inner = new Object[]{"jjj", "8787", "7686"};
Object[] outer = new Object[]{inner};
List<Object[]> resultList = new ArrayList<>();
resultList.add(outer);
List<Dto> dtos;
dtos = resultList.stream()
.flatMap((Object[] objArr) -> {
Object[] subArr = (Object[]) objArr[0];
return Arrays.asList(subArr).stream()
.map(obj -> obj.toString());
})
.map(name -> {
Dto dto = new Dto();
dto.setName(name);
return dto;
})
.collect(Collectors.toList());

Type in Array changed when Converting Objective-C Class to Swift

I post a request to get some json data and try to convert the data to Model
var newDrugs = Array<DxyDrugInfo>()
newDrugs = self.parseDrugJsonToModels(data) ?? []
private func parseDrugJsonToModels(data: NSData) -> Array<DxyDrugInfo>?{
let json = JSON(data: data)
if let drugsArray = json["data"].arrayObject as? Array<[String : AnyObject]>, success = json["success"].bool where success {
var sortedArray = Array<DxyDrugInfo>()
for drugDic in drugsArray {
let drugInfo = DxyDrugInfo()
drugInfo.setValuesForKeysWithDictionary(drugDic)
sortedArray.append(drugInfo)
}
return sortedArray
}
return nil
}
DxyDrugInfo is a Objective-C Model Class
My question is the sortedArray's type is Array of DxyDrugInfo, and it changed to
when assigned to newDrugs.
I want to convert it to DxyDrugInfo and I also want to what the type is.
Thank you
That type you're looking at, #lvalue [DxyDrugInfo], is "array of DxyDrugInfo". You're already done! The square brackets around a type are shorthand for "array", that is:
[DxyDrugInfo] == Array<DxyDrugInfo>
and #lvalue just means that newDrugs is a variable you can assign to. (It's the opposite of "r-value", which is a value you could assign to an "l-value".)

What local-storage tooling does WebSharper provide?

In looking at the documentation for WebSharper's local storage, the SetItem item is string * string -> unit (and GetItem is string -> string).
This means that I'll need to convert anything I want to store into strings and do the reverse to retrieve them. Or, to put it in another way, I'll need to serialize and de-serialize them. Is there a way to use the behind-the-scenes conversion that WebSharper already does for RPC calls, or am I stuck with using a server-side library like FsPicker?
Not built in yet, I have been using this helper module to make local storage usable the same way as a ref cell:
open IntelliFactory.WebSharper
// Helper for handling localstorage, making a stored value work like a ref cell.
[<JavaScript; AutoOpen>]
module LocalStorage =
open IntelliFactory.WebSharper.Html5
let localStorage = Window.Self.LocalStorage
type IValue<'T> =
abstract Value: 'T with get, set
let [<Inline>] ( ! ) (x: IValue<_>) = x.Value
let [<Inline>] ( := ) (x: IValue<_>) v = x.Value <- v
// Redefining Ref to use IValue
type Ref<'T> (value: 'T) =
let mutable v = value
interface IValue<'T> with
member this.Value
with get() = v
and set value = v <- value
let [<Inline>] ref v = Ref v
let incr i = i := !i + 1
let decr i = i := !i - 1
type IStorageItem<'T> =
inherit IValue<'T>
abstract Save: unit -> unit
abstract Delete: unit -> unit
type JSONStorageItem<'T>(key, defaultVal) =
let mutable value = None
let getValue() =
match value with
| Some v -> v
| _ ->
let v =
match localStorage.GetItem key with
| null -> defaultVal
| s ->
Json.Parse s :?> _
value <- Some v
v
interface IStorageItem<'T> with
member this.Value
with get() = getValue()
and set v =
try localStorage.SetItem(key, Json.Stringify v)
value <- Some v
with _ -> JavaScript.Alert "Saving data to storage failed."
member this.Save() =
try localStorage.SetItem(key, Json.Stringify (getValue()))
with _ -> JavaScript.Alert "Saving data to storage failed."
member this.Delete() =
localStorage.RemoveItem key
value <- None
let [<Inline>] getJSONStorage key defaultVal = JSONStorageItem<_>(key, defaultVal) :> IStorageItem<_>
However this can currently only stringify/parse straight data objects: record, list, array, tuple and union types are ok, but no prototypes are restored.

How to use a dynamically-determined Type as parameter to a Lambda<Func<>>?

I am dynamically creating a Lambda expression (based on user input but at the moment using dummy values for a proof-of-concept) for a type which I will only know at runtime. I therefore need to pass the T portion of the Func<T,TResult> as a dynamic type, since I won't know the type until runtime (TResult will always be a bool).
It seems that I cannot pass in a Type variable or use typeof with generics. Basically I'm trying to do something like this:
// (f => f.Baz == 1)
Type theType = Type.GetType("Foo");
ParameterExpression pe = Expression.Parameter(theType, "f");
Expression left = Expression.Property(pe, theType.GetProperty("Baz"));
Expression right = Expression.Constant(1);
Expression expr = Expression.Equal(left, right);
// This works fine but uses a hard-coded type, which I won't know until runtime:
// var lambda = Expression.Lambda<Func<Foo,bool>>(expr, new ParameterExpression[] { pe }).Compile();
var lambda = Expression.Lambda<Func<theType, bool>>(expr, new ParameterExpression[] { pe }).Compile();
However, I cannot use the variable theType as the T portion of the Func. How can I fix this?
No you can't.
For example, in C#, you can't:
Type t = typeof(int);
List<t> list = new List<t>();
or
object list = new List<t>();
Unless you use reflection, but then you have to put the list in an object, and you can use it only through reflection.
So if you want you can save your Func<> in an object (or a dynamic) but nothing more.
What you COULD do is always return Func<object, bool> and cast the object to the desidered type IN the lambda function (so use a Expression.Convert(pe, theType));
Or you could use the dynamic:
// lambda == Func<Foo, bool>
dynamic lamdba = Expression.Lambda(expr, new ParameterExpression[] { pe }).Compile();
bool res = lambda(myvalue);
or
// lambda == Func<Foo, bool>
Delegate lamdba = Expression.Lambda(expr, new ParameterExpression[] { pe }).Compile();
bool res = (bool)lambda2.DynamicInvoke(t);
To be taken "not as real" some benchmarks (in StopWatch.Ticks, look at them only for proportions) (Release Mode + Start Without Debugging + some useless cycles so that they are "hot"):
236384685 dynamic
56773593 Func<object, bool> + cast
10556024247 DynamicInvoke
as a note, Func<Foo, bool> has the same speed, so there isn't any speed lost in the extra cast.
You can see the code here http://ideone.com/qhnVP3

F# Serialization of Record types

I know how to serialize in F# using mutable objects, but is there a way to serialize/deserialize using record types using either XmlSerializer or the DataContractSerializer? looks like there is a way to do this for a discriminated union using the KnownType attribute, but i am looking for a way to use non-mutable records without default constructor...
Beginning with F# 3.0, serialization of record types is now supported by applying the CliMutableAttribute to the type. Example:
[<CLIMutable>]
type MyRecord = { Name : string; Age : int }
This example is taken from http://blogs.msdn.com/b/fsharpteam/archive/2012/07/19/more-about-fsharp-3.0-language-features.aspx, which includes a discussion of this feature and three other new features in F# 3.0: triple-quoted strings, automatic properties, and unused variable warnings.
The sample code for reading data from Freebase by Jomo Fisher uses DataContractJsonSerializer to load data into immutable F# records. The declaration of the record that he uses looks like this:
[<DataContract>]
type Result<'TResult> = { // '
[<field: DataMember(Name="code") >]
Code:string
[<field: DataMember(Name="result") >]
Result:'TResult // '
[<field: DataMember(Name="message") >]
Message:string }
The key point here is that the the DataMember attribute is attached to the underlying field that's actually used to store the data and not to the read-only property that the F# compiler generates (using the field: modifier on the attribute).
I'm not 100% sure if this is going to work with other types of serialization (probably not), but it may be a useful pointer to start with...
EDIT I'm not sure if I'm missing something here, but the following basic example works fine for me:
module Demo
#r "System.Runtime.Serialization.dll"
open System.IO
open System.Text
open System.Xml
open System.Runtime.Serialization
type Test =
{ Result : string[]
Title : string }
do
let sb = new StringBuilder()
let value = { Result = [| "Hello"; "World" |]; Title = "Hacking" }
let xmlSerializer = DataContractSerializer(typeof<Test>);
xmlSerializer.WriteObject(new XmlTextWriter(new StringWriter(sb)), value)
let sr = sb.ToString()
printfn "%A" sr
let xmlSerializer = DataContractSerializer(typeof<Test>);
let reader = new XmlTextReader(new StringReader(sr))
let obj = xmlSerializer.ReadObject(reader) :?> Test
printfn "Reading: %A" obj
EDIT 2 If you want to generate cleaner XML then you can add attributes like this:
[<XmlRoot("test")>]
type Test =
{ [<XmlArrayAttribute("results")>]
[<XmlArrayItem(typeof<string>, ElementName = "string")>]
Result : string[]
[<XmlArrayAttribute("title")>]
Title : string }
It doesn't use XmlSerializer or the DataContractSerializer, but Json.NET 6.0 includes nice F# support.
It looks like this:
type TestTarget =
{ a: string
b: int }
[<TestFixture>]
type JsonTests() =
[<Test>]
member x.``can serialize``() =
let objectUnderTest = { TestTarget.a = "isa"; b = 9 }
let jsonResult: string = Newtonsoft.Json.JsonConvert.SerializeObject(objectUnderTest)
printfn "json is:\n%s" jsonResult
let xmlResult = Newtonsoft.Json.JsonConvert.DeserializeXmlNode(jsonResult, "root")
printfn "xml is:\n%s" (xmlResult.OuterXml)
let jsonRoundtrip = Newtonsoft.Json.JsonConvert.DeserializeObject<TestTarget>(jsonResult)
printfn "json roundtrip: %A" jsonRoundtrip
let xmlAsJson = Newtonsoft.Json.JsonConvert.SerializeXmlNode(xmlResult, Newtonsoft.Json.Formatting.Indented, true)
printfn "object -> json -> xml -> json:\n%A" xmlAsJson
let xmlRoundtrip = Newtonsoft.Json.JsonConvert.DeserializeObject<TestTarget>(xmlAsJson)
printfn "xml roundtrip:\n%A" xmlRoundtrip
Assert.That(true, Is.False)
()
json is:
{"a":"isa","b":9}
xml is:
<root><a>isa</a><b>9</b></root>
json roundtrip: {a = "isa";
b = 9;}
object -> json -> xml -> json:
"{
"a": "isa",
"b": "9"
}"
xml roundtrip:
{a = "isa";
b = 9;}
You can use this series of annotations on the properties of classes to format the XML:
[XmlRoot("root")]
[XmlElement("some-element")]
[XmlAttribute("some-attribute")]
[XmlArrayAttribute("collections")]
[XmlArrayItem(typeof(SomeClass), ElementName = "item")]
I use the attributes on my c# classes, but deserialize in F# (c# classes are ina referenced lib).
in f#:
use file = new FileStream(filePath, FileMode.Open)
let serializer= XmlSerializer(typeof<SomeClass>)
let docs = serializer.Deserialize file :?> SomeClass