Flatbuffers: How to allow multiple types for a single field - schema

I'm writing a communication protocol schema for a list of parameters which can be of multiple values: uint64, float64, string or bool.
How can I set a table field to a union of multiple primitive scalar & non-scalar primitive type?
I've already tried using a union of those types, but I end up with the following error when building:
$ schemas/foobar.fbs:28: 0: error: type referenced but not defined
(check namespace): uint64, originally at: schemas/request.fbs:5
Here's the schema in its current state:
namespace Foobar;
enum RequestCode : uint16 { Noop, Get, Set, BulkGet, BulkSet }
union ParameterValue { uint64, float64, bool, string }
table Parameter {
name:string;
value:ParameterValue;
unit:string;
}
table Request {
code:RequestCode = Noop;
payload:[Parameter];
}
table Result {
request:Request;
success:bool = true;
payload:[Parameter];
}
The end result I'm looking for is the Request and Result tables to contain a list of parameters, where a parameter contains a name and value, and optionally the units.
Thx in advance!
Post-answer solution:
Here's what I came up with in the end, thx to Aardappel.
namespace foobar;
enum RequestCode : uint16 { Noop, Get, Set, BulkGet, BulkSet }
union ValueType { UnsignedInteger, SignedInteger, RealNumber, Boolean, Text }
table UnsignedInteger {
value:uint64 = 0;
}
table SignedInteger {
value:int64 = 0;
}
table RealNumber {
value:float64 = 0.0;
}
table Boolean {
value:bool = false;
}
table Text {
value:string (required);
}
table Parameter {
name:string (required);
valueType:ValueType;
unit:string;
}
table Request {
code:RequestCode = Noop;
payload:[Parameter];
}
table Result {
request:Request (required);
success:bool = true;
payload:[Parameter];
}

You currently can't put scalars directly in a union, so you'd have to wrap these in a table or a struct, where struct would likely be the most efficient, e.g.
struct UInt64 { u:uint64 }
union ParameterValue { UInt64, Float64, Bool, string }
This is because a union must be uniformly the same size, so it only allows types to which you can have an offset.
Generally though, FlatBuffers is a strongly typed system, and the schema you are creating here is undoing that by emulating dynamically typed data, since your data is essentially a list of (string, any type) pairs. You may be better off with a system designed for this particular use case, such as FlexBuffers (https://google.github.io/flatbuffers/flexbuffers.html, currently only C++) which explicitly has a map type that is all string -> any type pairs.
Of course, even better is to not store data so generically, but instead make a new schema for each type of request and response you have, and make parameter names into fields, rather than serialized data. This is by far the most efficient, and type safe.

Related

How do I use a variable as the data type for a different variable?

If I have the data type of something stored in the variable data_type, how can I create a new variable with the data type defined in this variable?
For example:
struct a {
var: String,
}
struct b {
var: String,
}
let var_type = "a";
let variable: var_type { var: "abc" }; // creates struct var_type
As long as you know all of your types at compile time, it is possible to transform unstructured data into typed data based on some value in the data. This is exactly what is done by the popular serde crate
Without knowing the use case, it's difficult to address the question precisely, yet the code below gives two examples about how to accomplish type-mapping using an enum (though match could be used to map any data to any type that is known at compile time).
enum VarType {
A(String),
B(String),
Unknown(String),
}
fn main() {
let _var1 = VarType::A("abc".to_string());
let _var2 = VarType::B("xyz".to_string());
let data = vec![("a", "abc"), ("b", "xyz")];
for item in data {
let (data_type, value) = item;
match data_type {
"a" => VarType::A(value.to_string()),
"b" => VarType::B(value.to_string()),
_ => VarType::Unknown(value.to_string()),
};
}
}
As Isak van Bakel, most said rust is static. However, if you have a list of all the possible structures, you can. (assuming your using serde here!). There is currently
a interesting question discussing polymorphic de-serialisation here, i suggest you take a look as it may help!
You can't. Rust is statically typed.

Golang database manager api concept, error with type assertion

The base concept creating a Database Manager API for getting data through an API. I am using the GORM for getting data of the instances of the strcuts. So there is 300-400 struct which represents the tables.
type Users struct {
ID int64
Name string
}
type Categories struct {
ID int64
Category string
}
The next step I implement a function which is return the correct instance of the struct by table name, what I get through the API endpoint param.
func GetModel(model string) interface{} {
switch model {
case "users":
return Users{}
case "categories"
return Categories{}
}
return false
}
After there is an operations struct where the only one field is the DB. There is methods, for example the GetLast() where I want to use the GORM db.Last(&users) function.
func (o Operations) GetLast(model string) interface{} {
modelStruct := GetModel(model)
.
.
.
return o.DB.Last(&modelStruct)
}
There is points so this is what I don't know. The current solution is not working because in this case it is an interface{} I need make a type assertion more info in this question. The type assertion is looks like:
func (o Operations) GetLast(model string) interface{} {
modelStruct := GetModel(model)
.
test := modelStruct.(Users)
.
return o.DB.Last(&test)
}
This solution working, but in this case I lost the modularity. I try using the reflect.TypeOf(modelStruct), but it is also not working because the result of the reflect.TypeOf is a reflect.Type, with is not a golang type.
Basically I solved the problem, for getting the model as a pointer, and after I return it back as a json file.
So my model is the following:
var Models = map[string]interface{}{
"users": new(Users),
"categories": new(Categories),
}
And it is return back a new model by table type. what I can use for gorm First() function. Then json Marshal it, and return.
func (o Operation) First(model string, query url.Values) string {
modelStruct := Models[model]
db := o.DB
db.First(modelStruct)
response, _ := json.Marshal(modelStruct)
clear(modelStruct)
return string(response)
}
Before the return I clear the model pointer because the First() function store callbacks for the latest queries.
func clear(v interface{}) {
p := reflect.ValueOf(v).Elem()
p.Set(reflect.Zero(p.Type()))
}

How can I tell Kotlin that an array or collection cannot contain nulls?

If I create an array, then fill it, Kotlin believes that there may be nulls in the array, and forces me to account for this
val strings = arrayOfNulls<String>(10000)
strings.fill("hello")
val upper = strings.map { it!!.toUpperCase() } // requires it!!
val lower = upper.map { it.toLowerCase() } // doesn't require !!
Creating a filled array doesn't have this problem
val strings = Array(10000, {"string"})
val upper = strings.map { it.toUpperCase() } // doesn't require !!
How can I tell the compiler that the result of strings.fill("hello") is an array of NonNull?
A rule of thumb: if in doubts, specify the types explicitly (there is a special refactoring for that):
val strings1: Array<String?> = arrayOfNulls<String>(10000)
val strings2: Array<String> = Array(10000, {"string"})
So you see that strings1 contains nullable items, while strings2 does not. That and only that determines how to work with these arrays:
// You can simply use nullability in you code:
strings2[0] = strings1[0]?.toUpperCase ?: "KOTLIN"
//Or you can ALWAYS cast the type, if you are confident:
val casted = strings1 as Array<String>
//But to be sure I'd transform the items of the array:
val asserted = strings1.map{it!!}
val defaults = strings1.map{it ?: "DEFAULT"}
Why the filled array works fine
The filled array infers the type of the array during the call from the lambda used as the second argument:
val strings = Array(10000, {"string"})
produces Array<String>
val strings = Array(10000, { it -> if (it % 2 == 0) "string" else null })
produces Array<String?>
Therefore changing the declaration to the left of the = that doesn't match the lambda does not do anything to help. If there is a conflict, there is an error.
How to make the arrayOfNulls work
For the arrayOfNulls problem, they type you specify to the call arrayOfNulls<String> is used in the function signature as generic type T and the function arrayOfNulls returns Array<T?> which means nullable. Nothing in your code changes that type. The fill method only sets values into the existing array.
To convert this nullable-element array to non-nullable-element list, use:
val nullableStrings = arrayOfNulls<String>(10000).apply { fill("hello") }
val strings = nullableStrings.filterNotNull()
val upper = strings.map { it.toUpperCase() } // no !! needed
Which is fine because your map call converts to a list anyway, so why not convert beforehand. Now depending on the size of the array this could be performant or not, the copy might be fast if in CPU cache. If it is large and no performant, you can make this lazy:
val nullableStrings = arrayOfNulls<String>(10000).apply { fill("hello") }
val strings = nullableStrings.asSequence().filterNotNull()
val upper = strings.map { it.toUpperCase() } // no !! needed
Or you can stay with arrays by doing a copy, but really this makes no sense because you undo it with the map:
val nullableStrings = arrayOfNulls<String>(10000).apply { fill("hello") }
val strings: Array<String> = Array(nullableStrings.size, { idx -> nullableStrings[idx]!! })
Arrays really are not that common in Java or Kotlin code (JetBrains studied the statistics) unless the code is doing really low level optimization. It could be better to use lists.
Given that you might end up with lists anyway, maybe start there too and give up the array.
val nullableStrings = listOf("a","b",null,"c",null,"d")
val strings = nullableStrings.filterNotNull()
But, if you can't stop the quest to use arrays, and really must cast one without a copy...
You can always write a function that does two things: First, check that all values are not null, and if so then return the array that is cast as not null. This is a bit hacky, but is safe only because the difference is nullability.
First, create an extension function on Array<T?>:
fun <T: Any> Array<T?>.asNotNull(): Array<T> {
if (this.any { it == null }) {
throw IllegalStateException("Cannot cast an array that contains null")
}
#Suppress("CAST_NEVER_SUCCEEDS")
return this as Array<T>
}
Then use this function new function to do the conversion (element checked as not null cast):
val nullableStrings = arrayOfNulls<String>(10000).apply { fill("hello") }
val strings = nullableStrings.asNotNull() // magic!
val upperStrings = strings.map { it.toUpperCase() } // no error
But I feel dirty even talking about this last option.
There is no way to tell this to the compiler. The type of the variable is determined when it is declared. In this case, the variable is declared as an array that can contain nulls.
The fill() method does not declare a new variable, it only modifies the contents of an existing one, so it cannot cause the variable type to change.

How can I work with SQL NULL values and JSON in a good way?

Go types like Int64 and String cannot store null values,
so I found I could use sql.NullInt64 and sql.NullString for this.
But when I use these in a Struct,
and generate JSON from the Struct with the json package,
then the format is different to when I use regular Int64 and String types.
The JSON has an additional level because the sql.Null*** is also a Struct.
Is there a good workaround for this,
or should I not use NULLs in my SQL database?
Types like sql.NullInt64 do not implement any special handling for JSON marshaling or unmarshaling, so the default rules apply. Since the type is a struct, it gets marshalled as an object with its fields as attributes.
One way to work around this is to create your own type that implements the json.Marshaller / json.Unmarshaler interfaces. By embedding the sql.NullInt64 type, we get the SQL methods for free. Something like this:
type JsonNullInt64 struct {
sql.NullInt64
}
func (v JsonNullInt64) MarshalJSON() ([]byte, error) {
if v.Valid {
return json.Marshal(v.Int64)
} else {
return json.Marshal(nil)
}
}
func (v *JsonNullInt64) UnmarshalJSON(data []byte) error {
// Unmarshalling into a pointer will let us detect null
var x *int64
if err := json.Unmarshal(data, &x); err != nil {
return err
}
if x != nil {
v.Valid = true
v.Int64 = *x
} else {
v.Valid = false
}
return nil
}
If you use this type in place of sql.NullInt64, it should be encoded as you expect.
You can test this example here: http://play.golang.org/p/zFESxLcd-c
If you use the null.v3 package, you won't need to implement any of the marshal or unmarshal methods. It's a superset of the sql.Null structs and is probably what you want.
package main
import "gopkg.in/guregu/null.v3"
type Person struct {
Name string `json:"id"`
Age int `json:"age"`
NickName null.String `json:"nickname"` // Optional
}
If you'd like to see a full Golang webserver that uses sqlite, nulls, and json you can consult this gist.

apache pig & piggybank avro union types

I have a record of union type of
union {TypeA, TypeB, TypeC, TypeD, TypeE} mydata;
I have the serialized data in avro format, however when I am trying to use piggybank.jar's AvroStorage function to load the avro data, it gives me the following error:
Caused by: java.io.IOException: We don't accept schema containing generic unions.
at org.apache.pig.piggybank.storage.avro.AvroSchema2Pig.convert(AvroSchema2Pig.java:54)
at org.apache.pig.piggybank.storage.avro.AvroStorage.getSchema(AvroStorage.java:384)
at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:174)
... 23 more
So, after reading the piggybank source code here https://github.com/triplel/pig/blob/branch-0.12/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/avro/AvroStorageUtils.java
/** determine whether a union is a nullable union;
* note that this function doesn't check containing
* types of the input union recursively. */
public static boolean isAcceptableUnion(Schema in) {
if (! in.getType().equals(Schema.Type.UNION))
return false;
List<Schema> types = in.getTypes();
if (types.size() <= 1) {
return true;
} else if (types.size() > 2) {
return false; /*contains more than 2 types */
} else {
/* one of two types is NULL */
return types.get(0).getType().equals(Schema.Type.NULL) || types.get(1) .getType().equals(Schema.Type.NULL);
}
}
basically piggybank's AvroStorage does not support more than 2 union types, I am wondering what is the idea behind this decision? Why not just make it compatible with Avro?