Golang database manager api concept, error with type assertion - sql

The base concept creating a Database Manager API for getting data through an API. I am using the GORM for getting data of the instances of the strcuts. So there is 300-400 struct which represents the tables.
type Users struct {
ID int64
Name string
}
type Categories struct {
ID int64
Category string
}
The next step I implement a function which is return the correct instance of the struct by table name, what I get through the API endpoint param.
func GetModel(model string) interface{} {
switch model {
case "users":
return Users{}
case "categories"
return Categories{}
}
return false
}
After there is an operations struct where the only one field is the DB. There is methods, for example the GetLast() where I want to use the GORM db.Last(&users) function.
func (o Operations) GetLast(model string) interface{} {
modelStruct := GetModel(model)
.
.
.
return o.DB.Last(&modelStruct)
}
There is points so this is what I don't know. The current solution is not working because in this case it is an interface{} I need make a type assertion more info in this question. The type assertion is looks like:
func (o Operations) GetLast(model string) interface{} {
modelStruct := GetModel(model)
.
test := modelStruct.(Users)
.
return o.DB.Last(&test)
}
This solution working, but in this case I lost the modularity. I try using the reflect.TypeOf(modelStruct), but it is also not working because the result of the reflect.TypeOf is a reflect.Type, with is not a golang type.

Basically I solved the problem, for getting the model as a pointer, and after I return it back as a json file.
So my model is the following:
var Models = map[string]interface{}{
"users": new(Users),
"categories": new(Categories),
}
And it is return back a new model by table type. what I can use for gorm First() function. Then json Marshal it, and return.
func (o Operation) First(model string, query url.Values) string {
modelStruct := Models[model]
db := o.DB
db.First(modelStruct)
response, _ := json.Marshal(modelStruct)
clear(modelStruct)
return string(response)
}
Before the return I clear the model pointer because the First() function store callbacks for the latest queries.
func clear(v interface{}) {
p := reflect.ValueOf(v).Elem()
p.Set(reflect.Zero(p.Type()))
}

Related

Get pointers to all fields of a struct dynamically using reflection

I'm trying to build a simple orm layer for golang.
Which would take a struct and generate the cols [] which can then be passed to sql function
rows.Scan(cols...) which takes pointers of fields in the struct corresponding to each of the columns it has found in the result set
Here is my example struct
type ExampleStruct struct {
ID int64 `sql:"id"`
aID string `sql:"a_id"`
UserID int64 `sql:"user_id"`
And this is my generic ORM function
func GetSqlColumnToFieldMap(model *ExampleStruct) map[string]interface{} {
typeOfModel := reflect.TypeOf(*model)
ValueOfModel := reflect.ValueOf(*model)
columnToDataPointerMap := make(map[string]interface{})
for i := 0; i < ValueOfModel.NumField(); i++ {
sql_column := typeOfModel.Field(i).Tag.Get("sql")
structValue := ValueOfModel.Field(i)
columnToDataPointerMap[sql_column] = structValue.Addr()
}
return columnToDataPointerMap
}
Once this method works fine i can use the map it generates to create an ordered list of sql pointers according to the column_names i get in rows() object
However i get below error on the .Addr() method call
panic: reflect.Value.Addr of unaddressable value [recovered]
panic: reflect.Value.Addr of unaddressable value
Is it not possible to do this ?
Also in an ideal scenario i would want the method to take an interface instead of *ExampleStruct so that it can be reused across different db models.
The error says the value whose address you want to get is unaddressable. This is because even though you pass a pointer to GetSqlColumnToFieldMap(), you immediately dereference it and work with a non-pointer value later on.
This value is wrapped in an interface{} when passed to reflect.ValueOf(), and values wrappped in interfaces are not addressable.
You must not dereference the pointer, but instead use Type.Elem() and Value.Elem() to get the element type and pointed value.
Something like this:
func GetSqlColumnToFieldMap(model *ExampleStruct) map[string]interface{} {
t := reflect.TypeOf(model).Elem()
v := reflect.ValueOf(model).Elem()
columnToDataPointerMap := make(map[string]interface{})
for i := 0; i < v.NumField(); i++ {
sql_column := t.Field(i).Tag.Get("sql")
structValue := v.Field(i)
columnToDataPointerMap[sql_column] = structValue.Addr()
}
return columnToDataPointerMap
}
With this simple change it works! And it doesn't depend on the parameter type, you may change it to interface{} and pass any struct pointers.
func GetSqlColumnToFieldMap(model interface{}) map[string]interface{} {
// ...
}
Testing it:
type ExampleStruct struct {
ID int64 `sql:"id"`
AID string `sql:"a_id"`
UserID int64 `sql:"user_id"`
}
type Point struct {
X int `sql:"x"`
Y int `sql:"y"`
}
func main() {
fmt.Println(GetSqlColumnToFieldMap(&ExampleStruct{}))
fmt.Println(GetSqlColumnToFieldMap(&Point{}))
}
Output (try it on the Go Playground):
map[a_id:<*string Value> id:<*int64 Value> user_id:<*int64 Value>]
map[x:<*int Value> y:<*int Value>]
Note that Value.Addr() returns the address wrapped in a reflect.Value. To "unwrap" the pointer, use Value.Interface():
func GetSqlColumnToFieldMap(model interface{}) map[string]interface{} {
t := reflect.TypeOf(model).Elem()
v := reflect.ValueOf(model).Elem()
m := make(map[string]interface{})
for i := 0; i < v.NumField(); i++ {
colName := t.Field(i).Tag.Get("sql")
field := v.Field(i)
m[colName] = field.Addr().Interface()
}
return m
}
This will output (try it on the Go Playground):
map[a_id:0xc00007e008 id:0xc00007e000 user_id:0xc00007e018]
map[x:0xc000018060 y:0xc000018068]
For an in-depth introduction to reflection, please read blog post: The Laws of Reflection

Selecting an int array from Postgres into struct and then marshalling it

I have the following struct:
type Payment struct {
...
PaymentMethods []int64 `json:"paymentMethods,omitempty" db:"payment_methods"`
}
In Postgres database, I have a simple []int column named payment_methods in payments table.
I need to make a simple Select query to get that struct and then marshall it to return it to the REST API in json format.
However, when I run the following:
payment := Payment{}
err := psql.db.Unsafe().Get(&payment, "select * from payments where id = $1", id)
I get an error :
sql: Scan error on column index 12, name "payment_methods": unsupported Scan, storing driver.Value t
Now, I know that I can use the pq.Int64Array type instead of []int.
However, the marshalling part won't work for that. I want to find a simpler solution that wouldn't add unnecessary overhead (unless it's impossible of course).
The marshalled struct then is encoded in this part:
func (handler *Handler) respond(w http.ResponseWriter, r *http.Request, data interface{}, status int) {
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.WriteHeader(status)
if data != nil {
err := json.NewEncoder(w).Encode(data)
if err != nil {
errors.Wrap(err, "Response Error while encoding data to json")
http.Error(w, http.StatusText(http.StatusNotFound), http.StatusNotFound)
}
}
}
How can I handle it in a generic way?
I'm using sqlx with pgx driver.

Flatbuffers: How to allow multiple types for a single field

I'm writing a communication protocol schema for a list of parameters which can be of multiple values: uint64, float64, string or bool.
How can I set a table field to a union of multiple primitive scalar & non-scalar primitive type?
I've already tried using a union of those types, but I end up with the following error when building:
$ schemas/foobar.fbs:28: 0: error: type referenced but not defined
(check namespace): uint64, originally at: schemas/request.fbs:5
Here's the schema in its current state:
namespace Foobar;
enum RequestCode : uint16 { Noop, Get, Set, BulkGet, BulkSet }
union ParameterValue { uint64, float64, bool, string }
table Parameter {
name:string;
value:ParameterValue;
unit:string;
}
table Request {
code:RequestCode = Noop;
payload:[Parameter];
}
table Result {
request:Request;
success:bool = true;
payload:[Parameter];
}
The end result I'm looking for is the Request and Result tables to contain a list of parameters, where a parameter contains a name and value, and optionally the units.
Thx in advance!
Post-answer solution:
Here's what I came up with in the end, thx to Aardappel.
namespace foobar;
enum RequestCode : uint16 { Noop, Get, Set, BulkGet, BulkSet }
union ValueType { UnsignedInteger, SignedInteger, RealNumber, Boolean, Text }
table UnsignedInteger {
value:uint64 = 0;
}
table SignedInteger {
value:int64 = 0;
}
table RealNumber {
value:float64 = 0.0;
}
table Boolean {
value:bool = false;
}
table Text {
value:string (required);
}
table Parameter {
name:string (required);
valueType:ValueType;
unit:string;
}
table Request {
code:RequestCode = Noop;
payload:[Parameter];
}
table Result {
request:Request (required);
success:bool = true;
payload:[Parameter];
}
You currently can't put scalars directly in a union, so you'd have to wrap these in a table or a struct, where struct would likely be the most efficient, e.g.
struct UInt64 { u:uint64 }
union ParameterValue { UInt64, Float64, Bool, string }
This is because a union must be uniformly the same size, so it only allows types to which you can have an offset.
Generally though, FlatBuffers is a strongly typed system, and the schema you are creating here is undoing that by emulating dynamically typed data, since your data is essentially a list of (string, any type) pairs. You may be better off with a system designed for this particular use case, such as FlexBuffers (https://google.github.io/flatbuffers/flexbuffers.html, currently only C++) which explicitly has a map type that is all string -> any type pairs.
Of course, even better is to not store data so generically, but instead make a new schema for each type of request and response you have, and make parameter names into fields, rather than serialized data. This is by far the most efficient, and type safe.

Go validator with sql null types?

I am having problems getting the golang validator to work with SQL null types. Here's an example of what I tried:
package main
import (
"database/sql"
"database/sql/driver"
"log"
"gopkg.in/go-playground/validator.v9"
)
// NullInt64
type NullInt64 struct {
sql.NullInt64
Set bool
}
func MakeNullInt64(valid bool, val int64) NullInt64 {
n := NullInt64{}
n.Set = true
n.Valid = valid
if valid {
n.Int64 = val
}
return n
}
func (n *NullInt64) Value() (driver.Value, error) {
if !n.NullInt64.Valid {
return nil, nil
}
return n.NullInt64.Int64, nil
}
type Thing struct {
N2 NullInt64 `validate:"min=10"`
N3 int64 `validate:"min=10"`
N4 *int64 `validate:"min=10"`
}
func main() {
validate := validator.New()
n := int64(6)
number := MakeNullInt64(true, n)
thing := Thing{number, n, &n}
e := validate.Struct(thing)
log.Println(e)
}
When I run this code, I only get this output:
Key: 'Thing.N3' Error:Field validation for 'N3' failed on the 'min'
tag
Key: 'Thing.N4' Error:Field validation for 'N4' failed on the
'min' tag
The problem is that I want it to also show that Thing.N2 failed for the same reasons as Thing.N3 and Thing.N4.
I tried introducing the func (n *NullInt64) Value() method because it was mentioned in the documentation. But I think I misunderstood something. Can anyone tell me what I did wrong?
UPDATE
There is an Example specifically for that. You may check it out. My other proposed solution should still work though.
Since the value you are trying to validate is Int64 inside sql.NullInt64, the easiest way would be to remove the validate tag and just register a Struct Level validation using:
validate.RegisterStructValidation(NullInt64StructLevelValidation, NullInt64{})
while NullInt64StructLevelValidation is a StructLevelFunc that looks like this:
func NullInt64StructLevelValidation(sl validator.StructLevel) {
ni := sl.Current().Interface().(NullInt64)
if ni.NullInt64.Int64 < 10 {
sl.ReportError(ni.NullInt64.Int64, "Int64", "", "min", "")
}
}
Note #1: this line thing := Thing{number,&number,n,&n} has one argument too many. I assume you meant thing := Thing{number, n, &n}
Note #2: Go tooling including gofmt is considered to be one of the most powerful features of the language. Please consider using it/them.
EDIT #1:
I don't think implementing Valuer interface is of any value in this context.

How can I work with SQL NULL values and JSON in a good way?

Go types like Int64 and String cannot store null values,
so I found I could use sql.NullInt64 and sql.NullString for this.
But when I use these in a Struct,
and generate JSON from the Struct with the json package,
then the format is different to when I use regular Int64 and String types.
The JSON has an additional level because the sql.Null*** is also a Struct.
Is there a good workaround for this,
or should I not use NULLs in my SQL database?
Types like sql.NullInt64 do not implement any special handling for JSON marshaling or unmarshaling, so the default rules apply. Since the type is a struct, it gets marshalled as an object with its fields as attributes.
One way to work around this is to create your own type that implements the json.Marshaller / json.Unmarshaler interfaces. By embedding the sql.NullInt64 type, we get the SQL methods for free. Something like this:
type JsonNullInt64 struct {
sql.NullInt64
}
func (v JsonNullInt64) MarshalJSON() ([]byte, error) {
if v.Valid {
return json.Marshal(v.Int64)
} else {
return json.Marshal(nil)
}
}
func (v *JsonNullInt64) UnmarshalJSON(data []byte) error {
// Unmarshalling into a pointer will let us detect null
var x *int64
if err := json.Unmarshal(data, &x); err != nil {
return err
}
if x != nil {
v.Valid = true
v.Int64 = *x
} else {
v.Valid = false
}
return nil
}
If you use this type in place of sql.NullInt64, it should be encoded as you expect.
You can test this example here: http://play.golang.org/p/zFESxLcd-c
If you use the null.v3 package, you won't need to implement any of the marshal or unmarshal methods. It's a superset of the sql.Null structs and is probably what you want.
package main
import "gopkg.in/guregu/null.v3"
type Person struct {
Name string `json:"id"`
Age int `json:"age"`
NickName null.String `json:"nickname"` // Optional
}
If you'd like to see a full Golang webserver that uses sqlite, nulls, and json you can consult this gist.