Go types like Int64 and String cannot store null values,
so I found I could use sql.NullInt64 and sql.NullString for this.
But when I use these in a Struct,
and generate JSON from the Struct with the json package,
then the format is different to when I use regular Int64 and String types.
The JSON has an additional level because the sql.Null*** is also a Struct.
Is there a good workaround for this,
or should I not use NULLs in my SQL database?
Types like sql.NullInt64 do not implement any special handling for JSON marshaling or unmarshaling, so the default rules apply. Since the type is a struct, it gets marshalled as an object with its fields as attributes.
One way to work around this is to create your own type that implements the json.Marshaller / json.Unmarshaler interfaces. By embedding the sql.NullInt64 type, we get the SQL methods for free. Something like this:
type JsonNullInt64 struct {
sql.NullInt64
}
func (v JsonNullInt64) MarshalJSON() ([]byte, error) {
if v.Valid {
return json.Marshal(v.Int64)
} else {
return json.Marshal(nil)
}
}
func (v *JsonNullInt64) UnmarshalJSON(data []byte) error {
// Unmarshalling into a pointer will let us detect null
var x *int64
if err := json.Unmarshal(data, &x); err != nil {
return err
}
if x != nil {
v.Valid = true
v.Int64 = *x
} else {
v.Valid = false
}
return nil
}
If you use this type in place of sql.NullInt64, it should be encoded as you expect.
You can test this example here: http://play.golang.org/p/zFESxLcd-c
If you use the null.v3 package, you won't need to implement any of the marshal or unmarshal methods. It's a superset of the sql.Null structs and is probably what you want.
package main
import "gopkg.in/guregu/null.v3"
type Person struct {
Name string `json:"id"`
Age int `json:"age"`
NickName null.String `json:"nickname"` // Optional
}
If you'd like to see a full Golang webserver that uses sqlite, nulls, and json you can consult this gist.
Related
I'm new to Go and still learning about optional struct fields.
I'm looking at the Gorm manual page where it gives an example of using pointers to indicate nullable fields (here)
If I strip down the example given so that it contains only a mandatory field and an optional field I'm left with something like this:
https://play.golang.com/p/lOLGWNVvq1l :
package main
import "fmt"
type User struct {
Name string
Email *string
}
func main() {
user := User{Name: "Example"}
// cannot use "example#example.com" (untyped string constant) as *string value in assignment
user.Email = "example#example.com"
// invalid operation: user.Email == "example#example.com" (mismatched types *string and untyped string)
if user.Email == "example#example.com" {
fmt.Println("foo")
}
}
How would I perform operations on a record that I've just retrieved from the database?
I need to be able to check if a nullable field is set to some value. I also can't assign a value to it.
One approach that I've thought of is to use some sort of wrapping function to try and make things safer, like at https://play.golang.com/p/4YlpPwaXMkm where I have:
func UnwrapString(x *string) string {
if x != nil {
return *x
}
return ""
}
func WrapString(x string) *string {
return &x
}
func main() {
user := User{Name: "Example"}
// can safely set an optional value that is currently null
if UnwrapString(user.Email) == "example#example.com" {
fmt.Println("hello world")
}
// can safely set a value if the existing Email is null
user.Email = WrapString("example#example.com")
// only safe because the value is set
if *user.Email == "example#example.com" {
fmt.Println("hello world")
}
}
Working with nullable fields in Gorm seems like such a basic and common thing that I don't expect to have to DIY. What's the idiomatic Gorm way to do this?
An idiomatic way to check if a field is non-nil, and if so, compare the value:
if user.Email != nil && *user.Email == "example#example.com" {
fmt.Println("foo")
}
The reason this works, even if user.Email is nil (and you get no nil-pointer dereference panic), is because Go has short circuit evaluation, meaning if the first comparison falls through in this AND statement, the second won't be evaluated, because there is no way this AND statement will ever be true if the first value is already false.
To do inline pointer assignments, the function you wrote is what I would do as well:
func StrPtr(s string) *string {
return &s
}
And you could then do:
user.Email = StrPtr("example#example.com")
I have the following struct:
type Payment struct {
...
PaymentMethods []int64 `json:"paymentMethods,omitempty" db:"payment_methods"`
}
In Postgres database, I have a simple []int column named payment_methods in payments table.
I need to make a simple Select query to get that struct and then marshall it to return it to the REST API in json format.
However, when I run the following:
payment := Payment{}
err := psql.db.Unsafe().Get(&payment, "select * from payments where id = $1", id)
I get an error :
sql: Scan error on column index 12, name "payment_methods": unsupported Scan, storing driver.Value t
Now, I know that I can use the pq.Int64Array type instead of []int.
However, the marshalling part won't work for that. I want to find a simpler solution that wouldn't add unnecessary overhead (unless it's impossible of course).
The marshalled struct then is encoded in this part:
func (handler *Handler) respond(w http.ResponseWriter, r *http.Request, data interface{}, status int) {
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.WriteHeader(status)
if data != nil {
err := json.NewEncoder(w).Encode(data)
if err != nil {
errors.Wrap(err, "Response Error while encoding data to json")
http.Error(w, http.StatusText(http.StatusNotFound), http.StatusNotFound)
}
}
}
How can I handle it in a generic way?
I'm using sqlx with pgx driver.
I am having problems getting the golang validator to work with SQL null types. Here's an example of what I tried:
package main
import (
"database/sql"
"database/sql/driver"
"log"
"gopkg.in/go-playground/validator.v9"
)
// NullInt64
type NullInt64 struct {
sql.NullInt64
Set bool
}
func MakeNullInt64(valid bool, val int64) NullInt64 {
n := NullInt64{}
n.Set = true
n.Valid = valid
if valid {
n.Int64 = val
}
return n
}
func (n *NullInt64) Value() (driver.Value, error) {
if !n.NullInt64.Valid {
return nil, nil
}
return n.NullInt64.Int64, nil
}
type Thing struct {
N2 NullInt64 `validate:"min=10"`
N3 int64 `validate:"min=10"`
N4 *int64 `validate:"min=10"`
}
func main() {
validate := validator.New()
n := int64(6)
number := MakeNullInt64(true, n)
thing := Thing{number, n, &n}
e := validate.Struct(thing)
log.Println(e)
}
When I run this code, I only get this output:
Key: 'Thing.N3' Error:Field validation for 'N3' failed on the 'min'
tag
Key: 'Thing.N4' Error:Field validation for 'N4' failed on the
'min' tag
The problem is that I want it to also show that Thing.N2 failed for the same reasons as Thing.N3 and Thing.N4.
I tried introducing the func (n *NullInt64) Value() method because it was mentioned in the documentation. But I think I misunderstood something. Can anyone tell me what I did wrong?
UPDATE
There is an Example specifically for that. You may check it out. My other proposed solution should still work though.
Since the value you are trying to validate is Int64 inside sql.NullInt64, the easiest way would be to remove the validate tag and just register a Struct Level validation using:
validate.RegisterStructValidation(NullInt64StructLevelValidation, NullInt64{})
while NullInt64StructLevelValidation is a StructLevelFunc that looks like this:
func NullInt64StructLevelValidation(sl validator.StructLevel) {
ni := sl.Current().Interface().(NullInt64)
if ni.NullInt64.Int64 < 10 {
sl.ReportError(ni.NullInt64.Int64, "Int64", "", "min", "")
}
}
Note #1: this line thing := Thing{number,&number,n,&n} has one argument too many. I assume you meant thing := Thing{number, n, &n}
Note #2: Go tooling including gofmt is considered to be one of the most powerful features of the language. Please consider using it/them.
EDIT #1:
I don't think implementing Valuer interface is of any value in this context.
The base concept creating a Database Manager API for getting data through an API. I am using the GORM for getting data of the instances of the strcuts. So there is 300-400 struct which represents the tables.
type Users struct {
ID int64
Name string
}
type Categories struct {
ID int64
Category string
}
The next step I implement a function which is return the correct instance of the struct by table name, what I get through the API endpoint param.
func GetModel(model string) interface{} {
switch model {
case "users":
return Users{}
case "categories"
return Categories{}
}
return false
}
After there is an operations struct where the only one field is the DB. There is methods, for example the GetLast() where I want to use the GORM db.Last(&users) function.
func (o Operations) GetLast(model string) interface{} {
modelStruct := GetModel(model)
.
.
.
return o.DB.Last(&modelStruct)
}
There is points so this is what I don't know. The current solution is not working because in this case it is an interface{} I need make a type assertion more info in this question. The type assertion is looks like:
func (o Operations) GetLast(model string) interface{} {
modelStruct := GetModel(model)
.
test := modelStruct.(Users)
.
return o.DB.Last(&test)
}
This solution working, but in this case I lost the modularity. I try using the reflect.TypeOf(modelStruct), but it is also not working because the result of the reflect.TypeOf is a reflect.Type, with is not a golang type.
Basically I solved the problem, for getting the model as a pointer, and after I return it back as a json file.
So my model is the following:
var Models = map[string]interface{}{
"users": new(Users),
"categories": new(Categories),
}
And it is return back a new model by table type. what I can use for gorm First() function. Then json Marshal it, and return.
func (o Operation) First(model string, query url.Values) string {
modelStruct := Models[model]
db := o.DB
db.First(modelStruct)
response, _ := json.Marshal(modelStruct)
clear(modelStruct)
return string(response)
}
Before the return I clear the model pointer because the First() function store callbacks for the latest queries.
func clear(v interface{}) {
p := reflect.ValueOf(v).Elem()
p.Set(reflect.Zero(p.Type()))
}
I realize I could pass the testing.T type to the functions that need to deal with the testing interfaces. But how about extending the struct and using it? I know I can type extend but this doesn't work unfortunately:
package test
import "testing"
type testingT testing.T
func (t *testingT) assert(val int) {
if val == 0 {
t.Errorf("%d is zero", val)
}
}
func TestSomething(t *testingT) {
t.assert(0)
}
One solution is embedding as you can see in eduncan911's answer. It's great if you want to add other fields to the new struct.
However, this is not a requirement. Creating a new type having testing.T as its underlying type (just as you did) is just enough:
type myT testing.T
func (t *myT) assert(val int) {
if val == 0 {
t.Errorf("%d is zero", val)
}
}
Using it
In TestSomething() we can't just call t.assert(0) because assert() is not a method of testing.T but our myT type. So we need a value of type myT on which we can call myT.assert().
We can't use type assertion to convert a testing.T to our myT type though, because type assertion requires the expression to be of interface type! And t is not an interface type, it's a concrete type: *testing.T.
The solution is really simple, we can use simple type conversion because myT and testing.T has the same underlying type:
my := myT(*t) // Conversion valid, but not yet enough, read on...
The above works, and we can call:
my.assert(0)
But when running tests, it won't fail! Why is that? It's because the above *t dereferencing, conversion and assignment will make a copy, and my will be of type myT but will be a copy of *t, and so when we call myT.assert(), it gets a pointer, but a pointer to the copy. And when myT.assert() calls t.Errorf(), it will register the error on a copy, so the testing framework will not see this Errorf() call.
The "final" solution is simple: use type conversion, but don't convert *t, but t itself (which is a pointer). Of course *t cannot be converted to myT, only to *myT. And this type needs to be parenthesized (to avoid ambiguity):
func TestSomething(t *testing.T) {
my := (*myT)(t)
my.assert(0)
}
And now we're happy, everything works as expected.
Note: You can also call myT.assert() in one line if you don't want to call multiple methods of myT:
func TestSomething(t *testing.T) {
(*myT)(t).assert(0)
}
You can use embedding which would allow you to add your owner methods:
package test
import "testing"
type testingT struct {
*testing.T
}
func (t *testingT) assert(val int) {
if val == 0 {
t.Errorf("%d is zero", val)
}
}
func TestSomething(t *testing.T) {
t2 := &testingT{t}
t2.assert(0)
}
Prepare custom testing struct:
type T testing.T
func (t *T) AssertEqualsInt(expected, actual int) {
if actual != expected {
// Correct point for file and line where happened error
t.Helper()
t.Errorf("Values not equal expected: %d, got: %d", expected, actual)
}
}
func (t *T) AssertEqualsString(expected, actual string) {
if actual != expected {
t.Helper()
t.Errorf("Values not equal expected: %s, got: %s", expected, actual)
}
}
Than use it in testing:
func TestFooBarBaz(t *testing.T) {
(*T)(t).AssertEqualsInt(0, 10)
(*T)(t).AssertEqualsString("foo", "bar")
}