I'm creating an API that will receive an object in a specific route. In this route I can receive a single object or a bulk of it.
Example:
[{"foo":"bar"}]
{"foo":"bar"}
How can I know if the body request is a slice or a single object before de json.Unmarshal. Moreover, if this is not possible, what is the best way to accept this two types of body requests and convert them to a list of objects?
I expect something like this:
type Foo struct {
Foo string `json:"foo"`
}
func Bla() []Foo {
fakeBody := []byte(`[{"foo":"bar"}]`)
fakeBody2 := []byte(`{"foo":"bar"}`)
var foo []Foo
// If fakeBody contains a array of objects
// then parse it to the foo slice variable normally
// So, if the fakeBody2 is a single object then
// parse this single object to the foo slice that will contain only
// one element.
return foo
}
This is what I would consider doing in this situation, in this order:
You can read the body, and check the first non-space character to see if it is '[' or '{', and unmarshal based on that.
You can first unmarshal as an array, then if that fails, as a single object.
You can unmarshal to an interface{}, do a type assertion, and parse the contents yourself.
Check the first non-whitespace byte to determine if the JSON document is an array or object. Decode accordingly.
func decode(body []byte) ([]Foo, error) {
b = bytes.TrimLeft(body, " \t\n\r")
if len(b) > 0 && b[0] == '[' {
var v []Foo
err := json.Unmarshal(body, &v)
return v, err
}
var v [1]Foo
err := json.Unmarshal(body, &v[0])
return v[:], err
}
Why not just add the [ the and ] if it's not their, and then always treat it as an array?
body := []byte(`{"foo":"bar"}`)
body = bytes.TrimSpace(body)
if len(body) > 0 && body[0] != '[' {
tmp := make([]byte, len(body)+2, len(body)+2)
tmp[0] = '['
tmp[len(tmp)-1] = ']'
copy(tmp[1:len(tmp)-1], body)
body = tmp
}
https://play.golang.org/p/YfnLgN9q64F
Or, create the array first, and then based on the first character either marshal into the array or the first item:
f := make([]Foo, 1)
body := []byte(`{"foo":"bar"}`)
if len(body) > 0 && body[0] != '[' {
json.Unmarshal(body, &f[0])
} else {
json.Unmarshal(body, &f)
}
fmt.Println(f)
https://play.golang.org/p/1fxBKH3ZJyH
Related
I'm trying to build a simple orm layer for golang.
Which would take a struct and generate the cols [] which can then be passed to sql function
rows.Scan(cols...) which takes pointers of fields in the struct corresponding to each of the columns it has found in the result set
Here is my example struct
type ExampleStruct struct {
ID int64 `sql:"id"`
aID string `sql:"a_id"`
UserID int64 `sql:"user_id"`
And this is my generic ORM function
func GetSqlColumnToFieldMap(model *ExampleStruct) map[string]interface{} {
typeOfModel := reflect.TypeOf(*model)
ValueOfModel := reflect.ValueOf(*model)
columnToDataPointerMap := make(map[string]interface{})
for i := 0; i < ValueOfModel.NumField(); i++ {
sql_column := typeOfModel.Field(i).Tag.Get("sql")
structValue := ValueOfModel.Field(i)
columnToDataPointerMap[sql_column] = structValue.Addr()
}
return columnToDataPointerMap
}
Once this method works fine i can use the map it generates to create an ordered list of sql pointers according to the column_names i get in rows() object
However i get below error on the .Addr() method call
panic: reflect.Value.Addr of unaddressable value [recovered]
panic: reflect.Value.Addr of unaddressable value
Is it not possible to do this ?
Also in an ideal scenario i would want the method to take an interface instead of *ExampleStruct so that it can be reused across different db models.
The error says the value whose address you want to get is unaddressable. This is because even though you pass a pointer to GetSqlColumnToFieldMap(), you immediately dereference it and work with a non-pointer value later on.
This value is wrapped in an interface{} when passed to reflect.ValueOf(), and values wrappped in interfaces are not addressable.
You must not dereference the pointer, but instead use Type.Elem() and Value.Elem() to get the element type and pointed value.
Something like this:
func GetSqlColumnToFieldMap(model *ExampleStruct) map[string]interface{} {
t := reflect.TypeOf(model).Elem()
v := reflect.ValueOf(model).Elem()
columnToDataPointerMap := make(map[string]interface{})
for i := 0; i < v.NumField(); i++ {
sql_column := t.Field(i).Tag.Get("sql")
structValue := v.Field(i)
columnToDataPointerMap[sql_column] = structValue.Addr()
}
return columnToDataPointerMap
}
With this simple change it works! And it doesn't depend on the parameter type, you may change it to interface{} and pass any struct pointers.
func GetSqlColumnToFieldMap(model interface{}) map[string]interface{} {
// ...
}
Testing it:
type ExampleStruct struct {
ID int64 `sql:"id"`
AID string `sql:"a_id"`
UserID int64 `sql:"user_id"`
}
type Point struct {
X int `sql:"x"`
Y int `sql:"y"`
}
func main() {
fmt.Println(GetSqlColumnToFieldMap(&ExampleStruct{}))
fmt.Println(GetSqlColumnToFieldMap(&Point{}))
}
Output (try it on the Go Playground):
map[a_id:<*string Value> id:<*int64 Value> user_id:<*int64 Value>]
map[x:<*int Value> y:<*int Value>]
Note that Value.Addr() returns the address wrapped in a reflect.Value. To "unwrap" the pointer, use Value.Interface():
func GetSqlColumnToFieldMap(model interface{}) map[string]interface{} {
t := reflect.TypeOf(model).Elem()
v := reflect.ValueOf(model).Elem()
m := make(map[string]interface{})
for i := 0; i < v.NumField(); i++ {
colName := t.Field(i).Tag.Get("sql")
field := v.Field(i)
m[colName] = field.Addr().Interface()
}
return m
}
This will output (try it on the Go Playground):
map[a_id:0xc00007e008 id:0xc00007e000 user_id:0xc00007e018]
map[x:0xc000018060 y:0xc000018068]
For an in-depth introduction to reflection, please read blog post: The Laws of Reflection
I have tried to find the solution for this problem, but keep running my head at the wall with this one.
This function is part of a Go SQL wrapper, and the function getJSON is called to extract the informations from the sql response.
The problem is, that the id parameter becomes jibberish and does not match the desired response, all the other parameters read are correct thou, so this really weirds me out.
Thank you in advance, for any attempt at figurring this problem out, it is really appreciated :-)
func getJSON(rows *sqlx.Rows) ([]byte, error) {
columns, err := rows.Columns()
rawResult := make([][]byte, len(columns))
dest := make([]interface{}, len(columns))
for i := range rawResult {
dest[i] = &rawResult[i]
}
defer rows.Close()
var results []map[string][]byte
for rows.Next() {
result := make(map[string][]byte, len(columns))
rows.Scan(dest...)
for i, raw := range rawResult {
if raw == nil {
result[columns[i]] = []byte("")
} else {
result[columns[i]] = raw
fmt.Println(columns[i] + " : " + string(raw))
}
}
results = append(results, result)
}
s, err := json.Marshal(results)
if err != nil {
panic(err)
}
rows.Close()
return s, nil
}
An example of the response, taking from the terminal:
id : r�b�X��M���+�2%
name : cat
issub : false
Expected result:
id : E262B172-B158-4DEF-8015-9BA12BF53225
name : cat
issub : false
That's not about type conversion.
An UUID (of any type; presently there are four) is defined to be a 128-bit-long lump of bytes, which is 128/8=16 bytes.
This means any bytes — not necessarily printable.
What you're after, is a string representation of an UUID value, which
Separates certain groups of bytes using dashes.
Formats each byte in these groups using hexadecimal (base-16) representation.
Since base-16 positional count represents values 0 through 15 using a single digit ('0' through 'F'), a single byte is represented by two such digits — a digit per each group of 4 bits.
I think any sensible UUID package should implement a "decoding" function/method which would produce a string representation out of those 16 bytes.
I have picked a random package produced by performing this search query, and it has github.com/google/uuid.FromBytes which produces an UUID from a given byte slice, and the type of the resulting value implements the String() method which produces what you're after.
Would it be possible to use Gob encoding for appending structs in series to the same file using append? It works for writing, but when reading with the decoder more than once I run into:
extra data in buffer
So I wonder if that's possible in the first place or whether I should use something like JSON to append JSON documents on a per line basis instead. Because the alternative would be to serialize a slice, but then again reading it as a whole would defeat the purpose of append.
The gob package wasn't designed to be used this way. A gob stream has to be written by a single gob.Encoder, and it also has to be read by a single gob.Decoder.
The reason for this is because the gob package not only serializes the values you pass to it, it also transmits data to describe their types:
A stream of gobs is self-describing. Each data item in the stream is preceded by a specification of its type, expressed in terms of a small set of predefined types.
This is a state of the encoder / decoder–about what types and how they have been transmitted–, a subsequent new encoder / decoder will not (cannot) analyze the "preceeding" stream to reconstruct the same state and continue where a previous encoder / decoder left off.
Of course if you create a single gob.Encoder, you may use it to serialize as many values as you'd like to.
Also you can create a gob.Encoder and write to a file, and then later create a new gob.Encoder, and append to the same file, but you must use 2 gob.Decoders to read those values, exactly matching the encoding process.
As a demonstration, let's follow an example. This example will write to an in-memory buffer (bytes.Buffer). 2 subsequent encoders will write to it, then we will use 2 subsequent decoders to read the values. We'll write values of this struct:
type Point struct {
X, Y int
}
For short, compact code, I use this "error handler" function:
func he(err error) {
if err != nil {
panic(err)
}
}
And now the code:
const n, m = 3, 2
buf := &bytes.Buffer{}
e := gob.NewEncoder(buf)
for i := 0; i < n; i++ {
he(e.Encode(&Point{X: i, Y: i * 2}))
}
e = gob.NewEncoder(buf)
for i := 0; i < m; i++ {
he(e.Encode(&Point{X: i, Y: 10 + i}))
}
d := gob.NewDecoder(buf)
for i := 0; i < n; i++ {
var p *Point
he(d.Decode(&p))
fmt.Println(p)
}
d = gob.NewDecoder(buf)
for i := 0; i < m; i++ {
var p *Point
he(d.Decode(&p))
fmt.Println(p)
}
Output (try it on the Go Playground):
&{0 0}
&{1 2}
&{2 4}
&{0 10}
&{1 11}
Note that if we'd use only 1 decoder to read all the values (looping until i < n + m, we'd get the same error message you posted in your question when the iteration reaches n + 1, because the subsequent data is not a serialized Point, but the start of a new gob stream.
So if you want to stick with the gob package for doing what you want to do, you have to slightly modify, enhance your encoding / decoding process. You have to somehow mark the boundaries when a new encoder is used (so when decoding, you'll know you have to create a new decoder to read subsequent values).
You may use different techniques to achieve this:
You may write out a number, a count before you proceed to write values, and this number would tell how many values were written using the current encoder.
If you don't want to or can't tell how many values will be written with the current encoder, you may opt to write out a special end-of-encoder value when you don't write more values with the current encoder. When decoding, if you encounter this special end-of-encoder value, you'll know you have to create a new decoder to be able to read more values.
Some things to note here:
The gob package is most efficient, most compact if only a single encoder is used, because each time you create and use a new encoder, the type specifications will have to be re-transmitted, causing more overhead, and making the encoding / decoding process slower.
You can't seek in the data stream, you can only decode any value if you read the whole file from the beginning up until the value you want. Note that this somewhat applies even if you use other formats (such as JSON or XML).
If you want seeking functionality, you'd need to manage an index file separately, which would tell at which positions new encoders / decoders start, so you could seek to that position, create a new decoder, and start reading values from there.
Check a related question: Efficient Go serialization of struct to disk
In addition to the above, I suggest using an intermediate structure to exclude the gob header:
package main
import (
"bytes"
"encoding/gob"
"fmt"
"io"
"log"
)
type Point struct {
X, Y int
}
func main() {
buf := new(bytes.Buffer)
enc, _, err := NewEncoderWithoutHeader(buf, new(Point))
if err != nil {
log.Fatal(err)
}
enc.Encode(&Point{10, 10})
fmt.Println(buf.Bytes())
}
type HeaderSkiper struct {
src io.Reader
dst io.Writer
}
func (hs *HeaderSkiper) Read(p []byte) (int, error) {
return hs.src.Read(p)
}
func (hs *HeaderSkiper) Write(p []byte) (int, error) {
return hs.dst.Write(p)
}
func NewEncoderWithoutHeader(w io.Writer, sample interface{}) (*gob.Encoder, *bytes.Buffer, error) {
hs := new(HeaderSkiper)
hdr := new(bytes.Buffer)
hs.dst = hdr
enc := gob.NewEncoder(hs)
// Write sample with header info
if err := enc.Encode(sample); err != nil {
return nil, nil, err
}
// Change writer
hs.dst = w
return enc, hdr, nil
}
func NewDecoderWithoutHeader(r io.Reader, hdr *bytes.Buffer, dummy interface{}) (*gob.Decoder, error) {
hs := new(HeaderSkiper)
hs.src = hdr
dec := gob.NewDecoder(hs)
if err := dec.Decode(dummy); err != nil {
return nil, err
}
hs.src = r
return dec, nil
}
Additionally to great icza answer, you could use the following trick to append to a gob file with already written data: when append the first time write and discard the first encode:
Create the file Encode gob as usual (first encode write headers)
Close file
Open file for append
Using and intermediate writer encode dummy struct (which write headers)
Reset the writer
Encode gob as usual (writes no headers)
Example:
package main
import (
"bytes"
"encoding/gob"
"fmt"
"io"
"io/ioutil"
"log"
"os"
)
type Record struct {
ID int
Body string
}
func main() {
r1 := Record{ID: 1, Body: "abc"}
r2 := Record{ID: 2, Body: "def"}
// encode r1
var buf1 bytes.Buffer
enc := gob.NewEncoder(&buf1)
err := enc.Encode(r1)
if err != nil {
log.Fatal(err)
}
// write to file
err = ioutil.WriteFile("/tmp/log.gob", buf1.Bytes(), 0600)
if err != nil {
log.Fatal()
}
// encode dummy (which write headers)
var buf2 bytes.Buffer
enc = gob.NewEncoder(&buf2)
err = enc.Encode(Record{})
if err != nil {
log.Fatal(err)
}
// remove dummy
buf2.Reset()
// encode r2
err = enc.Encode(r2)
if err != nil {
log.Fatal(err)
}
// open file
f, err := os.OpenFile("/tmp/log.gob", os.O_WRONLY|os.O_APPEND, 0600)
if err != nil {
log.Fatal(err)
}
// write r2
_, err = f.Write(buf2.Bytes())
if err != nil {
log.Fatal(err)
}
// decode file
data, err := ioutil.ReadFile("/tmp/log.gob")
if err != nil {
log.Fatal(err)
}
var r Record
dec := gob.NewDecoder(bytes.NewReader(data))
for {
err = dec.Decode(&r)
if err == io.EOF {
break
}
if err != nil {
log.Fatal(err)
}
fmt.Println(r)
}
}
since few days I was struggling on how to proceed with PATCH request in Go REST API until I have found an article about using pointers and omitempty tag which I have populated and is working fine. Fine until I have realized I still have to build an UPDATE SQL query.
My struct looks like this:
type Resource struct {
Name *string `json:"name,omitempty" sql:"resource_id"`
Description *string `json:"description,omitempty" sql:"description"`
}
I am expecting a PATCH /resources/{resource-id} request containing such a request body:
{"description":"Some new description"}
In my handler I will build the Resource object this way (ignoring imports, ignoring error handling):
var resource Resource
resourceID, _ := mux.Vars(r)["resource-id"]
d := json.NewDecoder(r.Body)
d.Decode(&resource)
// at this point our resource object should only contain
// the Description field with the value from JSON in request body
Now, for normal UPDATE (PUT request) I would do this (simplified):
stmt, _ := db.Prepare(`UPDATE resources SET description = ?, name = ? WHERE resource_id = ?`)
res, _ := stmt.Exec(resource.Description, resource.Name, resourceID)
The problem with PATCH and omitempty tag is that the object might be missing multiple properties, thus I cannot just prepare a statement with hardcoded fields and placeholders... I will have to build it dynamically.
And here comes my question: how can I build such UPDATE query dynamically? In the best case I'd need some solution with identifying the set properties, getting their SQL field names (probably from the tags) and then I should be able to build the UPDATE query. I know I can use reflection to get the object properties but have no idea hot to get their sql tag name and of course I'd like to avoid using reflection here if possible... Or I could simply check for each property it is not nil, but in real life the structs are much bigger than provided example here...
Can somebody help me with this one? Did somebody already have to solve the same/similar situation?
SOLUTION:
Based on the answers here I was able to come up with this abstract solution. The SQLPatches method builds the SQLPatch struct from the given struct (so no concrete struct specific):
import (
"fmt"
"encoding/json"
"reflect"
"strings"
)
const tagname = "sql"
type SQLPatch struct {
Fields []string
Args []interface{}
}
func SQLPatches(resource interface{}) SQLPatch {
var sqlPatch SQLPatch
rType := reflect.TypeOf(resource)
rVal := reflect.ValueOf(resource)
n := rType.NumField()
sqlPatch.Fields = make([]string, 0, n)
sqlPatch.Args = make([]interface{}, 0, n)
for i := 0; i < n; i++ {
fType := rType.Field(i)
fVal := rVal.Field(i)
tag := fType.Tag.Get(tagname)
// skip nil properties (not going to be patched), skip unexported fields, skip fields to be skipped for SQL
if fVal.IsNil() || fType.PkgPath != "" || tag == "-" {
continue
}
// if no tag is set, use the field name
if tag == "" {
tag = fType.Name
}
// and make the tag lowercase in the end
tag = strings.ToLower(tag)
sqlPatch.Fields = append(sqlPatch.Fields, tag+" = ?")
var val reflect.Value
if fVal.Kind() == reflect.Ptr {
val = fVal.Elem()
} else {
val = fVal
}
switch val.Kind() {
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
sqlPatch.Args = append(sqlPatch.Args, val.Int())
case reflect.String:
sqlPatch.Args = append(sqlPatch.Args, val.String())
case reflect.Bool:
if val.Bool() {
sqlPatch.Args = append(sqlPatch.Args, 1)
} else {
sqlPatch.Args = append(sqlPatch.Args, 0)
}
}
}
return sqlPatch
}
Then I can simply call it like this:
type Resource struct {
Description *string `json:"description,omitempty"`
Name *string `json:"name,omitempty"`
}
func main() {
var r Resource
json.Unmarshal([]byte(`{"description": "new description"}`), &r)
sqlPatch := SQLPatches(r)
data, _ := json.Marshal(sqlPatch)
fmt.Printf("%s\n", data)
}
You can check it at Go Playground. The only problem here I see is that I allocate both the slices with the amount of fields in the passed struct, which may be 10, even though I might only want to patch one property in the end resulting in allocating more memory than needed... Any idea how to avoid this?
I recently had same problem. about PATCH and looking around found this article. It also makes references to the RFC 5789 where it says:
The difference between the PUT and PATCH requests is reflected in the way the server processes the enclosed entity to modify the resource identified by the Request-URI. In a PUT request, the enclosed entity is considered to be a modified version of the resource stored on the origin server, and the client is requesting that the stored version be replaced. With PATCH, however, the enclosed entity contains a set of instructions describing how a resource currently residing on the origin server should be modified to produce a new version. The PATCH method affects the resource identified by the Request-URI, and it also MAY have side effects on other resources; i.e., new resources may be created, or existing ones modified, by the application of a PATCH.
e.g:
[
{ "op": "test", "path": "/a/b/c", "value": "foo" },
{ "op": "remove", "path": "/a/b/c" },
{ "op": "add", "path": "/a/b/c", "value": [ "foo", "bar" ] },
{ "op": "replace", "path": "/a/b/c", "value": 42 },
{ "op": "move", "from": "/a/b/c", "path": "/a/b/d" },
{ "op": "copy", "from": "/a/b/d", "path": "/a/b/e" }
]
This set of instructions should make it easier to build the update query.
EDIT
This is how you would obtain sql tags but you will have to use reflection:
type Resource struct {
Name *string `json:"name,omitempty" sql:"resource_id"`
Description *string `json:"description,omitempty" sql:"description"`
}
sp := "sort of string"
r := Resource{Description: &sp}
rt := reflect.TypeOf(r) // reflect.Type
rv := reflect.ValueOf(r) // reflect.Value
for i := 0; i < rv.NumField(); i++ { // Iterate over all the fields
if !rv.Field(i).IsNil() { // Check it is not nil
// Here you would do what you want to having the sql tag.
// Creating the query would be easy, however
// not sure you would execute the statement
fmt.Println(rt.Field(i).Tag.Get("sql")) // Output: description
}
}
I understand you don't want to use reflection, but still this may be a better answer than the previous one as you comment state.
EDIT 2:
About the allocation - read this guide lines of Effective Go about Data structures and Allocation:
// Here you are allocating an slice of 0 length with a capacity of n
sqlPatch.Fields = make([]string, 0, n)
sqlPatch.Args = make([]interface{}, 0, n)
With make(Type, Length, Capacity (optional))
Consider the following example:
// newly allocated zeroed value with Composite Literal
// length: 0
// capacity: 0
testSlice := []int{}
fmt.Println(len(testSlice), cap(testSlice)) // 0 0
fmt.Println(testSlice) // []
// newly allocated non zeroed value with make
// length: 0
// capacity: 10
testSlice = make([]int, 0, 10)
fmt.Println(len(testSlice), cap(testSlice)) // 0 10
fmt.Println(testSlice) // []
// newly allocated non zeroed value with make
// length: 2
// capacity: 4
testSlice = make([]int, 2, 4)
fmt.Println(len(testSlice), cap(testSlice)) // 2 4
fmt.Println(testSlice) // [0 0]
In your case, may want to the following:
// Replace this
sqlPatch.Fields = make([]string, 0, n)
sqlPatch.Args = make([]interface{}, 0, n)
// With this or simple omit the capacity in make above
sqlPatch.Fields = []string{}
sqlPatch.Args = []interface{}{}
// The allocation will go as follow: length - capacity
testSlice := []int{} // 0 - 0
testSlice = append(testSlice, 1) // 1 - 2
testSlice = append(testSlice, 1) // 2 - 2
testSlice = append(testSlice, 1) // 3 - 4
testSlice = append(testSlice, 1) // 4 - 4
testSlice = append(testSlice, 1) // 5 - 8
Alright, I think the solution I used back in 2016 was quite over-engineered for even more over-engineered problem and was completely unnecessary. The question asked here was very generalized, however we were building a solution that was able to build its SQL query on its own and based on the JSON object or query parameters and/or Headers sent in the request. And that to be as generic as possible.
Nowadays I think the best solution is to avoid PATCH unless truly necessary. And even then you still can use PUT and replace the whole resource with patched property/ies coming already from the client - i.e. not giving the client the option/possibility to send any PATCH request to your server and to deal with partial updates on their own.
However this is not always recommended, especially in cases of bigger objects to save some C02 by reducing the amount of redundant transmitted data. Whenever today if I need to enable a PATCH for the client I simply define what can be patched - this gives me clarity and the final struct.
Note that I am using a IETF documented JSON Merge Patch implementation. I consider that of JSON Patch (also documented by IETF) redundant as hypothetically we could replace the whole REST API by having one single JSON Patch endpoint and let clients control the resources via allowed operations. I also think the implementation of such JSON Patch on the server side is way more complicated. The only use-case I could think of using such implementation is if I was implementing a REST API over a file system...
So the struct may be defined as in my OP:
type ResourcePatch struct {
ResourceID some.UUID `json:"resource_id"`
Description *string `json:"description,omitempty"`
Name *string `json:"name,omitempty"`
}
In the handler func I'd decode the ID from the path into the ResourcePatch instance and unmarshall JSON from the request body into it, too.
Sending only this
{"description":"Some new description"}
to PATCH /resources/<UUID>
I should end up with with this object:
ResourcePatch
* ResourceID {"UUID"}
* Description {"Some new description"}
And now the magic: use a simple logic to build the query and exec parameters. For some it may seem tedious or repetitive or unclean for bigger PATCH objects, but my reply to this would be: if your PATCH object consists of more than 50% of the original resource' properties (or simply too many for your liking) use PUT and expect the clients to send (and replace) the whole resource instead.
It could look like this:
func (s Store) patchMyResource(r models.ResourcePatch) error {
q := `UPDATE resources SET `
qParts := make([]string, 0, 2)
args := make([]interface{}, 0, 2)
if r.Description != nil {
qParts = append(qParts, `description = ?`)
args = append(args, r.Description)
}
if r.Name != nil {
qParts = append(qParts, `name = ?`)
args = append(args, r.Name)
}
q += strings.Join(qParts, ',') + ` WHERE resource_id = ?`
args = append(args, r.ResourceID)
_, err := s.db.Exec(q, args...)
return err
}
I think there's nothing simpler and more effective. No reflection, no over-kills, reads quite good.
Struct tags are only visible through reflection, sorry.
If you don't want to use reflection (or, I think, even if you do), I think it is Go-like to define a function or method that "marshals" your struct into something that can easily be turned into a comma-separated list of SQL updates, and then use that. Build small things to help solve your problems.
For example given:
type Resource struct {
Name *string `json:"name,omitempty" sql:"resource_id"`
Description *string `json:"description,omitempty" sql:"description"`
}
You might define:
func (r Resource) SQLUpdates() SQLUpdates {
var s SQLUpdates
if (r.Name != nil) {
s.add("resource_id", *r.Name)
}
if (r.Description != nil) {
s.add("description", *r.Description)
}
}
where the type SQLUpdates looks something like this:
type SQLUpdates struct {
assignments []string
values []interface{}
}
func (s *SQLUpdates) add(key string, value interface{}) {
if (s.assignments == nil) {
s.assignments = make([]string, 0, 1)
}
if (s.values == nil) {
s.values = make([]interface{}, 0, 1)
}
s.assignments = append(s.assignments, fmt.Sprintf("%s = ?", key))
s.values = append(s.values, value)
}
func (s SQLUpdates) Assignments() string {
return strings.Join(s.assignments, ", ")
}
func (s SQLUpdates) Values() []interface{} {
return s.values
}
See it working (sorta) here: https://play.golang.org/p/IQAHgqfBRh
If you have deep structs-within-structs, it should be fairly easy to build on this. And if you change to an SQL engine that allows or encourages positional arguments like $1 instead of ?, it's easy to add that behavior to just the SQLUpdates struct without changing any code that used it.
For the purpose of getting arguments to pass to Exec, you would just expand the output of Values() with the ... operator.
I met a problem when query from Database and tried to insert into a slice(contains some map[string]interface{})
Even I already used make to create a new memory block, the slice seems always mapping to a same memory block.
type DBResult []map[string]interface{}
func ResultRows(rows *sql.Rows, limit int) (DBResult, error) {
cols, err := rows.Columns()
if err != nil {
return nil, err
}
vals := make([]sql.RawBytes, len(cols))
scanArgs := make([]interface{}, len(vals))
for i := range vals {
scanArgs[i] = &vals[i]
}
if limit > QUERY_HARD_LIMIT {
limit = QUERY_HARD_LIMIT
}
res := make(DBResult, 0, limit)
for rows.Next() {
err = rows.Scan(scanArgs...)
m := make(map[string]interface{})
for i := range vals {
m[cols[i]] = vals[i]
}
/* Append m to res */
res = append(res, m)
/* The value of m has been changed */
fmt.Printf("lib: m:\n\n%s\n\n", m)
/* When printing res, always mapping to the same memory block */
fmt.Printf("lib: res:\n\n%s\n\n", res)
}
return res, err
}
The following is the result, you can find the contents of res are the same
m = map[comment:first_comment id:0]
res = [map[id:0 comment:first_comment]]
m = map[id:1 comment:first_comment]
res = [map[id:1 comment:first_comment] map[id:1 comment:first_comment]]
m = map[id:2 comment:first_comment]
res = [map[id:2 comment:first_comment] map[id:2 comment:first_comment] map[id:2 comment:first_comment]]
My expect of res = [map[id:0 comment:first_comment] map[id:1 comment:first_comment] map[id:2 comment:first_comment]]
Thanks for watching
according to the document of Rows (https://golang.org/pkg/database/sql/#Rows.Scan):
Scan copies the columns in the current row into the values pointed at by dest.
If an argument has type *[]byte, Scan saves in that argument a copy of the corresponding data. The copy is owned by the caller and can be modified and held indefinitely. The copy can be avoided by using an argument of type *RawBytes instead; see the documentation for RawBytes for restrictions on its use.
If an argument has type *interface{}, Scan copies the value provided by the underlying driver without conversion. If the value is of type []byte, a copy is made and the caller owns the result.
in your case, you use RawBytes as argument of Scan. That might be the problem you got. Try other type as argument of Scan function.