I have tried to find the solution for this problem, but keep running my head at the wall with this one.
This function is part of a Go SQL wrapper, and the function getJSON is called to extract the informations from the sql response.
The problem is, that the id parameter becomes jibberish and does not match the desired response, all the other parameters read are correct thou, so this really weirds me out.
Thank you in advance, for any attempt at figurring this problem out, it is really appreciated :-)
func getJSON(rows *sqlx.Rows) ([]byte, error) {
columns, err := rows.Columns()
rawResult := make([][]byte, len(columns))
dest := make([]interface{}, len(columns))
for i := range rawResult {
dest[i] = &rawResult[i]
}
defer rows.Close()
var results []map[string][]byte
for rows.Next() {
result := make(map[string][]byte, len(columns))
rows.Scan(dest...)
for i, raw := range rawResult {
if raw == nil {
result[columns[i]] = []byte("")
} else {
result[columns[i]] = raw
fmt.Println(columns[i] + " : " + string(raw))
}
}
results = append(results, result)
}
s, err := json.Marshal(results)
if err != nil {
panic(err)
}
rows.Close()
return s, nil
}
An example of the response, taking from the terminal:
id : r�b�X��M���+�2%
name : cat
issub : false
Expected result:
id : E262B172-B158-4DEF-8015-9BA12BF53225
name : cat
issub : false
That's not about type conversion.
An UUID (of any type; presently there are four) is defined to be a 128-bit-long lump of bytes, which is 128/8=16 bytes.
This means any bytes — not necessarily printable.
What you're after, is a string representation of an UUID value, which
Separates certain groups of bytes using dashes.
Formats each byte in these groups using hexadecimal (base-16) representation.
Since base-16 positional count represents values 0 through 15 using a single digit ('0' through 'F'), a single byte is represented by two such digits — a digit per each group of 4 bits.
I think any sensible UUID package should implement a "decoding" function/method which would produce a string representation out of those 16 bytes.
I have picked a random package produced by performing this search query, and it has github.com/google/uuid.FromBytes which produces an UUID from a given byte slice, and the type of the resulting value implements the String() method which produces what you're after.
since few days I was struggling on how to proceed with PATCH request in Go REST API until I have found an article about using pointers and omitempty tag which I have populated and is working fine. Fine until I have realized I still have to build an UPDATE SQL query.
My struct looks like this:
type Resource struct {
Name *string `json:"name,omitempty" sql:"resource_id"`
Description *string `json:"description,omitempty" sql:"description"`
}
I am expecting a PATCH /resources/{resource-id} request containing such a request body:
{"description":"Some new description"}
In my handler I will build the Resource object this way (ignoring imports, ignoring error handling):
var resource Resource
resourceID, _ := mux.Vars(r)["resource-id"]
d := json.NewDecoder(r.Body)
d.Decode(&resource)
// at this point our resource object should only contain
// the Description field with the value from JSON in request body
Now, for normal UPDATE (PUT request) I would do this (simplified):
stmt, _ := db.Prepare(`UPDATE resources SET description = ?, name = ? WHERE resource_id = ?`)
res, _ := stmt.Exec(resource.Description, resource.Name, resourceID)
The problem with PATCH and omitempty tag is that the object might be missing multiple properties, thus I cannot just prepare a statement with hardcoded fields and placeholders... I will have to build it dynamically.
And here comes my question: how can I build such UPDATE query dynamically? In the best case I'd need some solution with identifying the set properties, getting their SQL field names (probably from the tags) and then I should be able to build the UPDATE query. I know I can use reflection to get the object properties but have no idea hot to get their sql tag name and of course I'd like to avoid using reflection here if possible... Or I could simply check for each property it is not nil, but in real life the structs are much bigger than provided example here...
Can somebody help me with this one? Did somebody already have to solve the same/similar situation?
SOLUTION:
Based on the answers here I was able to come up with this abstract solution. The SQLPatches method builds the SQLPatch struct from the given struct (so no concrete struct specific):
import (
"fmt"
"encoding/json"
"reflect"
"strings"
)
const tagname = "sql"
type SQLPatch struct {
Fields []string
Args []interface{}
}
func SQLPatches(resource interface{}) SQLPatch {
var sqlPatch SQLPatch
rType := reflect.TypeOf(resource)
rVal := reflect.ValueOf(resource)
n := rType.NumField()
sqlPatch.Fields = make([]string, 0, n)
sqlPatch.Args = make([]interface{}, 0, n)
for i := 0; i < n; i++ {
fType := rType.Field(i)
fVal := rVal.Field(i)
tag := fType.Tag.Get(tagname)
// skip nil properties (not going to be patched), skip unexported fields, skip fields to be skipped for SQL
if fVal.IsNil() || fType.PkgPath != "" || tag == "-" {
continue
}
// if no tag is set, use the field name
if tag == "" {
tag = fType.Name
}
// and make the tag lowercase in the end
tag = strings.ToLower(tag)
sqlPatch.Fields = append(sqlPatch.Fields, tag+" = ?")
var val reflect.Value
if fVal.Kind() == reflect.Ptr {
val = fVal.Elem()
} else {
val = fVal
}
switch val.Kind() {
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
sqlPatch.Args = append(sqlPatch.Args, val.Int())
case reflect.String:
sqlPatch.Args = append(sqlPatch.Args, val.String())
case reflect.Bool:
if val.Bool() {
sqlPatch.Args = append(sqlPatch.Args, 1)
} else {
sqlPatch.Args = append(sqlPatch.Args, 0)
}
}
}
return sqlPatch
}
Then I can simply call it like this:
type Resource struct {
Description *string `json:"description,omitempty"`
Name *string `json:"name,omitempty"`
}
func main() {
var r Resource
json.Unmarshal([]byte(`{"description": "new description"}`), &r)
sqlPatch := SQLPatches(r)
data, _ := json.Marshal(sqlPatch)
fmt.Printf("%s\n", data)
}
You can check it at Go Playground. The only problem here I see is that I allocate both the slices with the amount of fields in the passed struct, which may be 10, even though I might only want to patch one property in the end resulting in allocating more memory than needed... Any idea how to avoid this?
I recently had same problem. about PATCH and looking around found this article. It also makes references to the RFC 5789 where it says:
The difference between the PUT and PATCH requests is reflected in the way the server processes the enclosed entity to modify the resource identified by the Request-URI. In a PUT request, the enclosed entity is considered to be a modified version of the resource stored on the origin server, and the client is requesting that the stored version be replaced. With PATCH, however, the enclosed entity contains a set of instructions describing how a resource currently residing on the origin server should be modified to produce a new version. The PATCH method affects the resource identified by the Request-URI, and it also MAY have side effects on other resources; i.e., new resources may be created, or existing ones modified, by the application of a PATCH.
e.g:
[
{ "op": "test", "path": "/a/b/c", "value": "foo" },
{ "op": "remove", "path": "/a/b/c" },
{ "op": "add", "path": "/a/b/c", "value": [ "foo", "bar" ] },
{ "op": "replace", "path": "/a/b/c", "value": 42 },
{ "op": "move", "from": "/a/b/c", "path": "/a/b/d" },
{ "op": "copy", "from": "/a/b/d", "path": "/a/b/e" }
]
This set of instructions should make it easier to build the update query.
EDIT
This is how you would obtain sql tags but you will have to use reflection:
type Resource struct {
Name *string `json:"name,omitempty" sql:"resource_id"`
Description *string `json:"description,omitempty" sql:"description"`
}
sp := "sort of string"
r := Resource{Description: &sp}
rt := reflect.TypeOf(r) // reflect.Type
rv := reflect.ValueOf(r) // reflect.Value
for i := 0; i < rv.NumField(); i++ { // Iterate over all the fields
if !rv.Field(i).IsNil() { // Check it is not nil
// Here you would do what you want to having the sql tag.
// Creating the query would be easy, however
// not sure you would execute the statement
fmt.Println(rt.Field(i).Tag.Get("sql")) // Output: description
}
}
I understand you don't want to use reflection, but still this may be a better answer than the previous one as you comment state.
EDIT 2:
About the allocation - read this guide lines of Effective Go about Data structures and Allocation:
// Here you are allocating an slice of 0 length with a capacity of n
sqlPatch.Fields = make([]string, 0, n)
sqlPatch.Args = make([]interface{}, 0, n)
With make(Type, Length, Capacity (optional))
Consider the following example:
// newly allocated zeroed value with Composite Literal
// length: 0
// capacity: 0
testSlice := []int{}
fmt.Println(len(testSlice), cap(testSlice)) // 0 0
fmt.Println(testSlice) // []
// newly allocated non zeroed value with make
// length: 0
// capacity: 10
testSlice = make([]int, 0, 10)
fmt.Println(len(testSlice), cap(testSlice)) // 0 10
fmt.Println(testSlice) // []
// newly allocated non zeroed value with make
// length: 2
// capacity: 4
testSlice = make([]int, 2, 4)
fmt.Println(len(testSlice), cap(testSlice)) // 2 4
fmt.Println(testSlice) // [0 0]
In your case, may want to the following:
// Replace this
sqlPatch.Fields = make([]string, 0, n)
sqlPatch.Args = make([]interface{}, 0, n)
// With this or simple omit the capacity in make above
sqlPatch.Fields = []string{}
sqlPatch.Args = []interface{}{}
// The allocation will go as follow: length - capacity
testSlice := []int{} // 0 - 0
testSlice = append(testSlice, 1) // 1 - 2
testSlice = append(testSlice, 1) // 2 - 2
testSlice = append(testSlice, 1) // 3 - 4
testSlice = append(testSlice, 1) // 4 - 4
testSlice = append(testSlice, 1) // 5 - 8
Alright, I think the solution I used back in 2016 was quite over-engineered for even more over-engineered problem and was completely unnecessary. The question asked here was very generalized, however we were building a solution that was able to build its SQL query on its own and based on the JSON object or query parameters and/or Headers sent in the request. And that to be as generic as possible.
Nowadays I think the best solution is to avoid PATCH unless truly necessary. And even then you still can use PUT and replace the whole resource with patched property/ies coming already from the client - i.e. not giving the client the option/possibility to send any PATCH request to your server and to deal with partial updates on their own.
However this is not always recommended, especially in cases of bigger objects to save some C02 by reducing the amount of redundant transmitted data. Whenever today if I need to enable a PATCH for the client I simply define what can be patched - this gives me clarity and the final struct.
Note that I am using a IETF documented JSON Merge Patch implementation. I consider that of JSON Patch (also documented by IETF) redundant as hypothetically we could replace the whole REST API by having one single JSON Patch endpoint and let clients control the resources via allowed operations. I also think the implementation of such JSON Patch on the server side is way more complicated. The only use-case I could think of using such implementation is if I was implementing a REST API over a file system...
So the struct may be defined as in my OP:
type ResourcePatch struct {
ResourceID some.UUID `json:"resource_id"`
Description *string `json:"description,omitempty"`
Name *string `json:"name,omitempty"`
}
In the handler func I'd decode the ID from the path into the ResourcePatch instance and unmarshall JSON from the request body into it, too.
Sending only this
{"description":"Some new description"}
to PATCH /resources/<UUID>
I should end up with with this object:
ResourcePatch
* ResourceID {"UUID"}
* Description {"Some new description"}
And now the magic: use a simple logic to build the query and exec parameters. For some it may seem tedious or repetitive or unclean for bigger PATCH objects, but my reply to this would be: if your PATCH object consists of more than 50% of the original resource' properties (or simply too many for your liking) use PUT and expect the clients to send (and replace) the whole resource instead.
It could look like this:
func (s Store) patchMyResource(r models.ResourcePatch) error {
q := `UPDATE resources SET `
qParts := make([]string, 0, 2)
args := make([]interface{}, 0, 2)
if r.Description != nil {
qParts = append(qParts, `description = ?`)
args = append(args, r.Description)
}
if r.Name != nil {
qParts = append(qParts, `name = ?`)
args = append(args, r.Name)
}
q += strings.Join(qParts, ',') + ` WHERE resource_id = ?`
args = append(args, r.ResourceID)
_, err := s.db.Exec(q, args...)
return err
}
I think there's nothing simpler and more effective. No reflection, no over-kills, reads quite good.
Struct tags are only visible through reflection, sorry.
If you don't want to use reflection (or, I think, even if you do), I think it is Go-like to define a function or method that "marshals" your struct into something that can easily be turned into a comma-separated list of SQL updates, and then use that. Build small things to help solve your problems.
For example given:
type Resource struct {
Name *string `json:"name,omitempty" sql:"resource_id"`
Description *string `json:"description,omitempty" sql:"description"`
}
You might define:
func (r Resource) SQLUpdates() SQLUpdates {
var s SQLUpdates
if (r.Name != nil) {
s.add("resource_id", *r.Name)
}
if (r.Description != nil) {
s.add("description", *r.Description)
}
}
where the type SQLUpdates looks something like this:
type SQLUpdates struct {
assignments []string
values []interface{}
}
func (s *SQLUpdates) add(key string, value interface{}) {
if (s.assignments == nil) {
s.assignments = make([]string, 0, 1)
}
if (s.values == nil) {
s.values = make([]interface{}, 0, 1)
}
s.assignments = append(s.assignments, fmt.Sprintf("%s = ?", key))
s.values = append(s.values, value)
}
func (s SQLUpdates) Assignments() string {
return strings.Join(s.assignments, ", ")
}
func (s SQLUpdates) Values() []interface{} {
return s.values
}
See it working (sorta) here: https://play.golang.org/p/IQAHgqfBRh
If you have deep structs-within-structs, it should be fairly easy to build on this. And if you change to an SQL engine that allows or encourages positional arguments like $1 instead of ?, it's easy to add that behavior to just the SQLUpdates struct without changing any code that used it.
For the purpose of getting arguments to pass to Exec, you would just expand the output of Values() with the ... operator.
I'm trying to read all the rows from a table on a SQL server and store them in string slices to use for later. The issue I'm running into is that the previously scanned rows are getting overwritten every time I scan a new row, even though I've converted all the mutable byte slices to immutable strings and saved the result slices to another slice. Here is the code I'm using:
rawResult := make([]interface{}, len(cols)) // holds anything that could be in a row
result := make([]string, len(cols)) // will hold all row elements as strings
var results [][]string // will hold all the result string slices
dest := make([]interface{}, len(cols)) // temporary, to pass into scan
for i, _ := range rawResult {
dest[i] = &rawResult[i] // fill dest with pointers to rawResult to pass into scan
}
for rows.Next() { // for each row
err = rows.Scan(dest...) // scan the row
if err != nil {
log.Fatal("Failed to scan row", err)
}
for i, raw := range rawResult { // for each scanned byte slice in a row
switch rawtype := raw.(type){ // determine type, convert to string
case int64:
result[i] = strconv.FormatInt(raw.(int64), 10)
case float64:
result[i] = strconv.FormatFloat(raw.(float64), 'f', -1, 64)
case bool:
result[i] = strconv.FormatBool(raw.(bool))
case []byte:
result[i] = string(raw.([]byte))
case string:
result[i] = raw.(string)
case time.Time:
result[i] = raw.(time.Time).String()
case nil:
result[i] = ""
default: // shouldn't actually be reachable since all types have been covered
log.Fatal("Unexpected type %T", rawtype)
}
}
results = append(results, result) // append the result to our slice of results
}
I'm sure this has something to do with the way Go handles variables and memory, but I can't seem to fix it. Can somebody explain what I'm not understanding?
You should create new slice for each data row. Notice, that a slice has a pointer to underlying array, so every slice you added into results have same pointer on actual data array. That's why you have faced with that behaviour.
When you create a slice using func make() it return a type (Not a pointer to type). But it does not allocate new memory each time a element is reassigned. Hence
result := make([]string, 5)
will have fix memory to contain 5 strings. when a element is reassigned, it occupies same memory as before hence overriding the old value.
Hopefully following example make things clear.
http://play.golang.org/p/3w2NtEHRuu
Hence in your program you are changing the content of the same memory and appending it again and again. To solve this problem you should create your result slice inside the loop.
Move result := make([]string, len(cols)) into your for loop that loops over the available rows.
Is it valid to rebind a mutable variable in a while loop? I am having trouble getting the following trivial parser code to work. My intention is to replace the newslice binding with a progressively shorter slice as I copy characters out of the front of the array.
/// Test if a char is an ASCII digit
fn is_digit(c:u8) -> bool {
match c {
30|31|32|33|34|35|36|37|38|39 => true,
_ => false
}
}
/// Parse an integer from the front of an ascii string,
/// and return it along with the remainder of the string
fn parse_int(s:&[u8]) -> (u32, &[u8]) {
use std::str;
assert!(s.len()>0);
let mut newslice = s; // bytecopy of the fat pointer?
let mut n:Vec<u8> = vec![];
// Pull the leading digits into a separate array
while newslice.len()>0 && is_digit(newslice[0])
{
n.push(newslice[0]);
newslice = newslice.slice(1,newslice.len()-1);
//newslice = newslice[1..];
}
match from_str::<u32>(str::from_utf8(newslice).unwrap()) {
Some(i) => (i,newslice),
None => panic!("Could not convert string to int. Corrupted pgm file?"),
}
}
fn main(){
let s:&[u8] = b"12345";
assert!(s.len()==5);
let (i,newslice) = parse_int(s);
assert!(i==12345);
println!("length of returned slice: {}",newslice.len());
assert!(newslice.len()==0);
}
parse_int is failing to return a slice that is smaller than the one I passed in:
length of returned slice: 5
task '<main>' panicked at 'assertion failed: newslice.len() == 0', <anon>:37
playpen: application terminated with error code 101
Run this code in the rust playpen
As Chris Morgan mentioned, your call to slice passes the wrong value for the end parameter. newslice.slice_from(1) yields the correct slice.
is_digit tests for the wrong byte values. You meant to write 0x30, etc. instead of 30.
You call str::from_utf8 on the wrong value. You meant to call it on n.as_slice() rather than newslice.
Rebinding variables like that is perfectly fine. The general rule is simple: if the compiler doesn’t complain, it’s OK.
It’s a very simple error that you’ve made: your slice end point is incorrect.
slice produces the interval [start, end)—a half-open range, not closed. Therefore when you wish to just remove the first character, you should be writing newslice.slice(1, newslice.len()), not newslice.slice(1, newslice.len() - 1). You could also write newslice.slice_from(1).
I was wondering if it is possible to dynamically create variables in Go?
I have provided a pseudo-code below to illustrate what I mean. I am storing the newly created variables in a slice:
func method() {
slice := make([]type)
for(i=0;i<10;i++)
{
var variable+i=i;
slice := append(slice, variablei)
}
}
At the end of the loop, the slice should contain the variables: variable1, variable2...variable9
Go has no dynamic variables.
Dynamic variables in most languages are implemented as Map (Hashtable).
So you can have one of following maps in your code that will do what you want
var m1 map[string]int
var m2 map[string]string
var m3 map[string]interface{}
here is Go code that does what you what
http://play.golang.org/p/d4aKTi1OB0
package main
import "fmt"
func method() []int {
var slice []int
for i := 0; i < 10; i++ {
m1 := map[string]int{}
key := fmt.Sprintf("variable%d", i)
m1[key] = i
slice = append(slice, m1[key])
}
return slice
}
func main() {
fmt.Println(method())
}
No; you cannot refer to local variables if you don’t know their names at compile-time.
If you need the extra indirection you can do it using pointers instead.
func function() {
slice := []*int{}
for i := 0; i < 10; i++ {
variable := i
slice = append(slice, &variable)
}
// slice now contains ten pointers to integers
}
Also note that the parentheses in the for loop ought to be omitted, and putting the opening brace on a new line is a syntax error due to automatic semicolon insertion after ++. makeing a slice requires you to pass a length, hence I don’t use it since append is used anyway.