I have multiple structures with the same format as MapParameters that are passed to the encodeParams function. Unfortunately, using that function against these structures produces unwanted encoding including the embedded structure name. Is there anyway I can fix this using reflect without using a huge switch library of type assertions?
// Desired Encoding
&required_param=1
// Current Encoding
%5BRequired%5D&required_param=1
// Desired
type MapParameters struct {
Required struct { ... }
Optional struct { ... }
}
// Current
type MapParameters struct {
MapRequired
MapOptional
}
type MapRequired struct { ... }
type MapOptional struct { ... }
func encodeParams(s string, opt interface{}) (string, error) {
v := reflect.ValueOf(opt)
if v.Kind() == reflect.Ptr && v.IsNil() {
return s, nil
}
u, err := url.Parse(s)
if err != nil {
return s, err
}
// from github.com/google/go-querystring/query
qs, err := query.Values(opt)
if err != nil {
return s, err
}
u.RawQuery = u.RawQuery + qs.Encode()
return u.String(), nil
}
Anonymous not mean embed, they are completely different two thing. Embedding means the fields of nested struct will present to outer struct. Anonymous just missing the struct name. you are expecting anonymous structs to be embedded, it is not a good idea.
Anyhow, if you want encoding anonymous as embedded, change the code in url-encoding lib https://github.com/google/go-querystring/blob/master/query/encode.go
if /*sf.Anonymous &&*/ sv.Kind() == reflect.Struct {
// save embedded struct for later processing
embedded = append(embedded, sv)
continue
}
please note sf.Anonymous not mean anonymous in real, it means embedded as the comment saying
type StructField struct {
...
Index []int // index sequence for Type.FieldByIndex
Anonymous bool // is an embedded field
}
Related
type Apis struct {
Items []struct {
ID string `json:"id"`
Name string `json:"name"`
Description string `json:"description"`
CreatedDate int `json:"createdDate"`
APIKeySource string `json:"apiKeySource"`
EndpointConfiguration struct {
Types []string `json:"types"`
} `json:"endpointConfiguration"`
} `json:"items"`
}
This the struct I have defined to store the APIs i get in json format. How do I get a specific API by its name and then get it's ID. For example lets say, apiname == Shopping and i want Shopping API's ID assigned to id variable.
ps : I'm new to golang and a well explained answer will be very much appreciated.
Thanks guys
In your case Items is slice of custom structs, so you have to perform search over loop, like this:
package main
import (
"encoding/json"
"fmt"
)
type Apis struct {
Items []struct {
ID string `json:"id"`
Name string `json:"name"`
Description string `json:"description"`
CreatedDate int `json:"createdDate"`
APIKeySource string `json:"apiKeySource"`
EndpointConfiguration struct {
Types []string `json:"types"`
} `json:"endpointConfiguration"`
} `json:"items"`
}
func main() {
// Some JSON example:
jsonStr := `{"items": [{"id":"1","name":"foo"},{"id":"2","name":"bar"}]}`
// Unmarshal from JSON into Apis struct.
apis := Apis{}
err := json.Unmarshal([]byte(jsonStr), &apis)
if err != nil {
// error handling
}
// Actual search:
nameToFind := "bar"
for _, item := range apis.Items {
if item.Name == nameToFind {
fmt.Printf("found: %+v", item.ID)
break
}
}
}
It would be better to have map of custom structs instead of slice, so you could to do something like this:
package main
import (
"encoding/json"
"fmt"
)
type Apis struct {
Items map[string]struct {
ID string `json:"id"`
Name string `json:"name"`
Description string `json:"description"`
CreatedDate int `json:"createdDate"`
APIKeySource string `json:"apiKeySource"`
EndpointConfiguration struct {
Types []string `json:"types"`
} `json:"endpointConfiguration"`
} `json:"items"`
}
func main() {
// Some JSON example:
jsonStr := `{"items": {"foo":{"id":"1","name":"foo"},"bar":{"id":"2","name":"bar"}}}`
// Unmarshal from JSON into Apis struct.
apis := Apis{}
err := json.Unmarshal([]byte(jsonStr), &apis)
if err != nil {
// error handling
}
// Actual search:
nameToFind := "bar"
item, found := apis.Items[nameToFind]
if !found {
fmt.Printf("item not found")
}
fmt.Printf("found: %+v", item)
}
IMPORTANT: with slice you complexity of algorithm will be O(n) with map - O(1) which is way better (it's best what possible).
I'd like to ensure with a test, that for each APIErrorCode constant defined as below, the map APIErrorCodeMessages contains an entry. How can I enumerate all constants of a certain type in Go?
// APIErrorCode represents the API error code
type APIErrorCode int
const (
// APIErrorCodeAuthentication represents an authentication error and corresponds with HTTP 401
APIErrorCodeAuthentication APIErrorCode = 1000
// APIErrorCodeInternalError represents an unknown internal error and corresponds with HTTP 500
APIErrorCodeInternalError APIErrorCode = 1001
)
// APIErrorCodeMessages holds all error messages for APIErrorCodes
var APIErrorCodeMessages = map[APIErrorCode]string{
APIErrorCodeInternalError: "Internal Error",
}
I've looked into reflect and go/importer and tried tools/cmd/stringer without success.
Basic concept
The reflect package does not provide access to exported identifiers, as there is no guarantee they will be linked to the executable binary (and thus available at runtime); more on this: Splitting client/server code; and How to remove unused code at compile time?
This is a source-code level checking. What I would do is write a test that checks if the number of error code constants matches the map length. The solution below will only check the map length. An improved version (see below) may also check if the keys in the map match the values of the constant declarations too.
You may use the go/parser to parse the Go file containing the error code constants, which gives you an ast.File describing the file, containing the constant declarations. You just need to walk through it, and count the error code constant declarations.
Let's say your original file is named "errcodes.go", write a test file named "errcodes_test.go".
This is how the test function could look like:
func TestMap(t *testing.T) {
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "errcodes.go", nil, 0)
if err != nil {
t.Errorf("Failed to parse file: %v", err)
return
}
errCodeCount := 0
// Range through declarations:
for _, dd := range f.Decls {
if gd, ok := dd.(*ast.GenDecl); ok {
// Find constant declrations:
if gd.Tok == token.CONST {
for _, sp := range gd.Specs {
if valSp, ok := sp.(*ast.ValueSpec); ok {
for _, name := range valSp.Names {
// Count those that start with "APIErrorCode"
if strings.HasPrefix(name.Name, "APIErrorCode") {
errCodeCount++
}
}
}
}
}
}
}
if exp, got := errCodeCount, len(APIErrorCodeMessages); exp != got {
t.Errorf("Expected %d err codes, got: %d", exp, got)
}
}
Running go test will result in:
--- FAIL: TestMap (0.00s)
errcodes_test.go:39: Expected 2 err codes, got: 1
The test properly reveals that there are 2 constant error code declarations, but the APIErrorCodeMessages map contains only 1 entry.
If we now "complete" the map:
var APIErrorCodeMessages = map[APIErrorCode]string{
APIErrorCodeInternalError: "Internal Error",
APIErrorCodeAuthentication: "asdf",
}
And run go test again:
PASS
Note: it's a matter of style, but the big loop may be written this way to decrease nesting level:
// Range through declarations:
for _, dd := range f.Decls {
gd, ok := dd.(*ast.GenDecl)
if !ok {
continue
}
// Find constant declrations:
if gd.Tok != token.CONST {
continue
}
for _, sp := range gd.Specs {
valSp, ok := sp.(*ast.ValueSpec)
if !ok {
continue
}
for _, name := range valSp.Names {
// Count those that start with "APIErrorCode"
if strings.HasPrefix(name.Name, "APIErrorCode") {
errCodeCount++
}
}
}
}
Full, improved detection
This time we will check the exact type of the constants, not their names. We will also gather all the constant values, and in the end we will check each if that exact constant value is in the map. If something is missing, we will print the exact values of the missing codes.
So here it is:
func TestMap(t *testing.T) {
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "errcodes.go", nil, 0)
if err != nil {
t.Errorf("Failed to parse file: %v", err)
return
}
var keys []APIErrorCode
// Range through declarations:
for _, dd := range f.Decls {
gd, ok := dd.(*ast.GenDecl)
if !ok {
continue
}
// Find constant declrations:
if gd.Tok != token.CONST {
continue
}
for _, sp := range gd.Specs {
// Filter by APIErrorCode type:
valSp, ok := sp.(*ast.ValueSpec)
if !ok {
continue
}
if id, ok2 := valSp.Type.(*ast.Ident); !ok2 ||
id.Name != "APIErrorCode" {
continue
}
// And gather the constant values in keys:
for _, value := range valSp.Values {
bslit, ok := value.(*ast.BasicLit)
if !ok {
continue
}
keyValue, err := strconv.Atoi(bslit.Value)
if err != nil {
t.Errorf("Could not parse value from %v: %v",
bslit.Value, err)
}
keys = append(keys, APIErrorCode(keyValue))
}
}
}
for _, key := range keys {
if _, found := APIErrorCodeMessages[key]; !found {
t.Errorf("Could not found key in map: %v", key)
}
}
}
Running go test with an "incomplete" APIErrorCodeMessages map, we get the following output:
--- FAIL: TestMap (0.00s)
errcodes_test.go:58: Could not found key in map: 1000
Short of static code analysis, which generates your tests, you can't.
You'll just need to maintain a list of known types somewhere. The most obvious place is probably in your test:
func TestAPICodes(t *testing.T) {
for _, code := range []APIErrorCode{APIErrorCodeAuthentication, ...} {
// Do your test here
}
}
If you want the list defined closer to the code definitions, you could also put it in your main package:
// APIErrorCode represents the API error code
type APIErrorCode int
const (
// APIErrorCodeAuthentication represents an authentication error and corresponds with HTTP 401
APIErrorCodeAuthentication APIErrorCode = 1000
// APIErrorCodeInternalError represents an unknown internal error and corresponds with HTTP 500
APIErrorCodeInternalError APIErrorCode = 1001
)
var allCodes = []APIErrorCode{APIErrorCodeAuthentication, ...}
Or, if you're confident that your APIErrorCodeMessages map will be kept up-to-date, then you already have the solution. Just loop over that map in your test:
func TestAPICodes(t *testing.T) {
for code := range APIErrorCodeMessages {
// Do your tests...
}
}
Although I am aware that it might not be idiomatic to panic in Go, I would like to test to ensure a function panics under certain conditions and not in others.
An example of the function.
func PanicOnErr(potentialErr error) {
if potentialErr != nil {
panic(potentialErr)
}
}
The following is an implementation for checking if the function will panic.
func InvocationCausedPanic(f interface{}, params ...interface{}) bool {
// Obtain the function's signature.
reflectedFunc := reflect.ValueOf(f)
funcType := reflect.TypeOf(f)
if funcType.NumIn() != len(params) {
panic("InvocationCausedPanic called with a function and an incorrect number of parameter(s).")
}
reflectedParams := make([]reflect.Value, len(params))
for paramIndex, paramValue := range params {
expectedType := funcType.In(paramIndex)
actualType := reflect.TypeOf(paramValue)
if actualType != expectedType {
errStr := fmt.Sprintf("InvocationCausedPanic called with a mismatched parameter type [parameter #%v: expected %v; got %v].", paramIndex, expectedType, actualType)
panic(errStr)
}
reflectedParams[paramIndex] = reflect.ValueOf(paramValue)
}
return invoke(reflectedFunc, reflectedParams)
}
func invoke(reflectedFunc reflect.Value, reflectedParams []reflect.Value) (panicked bool) {
defer func() {
if r := recover(); r != nil {
panicked = true
}
}()
reflectedFunc.Call(reflectedParams)
return
}
Calling either of the following will cause the type-check to fail.
InvocationCausedPanic(PanicOnErr, errors.New("Some error."))
InvocationCausedPanic(PanicOnErr, nil)
However, it seems possible to call PanicOnErr using both nil and something generate by calling errors.New (seems to be of type *errors.errorString).
As such, is there a way to check if the type of some parameter is suitable for invoking some function?
While I know it is possible to use defer and recover to more simply test the function, I am curious as to whether it is possible to write a general function that can accept any function and parameters and determine whether it resulted in a panic (assuming the function completes).
Relevant Go Playground:
http://play.golang.org/p/qUG7OGuIbD
Use this function to determine if the parameter is compatible:
func compatible(actual, expected reflect.Type) bool {
if actual == nil {
k := expected.Kind()
return k == reflect.Chan ||
k == reflect.Func ||
k == reflect.Interface ||
k == reflect.Map ||
k == reflect.Ptr ||
k == reflect.Slice
}
return actual.AssignableTo(expected)
}
playground
Let's say I have a struct:
type User struct {
Name string
Id int
Score int
}
And a database table with the same schema. What's the easiest way to parse a database row into a struct? I've added an answer below but I'm not sure it's the best one.
Go package tests often provide clues as to ways of doing things. For example, from database/sql/sql_test.go,
func TestQuery(t *testing.T) {
/* . . . */
rows, err := db.Query("SELECT|people|age,name|")
if err != nil {
t.Fatalf("Query: %v", err)
}
type row struct {
age int
name string
}
got := []row{}
for rows.Next() {
var r row
err = rows.Scan(&r.age, &r.name)
if err != nil {
t.Fatalf("Scan: %v", err)
}
got = append(got, r)
}
/* . . . */
}
func TestQueryRow(t *testing.T) {
/* . . . */
var name string
var age int
var birthday time.Time
err := db.QueryRow("SELECT|people|age,name|age=?", 3).Scan(&age)
/* . . . */
}
Which, for your question, querying a row into a structure, would translate to something like:
var row struct {
age int
name string
}
err = db.QueryRow("SELECT|people|age,name|age=?", 3).Scan(&row.age, &row.name)
I know that looks similar to your solution, but it's important to show how to find a solution.
I recommend github.com/jmoiron/sqlx.
From the README:
sqlx is a library which provides a set of extensions on go's standard
database/sql library. The sqlx versions of sql.DB, sql.TX,
sql.Stmt, et al. all leave the underlying interfaces untouched, so
that their interfaces are a superset on the standard ones. This makes
it relatively painless to integrate existing codebases using
database/sql with sqlx.
Major additional concepts are:
Marshal rows into structs (with embedded struct support), maps, and slices
Named parameter support including prepared statements
Get and Select to go quickly from query to struct/slice
The README also includes a code snippet demonstrating scanning a row into a struct:
type Place struct {
Country string
City sql.NullString
TelephoneCode int `db:"telcode"`
}
// Loop through rows using only one struct
place := Place{}
rows, err := db.Queryx("SELECT * FROM place")
for rows.Next() {
err := rows.StructScan(&place)
if err != nil {
log.Fatalln(err)
}
fmt.Printf("%#v\n", place)
}
Note that we didn't have to manually map each column to a field of the struct. sqlx has some default mappings for struct fields to database columns, as well as being able to specify database columns using tags (note the TelephoneCode field of the Place struct above). You can read more about that in the documentation.
Here's one way to do it - just assign all of the struct values manually in the Scan function.
func getUser(name string) (*User, error) {
var u User
// this calls sql.Open, etc.
db := getConnection()
// note the below syntax only works for postgres
err := db.QueryRow("SELECT * FROM users WHERE name = $1", name).Scan(&u.Id, &u.Name, &u.Score)
if err != nil {
return &User{}, err
} else {
return &u, nil
}
}
rows, err := connection.Query("SELECT `id`, `username`, `email` FROM `users`")
if err != nil {
panic(err.Error())
}
for rows.Next() {
var user User
if err := rows.Scan(&user.Id, &user.Username, &user.Email); err != nil {
log.Println(err.Error())
}
users = append(users, user)
}
Full example
Here is a library just for that: scany.
You can use it like that:
type User struct {
Name string
Id int
Score int
}
// db is your *sql.DB instance
// ctx is your current context.Context instance
// Use sqlscan.Select to query multiple records.
var users []*User
sqlscan.Select(ctx, db, &users, `SELECT name, id, score FROM users`)
// Use sqlscan.Get to query exactly one record.
var user User
sqlscan.Get(ctx, db, &user, `SELECT name, id, score FROM users WHERE id=123`)
It's well documented and easy to work with.
Disclaimer: I am the author of this library.
there's package just for that: sqlstruct
unfortunately, last time I checked it did not support embedded structs (which are trivial to implement yourself - i had a working prototype in a few hours).
just committed the changes I made to sqlstruct
use :
go-models-mysql
sqlbuilder
val, err = m.ScanRowType(row, (*UserTb)(nil))
or the full code
import (
"database/sql"
"fmt"
lib "github.com/eehsiao/go-models-lib"
mysql "github.com/eehsiao/go-models-mysql"
)
// MyUserDao : extend from mysql.Dao
type MyUserDao struct {
*mysql.Dao
}
// UserTb : sql table struct that to store into mysql
type UserTb struct {
Name sql.NullString `TbField:"Name"`
Id int `TbField:"Id"`
Score int `TbField:"Score"`
}
// GetFirstUser : this is a data logical function, you can write more logical in there
// sample data logical function to get the first user
func (m *MyUserDao) GetFirstUser() (user *User, err error) {
m.Select("Name", "Id", "Score").From("user").Limit(1)
fmt.Println("GetFirstUser", m.BuildSelectSQL().BuildedSQL())
var (
val interface{}
row *sql.Row
)
if row, err = m.GetRow(); err == nil {
if val, err = m.ScanRowType(row, (*UserTb)(nil)); err == nil {
u, _ := val.(*UserTb)
user = &User{
Name: lib.Iif(u.Name.Valid, u.Nae.String, "").(string),
Id: u.Id,
Score: u.Score,
}
}
}
row, val = nil, nil
return
}
I am redigo to connect from Go to a redis database. How can I convert a type of []interface {}{[]byte{} []byte{}} to a set of strings? In this case I'd like to get the two strings Hello and World.
package main
import (
"fmt"
"github.com/garyburd/redigo/redis"
)
func main() {
c, err := redis.Dial("tcp", ":6379")
defer c.Close()
if err != nil {
fmt.Println(err)
}
c.Send("SADD", "myset", "Hello")
c.Send("SADD", "myset", "World")
c.Flush()
c.Receive()
c.Receive()
err = c.Send("SMEMBERS", "myset")
if err != nil {
fmt.Println(err)
}
c.Flush()
// both give the same return value!?!?
// reply, err := c.Receive()
reply, err := redis.MultiBulk(c.Receive())
if err != nil {
fmt.Println(err)
}
fmt.Printf("%#v\n", reply)
// $ go run main.go
// []interface {}{[]byte{0x57, 0x6f, 0x72, 0x6c, 0x64}, []byte{0x48, 0x65, 0x6c, 0x6c, 0x6f}}
// How do I get 'Hello' and 'World' from this data?
}
Look in module source code
// String is a helper that converts a Redis reply to a string.
//
// Reply type Result
// integer format as decimal string
// bulk return reply as string
// string return as is
// nil return error ErrNil
// other return error
func String(v interface{}, err error) (string, error) {
redis.String will convert (v interface{}, err error) in (string, error)
reply, err := redis.MultiBulk(c.Receive())
replace with
s, err := redis.String(redis.MultiBulk(c.Receive()))
Looking at the source code for the module, you can see the type signature returned from Receive will be:
func (c *conn) Receive() (reply interface{}, err error)
and in your case, you're using MultiBulk:
func MultiBulk(v interface{}, err error) ([]interface{}, error)
This gives a reply of multiple interface{} 's in a slice: []interface{}
Before an untyped interface{} you have to assert its type like so:
x.(T)
Where T is a type (eg, int, string etc.)
In your case, you have a slice of interfaces (type: []interface{}) so, if you want a string, you need to first assert that each one has type []bytes, and then cast them to a string eg:
for _, x := range reply {
var v, ok = x.([]byte)
if ok {
fmt.Println(string(v))
}
}
Here's an example: http://play.golang.org/p/ZifbbZxEeJ
You can also use a type switch to check what kind of data you got back:
http://golang.org/ref/spec#Type_switches
for _, y := range reply {
switch i := y.(type) {
case nil:
printString("x is nil")
case int:
printInt(i) // i is an int
etc...
}
}
Or, as someone mentioned, use the built in redis.String etc. methods which will check and convert them for you.
I think the key is, each one needs to be converted, you can't just do them as a chunk (unless you write a method to do so!).
Since redis.MultiBulk() now is deprecated, it might be a good way to use redis.Values() and convert the result into String:
import "github.com/gomodule/redigo/redis"
type RedisClient struct {
Conn redis.Conn
}
func (r *RedisClient) SMEMBERS(key string) interface{} {
tmp, err := redis.Values(r.Conn.Do("smembers", key))
if err != nil {
fmt.Println(err)
return nil
}
res := make([]string, 0)
for _, v := range tmp {
res = append(res, string(v.([]byte)))
}
return res
}