I'm trying to make a function that converts a struct in the way mysql rows.Scan function needs it, so I don't need to pass manually lots of parameters.
Note: I know the existence of sqlx and the alternative of writing manually in separate lines every pointer, but I'd like to solve it in this way as I'm learning go and want to understand what's going on.
The error I get with this solution is:
panic: sql: Scan error on column index 0: destination not a pointer
to me looks like valueField.Addr().Pointer() should be a Pointer to the value. The following is a simplification of my code.
type User struct {
Name string
Age int
}
func StrutForScan(u interface{}) []interface{} {
val := reflect.ValueOf(u).Elem()
v := make([]interface{}, val.NumField())
for i := 0; i < val.NumField(); i++ {
valueField := val.Field(i)
v[i] = valueField.Addr().Pointer()
}
return v
}
func ListUsers {
rows, err := db.Query("SELECT * FROM users")
PanicIf(err)
var user User
for rows.Next() {
err := rows.Scan(StrutForScan(&user)...)
PanicIf(err)
fmt.Printf("\nName: %s, Age: %s", user.Name, string(user.Age))
}
}
You need to use .Interface() not .Pointer()
func StrutForScan(u interface{}) []interface{} {
val := reflect.ValueOf(u).Elem()
v := make([]interface{}, val.NumField())
for i := 0; i < val.NumField(); i++ {
valueField := val.Field(i)
v[i] = valueField.Addr().Interface()
}
return v
}
The reason behind that is that .Pointer() returns an actual "pointer" to the data, you can't do much with it without using the unsafe package.
Related
I have searched for a solution on the net and in SO, but found nothing that apply to return values. It is a simple sql query with several rows that I want to return. Error handling not included:
func Fetch(query string) (string) {
type User struct{
id string
name string
}
rows, err := db.Query(query)
users := make([]*User, 0)
for rows.Next() {
user := new(User)
err := rows.Scan(&user.id, &user.name)
users = append(users, user)
}
return(users)
}
I get this error when compiling:
cannot use users (type []*User) as type string in return argument
How should I do to get a correct return value?
The expected input is
JD John Doe --OR-- {id:"JD",name:"John Doe"}
Add this to your code:
type userSlice []*User
func (us userSlice) String() string{
var s []string
for _, u := range us {
if u != nil {
s = append(s, fmt.Sprintf("%s %s", u.id, u.name))
}
}
return strings.Join(s, "\n")
}
type User struct{
id string
name string
}
In your Fetch function replace the last return statement like this:
func Fetch(query string) (string) {
// Note that we declare the User type outside the function.
rows, err := db.Query(query)
users := make([]*User, 0)
for rows.Next() {
user := new(User)
err := rows.Scan(&user.id, &user.name)
users = append(users, user)
}
return(userSlice(users).String()) // Replace this line in your code
}
If you have to return a String you can use the encoding/json package to serialize your user object, but you have to use fields that begin with capital letters for them to be exported. See the full example:
import (
"encoding/json"
)
func Fetch(query string) (string) {
type User struct{
Id string // <-- CHANGED THIS LINE
Name string // <-- CHANGED THIS LINE
}
rows, err := db.Query(query)
users := make([]*User, 0)
for rows.Next() {
user := new(User)
err := rows.Scan(&user.id, &user.name)
users = append(users, user)
}
return(ToJSON(users)) // <-- CHANGED THIS LINE
}
func ToJSON(obj interface{}) (string) {
res, err := json.Marshal(obj)
if err != nil {
panic("error with json serialization " + err.Error())
}
return string(res)
}
Let's say I have a postgres query like:
SELECT
id,
ARRAY_AGG(code) AS code
FROM
codes
WHERE id = '9252781'
GROUP BY id;
My return looks like:
id | codes
-----------+-------------
9252781 | {H01,H02}
Both id and codes are varchar.
In Golang when I scan along the rows for my result, it just freezes. No error, nothing.
If you're using the github.com/lib/pq postgres driver you can use their pq.Array helper function to scan and store postgres arrays.
var id string
var arr []string
row := db.QueryRow(`SELECT '9252781', ARRAY['H01','H02']`)
if err := row.Scan(&id, pq.Array(&arr)); err != nil {
log.Fatal(err)
}
log.Println(id, arr)
// 9252781 [H01 H02]
I don't want to use a additional driver (github.com/lib/pq), and I did not found any pgx way to do it other than creating my own type.
So I did one:
type Tags []string
func (t *Tags) Scan(v interface{}) error {
if v == nil {
*t = Tags{}
return nil
}
s, ok := v.(string)
if !ok {
return fmt.Errorf("Scan is expected to receive a string from database, but got [%+v]", v)
}
s = strings.TrimPrefix(s, "{")
s = strings.TrimSuffix(s, "}")
*t = strings.Split(s, ",")
return nil
}
func (t *Tags) Value() (driver.Value, error) {
s := fmt.Sprintf("{%v}", strings.Join(([]string)(*t), ","))
return s, nil
}
then you can scan the row returned from database like:
...
err := rows.Scan(
...
&Tags,
...
)
and you can use it directly you your Exec queries.
This code works well with constants in arrays, but you need more work if want to use it in commas and brackets.
I have a table t containing a lot of columns, and my sql is like this: select * from t. Now I only want to scan one column or two from the wide returned row set. However, the sql.Scan accepts dest ...interface{} as arguments. Does it mean I have to scan everything and use only the column I needed?
I know I could change the sql from select * to select my_favorite_rows, however, in this case, I have no way to change the sql.
You can make use of Rows.Columns, e.g.
package main
import (
"database/sql"
"fmt"
"github.com/lib/pq"
)
type Vehicle struct {
Id int
Name string
Wheels int
}
// VehicleCol returns a reference for a column of a Vehicle
func VehicleCol(colname string, vh *Vehicle) interface{} {
switch colname {
case "id":
return &vh.Id
case "name":
return &vh.Name
case "wheels":
return &vh.Wheels
default:
panic("unknown column " + colname)
}
}
func panicOnErr(err error) {
if err != nil {
panic(err.Error())
}
}
func main() {
conn, err := pq.ParseURL(`postgres://docker:docker#172.17.0.2:5432/pgsqltest?schema=public`)
panicOnErr(err)
var db *sql.DB
db, err = sql.Open("postgres", conn)
panicOnErr(err)
var rows *sql.Rows
rows, err = db.Query("select * from vehicle")
panicOnErr(err)
// get the column names from the query
var columns []string
columns, err = rows.Columns()
panicOnErr(err)
colNum := len(columns)
all := []Vehicle{}
for rows.Next() {
vh := Vehicle{}
// make references for the cols with the aid of VehicleCol
cols := make([]interface{}, colNum)
for i := 0; i < colNum; i++ {
cols[i] = VehicleCol(columns[i], &vh)
}
err = rows.Scan(cols...)
panicOnErr(err)
all = append(all, vh)
}
fmt.Printf("%#v\n", all)
}
For unknown length of columns but if you're sure about their type,
cols, err := rows.Columns()
if err != nil {
log.Fatal(err.Error())
}
colLen := len(cols)
vals := make([]interface{}, colLen)
for rows.Next() {
for i := 0; i < len(colLen); i++ {
vals[i] = new(string)
}
err := rows.Scan(vals...)
if err != nil {
log.Fatal(err.Error()) // if wrong type
}
fmt.Printf("Column 1: %s\n", *(vals[0].(*string))) // will panic if wrong type
}
PS: Not recommended for prod
Let's say I have a struct:
type User struct {
Name string
Id int
Score int
}
And a database table with the same schema. What's the easiest way to parse a database row into a struct? I've added an answer below but I'm not sure it's the best one.
Go package tests often provide clues as to ways of doing things. For example, from database/sql/sql_test.go,
func TestQuery(t *testing.T) {
/* . . . */
rows, err := db.Query("SELECT|people|age,name|")
if err != nil {
t.Fatalf("Query: %v", err)
}
type row struct {
age int
name string
}
got := []row{}
for rows.Next() {
var r row
err = rows.Scan(&r.age, &r.name)
if err != nil {
t.Fatalf("Scan: %v", err)
}
got = append(got, r)
}
/* . . . */
}
func TestQueryRow(t *testing.T) {
/* . . . */
var name string
var age int
var birthday time.Time
err := db.QueryRow("SELECT|people|age,name|age=?", 3).Scan(&age)
/* . . . */
}
Which, for your question, querying a row into a structure, would translate to something like:
var row struct {
age int
name string
}
err = db.QueryRow("SELECT|people|age,name|age=?", 3).Scan(&row.age, &row.name)
I know that looks similar to your solution, but it's important to show how to find a solution.
I recommend github.com/jmoiron/sqlx.
From the README:
sqlx is a library which provides a set of extensions on go's standard
database/sql library. The sqlx versions of sql.DB, sql.TX,
sql.Stmt, et al. all leave the underlying interfaces untouched, so
that their interfaces are a superset on the standard ones. This makes
it relatively painless to integrate existing codebases using
database/sql with sqlx.
Major additional concepts are:
Marshal rows into structs (with embedded struct support), maps, and slices
Named parameter support including prepared statements
Get and Select to go quickly from query to struct/slice
The README also includes a code snippet demonstrating scanning a row into a struct:
type Place struct {
Country string
City sql.NullString
TelephoneCode int `db:"telcode"`
}
// Loop through rows using only one struct
place := Place{}
rows, err := db.Queryx("SELECT * FROM place")
for rows.Next() {
err := rows.StructScan(&place)
if err != nil {
log.Fatalln(err)
}
fmt.Printf("%#v\n", place)
}
Note that we didn't have to manually map each column to a field of the struct. sqlx has some default mappings for struct fields to database columns, as well as being able to specify database columns using tags (note the TelephoneCode field of the Place struct above). You can read more about that in the documentation.
Here's one way to do it - just assign all of the struct values manually in the Scan function.
func getUser(name string) (*User, error) {
var u User
// this calls sql.Open, etc.
db := getConnection()
// note the below syntax only works for postgres
err := db.QueryRow("SELECT * FROM users WHERE name = $1", name).Scan(&u.Id, &u.Name, &u.Score)
if err != nil {
return &User{}, err
} else {
return &u, nil
}
}
rows, err := connection.Query("SELECT `id`, `username`, `email` FROM `users`")
if err != nil {
panic(err.Error())
}
for rows.Next() {
var user User
if err := rows.Scan(&user.Id, &user.Username, &user.Email); err != nil {
log.Println(err.Error())
}
users = append(users, user)
}
Full example
Here is a library just for that: scany.
You can use it like that:
type User struct {
Name string
Id int
Score int
}
// db is your *sql.DB instance
// ctx is your current context.Context instance
// Use sqlscan.Select to query multiple records.
var users []*User
sqlscan.Select(ctx, db, &users, `SELECT name, id, score FROM users`)
// Use sqlscan.Get to query exactly one record.
var user User
sqlscan.Get(ctx, db, &user, `SELECT name, id, score FROM users WHERE id=123`)
It's well documented and easy to work with.
Disclaimer: I am the author of this library.
there's package just for that: sqlstruct
unfortunately, last time I checked it did not support embedded structs (which are trivial to implement yourself - i had a working prototype in a few hours).
just committed the changes I made to sqlstruct
use :
go-models-mysql
sqlbuilder
val, err = m.ScanRowType(row, (*UserTb)(nil))
or the full code
import (
"database/sql"
"fmt"
lib "github.com/eehsiao/go-models-lib"
mysql "github.com/eehsiao/go-models-mysql"
)
// MyUserDao : extend from mysql.Dao
type MyUserDao struct {
*mysql.Dao
}
// UserTb : sql table struct that to store into mysql
type UserTb struct {
Name sql.NullString `TbField:"Name"`
Id int `TbField:"Id"`
Score int `TbField:"Score"`
}
// GetFirstUser : this is a data logical function, you can write more logical in there
// sample data logical function to get the first user
func (m *MyUserDao) GetFirstUser() (user *User, err error) {
m.Select("Name", "Id", "Score").From("user").Limit(1)
fmt.Println("GetFirstUser", m.BuildSelectSQL().BuildedSQL())
var (
val interface{}
row *sql.Row
)
if row, err = m.GetRow(); err == nil {
if val, err = m.ScanRowType(row, (*UserTb)(nil)); err == nil {
u, _ := val.(*UserTb)
user = &User{
Name: lib.Iif(u.Name.Valid, u.Nae.String, "").(string),
Id: u.Id,
Score: u.Score,
}
}
}
row, val = nil, nil
return
}
I am redigo to connect from Go to a redis database. How can I convert a type of []interface {}{[]byte{} []byte{}} to a set of strings? In this case I'd like to get the two strings Hello and World.
package main
import (
"fmt"
"github.com/garyburd/redigo/redis"
)
func main() {
c, err := redis.Dial("tcp", ":6379")
defer c.Close()
if err != nil {
fmt.Println(err)
}
c.Send("SADD", "myset", "Hello")
c.Send("SADD", "myset", "World")
c.Flush()
c.Receive()
c.Receive()
err = c.Send("SMEMBERS", "myset")
if err != nil {
fmt.Println(err)
}
c.Flush()
// both give the same return value!?!?
// reply, err := c.Receive()
reply, err := redis.MultiBulk(c.Receive())
if err != nil {
fmt.Println(err)
}
fmt.Printf("%#v\n", reply)
// $ go run main.go
// []interface {}{[]byte{0x57, 0x6f, 0x72, 0x6c, 0x64}, []byte{0x48, 0x65, 0x6c, 0x6c, 0x6f}}
// How do I get 'Hello' and 'World' from this data?
}
Look in module source code
// String is a helper that converts a Redis reply to a string.
//
// Reply type Result
// integer format as decimal string
// bulk return reply as string
// string return as is
// nil return error ErrNil
// other return error
func String(v interface{}, err error) (string, error) {
redis.String will convert (v interface{}, err error) in (string, error)
reply, err := redis.MultiBulk(c.Receive())
replace with
s, err := redis.String(redis.MultiBulk(c.Receive()))
Looking at the source code for the module, you can see the type signature returned from Receive will be:
func (c *conn) Receive() (reply interface{}, err error)
and in your case, you're using MultiBulk:
func MultiBulk(v interface{}, err error) ([]interface{}, error)
This gives a reply of multiple interface{} 's in a slice: []interface{}
Before an untyped interface{} you have to assert its type like so:
x.(T)
Where T is a type (eg, int, string etc.)
In your case, you have a slice of interfaces (type: []interface{}) so, if you want a string, you need to first assert that each one has type []bytes, and then cast them to a string eg:
for _, x := range reply {
var v, ok = x.([]byte)
if ok {
fmt.Println(string(v))
}
}
Here's an example: http://play.golang.org/p/ZifbbZxEeJ
You can also use a type switch to check what kind of data you got back:
http://golang.org/ref/spec#Type_switches
for _, y := range reply {
switch i := y.(type) {
case nil:
printString("x is nil")
case int:
printInt(i) // i is an int
etc...
}
}
Or, as someone mentioned, use the built in redis.String etc. methods which will check and convert them for you.
I think the key is, each one needs to be converted, you can't just do them as a chunk (unless you write a method to do so!).
Since redis.MultiBulk() now is deprecated, it might be a good way to use redis.Values() and convert the result into String:
import "github.com/gomodule/redigo/redis"
type RedisClient struct {
Conn redis.Conn
}
func (r *RedisClient) SMEMBERS(key string) interface{} {
tmp, err := redis.Values(r.Conn.Do("smembers", key))
if err != nil {
fmt.Println(err)
return nil
}
res := make([]string, 0)
for _, v := range tmp {
res = append(res, string(v.([]byte)))
}
return res
}