I'm using Go with the GORM ORM.
I have the following structs. The relation is simple. One Town has multiple Places and one Place belongs to one Town.
type Place struct {
ID int
Name string
Town Town
}
type Town struct {
ID int
Name string
}
Now i want to query all places and get along with all their fields the info of the corresponding town.
This is my code:
db, _ := gorm.Open("sqlite3", "./data.db")
defer db.Close()
places := []Place{}
db.Find(&places)
fmt.Println(places)
My sample database has this data:
/* places table */
id name town_id
1 Place1 1
2 Place2 1
/* towns Table */
id name
1 Town1
2 Town2
i'm receiving this:
[{1 Place1 {0 }} {2 Mares Place2 {0 }}]
But i'm expecting to receive something like this (both places belongs to the same town):
[{1 Place1 {1 Town1}} {2 Mares Place2 {1 Town1}}]
How can i do such query ? I tried using Preloads and Related without success (probably the wrong way). I can't get working the expected result.
TownID must be specified as the foreign key. The Place struct gets like this:
type Place struct {
ID int
Name string
Description string
TownID int
Town Town
}
Now there are different approach to handle this. For example:
places := []Place{}
db.Find(&places)
for i, _ := range places {
db.Model(places[i]).Related(&places[i].Town)
}
This will certainly produce the expected result, but notice the log output and the queries triggered.
[4.76ms] SELECT * FROM "places"
[1.00ms] SELECT * FROM "towns" WHERE ("id" = '1')
[0.73ms] SELECT * FROM "towns" WHERE ("id" = '1')
[{1 Place1 {1 Town1} 1} {2 Place2 {1 Town1} 1}]
The output is the expected but this approach has a fundamental flaw, notice that for every place there is the need to do another db query which produce a n + 1 problem issue. This could solve the problem but will quickly gets out of control as the amount of places grow.
It turns out that the good approach is fairly simple using preloads.
db.Preload("Town").Find(&places)
That's it, the query log produced is:
[22.24ms] SELECT * FROM "places"
[0.92ms] SELECT * FROM "towns" WHERE ("id" in ('1'))
[{1 Place1 {1 Town1} 1} {2 Place2 {1 Town1} 1}]
This approach will only trigger two queries, one for all places, and one for all towns that has places. This approach scales well regarding of the amount of places and towns (only two queries in all cases).
You do not specify the foreign key of towns in your Place struct. Simply add TownId to your Place struct and it should work.
package main
import (
"fmt"
"github.com/jinzhu/gorm"
_ "github.com/mattn/go-sqlite3"
)
type Place struct {
Id int
Name string
Town Town
TownId int //Foregin key
}
type Town struct {
Id int
Name string
}
func main() {
db, _ := gorm.Open("sqlite3", "./data.db")
defer db.Close()
db.CreateTable(&Place{})
db.CreateTable(&Town{})
t := Town{
Name: "TestTown",
}
p1 := Place{
Name: "Test",
TownId: 1,
}
p2 := Place{
Name: "Test2",
TownId: 1,
}
err := db.Save(&t).Error
err = db.Save(&p1).Error
err = db.Save(&p2).Error
if err != nil {
panic(err)
}
places := []Place{}
err = db.Find(&places).Error
for i, _ := range places {
db.Model(places[i]).Related(&places[i].Town)
}
if err != nil {
panic(err)
} else {
fmt.Println(places)
}
}
To optimize query I use "in condition" in the same situation
places := []Place{}
DB.Find(&places)
keys := []uint{}
for _, value := range places {
keys = append(keys, value.TownID)
}
rows := []Town{}
DB.Where(keys).Find(&rows)
related := map[uint]Town{}
for _, value := range rows {
related[value.ID] = value
}
for key, value := range places {
if _, ok := related[value.TownID]; ok {
res[key].Town = related[value.TownID]
}
}
First change your model:
type Place struct {
ID int
Name string
Description string
TownID int
Town Town
}
And second, make preloading:
https://gorm.io/docs/preload.html
Click For Full Docs
Summary: preloading one-to-one relation: has one, belongs to
eager preload:
db.Preload("Orders").Preload("Profile").Find(&users)
join preload using inner join:
db.Joins("Orders").Joins("Profile").Find(&users)
preload all associations:
db.Preload(clause.Associations).Find(&users)
No need to loop for ids, just pluck the ids
townIDs := []uint{}
DB.Model(&Place{}).Pluck("town_id", &placeIDs)
towns := []Town{}
DB.Where(townIDs).Find(&towns)
Related
I have tried to find the solution for this problem, but keep running my head at the wall with this one.
This function is part of a Go SQL wrapper, and the function getJSON is called to extract the informations from the sql response.
The problem is, that the id parameter becomes jibberish and does not match the desired response, all the other parameters read are correct thou, so this really weirds me out.
Thank you in advance, for any attempt at figurring this problem out, it is really appreciated :-)
func getJSON(rows *sqlx.Rows) ([]byte, error) {
columns, err := rows.Columns()
rawResult := make([][]byte, len(columns))
dest := make([]interface{}, len(columns))
for i := range rawResult {
dest[i] = &rawResult[i]
}
defer rows.Close()
var results []map[string][]byte
for rows.Next() {
result := make(map[string][]byte, len(columns))
rows.Scan(dest...)
for i, raw := range rawResult {
if raw == nil {
result[columns[i]] = []byte("")
} else {
result[columns[i]] = raw
fmt.Println(columns[i] + " : " + string(raw))
}
}
results = append(results, result)
}
s, err := json.Marshal(results)
if err != nil {
panic(err)
}
rows.Close()
return s, nil
}
An example of the response, taking from the terminal:
id : r�b�X��M���+�2%
name : cat
issub : false
Expected result:
id : E262B172-B158-4DEF-8015-9BA12BF53225
name : cat
issub : false
That's not about type conversion.
An UUID (of any type; presently there are four) is defined to be a 128-bit-long lump of bytes, which is 128/8=16 bytes.
This means any bytes — not necessarily printable.
What you're after, is a string representation of an UUID value, which
Separates certain groups of bytes using dashes.
Formats each byte in these groups using hexadecimal (base-16) representation.
Since base-16 positional count represents values 0 through 15 using a single digit ('0' through 'F'), a single byte is represented by two such digits — a digit per each group of 4 bits.
I think any sensible UUID package should implement a "decoding" function/method which would produce a string representation out of those 16 bytes.
I have picked a random package produced by performing this search query, and it has github.com/google/uuid.FromBytes which produces an UUID from a given byte slice, and the type of the resulting value implements the String() method which produces what you're after.
Trying to fix this problem i'm having with my api im building.
db:
DROP TABLE IF EXISTS contacts CASCADE;
CREATE TABLE IF NOT EXISTS contacts (
uuid UUID UNIQUE PRIMARY KEY,
first_name varchar(150),
);
DROP TABLE IF EXISTS workorders CASCADE;
CREATE TABLE IF NOT EXISTS workorders (
uuid UUID UNIQUE PRIMARY KEY,
work_date timestamp WITH time zone,
requested_by UUID REFERENCES contacts (uuid) ON UPDATE CASCADE ON DELETE CASCADE,
);
struct:
https://gopkg.in/guregu/null.v3
type WorkorderNew struct {
UUID string `json:"uuid"`
WorkDate null.Time `json:"work_date"`
RequestedBy null.String `json:"requested_by"`
}
api code:
workorder := &models.WorkorderNew{}
if err := json.NewDecoder(r.Body).Decode(workorder); err != nil {
log.Println("decoding fail", err)
}
// fmt.Println(NewUUID())
u2, err := uuid.NewV4()
if err != nil {
log.Fatalf("failed to generate UUID: %v", err)
}
q := `
INSERT
INTO workorders
(uuid,
work_date,
requested_by
)
VALUES
($1,$2,$3)
RETURNING uuid;`
statement, err := global.DB.Prepare(q)
global.CheckDbErr(err)
fmt.Println("requested by", workorder.RequestedBy)
lastInsertID := ""
err = statement.QueryRow(
u2,
workorder.WorkDate,
workorder.RequestedBy,
).Scan(&lastInsertID)
global.CheckDbErr(err)
json.NewEncoder(w).Encode(lastInsertID)
When I send an API request with null as the value it works as expected
but when i try to send a "" as the value for the null.String or the null.Time it fails
works:
{
"work_date":"2016-12-16T19:00:00Z",
"requested_by":null
}
not working:
{
"work_date":"2016-12-16T19:00:00Z",
"requested_by":""
}
Basically when i call the QueryRow and save to database the workorder.RequestedBy value should be a null and not the "" value im getting
thanks
If you want to treat empty strings as nulls you have at least two options.
"Extend" null.String:
type MyNullString struct {
null.String
}
func (ns *MyNullString) UnmarshalJSON(data []byte) error {
if string(data) == `""` {
ns.Valid = false
return nil
}
ns.String.UnmarshalJSON(data)
}
Or use NULLIF in the query:
INSERT INTO workorders (
uuid
, work_date
, requested_by
) VALUES (
$1
, $2
, NULLIF($3, '')
)
RETURNING uuid
Update:
To extend the null.Time you have to understand that the type of null.Time.Time is a struct. The builtin len function works on slices, arrays, pointers to arrays, maps, channels, and strings. Not structs. So in this case you can check the data argument, which is a byte slice, by converting it to a string and comparing it to a string that contains an empty string, i.e. it has two double quotes and nothing else.
type MyNullTime struct {
null.Time
}
func (ns *MyNullTime) UnmarshalJSON(data []byte) error {
if string(data) == `""` {
ns.Valid = false
return nil
}
return ns.Time.UnmarshalJSON(data)
}
I use https://github.com/jmoiron/sqlx to make queries to Postgres.
Is it possible to get back the whole row data when inserting a new row?
Here is the query I run:
result, err := Db.Exec("INSERT INTO users (name) VALUES ($1)", user.Name)
Or should I just use my existing user struct as the source of truth about the new entry in the database?
Here is docs about transaction of sqlx:
The result has two possible pieces of data: LastInsertId() or RowsAffected(), the availability of which is driver dependent. In MySQL, for instance, LastInsertId() will be available on inserts with an auto-increment key, but in PostgreSQL, this information can only be retrieved from a normal row cursor by using the RETURNING clause.
So I made a complete demo for how to execute transaction using sqlx, the demo will create an address row in addresses table and then create a user in users table using the new address_id PK as user_address_id FK of the user.
package transaction
import (
"database/sql"
"github.com/jmoiron/sqlx"
"log"
"github.com/pkg/errors"
)
import (
"github.com/icrowley/fake"
)
type User struct {
UserID int `db:"user_id"`
UserNme string `db:"user_nme"`
UserEmail string `db:"user_email"`
UserAddressId sql.NullInt64 `db:"user_address_id"`
}
type ITransactionSamples interface {
CreateUserTransaction() (*User, error)
}
type TransactionSamples struct {
Db *sqlx.DB
}
func NewTransactionSamples(Db *sqlx.DB) ITransactionSamples {
return &TransactionSamples{Db}
}
func (ts *TransactionSamples) CreateUserTransaction() (*User, error) {
tx := ts.Db.MustBegin()
var lastInsertId int
err := tx.QueryRowx(`INSERT INTO addresses (address_id, address_city, address_country, address_state) VALUES ($1, $2, $3, $4) RETURNING address_id`, 3, fake.City(), fake.Country(), fake.State()).Scan(&lastInsertId)
if err != nil {
tx.Rollback()
return nil, errors.Wrap(err, "insert address error")
}
log.Println("lastInsertId: ", lastInsertId)
var user User
err = tx.QueryRowx(`INSERT INTO users (user_id, user_nme, user_email, user_address_id) VALUES ($1, $2, $3, $4) RETURNING *;`, 6, fake.UserName(), fake.EmailAddress(), lastInsertId).StructScan(&user)
if err != nil {
tx.Rollback()
return nil, errors.Wrap(err, "insert user error")
}
err = tx.Commit()
if err != nil {
return nil, errors.Wrap(err, "tx.Commit()")
}
return &user, nil
}
Here is test result:
☁ transaction [master] ⚡ go test -v -count 1 ./...
=== RUN TestCreateUserTransaction
2019/06/27 16:38:50 lastInsertId: 3
--- PASS: TestCreateUserTransaction (0.01s)
transaction_test.go:28: &transaction.User{UserID:6, UserNme:"corrupti", UserEmail:"reiciendis_quam#Thoughtstorm.mil", UserAddressId:sql.NullInt64{Int64:3, Valid:true}}
PASS
ok sqlx-samples/transaction 3.254s
This is a sample code that works with named queries and strong type structures for inserted data and ID.
Query and struct included to cover used syntax.
const query = `INSERT INTO checks (
start, status) VALUES (
:start, :status)
returning id;`
type Row struct {
Status string `db:"status"`
Start time.Time `db:"start"`
}
func InsertCheck(ctx context.Context, row Row, tx *sqlx.Tx) (int64, error) {
return insert(ctx, row, insertCheck, "checks", tx)
}
// insert inserts row into table using query SQL command
// table used only for loging, actual table name defined in query
// should not be used from services directly - implement strong type wrappers
// function expects query with named parameters
func insert(ctx context.Context, row interface{}, query string, table string, tx *sqlx.Tx) (int64, error) {
// convert named query to native parameters format
query, args, err := tx.BindNamed(query, row)
if err != nil {
return 0, fmt.Errorf("cannot bind parameters for insert into %q: %w", table, err)
}
var id struct {
Val int64 `db:"id"`
}
err = sqlx.GetContext(ctx, tx, &id, query, args...)
if err != nil {
return 0, fmt.Errorf("cannot insert into %q: %w", table, err)
}
return id.Val, nil
}
PostgreSQL supports RETURNING syntax for INSERT statements.
Example:
INSERT INTO users(...) VALUES(...) RETURNING id, name, foo, bar
Documentaion: https://www.postgresql.org/docs/9.6/static/sql-insert.html
The optional RETURNING clause causes INSERT to compute and return value(s) based on each row actually inserted (or updated, if an ON CONFLICT DO UPDATE clause was used). This is primarily useful for obtaining values that were supplied by defaults, such as a serial sequence number. However, any expression using the table's columns is allowed. The syntax of the RETURNING list is identical to that of the output list of SELECT. Only rows that were successfully inserted or updated will be returned.
In Go, I would like to do something like this. I have a big object with many structs (using Google's protobuf). here is a contrived example:
person.name = "testing"
person.address.street = "123 test st"
person.address.city = "tester"
person.address.zip = 90210
person.billing.address.same = true
I would like to be able to dynamically reference things. for example:
key := "person.address.zip"
fmt.Println("the value of key: " + key) // would like to get 90210
key := "person.address.city"
fmt.Println("the value of key: " + key) // would like to get "tester"
Is this possible in Go? if so, how could I do that? essentially, I'm creating a report which only contains a subset of the object and I want to be able to create a mapping file where the user can map keys/values together and my program will output the value. I have this working in python, but wanted to try using Go :)
You may use func (v Value) FieldByName(name string) Value from reflect package:
FieldByName returns the struct field with the given name. It returns
the zero Value if no field was found. It panics if v's Kind is not
struct.
Like this working sample code:
package main
import "fmt"
import "reflect"
func main() {
person := Person{}
person.name = "testing"
person.address.street = "123 test st"
person.address.city = "tester"
person.address.zip = 90210
person.billing.address.same = true
v := reflect.ValueOf(person)
f := v.FieldByName("address")
key := f.FieldByName("zip")
fmt.Println(key) // 90210
fmt.Println(f.FieldByName("city")) // tester
}
type Person struct {
name string
address Address
billing Billing
}
type Billing struct {
address Address
}
type Address struct {
street, city string
zip int
same bool
}
output:
90210
tester
And for your special case, you may use fmt.Println(field(person, "person.address.zip")), like this working sample code (just for demonstration):
package main
import "fmt"
import "reflect"
import "strings"
func field(t interface{}, key string) reflect.Value {
strs := strings.Split(key, ".")
v := reflect.ValueOf(t)
for _, s := range strs[1:] {
v = v.FieldByName(s)
}
return v
}
func main() {
person := Person{}
person.name = "testing"
person.address.street = "123 test st"
person.address.city = "tester"
person.address.zip = 90210
person.billing.address.same = true
fmt.Println(field(person, "person.address.zip")) //90210
fmt.Println(field(person, "person.address.city")) //tester
}
type Person struct {
name string
address Address
billing Billing
}
type Billing struct {
address Address
}
type Address struct {
street, city string
zip int
same bool
}
output:
90210
tester
I'm not familiar with protobuf's internals or if it provides any means to do that.
But, (1) if you like to read the value in the way you described - by chaining fields dynamically and (2) you want to read it more than one time; I would just serialize it into json and use this package. It's very fast and gives you (almost) the same semantic you desire:
// assuming your object got marshaled to this for example
json := `{"name":{"first":"Janet","last":"Prichard"},"age":47}`
value := gjson.Get(json, "name.last")
println(value.String())
After reading many tutorials, I found that there are many ways to bind arguments on prepared statement in Go, some of them
SELECT * FROM bla WHERE x = ?col1 AND y = ?col2
SELECT * FROM bla WHERE x = ? AND y = ?
SELECT * FROM bla WHERE x = :col1 AND y = :col2
SELECT * FROM bla WHERE x = $1 AND y = $2
First question, what is the cross-database way to bind arguments? (that works on any database)
Second question, none of the tutorial I've read mention about LIKE statement, how to bind arguments for LIKE-statement correctly?
SELECT * FROM bla WHERE x LIKE /*WHAT?*/
Third question, also none of them give an example for IN statement, how to bind arguments for IN statement correctly?
`SELECT * FROM bla WHERE x IN ( /*WHAT?*/ )
What is the cross-database way to bind arguments?
With database/sql, there is none. Each database has its own way to represent parameter placeholders. The Go database/sql package does not provide any normalization facility for the prepared statements. Prepared statement texts are just passed to the underlying driver, and the driver typically just sends them unmodified to the database server (or library for embedded databases).
How to bind arguments for LIKE-statement correctly?
You can use parameter placeholders after a like statement and bind it as a string. For instance, you could write a prepared statement as:
SELECT a from bla WHERE b LIKE ?
Here is an example (error management handling omitted).
package main
import (
"database/sql"
"fmt"
_ "github.com/go-sql-driver/mysql"
)
// > select * from bla ;
// +------+------+
// | a | b |
// +------+------+
// | toto | titi |
// | bobo | bibi |
// +------+------+
func main() {
// Open connection
db, err := sql.Open("mysql", "root:XXXXXXX#/test")
if err != nil {
panic(err.Error()) // proper error handling instead of panic in your app
}
defer db.Close()
// Prepare statement for reading data
stmtOut, err := db.Prepare("SELECT a FROM bla WHERE b LIKE ?")
if err != nil {
panic(err.Error()) // proper error handling instead of panic in your app
}
defer stmtOut.Close()
var a string
b := "bi%" // LIKE 'bi%'
err = stmtOut.QueryRow(b).Scan(&a)
if err != nil {
panic(err.Error()) // proper error handling instead of panic in your app
}
fmt.Printf("a = %s\n", a)
}
Note that the % character is part of the bound string, not of the query text.
How to bind arguments for IN statement correctly?
None of the databases I know allows binding a list of parameters directly with a IN clause. This is not a limitation of database/sql or the drivers, but this is simply not supported by most database servers.
You have several ways to work the problem around:
you can build a query with a fixed number of placeholders in the IN clause. Only bind the parameters you are provided with, and complete the other placeholders by the NULL value. If you have more values than the fixed number you have chosen, just execute the query several times. This is not extremely elegant, but it can be effective.
you can build multiple queries with various number of placeholders. One query for IN ( ? ), a second query for IN (?, ?), a third for IN (?,?,?), etc ... Keep those prepared queries in a statement cache, and choose the right one at runtime depending on the number of input parameters. Note that it takes memory, and generally the maximum number of prepared statements is limited, so it cannot be used when the number of parameters is high.
if the number of input parameters is high, insert them in a temporary table, and replace the query with the IN clause by a join with the temporary table. It is effective if you manage to perform the insertion in the temporary table in one roundtrip. With Go and database/sql, it is not convenient because there is no way to batch queries.
Each of these solutions has drawbacks. None of them is perfect.
I'm a newbie to Go but just to answer the first part:
First question, what is the cross-database way to bind arguments? (that works on any database)
If you use sqlx, which is a superset of the built-in sql package, then you should be able to use sqlx.DB.Rebind to achieve that.
I had this same question, and after reading the answers started to look for other solution on how to bind arguments for the IN statement.
Here is an example of what I did, not the most elegant solution, but works for me.
What I did was to create a select query with the parameters statically set on the query, and not using the bind feature at all.
It could be a good idea to sanitize the string that comes from the Marshal command, to be sure and safe, but I don't need it now.
package main
import (
"database/sql"
"encoding/json"
"fmt"
"log"
_ "github.com/go-sql-driver/mysql"
)
type Result struct {
Identifier string
Enabled bool
}
func main() {
// Open connection
db, err := sql.Open("mysql", "username:password#tcp(server-host)/my-database")
if err != nil {
panic(err.Error()) // proper error handling instead of panic in your app
}
defer db.Close()
// this is an example of a variable list of IDs
idList := []string{"ID1", "ID2", "ID3", "ID4", "ID5", "IDx"}
// convert the list to a JSON string
formatted, _ := json.Marshal(idList)
// a JSON array starts and ends with '[]' respectivelly, so we replace them with '()'
formatted[0] = '('
formatted[len(formatted)-1] = ')'
// create a static select query
query := fmt.Sprintf("SELECT identifier, is_enabled FROM some_table WHERE identifier in %s", string(formatted))
// prepare que query
rows, err := db.Query(query)
if err != nil {
panic(err.Error()) // proper error handling instead of panic in your app
}
defer rows.Close()
var result []Result
// fetch rows
for rows.Next() {
var r0 Result
if err := rows.Scan(&r0.Identifier, &r0.Enabled); err != nil {
log.Fatal(err)
}
// append the row to the result
result = append(result, r0)
}
if err := rows.Err(); err != nil {
log.Fatal(err)
}
fmt.Printf("result = %v\n", result)
}