I have such code:
tx, _ := db.Begin()
defer tx.Rollback()
err := db.Insert(foo)
err = db.Delete(bar)
if !err {
tx.Commit()
}
and I don't have idea how to write 2 test cases:
successfull (data inserted and deleted)
error (nothing changes)
I was thinking about:
monkey patching by function injection to method which is doing db operations, and change this function in test
monkey patching by changing foo sql by making it global - I don't like it too much
make db not allowing delete operation for test time
Each of above options seems to be not ideal, how should I write this test cases?
Have a look at my library dbwrap https://github.com/metakeule/dbwrap
that implements a driver.Driver, wrapping around another driver.
It also has a fake driver that you can use like this.
package main
import (
"fmt"
"github.com/metakeule/dbwrap"
)
var fake, db = dbwrap.NewFake()
func q1() {
fake.SetNumInputs(1)
db.Query("Select ?", "hiho")
q, v := fake.LastQuery()
fmt.Println(q, v)
}
Use the source code of fake.go as a starting point.
Related
I have some SQL queries that do not change on every request (only it's parameter). So, instead of doing this for each request:
func HandleRequest() {
rows, err := db.Query(sqlQuery, params...)
// do something with data
}
Is it okay if for each reqest I do this instead:
// together with server initialization
stmt, err := db.Prepare(sqlQuery)
func HandleRequest() {
rows, err := stmt.Query(params...)
// do something with data
}
As the documentation of DB.Prepare() states:
Multiple queries or executions may be run concurrently from the returned statement.
It is safe for concurrent use, although the intended use for prepared statements is not to share them between multiple requests. The main reason is that a prepared statement (may) allocate resources in the DB server itself, and it's not freed until you call the Close() method of the returned statement. So I'd advise against it.
The typical use case is if you have to run the same statement multiple times with different parameters, such as the example in the documentation:
projects := []struct {
mascot string
release int
}{
{"tux", 1991},
{"duke", 1996},
{"gopher", 2009},
{"moby dock", 2013},
}
stmt, err := db.Prepare("INSERT INTO projects(id, mascot, release, category) VALUES( ?, ?, ?, ? )")
if err != nil {
log.Fatal(err)
}
defer stmt.Close() // Prepared statements take up server resources and should be closed after use.
for id, project := range projects {
if _, err := stmt.Exec(id+1, project.mascot, project.release, "open source"); err != nil {
log.Fatal(err)
}
}
Background
I am using the github.com/jmoiron/sqlx golang package with a Postgres database.
I have the following wrapper function to run SQL code in a transaction:
func (s *postgresStore) runInTransaction(ctx context.Context, fn func(*sqlx.Tx) error) error {
tx, err := s.db.Beginx()
if err != nil {
return err
}
defer func() {
if err != nil {
tx.Rollback()
return
}
err = tx.Commit()
}()
err = fn(tx)
return err
}
Given this, consider the following code:
func (s *store) SampleFunc(ctx context.Context) error {
err := s.runInTransaction(ctx,func(tx *sqlx.Tx) error {
// Point A: Do some database work
if err := tx.Commit(); err != nil {
return err
}
// Point B: Do some more database work, which may return an error
})
}
Desired behavior
If there is an error at Point A, then the transaction should have done zero work
If there is an error at Point B, then the transaction should still have completed the work at Point A.
Problem with current code
The code does not work as intended at the moment, because I am committing the transaction twice (once in runInTransaction, once in SampleFunc).
A Possible Solution
Where I commit the transaction, I could instead run something like tx.Exec("SAVEPOINT my_savepoint"), then defer tx.Exec("ROLLBACK TO SAVEPOINT my_savepoint")
After the code at Point B, I could run: tx.Exec("RELEASE SAVEPOINT my_savepoint")
So, if the code at Point B runs without error, I will fail to ROLLBACK to my savepoint.
Problems with Possible Solution
I'm not sure if using savepoints will mess with the database/sql package's behavior. Also, my solution seems a bit messy -- surely there is a cleaner way to do this!
Multiple transactions
You can split your work in two transactions:
func (s *store) SampleFunc(ctx context.Context) error {
err := s.runInTransaction(ctx,func(tx *sqlx.Tx) error {
// Point A: Do some database work
})
if err != nil {
return err
}
return s.runInTransaction(ctx,func(tx *sqlx.Tx) error {
// Point B: Do some more database work, which may return an error
})
}
I had the problem alike: I had a lots of steps in one transaction.
After starting transaction:
BEGIN
In loop:
SAVEPOINT s1
Some actions ....
If I get an error: ROLLBACK TO SAVEPOINT s1
If OK go to next step
Finally COMMIT
This approach gives me ability to perform all steps one-by-one. If some steps got failed I can throw away only them, keeping others. And finally commit all "good" work.
I am really new to Golang and I have a question regarding to testing.
I had a test where I wanted to check whether the persisting of a customer in elasticsearch works or not. I've reduced the code to the critical part and posted it on github: (https://github.com/fvosberg/elastic-go-testing)
The problem is, that I have to wait for elasticsearch to index the new document, before I can search for it. Is there another option than waiting a second for this to happen? This feels very ugly, but I don't know how I can test the integration (working with elasticsearch with lowercasing the email address ...) in another way.
Are there solutions for this problem?
package main
import (
"github.com/fvosberg/elastic-go-testing/customer"
"testing"
"time"
)
func TestRegistration(t *testing.T) {
testCustomer := customer.Customer{Email: "testing#test.de"}
testCustomer.Create()
time.Sleep(time.Second * 1)
_, err := customer.FindByEmail("testing#test.de")
if err != nil {
t.Logf("Error occured: %+v\n", err)
t.Fail()
} else {
t.Log("Found customer testing#test.de")
}
}
Elasticsearch has a flush command that is useful for this situation. Since you're using the elastic project as an interface, you can use the following (where client is your ES client):
...
testCustomer.Create()
res, err := client.Flush().Do()
if err != nil {
t.Fatal(err)
}
_, err := customer.FindByEmail("testing#test.de")
...
Disclaimer: I wish you a merry XMas and I hope my question does not disturb you!
sample.go:
package main
import(
"fmt"
"os"
)
type sample struct {
value int64
}
func (s sample) useful() {
if s.value == 0 {
fmt.Println("Error: something is wrong!")
os.Exit(1)
} else {
fmt.Println("May the force be with you!")
}
}
func main() {
s := sample{42}
s.useful()
s.value = 0
s.useful()
}
// output:
// May the force be with you!
// Error: something is wrong!
// exit status 1
I did a lot of research on how to use interfaces in golang testing. But so far I was not able to wrap my head around this completely. At least I can not see how interfaces help me when I need to "mock" (apologies for using this word) golang std. library packages like "fmt".
I came up with two scenarios:
use os/exec to test the command line interface
wrap fmt package so I have control and am able to check the output strings
I do not like both scenarios:
I experience going through the actual command line a convoluted and not-performant (see below). Might have portability issues, too.
I believe this is the way to go but I fear that wrapping the fmt package might be a lot of work (at least wrapping the time package for testing turned out a non-trivial task (https://github.com/finklabs/ttime)).
Actual Question here: Is there another (better/simpler/idiomatic) way?
Note: I want to do this in pure golang, I am not interested in the next testing framework.
cli_test.go:
package main
import(
"os/exec"
"testing"
)
func TestCli(t *testing.T) {
out, err := exec.Command("go run sample.go").Output()
if err != nil {
t.Fatal(err)
}
if string(out) != "May the force be with you!\nError: this is broken and not useful!\nexit status 1" {
t.Fatal("There is something wrong with the CLI")
}
}
Chapter 11 of Kerningham's Book gives a good solution to this question.
The trick is to change the calls to fmt.Printline() to calls to
fmt.Fprint(out, ...) where out is initialised to os.Stdout
This can be overwritten in the test harness to new(bytes.Buffer) allowing the
test to capture the output.
See https://github.com/adonovan/gopl.io/blob/master/ch11/echo/echo.go and
https://github.com/adonovan/gopl.io/blob/master/ch11/echo/echo_test.go
edited by OP...
sample.go:
package main
import(
"fmt"
"os"
"io"
)
var out io.Writer = os.Stdout // modified during testing
var exit func(code int) = os.Exit
type sample struct {
value int64
}
func (s sample) useful() {
if s.value == 0 {
fmt.Fprint(out, "Error: something is wrong!\n")
exit(1)
} else {
fmt.Fprint(out, "May the force be with you!\n")
}
}
func main() {
s := sample{42}
s.useful()
s.value = 0
s.useful()
}
// output:
// May the force be with you!
// Error: this is broken and not useful!
// exit status 1
cli_test.go:
package main
import(
"bytes"
"testing"
)
func TestUsefulPositive(t *testing.T) {
bak := out
out = new(bytes.Buffer)
defer func() { out = bak }()
s := sample{42}
s.useful()
if out.(*bytes.Buffer).String() != "May the force be with you!\n" {
t.Fatal("There is something wrong with the CLI")
}
}
func TestUsefulNegative(t *testing.T) {
bak := out
out = new(bytes.Buffer)
defer func() { out = bak }()
code := 0
osexit := exit
exit = func(c int) { code = c }
defer func() { exit = osexit }()
s := sample{0}
s.useful()
if out.(*bytes.Buffer).String() != "Error: something is wrong!\n" {
t.Fatal("There is something wrong with the CLI")
}
if code != 1 {
t.Fatal("Wrong exit code!")
}
}
Am I missing something here or are you talking of testable examples?
Basically, it works like this: In a *_test.go file, you need to adhere to the convention Example[[T][_M]] where T is a placeholder for the type and M a placeholder for the method you want to display the testable example as example code in the Godoc. If the function is just called Example(), the code will be shown as a package example.
Below the last line of the code of your example, you can put a comment like this
// Output:
// Foo
Now go test will make sure that the testable example function either exactly puts out everything below // Output: (including whitespace) or it will make the test fail.
Here is an actual example for an testable example
func ExampleMongoStore_Get() {
sessionId := "ExampleGetSession"
data, err := ms.Get(sessionId)
if err == sessionmw.ErrSessionNotFound {
fmt.Printf("Session '%s' not found\n", sessionId)
data = make(map[string]interface{})
data["foo"] = "bar"
ms.Save(sessionId, data)
}
loaded, _ := ms.Get(sessionId)
fmt.Printf("Loaded value '%s' for key '%s' in session '%s'",
loaded["foo"],
"foo", sessionId)
// Output:
// Session 'ExampleGetSession' not found
// Loaded value 'bar' for key 'foo' in session 'ExampleGetSession'
}
Edit: Have a look at the output of above example at godoc.org
After reading many tutorials, I found that there are many ways to bind arguments on prepared statement in Go, some of them
SELECT * FROM bla WHERE x = ?col1 AND y = ?col2
SELECT * FROM bla WHERE x = ? AND y = ?
SELECT * FROM bla WHERE x = :col1 AND y = :col2
SELECT * FROM bla WHERE x = $1 AND y = $2
First question, what is the cross-database way to bind arguments? (that works on any database)
Second question, none of the tutorial I've read mention about LIKE statement, how to bind arguments for LIKE-statement correctly?
SELECT * FROM bla WHERE x LIKE /*WHAT?*/
Third question, also none of them give an example for IN statement, how to bind arguments for IN statement correctly?
`SELECT * FROM bla WHERE x IN ( /*WHAT?*/ )
What is the cross-database way to bind arguments?
With database/sql, there is none. Each database has its own way to represent parameter placeholders. The Go database/sql package does not provide any normalization facility for the prepared statements. Prepared statement texts are just passed to the underlying driver, and the driver typically just sends them unmodified to the database server (or library for embedded databases).
How to bind arguments for LIKE-statement correctly?
You can use parameter placeholders after a like statement and bind it as a string. For instance, you could write a prepared statement as:
SELECT a from bla WHERE b LIKE ?
Here is an example (error management handling omitted).
package main
import (
"database/sql"
"fmt"
_ "github.com/go-sql-driver/mysql"
)
// > select * from bla ;
// +------+------+
// | a | b |
// +------+------+
// | toto | titi |
// | bobo | bibi |
// +------+------+
func main() {
// Open connection
db, err := sql.Open("mysql", "root:XXXXXXX#/test")
if err != nil {
panic(err.Error()) // proper error handling instead of panic in your app
}
defer db.Close()
// Prepare statement for reading data
stmtOut, err := db.Prepare("SELECT a FROM bla WHERE b LIKE ?")
if err != nil {
panic(err.Error()) // proper error handling instead of panic in your app
}
defer stmtOut.Close()
var a string
b := "bi%" // LIKE 'bi%'
err = stmtOut.QueryRow(b).Scan(&a)
if err != nil {
panic(err.Error()) // proper error handling instead of panic in your app
}
fmt.Printf("a = %s\n", a)
}
Note that the % character is part of the bound string, not of the query text.
How to bind arguments for IN statement correctly?
None of the databases I know allows binding a list of parameters directly with a IN clause. This is not a limitation of database/sql or the drivers, but this is simply not supported by most database servers.
You have several ways to work the problem around:
you can build a query with a fixed number of placeholders in the IN clause. Only bind the parameters you are provided with, and complete the other placeholders by the NULL value. If you have more values than the fixed number you have chosen, just execute the query several times. This is not extremely elegant, but it can be effective.
you can build multiple queries with various number of placeholders. One query for IN ( ? ), a second query for IN (?, ?), a third for IN (?,?,?), etc ... Keep those prepared queries in a statement cache, and choose the right one at runtime depending on the number of input parameters. Note that it takes memory, and generally the maximum number of prepared statements is limited, so it cannot be used when the number of parameters is high.
if the number of input parameters is high, insert them in a temporary table, and replace the query with the IN clause by a join with the temporary table. It is effective if you manage to perform the insertion in the temporary table in one roundtrip. With Go and database/sql, it is not convenient because there is no way to batch queries.
Each of these solutions has drawbacks. None of them is perfect.
I'm a newbie to Go but just to answer the first part:
First question, what is the cross-database way to bind arguments? (that works on any database)
If you use sqlx, which is a superset of the built-in sql package, then you should be able to use sqlx.DB.Rebind to achieve that.
I had this same question, and after reading the answers started to look for other solution on how to bind arguments for the IN statement.
Here is an example of what I did, not the most elegant solution, but works for me.
What I did was to create a select query with the parameters statically set on the query, and not using the bind feature at all.
It could be a good idea to sanitize the string that comes from the Marshal command, to be sure and safe, but I don't need it now.
package main
import (
"database/sql"
"encoding/json"
"fmt"
"log"
_ "github.com/go-sql-driver/mysql"
)
type Result struct {
Identifier string
Enabled bool
}
func main() {
// Open connection
db, err := sql.Open("mysql", "username:password#tcp(server-host)/my-database")
if err != nil {
panic(err.Error()) // proper error handling instead of panic in your app
}
defer db.Close()
// this is an example of a variable list of IDs
idList := []string{"ID1", "ID2", "ID3", "ID4", "ID5", "IDx"}
// convert the list to a JSON string
formatted, _ := json.Marshal(idList)
// a JSON array starts and ends with '[]' respectivelly, so we replace them with '()'
formatted[0] = '('
formatted[len(formatted)-1] = ')'
// create a static select query
query := fmt.Sprintf("SELECT identifier, is_enabled FROM some_table WHERE identifier in %s", string(formatted))
// prepare que query
rows, err := db.Query(query)
if err != nil {
panic(err.Error()) // proper error handling instead of panic in your app
}
defer rows.Close()
var result []Result
// fetch rows
for rows.Next() {
var r0 Result
if err := rows.Scan(&r0.Identifier, &r0.Enabled); err != nil {
log.Fatal(err)
}
// append the row to the result
result = append(result, r0)
}
if err := rows.Err(); err != nil {
log.Fatal(err)
}
fmt.Printf("result = %v\n", result)
}