What's the best way to delete all database records used for a Test suite? - sql

I have a test suite that pollute my database using a seed read from a YAML file.
I'm wondering is there a way to clean my database (delete all records used for the test suite) after running my tests.
// Open db and returns pointer and closer func
func prepareMySQLDB(t *testing.T) (db *sql.DB, closer func() error) {
db, err := sql.Open("mysql", "user:pass#/database")
if err != nil {
t.Fatalf("open mysql connection: %s", err)
}
return db, db.Close
}
// Pollute my database
func polluteDb(db *sql.DB, t *testing.T) {
seed, err := os.Open("seed.yml")
if err != nil {
t.Fatalf("failed to open seed file: %s", err)
}
defer seed.Close()
p := polluter.New(polluter.MySQLEngine(db))
if err := p.Pollute(seed); err != nil {
t.Fatalf("failed to pollute: %s", err)
}
}
func TestAllUsers(t *testing.T) {
t.Parallel()
db, closeDb := prepareMySQLDB(t)
defer closeDb()
polluteDb(db, t)
users, err := AllUsersD(db)
if err != nil {
t.Fatal("AllUsers() failed")
}
got := users[0].Email
if got != "myemail#gmail.com" {
t.Errorf("AllUsers().Email = %s; want myemail#gmail.com", got)
}
got1 := len(users)
if got1 != 1 {
t.Errorf("len(AllUsers()) = %d; want 1", got1)
}
}
// Test I'm interested in
func TestAddUser(t *testing.T) {
t.Parallel()
db, closeDb := prepareMySQLDB(t)
defer closeDb()
polluteDb(db, t)
user, err := AddUser(...)
if err != nil {
t.Fatal("AddUser() failed")
}
//how can I clean my database after this?
}
Should I retrieve the last ID inserted in TestAddUser() and just delete that line manually or there's any other way to save my database state and retrieve it after?
As I said I'm new to Go so any other comments on my code or what so ever are strongly appreciated.

The best way is usually to use a transaction, then ROLLBACK, so they are never committed in the first place.
The github.com/DATA-DOG/go-txdb package can help a lot with that.
Final code:
import (
"database/sql"
"os"
"testing"
txdb "github.com/DATA-DOG/go-txdb"
"github.com/romanyx/polluter"
)
//mostly sql tests
func init() {
txdb.Register("txdb", "mysql", "root:root#/betell_rest")
}
func TestAddUser(t *testing.T) {
db, err := sql.Open("txdb", "root:root#/betell_rest")
if err != nil {
t.Fatal(err)
}
defer db.Close()
users, _ := AllUsers(db)
userscount := len(users)
err = AddUser(db, "bla#gmail.com", "pass")
if err != nil {
t.Fatal("AddUser() failed")
}
users, _ = AllUsers(db)
if (userscount + 1) != len(users) {
t.Fatal("AddUser() failed to write in database")
}
}
Note: Also you can pass db into your polluter so you don't affect your database at all.

Related

VS Code how to format a few lines of SQL

I have a main.go file with a sql query saved as a string.
defer db.Close()
id := 1
// format this query
row := db.QueryRow("SELECT f_name, l_name from users WHERE id = ?", id)
var fname, lname string
err = row.Scan(&fname, &lname)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%v %v\n", fname, lname)
Is there a way to format that line?
I have also had the same problem. For me, the best solution is to move the SQL query to an embed file and format it there. This also has the advantage of making more complex or larger SQL queries more readable.
For *.sql files can than be a plugin used. For example Language PL/SQL.
Here is an example:
Excerpt of a repository (sqlite3,postgres,...)
package repository
import (
"context"
"database/sql"
_ "embed"
)
//go:embed dml/selectUsers.sql
var selectUsers string
type Repository struct{
sqlDB *sql.DB
}
func (r *Repository) GetUsers()([]*User,error){
tx, err := s.sqlDb.BeginTx(ctx, &sql.TxOptions{ReadOnly: false})
if err != nil {
return nil, fmt.Errorf("failed to begin new transaction: %w", err)
}
defer tx.Close()
selectedUsers, err := r.selectUsers(tx, selectUsers)
if err != nil {
return nil, fmt.Errorf("failed to select users: %w", err)
}
err = tx.Commit()
if err != nil {
return nil, fmt.Errorf("failed to commit transaction: %w", err)
}
return selectedUsers, nil
}
func (r *Repository) selectUsers(tx *sql.Tx, query string, args ...interface{}) ([]*User,error) {
stmt, err := tx.Prepare(insertCompany)
if err != nil {
return nil, fmt.Errorf("failed to prepare statement: %v", err)
}
defer stmt.Close()
rows, err := stmt.Query(args...)
if err != nil {
return nil, fmt.Errorf("failed to query statement: %v", err)
}
selectedUsers := make([]*User, 0)
for rows.Next() {
selectedUser := new(User)
err = rows.Scan(
&selectedUser.ID,
&selectedUser.Surename,
&selectedUser.Familyname,
)
if err != nil {
return nil, fmt.Errorf("failed to scan row: %v", err)
}
selectedUsers = append(selectedUsers, selectedUser)
}
return selectedUsers, nil
}
dml/selectUsers.sql
SELECT
id,
surename,
familyname
FROM
users;

How to optimize database connections

In my Go application I use crontab package to run Tracker function every minute. As you can notice from the code I call PostgreSQL function. To interact with the PostgreSQL database, I use the gorm package. Application worked several days without any problem but now I notice an error in logs: pq: sorry, too many clients already. I know that same questions was asked several times in StackOverflow before. For example in this post people advice to use Exec or Scan methods. In my case as you can see I use Exec method but anyway I have error. As far as I understand, each database request makes a separate connection and does not close it. I can't figure out what I'm doing wrong.
main.go:
package main
import (
"github.com/mileusna/crontab"
)
func main() {
database.ConnectPostgreSQL()
defer database.DisconnectPostgreSQL()
err = crontab.New().AddJob("* * * * *", controllers.Tracker); if err != nil {
utils.Logger().Fatal(err)
return
}
}
tracker.go:
package controllers
import (
"questionnaire/database"
"time"
)
var Tracker = func() {
err := database.DBGORM.Exec("CALL tracker($1)", time.Now().Format("2006-01-02 15:04:05")).Error; if err != nil {
utils.Logger().Println(err) // ERROR: pq: sorry, too many clients already
return
}
}
PostgreSQL.go:
package database
import (
"fmt"
"github.com/jinzhu/gorm"
_ "github.com/jinzhu/gorm/dialects/postgres"
"github.com/joho/godotenv"
"questionnaire/utils"
)
var DBGORM *gorm.DB
func ConnectPostgreSQL() {
err := godotenv.Load(".env")
if err != nil {
utils.Logger().Println(err)
panic(err)
}
databaseUser := utils.CheckEnvironmentVariable("PostgreSQL_USER")
databasePassword := utils.CheckEnvironmentVariable("PostgreSQL_PASSWORD")
databaseHost := utils.CheckEnvironmentVariable("PostgreSQL_HOST")
databaseName := utils.CheckEnvironmentVariable("PostgreSQL_DATABASE_NAME")
databaseURL:= fmt.Sprintf("host=%s user=%s dbname=%s password=%s sslmode=disable", databaseHost, databaseUser, databaseName, databasePassword)
DBGORM, err = gorm.Open("postgres", databaseURL)
if err != nil {
utils.Logger().Println(err)
panic(err)
}
err = DBGORM.DB().Ping()
if err != nil {
utils.Logger().Println(err)
panic(err)
}
DBGORM.LogMode(true)
}
func DisconnectPostgreSQL() error {
return DBGORM.Close()
}

golang sql pointer values keep repeating itself

Below is a golang function which is being called with an input channel
func getOptions(inChannel <-chan Param) <-chan ParamOptions {
paramOptions := make(chan ParamOptions )
go func() {
defer close(paramOptions )
var wg sync.WaitGroup
conn, err := sql.Open("mssql", wellConnStr)
if err != nil {
log.Fatal("open connection failed:", err.Error())
}
defer conn.Close()
getParamOptions := func(db *sql.DB, param *Param) {
defer wg.Done()
fmt.Println("querying options for ", param.Code, param.Name)
rows, err := db.Query(`select *
from dbo.ParamOptions where code=? and name=?`, &param.Code, &param.Name)
fmt.Println("results for ", param.Name, param.Code)
if err != nil {
log.Fatal("query failed:", err.Error())
}
defer rows.Close()
found := false
...
paramOptions <- ParamOptions...
break
}
if found == false {
fmt.Println("did not find options for ", param.Code, param.Name)
}
}
for paramInChannel := range paramChannel {
wg.Add(1)
fmt.Println("retrieving inputs for ", paramInChannel.Code, paramInChannel.Name)
**go** getParamOptions(conn, &wellInChannel)
}
wg.Wait()
}()
return paramOptions
}
If i remove the go keyword before calling the function getParamOptions it works without any problems. However if I use go then the last code and name keeps repeating within the the getParamOptions function, even though the options retrieved seems to be of the correct Param, the Code and Name values are being repeated

Go ioutil using too many file descriptors/leak?

I am going through a list of files and Unmarshalling the xml data in them into an array of structs rArray. I intend to process about 18000 files. When I get to about 1300 files processed, the program panics and says that too many files are open. If I limit the amount of files processed to a safe amount of 1000, the program does not crash.
As seen below, I am using ioutil.ReadFile to read the file data.
for _, f := range files {
func() {
data, err := ioutil.ReadFile("./" + recordDir + "/" + f.Name())
if err != nil {
fmt.Println("error reading %v", err)
return
} else {
if (strings.Contains(filepath.Ext(f.Name()), "xml")) {
//unmarshal data and put into struct array
err = xml.Unmarshal([]byte(data), &rArray[a])
if err != nil {
fmt.Println("error decoding %v: %v",f.Name(), err)
return
}
}
}
}()
}
I am not sure if Go is using too many file descriptors or not closing the files fast enough.
After reading https://groups.google.com/forum/#!topic/golang-nuts/7yXXjgcOikM and viewing the ioutil source in http://golang.org/src/pkg/io/ioutil/ioutil.go, the code for ioutil.ReadFile shows that it uses defer to close the file. defer runs when calling function is returned and ReadFile() is the calling function. Am I correct in this understanding?
I also tried wrapping the ioutil.ReadFile part of my code in a function, but it makes no difference.
My ulimit is set to unlimited.
UPDATE:
I believe that the error of too many files is actually occurring during my Unzip function.
func Unzip(src, dest string) error {
r, err := zip.OpenReader(src)
if err != nil {
return err
}
for _, f := range r.File {
rc, err := f.Open()
if err != nil {
panic(err)
}
path := filepath.Join(dest, f.Name)
if f.FileInfo().IsDir() {
os.MkdirAll(path, f.Mode())
} else {
f, err := os.OpenFile(
path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, f.Mode())
if err != nil {
panic(err)
}
_, err = io.Copy(f, rc)
if err != nil {
panic(err)
}
f.Close()
}
rc.Close()
}
r.Close()
return nil
}
I initially got the Unzip function from https://gist.github.com/hnaohiro/4572580, but upon further inspection, the use of defer in the gist author's function seemed wrong as the file would only be closed after the Unzip() function returned which is too late becuase then 18000 file descriptors will be open. ;)
I replaced the deferred Closes with explicit Close() as shown above, but am still getting the same "too many open files" error. Is there a problem with my modified Unzip function?
UPDATE # 2
Oops, I was running this on Heroku and was pushing to the wrong app with my changes this entire time. Lesson learned: verify target app in heroku toolbelt.
Unzip code from https://gist.github.com/hnaohiro/4572580 does not work as it does not close files until all files processed.
My unzip code with explicit close above works and so does the defer version in #peterSO's answer.
I would modify the Unzip function from https://gist.github.com/hnaohiro/4572580 to the following:
package main
import (
"archive/zip"
"io"
"log"
"os"
"path/filepath"
)
func unzipFile(f *zip.File, dest string) error {
rc, err := f.Open()
if err != nil {
return err
}
defer rc.Close()
path := filepath.Join(dest, f.Name)
if f.FileInfo().IsDir() {
err := os.MkdirAll(path, f.Mode())
if err != nil {
return err
}
} else {
f, err := os.OpenFile(
path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, f.Mode())
if err != nil {
return err
}
defer f.Close()
_, err = io.Copy(f, rc)
if err != nil {
return err
}
}
return nil
}
func Unzip(src, dest string) error {
r, err := zip.OpenReader(src)
if err != nil {
return err
}
defer r.Close()
for _, f := range r.File {
err := unzipFile(f, dest)
if err != nil {
return err
}
}
return nil
}
func main() {
err := Unzip("./sample.zip", "./out")
if err != nil {
log.Fatal(err)
}
}

GO lang : Communicate with shell process

I want to execute a shell script from Go.
The shell script takes standard input and echoes the result.
I want to supply this input from GO and use the result.
What I am doing is:
cmd := exec.Command("python","add.py")
in, _ := cmd.StdinPipe()
But how do I read from in?
Here is some code writing to a process, and reading from it:
package main
import (
"bufio"
"fmt"
"os/exec"
)
func main() {
// What we want to calculate
calcs := make([]string, 2)
calcs[0] = "3*3"
calcs[1] = "6+6"
// To store the results
results := make([]string, 2)
cmd := exec.Command("/usr/bin/bc")
in, err := cmd.StdinPipe()
if err != nil {
panic(err)
}
defer in.Close()
out, err := cmd.StdoutPipe()
if err != nil {
panic(err)
}
defer out.Close()
// We want to read line by line
bufOut := bufio.NewReader(out)
// Start the process
if err = cmd.Start(); err != nil {
panic(err)
}
// Write the operations to the process
for _, calc := range calcs {
_, err := in.Write([]byte(calc + "\n"))
if err != nil {
panic(err)
}
}
// Read the results from the process
for i := 0; i < len(results); i++ {
result, _, err := bufOut.ReadLine()
if err != nil {
panic(err)
}
results[i] = string(result)
}
// See what was calculated
for _, result := range results {
fmt.Println(result)
}
}
You might want to read/write from/to the process in different goroutines.