Given that db is of type *sql.DB (uging lib/pq driver), the following code causes connection leak:
rows, err := db.Query(
"select 1 from things where id = $1",
thing,
)
if err != nil {
return nil, fmt.Errorf("can't select thing (%d): %w", thing, err)
}
found := false
for rows.Next() {
found = true
break
}
Calling this code repeatedly increases the number of open connections, until exhausted:
select sum(numbackends) from pg_stat_database;
// 5
// 6
// 7
// ...
// 80
How do I fix it?
There are a couple problems with your code as written. The direct answer to your question of avoiding connection leaks is to close the rows iterator as mentioned in the documentation. The normal way to call it is in a defer statement:
rows, err := db.Query(
"select 1 from things where id = $1",
thing,
)
if err != nil {
return nil, fmt.Errorf("can't select thing (%d): %w", thing, err)
}
defer rows.Close()
found := false
for rows.Next() {
found = true
break
}
Second, since all you ever care about is a single result, there's no reason to fetch a multi-row result set at all, which will implicitly solve the connection leak issue, as well. See this post for a discussion on the quickest way to check for existence in Postgres. If we adapt that here:
row, err := db.QueryRow(
"select EXISTS(SELECT 1 from things where id = $1)",
thing,
)
if err != nil {
return nil, fmt.Errorf("can't select thing (%d): %w", thing, err)
}
var found bool
if err := row.Scan(&found); err != nil {
return nil, fmt.Errorf("Failed to scan result: %w", err)
}
Related
I have mini project using Golang, my plan is make a base function which it will be called from Model to execute sql query, then return the rows result without Scan it first. I'm using this way to prevent forget write defer rows.Close() and the code for execute the Query in model more simple. I had tried this way, but when try to print the result, I got nil without any error. here my screenshoot. The result exists when the query executed and the rows result scanned are in same function. Maybe I miss something? This is my first question, sorry it's too long. Thank you
The base model where the SQL query will be executed and return the result
package model
import "database/sql"
import "hb-backend-v1/config/database"
import "fmt"
func Query(query string) (*sql.Rows, error){
connect, err := database.Connect()
if err != nil{
fmt.Println("Connection Failed")
return nil, err
}
fmt.Println("Connection Success")
defer connect.Close()
rows, err := connect.Query(query)
defer rows.Close()
if err != nil{
return nil, err
}
return rows, nil
}
This is where the base model will be called and give the result
package product
import "database/sql"
import _"fmt"
import "hb-backend=v1/model"
type Hasil struct{
Id_alamat_store int
Id_tk int
Alamat string
Id_wil int
Latitude sql.NullString
Longitude sql.NullString
}
func ProductList() ([]Hasil, error){
rows, err := model.Query("SELECT * FROM alamat_store")
if err != nil{
return nil, err
}
var result []Hasil
for rows.Next(){
var each = Hasil{}
var err = rows.Scan(&each.Id_alamat_store, &each.Id_tk, &each.Alamat, &each.Id_wil, &each.Latitude, &each.Longitude)
if err != nil{
return nil, err
}
result = append(result, each)
}
return result, nil
}
Both connection and rows will be closed once Query exits, after those two are closed you can't use rows anymore.
One approach to get around that would be to pass a closure to Query and have Query execute it before closing the two resources:
func Query(query string, scan func(*sql.Rows) error) error {
connect, err := database.Connect()
if err != nil{
return err
}
defer connect.Close()
rows, err := connect.Query(query)
if err != nil{
return err
}
defer rows.Close()
return scan(rows)
}
func ProductList() ([]Hasil, error) {
var result []Hasil
err := model.Query("SELECT * FROM alamat_store", func(rows *sql.Rows) error {
for rows.Next() {
var each = Hasil{}
var err = rows.Scan(&each.Id_alamat_store, &each.Id_tk, &each.Alamat, &each.Id_wil, &each.Latitude, &each.Longitude)
if err != nil {
return err
}
result = append(result, each)
}
return nil
})
if err != nil {
return nil, err
}
return result, nil
}
I have a function that I used to iterate over a result set from a query:
func readRows(rows *sql.Rows, translator func(*sql.Rows) error) error {
defer rows.Close()
// Iterate over each row in the rows and scan each; if an error occurs then return
for shouldScan := rows.Next(); shouldScan; {
if err := translator(rows); err != nil {
return err
}
}
// Check if the rows had an error; if they did then return them. Otherwise,
// close the rows and return an error if the close function fails
if err := rows.Err(); err != nil {
return err
}
return nil
}
The translator function is primarily responsible for calling Scan on the *sql.Rows object. An example of this is:
readRows(rows, func(scanner *sql.Rows) error {
var entry gopb.TestObject
// Embed the variables into a list that we can use to pull information out of the rows
scanned := []interface{}{...}
if err := scanner.Scan(scanned...); err != nil {
return err
}
entries = append(entries, &entry)
return nil
})
I wrote a unit test for this code:
// Create the SQL mock and the RDS reqeuster
db, mock, _ := sqlmock.New()
requester := Requester{conn: db}
defer db.Close()
// Create the rows we'll use for testing the query
rows := sqlmock.NewRows([]string{"id", "data"}).
AddRow(0, "data")
// Verify the command order for the transaction
mock.ExpectBegin()
mock.ExpectQuery(regexp.QuoteMeta("SELECT `id`, `data`, FROM `data`")).WillReturnRows(rows)
mock.ExpectRollback()
// Attempt to get the data
data, err := requester.GetData(context.TODO())
However, it appears that Next is being called infinitely. I'm not sure if this is an sqlmock issue or an issue with my code. Any help would be appreciated.
In my Go application I use crontab package to run Tracker function every minute. As you can notice from the code I call PostgreSQL function. To interact with the PostgreSQL database, I use the gorm package. Application worked several days without any problem but now I notice an error in logs: pq: sorry, too many clients already. I know that same questions was asked several times in StackOverflow before. For example in this post people advice to use Exec or Scan methods. In my case as you can see I use Exec method but anyway I have error. As far as I understand, each database request makes a separate connection and does not close it. I can't figure out what I'm doing wrong.
main.go:
package main
import (
"github.com/mileusna/crontab"
)
func main() {
database.ConnectPostgreSQL()
defer database.DisconnectPostgreSQL()
err = crontab.New().AddJob("* * * * *", controllers.Tracker); if err != nil {
utils.Logger().Fatal(err)
return
}
}
tracker.go:
package controllers
import (
"questionnaire/database"
"time"
)
var Tracker = func() {
err := database.DBGORM.Exec("CALL tracker($1)", time.Now().Format("2006-01-02 15:04:05")).Error; if err != nil {
utils.Logger().Println(err) // ERROR: pq: sorry, too many clients already
return
}
}
PostgreSQL.go:
package database
import (
"fmt"
"github.com/jinzhu/gorm"
_ "github.com/jinzhu/gorm/dialects/postgres"
"github.com/joho/godotenv"
"questionnaire/utils"
)
var DBGORM *gorm.DB
func ConnectPostgreSQL() {
err := godotenv.Load(".env")
if err != nil {
utils.Logger().Println(err)
panic(err)
}
databaseUser := utils.CheckEnvironmentVariable("PostgreSQL_USER")
databasePassword := utils.CheckEnvironmentVariable("PostgreSQL_PASSWORD")
databaseHost := utils.CheckEnvironmentVariable("PostgreSQL_HOST")
databaseName := utils.CheckEnvironmentVariable("PostgreSQL_DATABASE_NAME")
databaseURL:= fmt.Sprintf("host=%s user=%s dbname=%s password=%s sslmode=disable", databaseHost, databaseUser, databaseName, databasePassword)
DBGORM, err = gorm.Open("postgres", databaseURL)
if err != nil {
utils.Logger().Println(err)
panic(err)
}
err = DBGORM.DB().Ping()
if err != nil {
utils.Logger().Println(err)
panic(err)
}
DBGORM.LogMode(true)
}
func DisconnectPostgreSQL() error {
return DBGORM.Close()
}
In code which tries to be database agnostic, I would like to perform some database specific queries, so I need to know name of Database Driver in Go language:
db,err := sql.Open(dbstr, dbconnstr)
if err != nil {
log.Fatal(err)
}
errp := db.Ping()
if errp != nil {
log.Fatal(errp)
}
log.Printf("%s\n", db.Driver())
How I can determine name of database driver I'm using?
Give your database string in url format like postgres://postgres#localhost:5432/db_name?sslmode=disable.
And then find the database type you are using Parse function of url package. Based on the database type, run db specific queries.
func New(url string) (Driver, error) {
u, err := neturl.Parse(url)
if err != nil {
return nil, err
}
switch u.Scheme {
case "postgres":
d := &postgres.Driver{}
if err := d.Initialize(url); err != nil {
return nil, err
}
return d, nil
case "mysql":
d := &mysql.Driver{}
if err := d.Initialize(url); err != nil {
return nil, err
}
return d, nil
case "bash":
d := &bash.Driver{}
if err := d.Initialize(url); err != nil {
return nil, err
}
return d, nil
case "cassandra":
d := &cassandra.Driver{}
if err := d.Initialize(url); err != nil {
return nil, err
}
return d, nil
case "sqlite3":
d := &sqlite3.Driver{}
if err := d.Initialize(url); err != nil {
return nil, err
}
return d, nil
default:
return nil, errors.New(fmt.Sprintf("Driver '%s' not found.", u.Scheme))
}
}
You should already know the name of the database driver because its represented by the parameter you identified with the dbstr variable.
db, err := sql.Open("postgres", "user= ... ")
if err != nil {
log.Fatal(err)
}
db.Driver() correctly returns the underlying driver in use, but you are formatting it as string (because of %s). If you change %s with %T you will see that it correctly prints out the type:
log.Printf("%T\n", db.Driver())
For example, if you use github.com/lib/pq, the output is *pq.drv. This is the same of using the reflect package:
log.Printf("%s\n", reflect.TypeOf(db.Driver()))
It may be impractical to use that value for performing conditional executions. Moreover, the Driver interface doesn't specify any way to get the specific driver information, except the Open() function.
If you have specific needs, you may want to either use the driver name passed when you open the connection, or create specific drivers that delegate to the original ones and handle your custom logic.
Using the database/sql and driver packages and Tx, it is not possible it appears to detect whether a transaction has been committed or rolled-back without attempting another and receiving an error as a result, and then examining the error to determine the type of error. I would like to be able to determine from the Tx object whether committed or not. Sure, I can define and set another variable in the function that uses Tx, but I have quite a number of them, and it is times 2 every time (variable and assignment). I also have a deferred function to do a Rollback if needed, and it needs to be passed the bool variable.
Would it be acceptable to set the Tx variable to nil after a Commit or Rollback, and will the GC recover any memory, or is that a no-no, or is there a better alternative?
You want to make sure that Begin(), Commit(), and Rollback() appear within the same function. It makes transactions easier to track, and lets you ensure they are closed properly by using a defer.
Here is an example of this, which does a Commit or Rollback depending on whether an error is returned:
func (s Service) DoSomething() (err error) {
tx, err := s.db.Begin()
if err != nil {
return
}
defer func() {
if err != nil {
tx.Rollback()
return
}
err = tx.Commit()
}()
if _, err = tx.Exec(...); err != nil {
return
}
if _, err = tx.Exec(...); err != nil {
return
}
// ...
return
}
This can get a bit repetitive. Another way of doing this is by wrapping your transactions using a transaction handler:
func Transact(db *sql.DB, txFunc func(*sql.Tx) error) (err error) {
tx, err := db.Begin()
if err != nil {
return
}
defer func() {
if p := recover(); p != nil {
tx.Rollback()
panic(p) // re-throw panic after Rollback
} else if err != nil {
tx.Rollback() // err is non-nil; don't change it
} else {
err = tx.Commit() // err is nil; if Commit returns error update err
}
}()
err = txFunc(tx)
return err
}
Using the transaction hander above, I can do this:
func (s Service) DoSomething() error {
return Transact(s.db, func (tx *sql.Tx) error {
if _, err := tx.Exec(...); err != nil {
return err
}
if _, err := tx.Exec(...); err != nil {
return err
}
return nil
})
}
This keeps my transactions succinct and ensures by transactions are properly handled.
In my transaction handler I use recover() to catch panics to ensure a Rollback happens right away. I re-throw the panic to allow my code to catch it if a panic is expected. Under normal circumstances a panic should not occur. Errors should be returned instead.
If we did not handle panics the transaction would be rolled back eventually. A non-commited transaction gets rolled back by the database when the client disconnects or when the transaction gets garbage collected. However, waiting for the transaction to resolve on its own could cause other (undefined) issues. So it's better to resolve it as quickly as possible.
One thing that may not be immediately clear is that defer can change the return value within a closure if the return variable is captured. In the transaction handler the transaction is committed when err (the return value) is nil. The call to Commit can also return an error, so we set its return to err with err = tx.Commit(). We do not do the same with Rollback because err is non-nil and we do not want to overwrite the existing error.