Context Timeout in GoLang - api

Implemented a context timeout in Go APIs using
context.WithTimeout(context,time)
Expectation is that if I set it to 10 seconds and if the API happens to run longer, it should stop and return Timeout error. However, if I add a sleep in between , the API still runs and gives 200 output. Is there any solution to this other than manually checking at random points for the time elapsed ? Looks like context.WithTimeout() doesnt solve its purpose.

So, this is a minimum code to exemplify how the context.WithTimeout can work:
server.go
package main
import (
"fmt"
"net/http"
"time"
)
func main() {
http.HandleFunc("/bad", badResponse)
http.HandleFunc("/nice", niceResponse)
http.ListenAndServe(":8099", nil)
}
// badResponse is a procedure that takes too much time
func badResponse(w http.ResponseWriter, r *http.Request) {
fmt.Println("got request, it may take too long to answer")
time.Sleep(3 * time.Second)
fmt.Fprintf(w, "bad response!")
}
// niceResponse is a procedure that respond in time
func niceResponse(w http.ResponseWriter, r *http.Request) {
fmt.Println("got request, will return fast")
fmt.Fprintf(w, "nice response!")
}
client.go
package main
import (
"context"
"io/ioutil"
"log"
"net/http"
"time"
)
func main () {
req, err := http.NewRequest(http.MethodGet, "http://localhost:8099/bad", nil)
if err != nil {
log.Fatal(err)
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Millisecond)
defer cancel()
req = req.WithContext(ctx)
c := &http.Client{}
res, err := c.Do(req)
if err != nil {
log.Fatal(err)
}
defer res.Body.Close()
out, err := ioutil.ReadAll(res.Body)
if err != nil {
log.Fatal(err)
}
log.Println(string(out))
}
In the example above the client must fail because the timeout is lower than the client server delay.

Related

Issue with go-sqlmock testing in the expected query part

I am using go-sqlmock for the first time and I am trying to write a test for post operation. I am using gorm and gin.
The test is giving me an error where s.mock.ExpectQuery(regexp.QuoteMeta(.... I am not what is the issue here. I have posted both the test and the output.
Also, (this has nothing to do with 1) in this test I really do not know what the code will be as it is randomly generated in the api controller. Is there a way to assign a generic number in the code field.
The test file
package unit
import (
"net/http"
"net/http/httptest"
"regexp"
"testing"
"github.com/DATA-DOG/go-sqlmock"
"github.com/SamiAlsubhi/go/controllers"
"github.com/SamiAlsubhi/go/routes"
"github.com/gin-gonic/gin"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/stretchr/testify/suite"
"gorm.io/driver/postgres"
"gorm.io/gorm"
)
type Suite struct {
suite.Suite
DB *gorm.DB
mock sqlmock.Sqlmock
router *gin.Engine
}
func (s *Suite) SetupSuite(t *testing.T) {
conn, mock, err := sqlmock.New()
if err != nil || conn == nil {
t.Errorf("Failed to open mock sql db, got error: %v", err)
}
s.mock = mock
dialector := postgres.New(postgres.Config{
DSN: "sqlmock_db_0",
DriverName: "postgres",
Conn: conn,
PreferSimpleProtocol: true,
})
if db, err := gorm.Open(dialector, &gorm.Config{}); err != nil || db == nil {
t.Errorf("Failed to open gorm v2 db, got error: %v", err)
} else {
s.DB = db
}
api := &controllers.API{Db: s.DB}
s.router = routes.SetupRouter(api)
}
func TestSetup(t *testing.T) {
suite.Run(t, new(Suite))
}
func (s *Suite) AfterTest(_, _ string) {
require.NoError(s.T(), s.mock.ExpectationsWereMet())
}
func (s *Suite) Test_GetOTP() {
var (
phone = "99999999"
code = "123456"
)
s.mock.ExpectQuery(regexp.QuoteMeta(
`INSERT INTO "otps" ("phone","code") VALUES ($1,$2) RETURNING "otps"."id"`)).
WithArgs(phone, code).
WillReturnRows(sqlmock.NewRows([]string{"id"}).
AddRow(1))
s.mock.ExpectCommit()
w := httptest.NewRecorder()
req, err := http.NewRequest("GET", "/api/auth/get-otp/"+phone, nil)
require.NoError(s.T(), err)
s.router.ServeHTTP(w, req)
assert.Equal(s.T(), 200, w.Code)
//require.Nil(s.T(), deep.Equal(&model.Person{ID: id, Name: name}, w.Body))
}
the output.
--- FAIL: TestSetup (0.00s)
--- FAIL: TestSetup/Test_GetOTP (0.00s)
/Users/sami/Desktop/SamiAlsubhi/go/test/unit/suite.go:63: test panicked: runtime error: invalid memory address or nil pointer dereference
goroutine 26 [running]:
runtime/debug.Stack()
/usr/local/go/src/runtime/debug/stack.go:24 +0x65
github.com/stretchr/testify/suite.failOnPanic(0xc000001a00)
/Users/sami/Desktop/golang/pkg/mod/github.com/stretchr/testify#v1.7.0/suite/suite.go:63 +0x3e
panic({0x49e96a0, 0x5193810})
/usr/local/go/src/runtime/panic.go:1038 +0x215
github.com/SamiAlsubhi/go/test/unit.(*Suite).AfterTest(0x4abe61b, {0x4becfd0, 0xc000468940}, {0x0, 0x0})
/Users/sami/Desktop/SamiAlsubhi/go/test/unit/setup_test.go:60 +0x1c
github.com/stretchr/testify/suite.Run.func1.1()
/Users/sami/Desktop/golang/pkg/mod/github.com/stretchr/testify#v1.7.0/suite/suite.go:137 +0x1b7
panic({0x49e96a0, 0x5193810})
/usr/local/go/src/runtime/panic.go:1038 +0x215
github.com/SamiAlsubhi/go/test/unit.(*Suite).Test_GetOTP(0xc000468940)
/Users/sami/Desktop/SamiAlsubhi/go/test/unit/setup_test.go:69 +0x4f
reflect.Value.call({0xc000049140, 0xc000010308, 0x13}, {0x4abf50c, 0x4}, {0xc000080e70, 0x1, 0x1})
/usr/local/go/src/reflect/value.go:543 +0x814
reflect.Value.Call({0xc000049140, 0xc000010308, 0xc000468940}, {0xc0003c9e70, 0x1, 0x1})
/usr/local/go/src/reflect/value.go:339 +0xc5
github.com/stretchr/testify/suite.Run.func1(0xc000001a00)
/Users/sami/Desktop/golang/pkg/mod/github.com/stretchr/testify#v1.7.0/suite/suite.go:158 +0x4b6
testing.tRunner(0xc000001a00, 0xc000162000)
/usr/local/go/src/testing/testing.go:1259 +0x102
created by testing.(*T).Run
/usr/local/go/src/testing/testing.go:1306 +0x35a
FAIL
coverage: [no statements]
FAIL github.com/SamiAlsubhi/go/test/unit 0.912s
FAIL
Solution to the first issue:
when using testify/suite, There are bunch of methods if created for the Suite struct, they will be automatically executed when running the test. That being said, These methods will pass through an interface filter. In the case of .SetupSuite, it has to have NO arguments and No return, in order to run.
Solution to the second issue:
There is a way in go-sqlmock to match any kind of data by using sqlmock.AnyArg().
Fixed code:
package unit
import (
"encoding/json"
"fmt"
"math/rand"
"net/http"
"net/http/httptest"
"regexp"
"testing"
"github.com/DATA-DOG/go-sqlmock"
"github.com/SamiAlsubhi/go/controllers"
"github.com/SamiAlsubhi/go/routes"
"github.com/gin-gonic/gin"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/stretchr/testify/suite"
"gorm.io/driver/postgres"
"gorm.io/gorm"
)
type Suite struct {
suite.Suite
DB *gorm.DB
mock sqlmock.Sqlmock
router *gin.Engine
}
func (s *Suite) SetupSuite() {
//t.Logf("setup start")
conn, mock, err := sqlmock.New(sqlmock.QueryMatcherOption(sqlmock.QueryMatcherRegexp))
if err != nil || conn == nil {
panic(fmt.Sprintf("Failed to open mock sql db, got error: %v", err))
}
s.mock = mock
dialector := postgres.New(postgres.Config{
DSN: "sqlmock_db_0",
DriverName: "postgres",
Conn: conn,
PreferSimpleProtocol: true,
})
if db, err := gorm.Open(dialector, &gorm.Config{SkipDefaultTransaction: true}); err != nil || db == nil {
panic(fmt.Sprintf("Failed to open gorm v2 db, got error: %v", err))
} else {
s.DB = db
}
api := &controllers.API{Db: s.DB, IsTesting: true}
s.router = routes.SetupRouter(api)
}
func TestSetup(t *testing.T) {
suite.Run(t, new(Suite))
}
// func (s *Suite) AfterTest(_, _ string) {
// require.NoError(s.T(), s.mock.ExpectationsWereMet())
// }
func (s *Suite) Test_GetOTP_Non_Existing_Phone() {
/*
This to test getting OTP for a phone number that does not exist in the otps table
*/
phone := fmt.Sprintf("%v", 90000000+rand.Intn(99999999-90000000))
s.mock.MatchExpectationsInOrder(false)
s.mock.ExpectQuery(regexp.QuoteMeta(`SELECT count(*) FROM "otps" WHERE phone = $1 AND "otps"."deleted_at" IS NULL`)).
WithArgs(phone).
WillReturnRows(sqlmock.NewRows([]string{"count"}).
AddRow(0))
s.mock.ExpectQuery(regexp.QuoteMeta(
`INSERT INTO "otps" ("created_at","updated_at","deleted_at","phone","code") VALUES ($1,$2,$3,$4,$5) RETURNING "id"`)).
WithArgs(sqlmock.AnyArg(), sqlmock.AnyArg(), sqlmock.AnyArg(), phone, sqlmock.AnyArg()).
WillReturnRows(sqlmock.NewRows([]string{"id"}).
AddRow(1))
w := httptest.NewRecorder()
req, err := http.NewRequest("GET", "/api/auth/get-otp/"+phone, nil)
require.NoError(s.T(), err)
s.router.ServeHTTP(w, req)
assert.Equal(s.T(), 200, w.Code)
//parse response
var response gin.H
err = json.Unmarshal(w.Body.Bytes(), &response)
require.NoError(s.T(), err)
_, ok := response["expiry_in"]
assert.True(s.T(), ok)
require.NoError(s.T(), s.mock.ExpectationsWereMet())
}

How to optimize database connections

In my Go application I use crontab package to run Tracker function every minute. As you can notice from the code I call PostgreSQL function. To interact with the PostgreSQL database, I use the gorm package. Application worked several days without any problem but now I notice an error in logs: pq: sorry, too many clients already. I know that same questions was asked several times in StackOverflow before. For example in this post people advice to use Exec or Scan methods. In my case as you can see I use Exec method but anyway I have error. As far as I understand, each database request makes a separate connection and does not close it. I can't figure out what I'm doing wrong.
main.go:
package main
import (
"github.com/mileusna/crontab"
)
func main() {
database.ConnectPostgreSQL()
defer database.DisconnectPostgreSQL()
err = crontab.New().AddJob("* * * * *", controllers.Tracker); if err != nil {
utils.Logger().Fatal(err)
return
}
}
tracker.go:
package controllers
import (
"questionnaire/database"
"time"
)
var Tracker = func() {
err := database.DBGORM.Exec("CALL tracker($1)", time.Now().Format("2006-01-02 15:04:05")).Error; if err != nil {
utils.Logger().Println(err) // ERROR: pq: sorry, too many clients already
return
}
}
PostgreSQL.go:
package database
import (
"fmt"
"github.com/jinzhu/gorm"
_ "github.com/jinzhu/gorm/dialects/postgres"
"github.com/joho/godotenv"
"questionnaire/utils"
)
var DBGORM *gorm.DB
func ConnectPostgreSQL() {
err := godotenv.Load(".env")
if err != nil {
utils.Logger().Println(err)
panic(err)
}
databaseUser := utils.CheckEnvironmentVariable("PostgreSQL_USER")
databasePassword := utils.CheckEnvironmentVariable("PostgreSQL_PASSWORD")
databaseHost := utils.CheckEnvironmentVariable("PostgreSQL_HOST")
databaseName := utils.CheckEnvironmentVariable("PostgreSQL_DATABASE_NAME")
databaseURL:= fmt.Sprintf("host=%s user=%s dbname=%s password=%s sslmode=disable", databaseHost, databaseUser, databaseName, databasePassword)
DBGORM, err = gorm.Open("postgres", databaseURL)
if err != nil {
utils.Logger().Println(err)
panic(err)
}
err = DBGORM.DB().Ping()
if err != nil {
utils.Logger().Println(err)
panic(err)
}
DBGORM.LogMode(true)
}
func DisconnectPostgreSQL() error {
return DBGORM.Close()
}

Golang test with channels does not exit

The following Golang test never exits. I suspect it has something to do with a channel deadlock but being a go-noob, I am not very certain.
const userName = "xxxxxxxxxxxx"
func TestSynchroninze(t *testing.T) {
c, err := channel.New(github.ChannelName, authToken)
if err != nil {
t.Fatalf("Could not create channel: %s", err)
return
}
state := channel.NewState(nil)
ctx := context.Background()
ctx = context.WithValue(ctx, "userId", userName)
user := api.User{}
output, errs := c.Synchronize(state, ctx)
if err = <-errs; err != nil {
t.Fatalf("Error performing synchronize: %s", err)
return
}
for o := range output {
switch oo := o.Data.(type) {
case api.User:
user = oo
glog.Infof("we have a USER %s\n", user)
default:
t.Errorf("Encountered unexpected data type: %T", oo)
}
}
}
Here are the methods being tested.
type github struct {
client *api.Client
}
func newImplementation(t auth.UserToken) implementation.Implementation {
return &github{client: api.NewClient(t)}
}
// -------------------------------------------------------------------------------------
const (
kLastUserFetch = "lastUserFetch"
)
type synchronizeFunc func(implementation.MutableState, chan *implementation.Output, context.Context) error
// -------------------------------------------------------------------------------------
func (g *github) Synchronize(state implementation.MutableState, ctx context.Context) (<-chan *implementation.Output, <-chan error) {
output := make(chan *implementation.Output)
errors := make(chan error, 1) // buffer allows preflight errors
// Close output channels once we're done
defer func() {
go func() {
// wg.Wait()
close(errors)
close(output)
}()
}()
err := g.fetchUser(state, output, ctx)
if err != nil {
errors <- err
}
return output, errors
}
func (g *github) fetchUser(state implementation.MutableState, output chan *implementation.Output, ctx context.Context) error {
var err error
var user = api.User{}
userId, _ := ctx.Value("userId").(string)
user, err = g.client.GetUser(userId, ctx.Done())
if err == nil {
glog.Info("No error in fetchUser")
output <- &implementation.Output{Data: user}
state.SetTime(kLastUserFetch, time.Now())
}
return err
}
func (c *Client) GetUser(id string, quit <-chan struct{}) (user User, err error) {
// Execute request
var data []byte
data, err = c.get("users/"+id, nil, quit)
glog.Infof("USER DATA %s", data)
// Parse response
if err == nil && len(data) > 0 {
err = json.Unmarshal(data, &user)
data, _ = json.Marshal(user)
}
return
}
Here is what I see in the console (most of the user details removed)
I1228 13:25:05.291010 21313 client.go:177] GET https://api.github.com/users/xxxxxxxx
I1228 13:25:06.010085 21313 client.go:36] USER DATA {"login":"xxxxxxxx","id":00000000,"avatar_url":"https://avatars.githubusercontent.com/u/0000000?v=3",...}
I1228 13:25:06.010357 21313 github.go:90] No error in fetchUser
==========EDIT=============
Here is the relevant portion of the api package.
package api
type Client struct {
authToken auth.UserToken
http *http.Client
}
func NewClient(authToken auth.UserToken) *Client {
return &Client{
authToken: authToken,
http: auth.NewClient(authToken),
}
}
// -------------------------------------------------------------------------------------
type User struct {
Id int `json:"id,omitempty"`
Username string `json:"login,omitempty"`
Email string `json:"email,omitempty"`
FullName string `json:"name,omitempty"`
ProfilePicture string `json:"avatar_url,omitempty"`
Bio string `json:"bio,omitempty"`
Website string `json:"blog,omitempty"`
Company string `json:"company,omitempty"`
}
And the channel package
package channel
type Channel struct {
implementation.Descriptor
imp implementation.Implementation
}
// New returns a channel implementation with a given name and auth token.
func New(name string, token auth.UserToken) (*Channel, error) {
if desc, ok := implementation.Lookup(name); ok {
if imp := implementation.New(name, token); imp != nil {
return &Channel{Descriptor: desc, imp: imp}, nil
}
}
return nil, ErrInvalidChannel
}
and the implementation package...
package implementation
import "golang.org/x/net/context"
// -------------------------------------------------------------------------------------
// Implementation is the interface implemented by subpackages.
type Implementation interface {
// Synchronize performs a synchronization using the given state. A context parameters
// is provided to provide cancellation as well as implementation-specific behaviors.
//
// If a fatal error occurs (see package error definitions), the state can be discarded
// to prevent the persistence of an invalid state.
Synchronize(state MutableState, ctx context.Context) (<-chan *Output, <-chan error)
// FetchDetails gets details for a given timeline item. Any changes to the TimelineItem
// (including the Meta value) will be persisted.
FetchDetails(item *TimelineItem, ctx context.Context) (interface{}, error)
}
======Edit #2=======
This is the original Synchronize method. I removed some details in my testing to try and simplify the problem. By removing a go func call, I believe I introduced a new problem which could be confusing things.
Here is the original Synchronize method. There are some things with Wait Groups and a function array containing a single function because this method will eventually be synchronizing multiple functions.
func (g *github) Synchronize(state implementation.MutableState, ctx context.Context) (<-chan *implementation.Output, <-chan error) {
wg := sync.WaitGroup{}
output := make(chan *implementation.Output)
errors := make(chan error, 1) // buffer allows preflight errors
// Close output channels once we're done
defer func() {
go func() {
wg.Wait()
close(errors)
close(output)
}()
}()
// Perform fetch functions in separate routines
funcs := []synchronizeFunc{
g.fetchUser,
}
for _, f := range funcs {
wg.Add(1)
go func(f synchronizeFunc) {
defer wg.Done()
if err := f(state, output, ctx); err != nil {
errors <- err
}
}(f)
}
glog.Info("after go sync...")
return output, errors
}
I think the two problems are in
output <- &implementation.Output{Data: user}
the channel does not have a buffer. It will block until some other goroutine reads from it. But in your code is the same goroutine so it will block.
and second:
// Close output channels once we're done
defer func() {
go func() {
// wg.Wait()
close(errors)
close(output)
}()
}()
you launch a go routine when the routine exits. The goroutine is scheduled, the function returns but it never calls the goroutine.
I would suggest to unify all that logic in one:
func (g *github) Synchronize(state implementation.MutableState, ctx context.Context) (<-chan *implementation.Output, <-chan error) {
output := make(chan *implementation.Output)
errors := make(chan error, 1) // buffer allows preflight errors
go func() {
defer close(output)
defer close(errors)
err := g.fetchUser(state, output, ctx)
if err != nil {
errors <- err
}
}()
return output, errors
}

Go ioutil using too many file descriptors/leak?

I am going through a list of files and Unmarshalling the xml data in them into an array of structs rArray. I intend to process about 18000 files. When I get to about 1300 files processed, the program panics and says that too many files are open. If I limit the amount of files processed to a safe amount of 1000, the program does not crash.
As seen below, I am using ioutil.ReadFile to read the file data.
for _, f := range files {
func() {
data, err := ioutil.ReadFile("./" + recordDir + "/" + f.Name())
if err != nil {
fmt.Println("error reading %v", err)
return
} else {
if (strings.Contains(filepath.Ext(f.Name()), "xml")) {
//unmarshal data and put into struct array
err = xml.Unmarshal([]byte(data), &rArray[a])
if err != nil {
fmt.Println("error decoding %v: %v",f.Name(), err)
return
}
}
}
}()
}
I am not sure if Go is using too many file descriptors or not closing the files fast enough.
After reading https://groups.google.com/forum/#!topic/golang-nuts/7yXXjgcOikM and viewing the ioutil source in http://golang.org/src/pkg/io/ioutil/ioutil.go, the code for ioutil.ReadFile shows that it uses defer to close the file. defer runs when calling function is returned and ReadFile() is the calling function. Am I correct in this understanding?
I also tried wrapping the ioutil.ReadFile part of my code in a function, but it makes no difference.
My ulimit is set to unlimited.
UPDATE:
I believe that the error of too many files is actually occurring during my Unzip function.
func Unzip(src, dest string) error {
r, err := zip.OpenReader(src)
if err != nil {
return err
}
for _, f := range r.File {
rc, err := f.Open()
if err != nil {
panic(err)
}
path := filepath.Join(dest, f.Name)
if f.FileInfo().IsDir() {
os.MkdirAll(path, f.Mode())
} else {
f, err := os.OpenFile(
path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, f.Mode())
if err != nil {
panic(err)
}
_, err = io.Copy(f, rc)
if err != nil {
panic(err)
}
f.Close()
}
rc.Close()
}
r.Close()
return nil
}
I initially got the Unzip function from https://gist.github.com/hnaohiro/4572580, but upon further inspection, the use of defer in the gist author's function seemed wrong as the file would only be closed after the Unzip() function returned which is too late becuase then 18000 file descriptors will be open. ;)
I replaced the deferred Closes with explicit Close() as shown above, but am still getting the same "too many open files" error. Is there a problem with my modified Unzip function?
UPDATE # 2
Oops, I was running this on Heroku and was pushing to the wrong app with my changes this entire time. Lesson learned: verify target app in heroku toolbelt.
Unzip code from https://gist.github.com/hnaohiro/4572580 does not work as it does not close files until all files processed.
My unzip code with explicit close above works and so does the defer version in #peterSO's answer.
I would modify the Unzip function from https://gist.github.com/hnaohiro/4572580 to the following:
package main
import (
"archive/zip"
"io"
"log"
"os"
"path/filepath"
)
func unzipFile(f *zip.File, dest string) error {
rc, err := f.Open()
if err != nil {
return err
}
defer rc.Close()
path := filepath.Join(dest, f.Name)
if f.FileInfo().IsDir() {
err := os.MkdirAll(path, f.Mode())
if err != nil {
return err
}
} else {
f, err := os.OpenFile(
path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, f.Mode())
if err != nil {
return err
}
defer f.Close()
_, err = io.Copy(f, rc)
if err != nil {
return err
}
}
return nil
}
func Unzip(src, dest string) error {
r, err := zip.OpenReader(src)
if err != nil {
return err
}
defer r.Close()
for _, f := range r.File {
err := unzipFile(f, dest)
if err != nil {
return err
}
}
return nil
}
func main() {
err := Unzip("./sample.zip", "./out")
if err != nil {
log.Fatal(err)
}
}

Set UserAgent in http request

I'm trying to make my Go application specify itself as a specific UserAgent, but can't find anything on how to go about doing this with net/http. I'm creating an http.Client, and using it to make Get requests, via client.Get().
Is there a way to set the UserAgent in the Client, or at all?
When creating your request use request.Header.Set("key", "value"):
package main
import (
"io/ioutil"
"log"
"net/http"
)
func main() {
client := &http.Client{}
req, err := http.NewRequest("GET", "http://httpbin.org/user-agent", nil)
if err != nil {
log.Fatalln(err)
}
req.Header.Set("User-Agent", "Golang_Spider_Bot/3.0")
resp, err := client.Do(req)
if err != nil {
log.Fatalln(err)
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatalln(err)
}
log.Println(string(body))
}
Result:
2012/11/07 15:05:47 {
"user-agent": "Golang_Spider_Bot/3.0"
}
P.S. http://httpbin.org is amazing for testing this kind of thing!