How to mock external http request api when integration test in golang - api

I have a service to get db data and get others data from third party api.
Like this:
type Service interface {
GetDataFromDB(params apiParams, thirdClient ApiCient)
}
type Repository interface {
GetDataFromDB(orm *gorm.DB)
}
type DataService struct {
repo Repository
}
func (s *DataService) GetDataFromDB(params apiParams, thirdClient ApiClient) []interface{} {
var result []interface{}
dataFromDb := s.repo.GetDataFromDB()
dataFromAPI := thirdClient.Do(url)
result = append(result, dataFromDb)
result = append(result, dataFromAPI)
return result
}
func getData(c *gin.Context) {
//already implement interface
repo := NewRepository(orm)
srv := NewService(repo)
thirdPartyClient := NewApiClient()
params := &params{Id:1,Name:"hello world"}
res := srv.GetDataFromDB(params, thirdPartyClient)
c.JSON(200,res)
}
func TestGetData(t *testing.T) {
w := httptest.NewRecorder()
request := http.NewRequest(http.MethodGet, "/v1/get_data", nil)
route.ServeHTTP(w, request)
}
And third party api client will return random data.
In this situation, what should I do ?
If I want to mock client to get stable data to test, how to fake it in integration test ?

I am assuming "integration test" means you will be running your entire application and then testing the running instance together with its dependencies (in your case a database & third party service). I assume you do not mean unit testing.
For integration tests you have a few options. In my case, usually I would integration test including whatever the third party client is connecting to (no mocking) because I want to test the integration of my service with the third party one. Or if that is not possible I might write a simple stub application with the same public interface as the third party service and run it on localhost (or somewhere) for my application to connect to during testing.
If you don't want to or can't do either of those and want to stub the external dependency inside your Go application, you can write an interface for the third party client and provide a stubbed implementation of the interface when running integration tests (using a flag on your application to tell it to run in "stubbed" mode or something of that nature).
Here's an example of what this might look like. Here's the source code file you posted - but using an interface for getting the third party data:
type Service interface {
GetDataFromDB(params apiParams, thirdClient ApiCient)
}
type Repository interface {
GetDataFromDB(orm *gorm.DB)
}
type ThirdPartyDoer interface {
Do(url string) interface{}
}
type DataService struct {
repo Repository
thirdParty ThirdPartyDoer
}
func (s *DataService) GetDataFromDB(params apiParams, thirdClient ApiClient) []interface{} {
var result []interface{}
dataFromDb := s.repo.GetDataFromDB()
dataFromAPI := s.thirdParty.Do(url)
result = append(result, dataFromDb)
result = append(result, dataFromAPI)
return result
}
Then you can write a stub implementation for ThirdPartyDoer and use it when testing. When running in Production you can use the real third party client as ThirdPartyDoer's implementation:
type thirdPartyDoerStub struct {}
func (s *thirdPartyDoerStub) Do(url string) interface{} {
return "some static test data"
}
// ...
// Test setup:
integrationTestDataService := &DataService{repo: realRepository, thirdParty: &thirdPartyDoerStub{}}
// Production setup:
integrationTestDataService := &DataService{repo: realRepository, thirdParty: realThirdParty}
You would need to have a flag to select between "test setup" and "production setup" when starting your application.

Related

How would I test this method?

Essentially I've begun to work on a wrapper for the Riot Games API and I'm struggling with how to test it. I've got the repository plugged into Travis so on push it runs go test but I'm not sure how to go about testing it since the API_KEY required for the requests changes daily and I can't auto-regenerate it, i'd have to manually add it every day if I tested the endpoints directly.
So I was wondering if it was possible to mock the responses, but in that case I'm guessing I'd need to refactor my code?
So i've made a struct to represent their SummonerDTO
type Summoner struct {
ID int64 `json:"id"`
AccountID int64 `json:"accountId"`
ProfileIconID int `json:"profileIconId"`
Name string `json:"name"`
Level int `json:"summonerLevel"`
RevisionDate int64 `json:"revisionDate"`
}
That struct has a method:
func (s Summoner) ByName(name string, region string) (summoner *Summoner, err error) {
endpoint := fmt.Sprintf("https://%s.api.riotgames.com/lol/summoner/%s/summoners/by-name/%s", REGIONS[region], VERSION, name)
client := &http.Client{}
req, err := http.NewRequest("GET", endpoint, nil)
if err != nil {
return nil, fmt.Errorf("unable to create new client for request: %v", err)
}
req.Header.Set("X-Riot-Token", API_KEY)
resp, err := client.Do(req)
if err != nil {
return nil, fmt.Errorf("unable to complete request to endpoint: %v", err)
}
defer resp.Body.Close()
if resp.StatusCode != 200 {
return nil, fmt.Errorf("request to api failed with: %v", resp.Status)
}
respBody, err := ioutil.ReadAll(resp.Body)
if err != nil {
return nil, fmt.Errorf("unable to read response body: %v", err)
}
if err := json.Unmarshal([]byte(respBody), &summoner); err != nil {
return nil, fmt.Errorf("unable to unmarshal response body to summoner struct: %v", err)
}
return summoner, nil
}
Is it a case that the struct method doesn't have a single responsibility, in a sense it's building the endpoint, firing off the request and parsing the response. Do I need to refactor it in order to make it testable, and in which case what's the best approach for that? Should I make a Request and Response struct and then test those?
To clarify the API Keys used are rate limited and need to be regenerated daily, Riot Games do not allow you to use a crawler to auto-regenerate your keys. I'm using Travis for continuous integration so I'm wondering if there's a way to mock the request/response.
Potentially my approach is wrong, still learning.
Hopefully that all makes some form of sense, happy to clarify if not.
Writing unit tests consists of:
Providing known state for all of your inputs.
Testing that, given all meaning combinations of those inputs, you receive the expected outputs.
So you need to first identify your inputs:
s Summoner
name string
region string
Plus any "hidden" inputs, by way of globals:
client := &http.Client{}
And your outputs are:
summoner *Summoner
err error
(There can also be "hidden" outputs, if you write files, or change global variables, but you don't appear to do that here).
Now the first three inputs are easy to create from scratch for your tests: Just provide an empty Summoner{} (since you don't read or set that at all in your function, there's no need to set it other than to an empty value). name and region can simply be set to strings.
The only part remaining is your http.Client. At minimum, you should probably pass that in as an argument. Not only does this give you control over your tests, but it allows you to use easily use different client in production in the future.
But to ease testing, you might consider actually passing in a client-like interface, which you can easily mock. The only method you call on client is Do, so you could easily create a Doer interface:
type doer interface {
Do(req *Request) (*Response, error)
}
Then change your function signature to take that as one argument:
func (s Summoner) ByName(client doer, name string, region string) (summoner *Summoner, err error) {
Now, in your test you can create a custom type that fulfills the doer interface, and responds with any http.Response you like, without needing to use a server in your tests.

Go testing customization with testing.TB

I'm trying to customize the testing.T with my own assert method to lower the number of lines I'm writing. I tried the following, ended with an error: "wrong signature for TestCustom, must be: func TestCustom(t *testing.T)".
How can I make TestCustom use CustomTester interface with a new method, assert?
I don't want to use a 3rd-party framework.
custom_testing.go
type CustomTester struct {
testing.TB
}
func (t *CustomTester) assert(exp interface{}, act interface{}) {
if exp != act {
t.Errorf("expected: %v. got: %v\n", exp, act)
}
}
// I want testing package inject testing.T here
// But, by using my own wrapper: CustomTester struct with,
// my own assert method to get rid of using t as an argument,
// in each assert like: assert(t, exp, act)
func TestCustom(t *testing.TB) {
t.assert(3, len(foo))
}
NOTE: I also tried this, it works but, I don't want to pass t each time when I'm testing:
working_not_wanted.go
func assert(t *testing.TB, exp interface{}, act interface{}) {
if exp != act {
t.Errorf("expected: %v. got: %v\n", exp, act)
}
}
func TestCustom(t *testing.T) {
assert(t, 3, len(foo))
}
The Go testing framework executes test functions of a specific signature, and that signature takes a *testing.T. If you want to use the stdlib testing system, your test functions have to have the required signature.
You could wrap it with one line in every test function:
func MyTest(stdt *testing.T) {
// This line:
t := &CustomTester{stdt}
t.assert(true)
t.Error("An error done happened")
}
There are other ways to do it, but there is no way to have a testing function, run by go test, using the stdlib testing package, that takes anything other than *testing.T as its sole parameter.

Golang - API Server and Socket at the same time

I try to make sockets to communicate with my clients.
A socket would be created after some requests to my API. It means, a client connects itself (only by request), but then, he joins a chat, so a socket is created and linked to the good channel.
I already used sockets so I understand how it works (C, C++, C#, Java), but what I want to make, with what I saw on the web, I think it's possible, but I don't understand how to handle it with golang.
I create a first server:
func main() {
r := mux.NewRouter()
r.HandleFunc("/", HomeHandler)
r.HandleFunc("/products", ProductsHandler)
r.HandleFunc("/articles", ArticlesHandler)
http.Handle("/", r)
}
But for socket, I need another one?
package main
import "net"
import "fmt"
import "bufio"
import "strings" // only needed below for sample processing
func main() {
fmt.Println("Launching server...")
// listen on all interfaces
ln, _ := net.Listen("tcp", ":8081")
// accept connection on port
conn, _ := ln.Accept()
// run loop forever (or until ctrl-c)
for {
// will listen for message to process ending in newline (\n)
message, _ := bufio.NewReader(conn).ReadString('\n')
// output message received
fmt.Print("Message Received:", string(message))
// sample process for string received
newmessage := strings.ToUpper(message)
// send new string back to client
conn.Write([]byte(newmessage + "\n"))
}
}
Thank for help !
Based on our chat discussion.
OVERsimplified example with lots of pseudocode
import (
"net"
"encoding/json"
"errors"
)
type User struct {
name string
}
type Message {
Action string
Params map[string]string
}
type Server struct {
connected_users map[*User]net.Conn
users_connected_with_each_other map[*User][]*User
good_users map[string]*User
}
func (srv *Server) ListenAndServe(addr string) error {
ln, err := net.Listen("tcp", addr)
if err != nil {
return err
}
return srv.Serve(tcpKeepAliveListener{ln.(*net.TCPListener)})
}
func (srv *Server) Serve(l net.Listener) error {
defer l.Close()
for {
rw, e := l.Accept()
if e != nil {
return e
}
// you want to create server_conn here with buffers, channels and stuff
// to use async thread safe read/write from it
go srv.serve_conn(rw)
}
}
func (srv *Server) serve_conn(rw net.Conn) error {
dec := json.NewDecoder(rw)
var message Message
//read 1st message he sent, should be token to connect
dec.Decode(&message)
token := get_token(Message)
user, ok := srv.good_users[token]
if !ok {
return errors.New("BAD USER!")
}
// store connected user
srv.connected_users[user] = rw
for {
// async reader will be nice
dec.Decode(&message)
switch message.Action {
case "Message":
// find users to send message to
if chats_with, err := users_connected_with_each_other[user]; err == nil {
for user_to_send_message_to := range chats_with {
// find connections to send message to
if conn, err := srv.connected_users[user_to_send_message_to]; err == nil {
// send json encoded message
err := json.NewEncoder(conn).Encode(message)
//if write failed store message for later
}
}
}
//other cases
default:
// log?
}
}
}
func main() {
known_users_with_tokens := make(map[string]*User)
srv := &Server{
connected_users: make(map[*User]net.Conn),
users_connected_with_each_other: make(map[*User][]*User),
good_users: known_users_with_tokens, // map is reference type, so treat it like pointer
}
// start our server
go srv.ListenAndServe(":54321")
ConnRequestHandler := function(w http.ResponseWriter, r *http.Request) {
user := create_user_based_on_request(r)
token := create_token(user)
// now user will be able to connect to server with token
known_users_with_tokens[token] = user
}
ConnectUsersHandler := function(user1,user2) {
// you should guard your srv.* members to avoid concurrent read/writes to map
srv.users_connected_with_each_other[user1] = append(srv.users_connected_with_each_other[user1], user2)
srv.users_connected_with_each_other[user2] = append(srv.users_connected_with_each_other[user2], user1)
}
//initialize your API http.Server
r := mux.NewRouter()
r.HandleFunc("/", HomeHandler)
r.HandleFunc("/products", ProductsHandler)
r.HandleFunc("/articles", ArticlesHandler)
r.HandleFunc("/connection_request", ConnRequestHandler) // added
http.Handle("/", r)
}
Call ConnectUsersHandler(user1, user2) to allow them communicate with each other.
known_users_with_tokens[token] = user to allow user to connect to server
You need to implement async reader/writer for connections to your server. Usefull structs to keep good Users.
Guard Server struct members and provide thread safe access to update it.
UDP
Looks like json.NewEncoder(connection).Encode(&message) and json.NewDecoder(connection).Decode(&message) is async and thread safe. So you can write simultaneously from different goroutines. No need to manual sync, YAY!
default http server accepts connection on one "host:port" only
Answer depends on what protocol you are going to use to communicate via your sockets.
I suggest to do it this way: (much simplified)
Leave http.Server alone to serve your API (it implements protocols HTTP 1.*/2 so you dont need to worry about it)
Implement your own "MultiSocketServer", do to so:
2.1 Implement GracefulListener (must implement net.Listener) (you need to shutdown your sockets when you dont need them anymore, right?)
2.2 Implement MultiSocketServer.Serve(l GracefulListener) (hello http.Server.Serve() ) to serve individual connection (your protocol to communicate with client via sockets goes here. something like net.textproto will be easy to implement since you GracefulListener.Accept() will return net.Conn)
2.3 Add methods MultiSocketServer.ListenAndServe(addr), MultiSocketServer.StopServe(l GracefulListener) to your MultiSocketServer
type MultiSocketServer struct {
listeners GracefulListener[] //or map?
// lots of other stuff
}
// looks familiar? (http.Server.ListenAndServe)
func (s *MultiSocketServer) ListenAndServe(addr string) {
ln, err := net.Listen("tcp", addr)
graceful_listner = &GracefulListener(ln)
s.listeners = append(s.listeners, graceful_listner)
go s.Serve(graceful_listner)
return graceful_listner
}
func (s *MultiSocketServer) StopServe(graceful_listner GracefulListener) {
graceful_listner.Stop()
//pseudocode
remove_listener_from_slice(s.listeners, graceful_listner)
}
Ofcourse, you need to add error checking and mutex (propably) to guard MultiSocketServer.listeners to make it thread safe.
In your main() start your API http.Server, and initialize your MultiSocketServer. Now from your http.Handler/http.HandlerFunc of http.Server you should be able to call MultiSocketServer.ListenAndServe(addr) to listen and serve your sockets connections.
UPDATE based on question
however, I'm not sure to understand the part "In your main()". If I understand it good, you said I have my API, and after starting it, I initialize MultiSocketServer. But where? after the starting of my API? Or you mean it would be better that I use the logic of your code as an API? Every request trough a socket
BTW: updated MultiSocketServer.ListenAndServe to start Listen and return graceful_listner
func main() {
//init MultiSocketServer
multi_socket_server = &MultiSocketServer{} //nil for GracefulListener[] is fine for now, complex initialization will be added later
// no listners yet, serves nothing
// create new Handeler for your "socket requests"
SocketRequestHandler := function(w http.ResponseWriter, r *http.Request) {
// identify client, assign him an address to connect
addr_to_listen := parse_request(r) //pseudocode
listener := multi_socket_server.ListenAndServe(addr_to_listen)
// TODO: handle errors
// now your multi_socket_server listen to addr_to_listen and serves it with multi_socket_server.Serve method in its own goroutine
// as i said MultiSocketServer.Serve method must implement your protocol (plaintext Reader/Writer on listener for now?)
save_listener_in_context_or_whatever_you_like_to_track_it(listener) //pseudo
}
SocketDisconnectHandler := function(w http.ResponseWriter, r *http.Request) {
// identify client
some_client := parse_request(r) //pseudocode
// get listener based on info
listener := get_listener_from_context_or_whatever(some_client) //pseudo
multi_socket_server.StopServe(listener)
// TODO: handle errors
}
//initialize your API http.Server
r := mux.NewRouter()
r.HandleFunc("/", HomeHandler)
r.HandleFunc("/products", ProductsHandler)
r.HandleFunc("/articles", ArticlesHandler)
r.HandleFunc("/socket_request", SocketRequestHandler) // added
r.HandleFunc("/socket_disconnect", SocketDisconnectHandler) //added
http.Handle("/", r)
// it creates new http.Server with DefaultServeMux as Handler (which is configured with your http.Handle("/", r) call)
http.ListenAndServe(":8080") // start serving API via HTTP proto
}
Actually, you may call multi_socket_server.ListenAndServe(addr_to_listen) and multi_socket_server.StopServe(listener) from any handler in your API server.
Every time you call multi_socket_server.ListenAndServe(addr_to_listen) it will create new listener and serve on it, you have to control it (dont listen on the same address more then once (i think it will error out anyway))
Your MultiSocketServer.Serve may looks like:
func (s *MultiSocketServer) Serve(l net.Listener) {
defer l.Close()
for {
// will listen for message to process ending in newline (\n)
message, _ := bufio.NewReader(conn).ReadString('\n')
// output message received
fmt.Print("Message Received:", string(message))
// sample process for string received
newmessage := strings.ToUpper(message)
// send new string back to client
conn.Write([]byte(newmessage + "\n"))
}
}
Possible GracefulListener implementation github
Or are you trying to achieve something completely different? =)

How should I unit test middleware packages with gorilla context

I have this net/http server setup with several middleware in a chain and I can't find examples on how I should test these...
I am using basic net/http with the gorilla/mux router and one Handle looks somewhat like this:
r.Handle("/documents", addCors(checkAPIKey(getDocuments(sendJSON)))).Methods("GET")
In these I aggregate some data and supply them via Gorilla Context context.Set methods.
Usually I test my http functions with httptest, and I hope to do it with these as well but I can't figure out how and I am curious as to what is the best way. Should I test each middleware seperately? Should I prefill the appropriate context values then when they are needed? Can I test this entire chain at once so I can just check desired states on input?
I would not test anything involving Gorilla or any other 3rd party package. If you want to test to make sure it works, i'd setup some external test runner or integration suite for the endpoints of a running version of your app (e.g. a C.I. server).
Instead, test your Middleware and Handlers individually - as those you have control over.
But, if you are set on testing the stack (mux -> handler -> handler -> handler -> MyHandler), this is where defining the middleware globally using functions as vars could help:
var addCors = func(h http.Handler) http.Handler {
...
}
var checkAPIKey = func(h http.Handler) http.Handler {
...
}
During normal use, their implementation remains the same with change.
r.Handle("/documents", addCors(checkAPIKey(getDocuments(sendJSON)))).Methods("GET")
But for unit testing, you can override them:
// important to keep the same package name for
// your test file, so you can get to the private
// vars.
package main
import (
"testing"
)
func TestXYZHandler(t *testing.T) {
// save the state, so you can restore at the end
addCorsBefore := addCors
checkAPIKeyBefore := checkAPIKey
// override with whatever customization you want
addCors = func(h http.Handler) http.Handler {
return h
}
checkAPIKey = func(h http.Handler) http.Handler {
return h
}
// arrange, test, assert, etc.
//
// when done, be a good dev and restore the global state
addCors = addCorsBefore
checkAPIKey = checkAPIKeyBefore
}
If you find yourself copy-n-pasting this boiler plate code often, move it to a global pattern within your unit tests:
package main
import (
"testing"
)
var (
addCorsBefore = addCors
checkAPIKeyBefore = checkAPIKey
)
func clearMiddleware() {
addCors = func(h http.Handler) http.Handler {
return h
}
checkAPIKey = func(h http.Handler) http.Handler {
return h
}
}
func restoreMiddleware() {
addCors = addCorsBefore
checkAPIKey = checkAPIKeyBefore
}
func TestXYZHandler(t *testing.T) {
clearMiddleware()
// arrange, test, assert, etc.
//
restoreMiddleware()
}
A side note on unit testing end points...
Since middleware should operate with sensible defaults (expected to pass normally and not mutex state of the underlying stream of data you want to test in func), I advise to unit test the middleware outside of the context of your actual main Handler function.
That way, you have one set of unit tests strictly for your middleware. And another set of tests focusing purely on the primary Handler of the url you are calling. It makes discovering the code much easier for newcomers.

Golang test mock functions best practices

I am developing some tests for my code (using the testing package), and I am wondering what's the best way to mock functions inside the tested function:
Should I pass the function as parameter?
In that case, what if that function calls another function? Should I pass both the first and second function as parameters in the tested one?
Note: some of the functions are called on objects (i.e. someObj.Create()) and use HTTP API calls.
UPDATE for clarification:
Example: functions
func f1() error {
... //some API call
}
func (s *SomeStruct) f2() error {
return f1
}
func f3() error {
return nil
}
func f4() error {
...
err = obj.f2()
...
err = f3()
...
}
For the above: if I want to test f4, what's the best way to mock f2 and f3?
If I pass f2 and f3 to f4 as parameters it would work, but then what for the f2 test? Should I pass f1 to f2 as parameter?
And if that's it, should then f4 have f1 as well in the parameters?
As a general guideline, functions aren't very mockable so its in our best interests to mock structs that implement a certain interface that may be passed into functions to test the different branches of code. See below for a basic example.
package a
type DoSomethingInterface interface {
DoSomething() error
}
func DoSomething(a DoSomethingInterface) {
if err := a.DoSomething(); err != nil {
fmt.Println("error occurred")
return
}
fmt.Println("no error occurred")
return
}
package a_test
import (
"testing"
"<path to a>/a"
)
type simpleMock struct {
err error
}
func (m *simpleMock) DoSomething() error {
return m.err
}
func TestDoSomething(t *testing.T) {
errorMock := &simpleMock{errors.New("some error")}
a.DoSomething(errorMock)
// test that "an error occurred" is logged
regularMock := &simpleMock{}
a.DoSomething(regularMock)
// test "no error occurred" is logged
}
In the above example, you would test the DoSomething function and the branches that happens eg. you would create an instance of the mock with an error for one test case and create another instance of the mock without the error to test the other case. The respective cases are to test a certain string has been logged to standard out; in this case it would be "error occurred" when simpleMock is instantiated with an error and "no error occurred" when there simpleMock is not instantiated with an error.
This can of course be expanded to other cases eg. the DoSomething function actually returns some kind of value and you want to make an assertion on the value.
Edit:
I updated the code with the concern that the interface lives in another package. Note that the new updated code has a package a that contains the interface and the function under test and a package a_test that is merely a template of how to approach testing a.DoSomething.
I'm not sure what you're trying to do here but I'll explain how testing should be done in Go.
Lets say we have an application with the following directory hierarchy:
root/
pack1/
pack1.go
pack1_test.go
pack2/
pack2.go
pack2_test.go
main.go
main_test.go
We'll assume that pack2.go has the functions you want to test:
package pack2
func f1() error {
... //some API call
}
func (s *SomeStruct) f2() error {
return f1
}
func f3() error {
return nil
}
func f4() error {
...
err = obj.f2()
...
err = f3()
...
}
Looks good so far. Now if you want to test the functions in pack2, you would create a file called pack2_test.go. All test files in go are named similarly (packagename_test.go). Now lets see the inside of a typical test for a package (pack2_test.go in this example):
package pack2
import (
"testing"
"fmt"
)
TestF1(*testing.T) {
x := "something for testing"
f1() // This tests f1 from the package "pact2.go"
}
TestF2(*testing.T) {
y := new(somestruct)
y.f2() // tests f2 from package "pact2.go"
}
TestF3(*testing.T) {
/// some code
f3() // tests f3
}
TestF4(*testing.T) {
/// code
f3() // you get the gist
}
Let me explain. Notice how in pack2_test.go, the first line says that the package is pack2. In a nutshell, this means that we're in the "scope" of the package pack2 and thus all the functions found in pack2 can be called as if you're within pack2. Thats why, within the Testf* functions, we could've called the functions from pack2. Another thing to note is the imported package "testing". This helps with two things:
First, it provides some functionality for running tests. I won't go into that.
Second, it helps identify the functions that go test should run.
Now to the functions. Any function within a test package that has the prefix "Test" and the parameters "t *testing.T" (you can use "*testing.T" when you don't need to use the testing functionality) will be executed when you run go test. You use the variable t to reference the testing functionality I mentioned. You can also declare functions without the prefix and call them within the prefixed functions.
So, if I go to my terminal and run go test, it will execute the functions you want to test, specified in pack2_test.go
You can learn more about testing here and here