I'm working on a app using gorm to deal with database access.
I'm considering using an existing package for this and the package needs a sql.Tx struct value.
After starting a gorm transaction, I can get the sql.DB out of the gorm.DB but didn't find a way to extract the sql.Tx.
How do I get the sql.Tx from a gorm transaction db struct?
sql.Tx implement gorm.ConnPool, if *gorm.DB not in Transaction, db.Statement.ConnPool is *sql.DB, if *gorm.DB in Transaction, db.Statement.ConnPool is *sql.Tx
err = DB.Transaction(func(tx *gorm.DB) error {
pool := tx.Statement.ConnPool
fmt.Printf("%T \n", pool) // *sql.Tx
sqlTx, ok := pool.(*sql.Tx)
if !ok {
// todo
return fmt.Errorf("todo")
}
// ...
return nil
})
Related
I am trying to upload bunch of files using the company's api to the storage service they provide. (basically to my account). I have got lots of files like 40-50 or something.
I got the full path of the files and utilize the os.Open, so that, I can pass the io.Reader. I did try to use client.Files.Upload() without goroutines but it took so much time to upload them and decided to use goroutines. Here the implementation that I tried. When I run the program it just uploads one file which is the one that has the lowest size or something that it waits for a long time. What is wrong with it? Is it not like every time for loops run it creates a goroutine continue its cycle and creates for every file? How to make it as fast as possible with goroutines?
var filePaths []string
var wg sync.WaitGroup
// fills the string of slice with fullpath of files.
func fill() {
filepath.Walk(rootpath, func(path string, info os.FileInfo, err error) error {
if !info.IsDir() {
filePaths = append(filePaths, path)
}
if err != nil {
fmt.Println("ERROR:", err)
}
return nil
})
}
func main() {
fill()
tokenSource := oauth2.StaticTokenSource(&oauth2.Token{AccessToken: token})
oauthClient := oauth2.NewClient(context.TODO(), tokenSource)
client := putio.NewClient(oauthClient)
for _, path := range filePaths {
wg.Add(1)
go func() {
defer wg.Done()
f, err := os.Open(path)
if err != nil {
log.Println("err:OPEN", err)
}
upload, err := client.Files.Upload(context.TODO(), f, path, 0)
if err != nil {
log.Println("error uploading file:", err)
}
fmt.Println(upload)
}()
}
wg.Wait()
}
Consider a worker pool pattern like this: https://go.dev/play/p/p6SErj3L6Yc
In this example application, I've taken out the API call and just list the file names. That makes it work on the playground.
A fixed number of worker goroutines are started. We'll use a channel to distribute their work and we'll close the channel to communicate the end of the work. This number could be 1 or 1000 routines, or more. The number should be chosen based on how many concurrent API operations your putio API can reasonably be expected to support.
paths is a chan string we'll use for this purpose.
workers range over paths channel to receive new file paths to upload
package main
import (
"fmt"
"os"
"path/filepath"
"sync"
)
func main() {
paths := make(chan string)
var wg = new(sync.WaitGroup)
for i := 0; i < 10; i++ {
wg.Add(1)
go worker(paths, wg)
}
if err := filepath.Walk("/usr", func(path string, info os.FileInfo, err error) error {
if err != nil {
return fmt.Errorf("Failed to walk directory: %T %w", err, err)
}
if info.IsDir() {
return nil
}
paths <- path
return nil
}); err != nil {
panic(fmt.Errorf("failed Walk: %w", err))
}
close(paths)
wg.Wait()
}
func worker(paths <-chan string, wg *sync.WaitGroup) {
defer wg.Done()
for path := range paths {
// do upload.
fmt.Println(path)
}
}
This pattern can handle an indefinitely large amount of files without having to load the entire list in memory before processing it. As you can see, this doesn't make the code more complicated - actually, it's simpler.
When I run the program it just uploads one file which is the one
Function literals inherit the scope in which they were defined. This is why our code only listed one path - the path variable scope in the for loop was shared to each go routine, so when that variable changed, all routines picked up the change.
Avoid function literals unless you actually want to inherit scope. Functions defined at the global scope don't inherit any scope, and you must pass all relevant variables to those functions instead. This is a good thing - it makes the functions more straightforward to understand and makes variable "ownership" transitions more explicit.
An appropriate case to use a function literal could be for the os.Walk parameter; its arguments are defined by os.Walk so definition scope is one way to access other values - such as paths channel, in our case.
Speaking of scope, global variables should be avoided unless their scope of usage is truly global. Prefer passing variables between functions to sharing global variables. Again, this makes variable ownership explicit and makes it easy to understand which functions do and don't access which variables. Neither your wait group nor your filePaths have any cause to be global.
f, err := os.Open(path)
Don't forget to close any files you open. When you're dealing with 40 or 50 files, letting all those open file handles pile up until the program ends isn't so bad, but it's a time bomb in your program that will go off when the number of files exceeds the ulimit of allowed open files. Because the function execution greatly exceeds the part where the file needs to be open, defer doesn't make sense in this case. I would use an explicit f.Close() after uploading the file.
db.ping() function takes about 2s to return a error if db gone offline while execution of programe. is there any way to reduce that time taken
Really no,
If you are waiting for a successful ping to happen before executing operations against the db then that's the price you have to pay!
It entirely depends on which db, which library, what's the network like etc.
From a top level, you could re-architect so that db healthchecks are not performed during user requests and then it won't really matter that the healthcheck is slow!
Or a more aggressive approach would be to add timeouts. If you know that a success ping takes 20ms then add a 40ms or 50ms timeout to the operation, and consider that to be a failure! Fail fast!
I used go-sql-driver/mysql library:
https://github.com/go-sql-driver/mysql
has this configuration options
TimeOut
ReadTimeOut
WriteTimeOut
by changing its values actually in my case TimeOut. I could reduce timeOut time to user defined time
Resolve the mysql bad connection error
func checkPing(db *sql.DB, globalConf string) *sql.DB {
for i := 0; i < 5; i++ {
err := db.Ping()
if err == nil {
break
}
if err != nil {
log.Printf("mysql connection failed!", err)
time.Sleep(time.Second * 5)
log.Print("Retrying to make mysql connection...")
/* mysql connection */
db := connectToMysql(globalConf) // making reconnection..
return db
}
}
return db
}
Essentially I've begun to work on a wrapper for the Riot Games API and I'm struggling with how to test it. I've got the repository plugged into Travis so on push it runs go test but I'm not sure how to go about testing it since the API_KEY required for the requests changes daily and I can't auto-regenerate it, i'd have to manually add it every day if I tested the endpoints directly.
So I was wondering if it was possible to mock the responses, but in that case I'm guessing I'd need to refactor my code?
So i've made a struct to represent their SummonerDTO
type Summoner struct {
ID int64 `json:"id"`
AccountID int64 `json:"accountId"`
ProfileIconID int `json:"profileIconId"`
Name string `json:"name"`
Level int `json:"summonerLevel"`
RevisionDate int64 `json:"revisionDate"`
}
That struct has a method:
func (s Summoner) ByName(name string, region string) (summoner *Summoner, err error) {
endpoint := fmt.Sprintf("https://%s.api.riotgames.com/lol/summoner/%s/summoners/by-name/%s", REGIONS[region], VERSION, name)
client := &http.Client{}
req, err := http.NewRequest("GET", endpoint, nil)
if err != nil {
return nil, fmt.Errorf("unable to create new client for request: %v", err)
}
req.Header.Set("X-Riot-Token", API_KEY)
resp, err := client.Do(req)
if err != nil {
return nil, fmt.Errorf("unable to complete request to endpoint: %v", err)
}
defer resp.Body.Close()
if resp.StatusCode != 200 {
return nil, fmt.Errorf("request to api failed with: %v", resp.Status)
}
respBody, err := ioutil.ReadAll(resp.Body)
if err != nil {
return nil, fmt.Errorf("unable to read response body: %v", err)
}
if err := json.Unmarshal([]byte(respBody), &summoner); err != nil {
return nil, fmt.Errorf("unable to unmarshal response body to summoner struct: %v", err)
}
return summoner, nil
}
Is it a case that the struct method doesn't have a single responsibility, in a sense it's building the endpoint, firing off the request and parsing the response. Do I need to refactor it in order to make it testable, and in which case what's the best approach for that? Should I make a Request and Response struct and then test those?
To clarify the API Keys used are rate limited and need to be regenerated daily, Riot Games do not allow you to use a crawler to auto-regenerate your keys. I'm using Travis for continuous integration so I'm wondering if there's a way to mock the request/response.
Potentially my approach is wrong, still learning.
Hopefully that all makes some form of sense, happy to clarify if not.
Writing unit tests consists of:
Providing known state for all of your inputs.
Testing that, given all meaning combinations of those inputs, you receive the expected outputs.
So you need to first identify your inputs:
s Summoner
name string
region string
Plus any "hidden" inputs, by way of globals:
client := &http.Client{}
And your outputs are:
summoner *Summoner
err error
(There can also be "hidden" outputs, if you write files, or change global variables, but you don't appear to do that here).
Now the first three inputs are easy to create from scratch for your tests: Just provide an empty Summoner{} (since you don't read or set that at all in your function, there's no need to set it other than to an empty value). name and region can simply be set to strings.
The only part remaining is your http.Client. At minimum, you should probably pass that in as an argument. Not only does this give you control over your tests, but it allows you to use easily use different client in production in the future.
But to ease testing, you might consider actually passing in a client-like interface, which you can easily mock. The only method you call on client is Do, so you could easily create a Doer interface:
type doer interface {
Do(req *Request) (*Response, error)
}
Then change your function signature to take that as one argument:
func (s Summoner) ByName(client doer, name string, region string) (summoner *Summoner, err error) {
Now, in your test you can create a custom type that fulfills the doer interface, and responds with any http.Response you like, without needing to use a server in your tests.
I try to make sockets to communicate with my clients.
A socket would be created after some requests to my API. It means, a client connects itself (only by request), but then, he joins a chat, so a socket is created and linked to the good channel.
I already used sockets so I understand how it works (C, C++, C#, Java), but what I want to make, with what I saw on the web, I think it's possible, but I don't understand how to handle it with golang.
I create a first server:
func main() {
r := mux.NewRouter()
r.HandleFunc("/", HomeHandler)
r.HandleFunc("/products", ProductsHandler)
r.HandleFunc("/articles", ArticlesHandler)
http.Handle("/", r)
}
But for socket, I need another one?
package main
import "net"
import "fmt"
import "bufio"
import "strings" // only needed below for sample processing
func main() {
fmt.Println("Launching server...")
// listen on all interfaces
ln, _ := net.Listen("tcp", ":8081")
// accept connection on port
conn, _ := ln.Accept()
// run loop forever (or until ctrl-c)
for {
// will listen for message to process ending in newline (\n)
message, _ := bufio.NewReader(conn).ReadString('\n')
// output message received
fmt.Print("Message Received:", string(message))
// sample process for string received
newmessage := strings.ToUpper(message)
// send new string back to client
conn.Write([]byte(newmessage + "\n"))
}
}
Thank for help !
Based on our chat discussion.
OVERsimplified example with lots of pseudocode
import (
"net"
"encoding/json"
"errors"
)
type User struct {
name string
}
type Message {
Action string
Params map[string]string
}
type Server struct {
connected_users map[*User]net.Conn
users_connected_with_each_other map[*User][]*User
good_users map[string]*User
}
func (srv *Server) ListenAndServe(addr string) error {
ln, err := net.Listen("tcp", addr)
if err != nil {
return err
}
return srv.Serve(tcpKeepAliveListener{ln.(*net.TCPListener)})
}
func (srv *Server) Serve(l net.Listener) error {
defer l.Close()
for {
rw, e := l.Accept()
if e != nil {
return e
}
// you want to create server_conn here with buffers, channels and stuff
// to use async thread safe read/write from it
go srv.serve_conn(rw)
}
}
func (srv *Server) serve_conn(rw net.Conn) error {
dec := json.NewDecoder(rw)
var message Message
//read 1st message he sent, should be token to connect
dec.Decode(&message)
token := get_token(Message)
user, ok := srv.good_users[token]
if !ok {
return errors.New("BAD USER!")
}
// store connected user
srv.connected_users[user] = rw
for {
// async reader will be nice
dec.Decode(&message)
switch message.Action {
case "Message":
// find users to send message to
if chats_with, err := users_connected_with_each_other[user]; err == nil {
for user_to_send_message_to := range chats_with {
// find connections to send message to
if conn, err := srv.connected_users[user_to_send_message_to]; err == nil {
// send json encoded message
err := json.NewEncoder(conn).Encode(message)
//if write failed store message for later
}
}
}
//other cases
default:
// log?
}
}
}
func main() {
known_users_with_tokens := make(map[string]*User)
srv := &Server{
connected_users: make(map[*User]net.Conn),
users_connected_with_each_other: make(map[*User][]*User),
good_users: known_users_with_tokens, // map is reference type, so treat it like pointer
}
// start our server
go srv.ListenAndServe(":54321")
ConnRequestHandler := function(w http.ResponseWriter, r *http.Request) {
user := create_user_based_on_request(r)
token := create_token(user)
// now user will be able to connect to server with token
known_users_with_tokens[token] = user
}
ConnectUsersHandler := function(user1,user2) {
// you should guard your srv.* members to avoid concurrent read/writes to map
srv.users_connected_with_each_other[user1] = append(srv.users_connected_with_each_other[user1], user2)
srv.users_connected_with_each_other[user2] = append(srv.users_connected_with_each_other[user2], user1)
}
//initialize your API http.Server
r := mux.NewRouter()
r.HandleFunc("/", HomeHandler)
r.HandleFunc("/products", ProductsHandler)
r.HandleFunc("/articles", ArticlesHandler)
r.HandleFunc("/connection_request", ConnRequestHandler) // added
http.Handle("/", r)
}
Call ConnectUsersHandler(user1, user2) to allow them communicate with each other.
known_users_with_tokens[token] = user to allow user to connect to server
You need to implement async reader/writer for connections to your server. Usefull structs to keep good Users.
Guard Server struct members and provide thread safe access to update it.
UDP
Looks like json.NewEncoder(connection).Encode(&message) and json.NewDecoder(connection).Decode(&message) is async and thread safe. So you can write simultaneously from different goroutines. No need to manual sync, YAY!
default http server accepts connection on one "host:port" only
Answer depends on what protocol you are going to use to communicate via your sockets.
I suggest to do it this way: (much simplified)
Leave http.Server alone to serve your API (it implements protocols HTTP 1.*/2 so you dont need to worry about it)
Implement your own "MultiSocketServer", do to so:
2.1 Implement GracefulListener (must implement net.Listener) (you need to shutdown your sockets when you dont need them anymore, right?)
2.2 Implement MultiSocketServer.Serve(l GracefulListener) (hello http.Server.Serve() ) to serve individual connection (your protocol to communicate with client via sockets goes here. something like net.textproto will be easy to implement since you GracefulListener.Accept() will return net.Conn)
2.3 Add methods MultiSocketServer.ListenAndServe(addr), MultiSocketServer.StopServe(l GracefulListener) to your MultiSocketServer
type MultiSocketServer struct {
listeners GracefulListener[] //or map?
// lots of other stuff
}
// looks familiar? (http.Server.ListenAndServe)
func (s *MultiSocketServer) ListenAndServe(addr string) {
ln, err := net.Listen("tcp", addr)
graceful_listner = &GracefulListener(ln)
s.listeners = append(s.listeners, graceful_listner)
go s.Serve(graceful_listner)
return graceful_listner
}
func (s *MultiSocketServer) StopServe(graceful_listner GracefulListener) {
graceful_listner.Stop()
//pseudocode
remove_listener_from_slice(s.listeners, graceful_listner)
}
Ofcourse, you need to add error checking and mutex (propably) to guard MultiSocketServer.listeners to make it thread safe.
In your main() start your API http.Server, and initialize your MultiSocketServer. Now from your http.Handler/http.HandlerFunc of http.Server you should be able to call MultiSocketServer.ListenAndServe(addr) to listen and serve your sockets connections.
UPDATE based on question
however, I'm not sure to understand the part "In your main()". If I understand it good, you said I have my API, and after starting it, I initialize MultiSocketServer. But where? after the starting of my API? Or you mean it would be better that I use the logic of your code as an API? Every request trough a socket
BTW: updated MultiSocketServer.ListenAndServe to start Listen and return graceful_listner
func main() {
//init MultiSocketServer
multi_socket_server = &MultiSocketServer{} //nil for GracefulListener[] is fine for now, complex initialization will be added later
// no listners yet, serves nothing
// create new Handeler for your "socket requests"
SocketRequestHandler := function(w http.ResponseWriter, r *http.Request) {
// identify client, assign him an address to connect
addr_to_listen := parse_request(r) //pseudocode
listener := multi_socket_server.ListenAndServe(addr_to_listen)
// TODO: handle errors
// now your multi_socket_server listen to addr_to_listen and serves it with multi_socket_server.Serve method in its own goroutine
// as i said MultiSocketServer.Serve method must implement your protocol (plaintext Reader/Writer on listener for now?)
save_listener_in_context_or_whatever_you_like_to_track_it(listener) //pseudo
}
SocketDisconnectHandler := function(w http.ResponseWriter, r *http.Request) {
// identify client
some_client := parse_request(r) //pseudocode
// get listener based on info
listener := get_listener_from_context_or_whatever(some_client) //pseudo
multi_socket_server.StopServe(listener)
// TODO: handle errors
}
//initialize your API http.Server
r := mux.NewRouter()
r.HandleFunc("/", HomeHandler)
r.HandleFunc("/products", ProductsHandler)
r.HandleFunc("/articles", ArticlesHandler)
r.HandleFunc("/socket_request", SocketRequestHandler) // added
r.HandleFunc("/socket_disconnect", SocketDisconnectHandler) //added
http.Handle("/", r)
// it creates new http.Server with DefaultServeMux as Handler (which is configured with your http.Handle("/", r) call)
http.ListenAndServe(":8080") // start serving API via HTTP proto
}
Actually, you may call multi_socket_server.ListenAndServe(addr_to_listen) and multi_socket_server.StopServe(listener) from any handler in your API server.
Every time you call multi_socket_server.ListenAndServe(addr_to_listen) it will create new listener and serve on it, you have to control it (dont listen on the same address more then once (i think it will error out anyway))
Your MultiSocketServer.Serve may looks like:
func (s *MultiSocketServer) Serve(l net.Listener) {
defer l.Close()
for {
// will listen for message to process ending in newline (\n)
message, _ := bufio.NewReader(conn).ReadString('\n')
// output message received
fmt.Print("Message Received:", string(message))
// sample process for string received
newmessage := strings.ToUpper(message)
// send new string back to client
conn.Write([]byte(newmessage + "\n"))
}
}
Possible GracefulListener implementation github
Or are you trying to achieve something completely different? =)
What is the best practice for storing a connection to a database in Go language?
In Java for example you can use singletons, some IoC containers like Spring.
What is the best practice in it's lifecycle?
How to release it after application close?
There is nothing wrong about using a Singleton pattern here too.
I would use something like this:
var db *sql.DB = nil
func GetDB() (*sql.DB, error) {
if db == nil {
conn := fmt.Sprintf("host=%s user=%s password=%s dbname=%s sslmode=require",
DB_HOST, DB_USER, DB_PASSWORD, DB_NAME)
log.Println("Creating a new connection: %v", conn)
d, err := sql.Open("postgres", conn)
if err != nil {
return nil, err
}
db = d
}
return db, nil
}
With this exported function you can receive a connection from all other packages.
Update of the answer according to the comments (thanks #all for the information)!:
The returned DB is safe for concurrent use by multiple goroutines and
maintains its own pool of idle connections. Thus, the Open function
should be called just once. It is rarely necessary to close a DB.¹
It is rare to Close a DB, as the DB handle is meant to be long-lived
and shared between many goroutines.²
I would say that there is no forcing reason to call close on the database connection. I found no other statements. Despite this I would use a defer GetDB().close() in the main function - just for the completeness of the code.
Another thing I would like to note is that the connection should be verified by a db.Ping() otherwise the connection could be established but the database may not exist.
With this new information I wouldn't bother using some mutexes to ensure that the database is established. I would create a new DBInit() and run it inside the init() function of the main package.