db.ping() function takes about 2s to return a error if db gone offline while execution of programe. is there any way to reduce that time taken
Really no,
If you are waiting for a successful ping to happen before executing operations against the db then that's the price you have to pay!
It entirely depends on which db, which library, what's the network like etc.
From a top level, you could re-architect so that db healthchecks are not performed during user requests and then it won't really matter that the healthcheck is slow!
Or a more aggressive approach would be to add timeouts. If you know that a success ping takes 20ms then add a 40ms or 50ms timeout to the operation, and consider that to be a failure! Fail fast!
I used go-sql-driver/mysql library:
https://github.com/go-sql-driver/mysql
has this configuration options
TimeOut
ReadTimeOut
WriteTimeOut
by changing its values actually in my case TimeOut. I could reduce timeOut time to user defined time
Resolve the mysql bad connection error
func checkPing(db *sql.DB, globalConf string) *sql.DB {
for i := 0; i < 5; i++ {
err := db.Ping()
if err == nil {
break
}
if err != nil {
log.Printf("mysql connection failed!", err)
time.Sleep(time.Second * 5)
log.Print("Retrying to make mysql connection...")
/* mysql connection */
db := connectToMysql(globalConf) // making reconnection..
return db
}
}
return db
}
Related
I'm currently working on creating tests for specific use cases one of which is Init WF200 -> connect to AP -> send TCP data -> Deinit WF200. The application is very energy critical so I have to ensure that the WF200 is enabled as short as possible.
The hardware I use is a EFM32GG11 MCU together with a WF200 WIFI transceiver, both from SiliconLabs. I'm using an RTOS and the lwip stack with the netconn API for TCP communication.
The problem is that I can't find a way to know if the TCP transaction was completed, which I need to know before putting the WF200 into shutdown. Currently the task is faster then the actual transmission which leads to data loss and incomplete TCP communication.
Currently I have a working work around which is adding a delay. But that doesn't seem like an elegant solution to me, especially because the delay is dependent on the amount of data being sent.
I have already tried checking the tcp pcb state but with no success. Is there some way I can block the thread until the transaction is completed?
Thanks in advance!
static void tcp_thread(void *p_arg) {
struct netconn *conn;
err_t err;
LWIP_UNUSED_ARG(p_arg);
// needed, otherwise netconn_connect fails
KAL_Dly(1);
conn = netconn_new(NETCONN_TCP);
if (conn != NULL) {
struct ip4_addr broker_ip;
IP_ADDR4(&broker_ip, SERVER_IP_0, SERVER_IP_1, SERVER_IP_2, SERVER_IP_3);
err = netconn_connect(conn, &broker_ip, 65432);
if (err == ERR_OK) {
// NOCOPY only safe when data is static and const
err = netconn_write(conn, test_data, strlen(test_data), NETCONN_NOCOPY);
printf("Data sent\n");
netconn_close(conn);
netconn_delete(conn);
} else {
printf("No TCP connection\n");
}
} else {
printf("No netconn\n");
}
KAL_Dly(200);
sl_wfx_deinit();
OSTaskDel(0, &err);
}
netconn will execute a call back, you can add while(1){osDelay(2);} to your code and wait for that callback to finish. Maybe post a flag in the callback you can check for. For example you can use that data RX callback (or error callback, etc), inspect for an ack, then write to a var you can check in your code's while loop.
if you look at the code of RedisLock.cs class you can see that it is reading the lock value to validate timeout itself outside Redis and it is also using Watch and Unwatch to overwrite timeout value if nobody touched it.
In another words, what are the exact points of using watch and unwatch and commit while we can use Redis internal set timeouts which are even more reliable?
ExecUtils.RetryUntilTrue(() =>{
//This pattern is taken from the redis command for SETNX http://redis.io/commands/setnx
//Calculate a unix time for when the lock should expire
var realSpan = timeOut ?? new TimeSpan(365, 0, 0, 0); //if nothing is passed in the timeout hold for a year
var expireTime = DateTime.UtcNow.Add(realSpan);
var lockString = (expireTime.ToUnixTimeMs() + 1).ToString();
//Try to set the lock, if it does not exist this will succeed and the lock is obtained
var nx = redisClient.SetValueIfNotExists(key, lockString);
if (nx)
return true;
//If we've gotten here then a key for the lock is present. This could be because the lock is
//correctly acquired or it could be because a client that had acquired the lock crashed (or didn't release it properly).
//Therefore we need to get the value of the lock to see when it should expire
redisClient.Watch(key);
var lockExpireString = redisClient.Get<string>(key);
if (!long.TryParse(lockExpireString, out var lockExpireTime))
{
redisClient.UnWatch(); // since the client is scoped externally
return false;
}
//If the expire time is greater than the current time then we can't let the lock go yet
if (lockExpireTime > DateTime.UtcNow.ToUnixTimeMs())
{
redisClient.UnWatch(); // since the client is scoped externally
return false;
}
//If the expire time is less than the current time then it wasn't released properly and we can attempt to
//acquire the lock. The above call to Watch(_lockKey) enrolled the key in monitoring, so if it changes
//before we call Commit() below, the Commit will fail and return false, which means that another thread
//was able to acquire the lock before we finished processing.
using (var trans = redisClient.CreateTransaction()) // we started the "Watch" above; this tx will succeed if the value has not moved
{
trans.QueueCommand(r => r.Set(key, lockString));
return trans.Commit(); //returns false if Transaction failed
}
},timeOut);
Because StackExchange.Redis Library equivalent only use string.set capabilities. despite the fact that that ServiceStackRedis locking mechanism is more advanced and will wait and retry until get a lock but still not using timeouts of REDIS SET command itself is not justified.
public bool LockTake(RedisKey key, RedisValue value, TimeSpan expiry, CommandFlags flags = CommandFlags.None) {
if (value.IsNull) throw new ArgumentNullException(nameof(value));
return StringSet(key, value, expiry, When.NotExists, flags);
}
I'm running some tests in golang and I want to avoid running the slow ones, for example this one uses bcrypt so it's slow:
// +build slow
package services
import (
"testing"
"testing/quick"
)
// using bcrypt takes too much time, reduce the number of iterations.
var config = &quick.Config{MaxCount: 20}
func TestSignaturesAreSame(t *testing.T) {
same := func(simple string) bool {
result, err := Encrypt(simple)
success := err == nil && ComparePassWithHash(simple, result)
return success
}
if err := quick.Check(same, config); err != nil {
t.Error(err)
}
}
To avoid running this in every iteration I've set up the // +build slow flag. This should only run when doing go test -tags slow but unfortunately it's running every time (the -v flag shows it's running).
Any idea what's wrong?
Your // +build slow needs to be followed by a blank line
To distinguish build constraints from package documentation, a series of build constraints must be followed by a blank line.
visit Build Constraints
What is the best practice for storing a connection to a database in Go language?
In Java for example you can use singletons, some IoC containers like Spring.
What is the best practice in it's lifecycle?
How to release it after application close?
There is nothing wrong about using a Singleton pattern here too.
I would use something like this:
var db *sql.DB = nil
func GetDB() (*sql.DB, error) {
if db == nil {
conn := fmt.Sprintf("host=%s user=%s password=%s dbname=%s sslmode=require",
DB_HOST, DB_USER, DB_PASSWORD, DB_NAME)
log.Println("Creating a new connection: %v", conn)
d, err := sql.Open("postgres", conn)
if err != nil {
return nil, err
}
db = d
}
return db, nil
}
With this exported function you can receive a connection from all other packages.
Update of the answer according to the comments (thanks #all for the information)!:
The returned DB is safe for concurrent use by multiple goroutines and
maintains its own pool of idle connections. Thus, the Open function
should be called just once. It is rarely necessary to close a DB.¹
It is rare to Close a DB, as the DB handle is meant to be long-lived
and shared between many goroutines.²
I would say that there is no forcing reason to call close on the database connection. I found no other statements. Despite this I would use a defer GetDB().close() in the main function - just for the completeness of the code.
Another thing I would like to note is that the connection should be verified by a db.Ping() otherwise the connection could be established but the database may not exist.
With this new information I wouldn't bother using some mutexes to ensure that the database is established. I would create a new DBInit() and run it inside the init() function of the main package.
On a plain Go HTTP handler, if I disconnect a client while still writting to the response, http.ResponseWritter.Write will return an error with a message like write tcp 127.0.0.1:60702: connection reset by peer.
Now from the syscall package, I have sysca.ECONNRESET, which has the message connection reset by peer, so they're not exactly the same.
How can I match them, so I know not to panic if it occurs ? On other ocasions I have been doing
if err == syscall.EAGAIN {
/* handle error differently */
}
for instance, and that worked fine, but I can't do it with syscall.ECONNRESET.
Update:
Because I'm desperate for a solution, for the time being I'll be doing this very dirty hack:
if strings.Contains(err.Error(), syscall.ECONNRESET.Error()) {
println("it's a connection reset by peer!")
return
}
The error you get has the underlying type *net.OpError, built here, for example.
You should be able to type-assert the error to its concrete type like this:
operr, ok := err.(*net.OpError)
And then access its Err field, which should correspond to the syscall error you need:
operr.Err.Error() == syscall.ECONNRESET.Error()
The answer by #zian is more useful than the accepted answer, but now on Go 1.13+ it is preferable to avoid manually unwrapping the errors:
if errors.Is(opErr,syscall.ECONNRESET) {
fmt.Println("Found a ECONNRESET")
}
This has the benefit that you can also use it more generally, such as after:
resp, err := http.Get("http://127.0.0.1:4444")
Here this err would otherwise have an extra layer of wrapping (*url.Error) and would be missed by the condition #zian used without explicitly unwrapping it a third time.
I came across this issue and the accepted answer was sufficient to point me in the right direction. However, the code it provides to check if the Error embedded inside *net.OpError is ECONNRESET is not complete, at least not for Golang 1.9.
The error embedded at OpError.Err is actually of type *os.SyscallError (https://golang.org/pkg/os/#SyscallError). The Write() function implemented by struct *net.netFD (which is what's being written to when sending a response over the network) looks like this:
func (fd *netFD) Write(p []byte) (nn int, err error) {
nn, err = fd.pfd.Write(p)
runtime.KeepAlive(fd)
return nn, wrapSyscallError("write", err)
}
And wrapSyscallError:
func wrapSyscallError(name string, err error) error {
if _, ok := err.(syscall.Errno); ok {
err = os.NewSyscallError(name, err)
}
return err
}
The error inside the *os.SyscallError struct can be directly compared against syscall.ECONNRESET.
So, given an error returned from a network write (e.g. a call to http.ResponseWritter.Write), the full code block to determine if that error is ECONNRESET is:
if opErr, ok := err.(*net.OpError); ok {
if syscallErr, ok := opErr.Err.(*os.SyscallError); ok {
if syscallErr.Err == syscall.ECONNRESET {
fmt.Println("Found a ECONNRESET")
}
}
}
#zian - thanks for your good solution to João Pinto's (and my) question : How can I match them, so I know not to panic if it occurs ?
As at go version 1.13, an improvement is to use the errors.Is function which does error unwrapping and testing sequentially 'under the hood'. For example :
if errors.Is(opErr,syscall.ECONNRESET) {
fmt.Println("Found a ECONNRESET")
}
#SteveCoffman - adding to your good answer, cheers!
Working with Errors in Go 1.13 - The Go Blog - Golang