What is the best practice for storing a connection to a database in Go language?
In Java for example you can use singletons, some IoC containers like Spring.
What is the best practice in it's lifecycle?
How to release it after application close?
There is nothing wrong about using a Singleton pattern here too.
I would use something like this:
var db *sql.DB = nil
func GetDB() (*sql.DB, error) {
if db == nil {
conn := fmt.Sprintf("host=%s user=%s password=%s dbname=%s sslmode=require",
DB_HOST, DB_USER, DB_PASSWORD, DB_NAME)
log.Println("Creating a new connection: %v", conn)
d, err := sql.Open("postgres", conn)
if err != nil {
return nil, err
}
db = d
}
return db, nil
}
With this exported function you can receive a connection from all other packages.
Update of the answer according to the comments (thanks #all for the information)!:
The returned DB is safe for concurrent use by multiple goroutines and
maintains its own pool of idle connections. Thus, the Open function
should be called just once. It is rarely necessary to close a DB.¹
It is rare to Close a DB, as the DB handle is meant to be long-lived
and shared between many goroutines.²
I would say that there is no forcing reason to call close on the database connection. I found no other statements. Despite this I would use a defer GetDB().close() in the main function - just for the completeness of the code.
Another thing I would like to note is that the connection should be verified by a db.Ping() otherwise the connection could be established but the database may not exist.
With this new information I wouldn't bother using some mutexes to ensure that the database is established. I would create a new DBInit() and run it inside the init() function of the main package.
Related
I have configured TcpClient as described in the examples here. I am trying to make the following code resilient in situations where the server unexpectedly closes the connection:
TcpClient tcpClient = getTcpClient();
public Mono<String> sendMessage(Mono<bytes[]> request) {
Connection connection = getConnectionFromPool(tcpClient);
return connection
.outbound()
.sendByteArray(request)
.then()
.then(connection.inbound().receive().asString().as(Mono::from));
}
In such an event, I expect the method "getConnectionFromPool" to be able to retrieve a connection from the pool or open a new one if none are available.
After noticing .connect() eventually defers to ConnectionProvider.acquire(), I tried to use tcpClient.connect(), but it becomes necessary to change the method return type as follows:
public Mono<Mono<String>> sendMessage1(Mono<String> request) {
return this.tcpClient
.connect()
.map(connection ->
connection
.outbound()
.sendByteArray(request.map(String::getBytes))
.then()
.then(connection.inbound().receive().asString().as(Mono::from))
);
}
Clearly this is undesirable. How do I acquire a Connection instance directly from the pool? Is there a simple Mono operator I am missing, or am I using the TcpClient API incorrectly?
Thanks very much for any help.
What about using flatMap instead of map?
public Mono<String> sendMessage1(Mono<String> request) {
return this.tcpClient
.connect()
.flatMap(connection ->
connection
.outbound()
.sendByteArray(request.map(String::getBytes))
.then()
.then(connection.inbound().receive().asString().as(Mono::from))
);
}
db.ping() function takes about 2s to return a error if db gone offline while execution of programe. is there any way to reduce that time taken
Really no,
If you are waiting for a successful ping to happen before executing operations against the db then that's the price you have to pay!
It entirely depends on which db, which library, what's the network like etc.
From a top level, you could re-architect so that db healthchecks are not performed during user requests and then it won't really matter that the healthcheck is slow!
Or a more aggressive approach would be to add timeouts. If you know that a success ping takes 20ms then add a 40ms or 50ms timeout to the operation, and consider that to be a failure! Fail fast!
I used go-sql-driver/mysql library:
https://github.com/go-sql-driver/mysql
has this configuration options
TimeOut
ReadTimeOut
WriteTimeOut
by changing its values actually in my case TimeOut. I could reduce timeOut time to user defined time
Resolve the mysql bad connection error
func checkPing(db *sql.DB, globalConf string) *sql.DB {
for i := 0; i < 5; i++ {
err := db.Ping()
if err == nil {
break
}
if err != nil {
log.Printf("mysql connection failed!", err)
time.Sleep(time.Second * 5)
log.Print("Retrying to make mysql connection...")
/* mysql connection */
db := connectToMysql(globalConf) // making reconnection..
return db
}
}
return db
}
Is it a good idea to initialize databaze as global variable? Can it work?
I'm thinking about something like that:
func MustDB(d *sql.DB, err error) *sql.DB {
if err != nil {
log.Panic(err)
}
return d
}
// what I don't know - is how to call db.Close()
// username and password can also be read similar way
var db *DB = MustDB(db.Open(...))
func MustPrepare(db *sql.DB, query string) *sql.Stmt {
res, err := sql.Prepare(db, query)
if err!=nil {
log.Panic(err)
}
return ret;
}
The advantage is, I can simple create prepared sql statements as global variables. I don't have to create and manage a storage, where all sql commands will be put. Only I write:
var s1 *sql.Stmt = MustPrepare(db, "SELECT * FROM MyTable")
var s2 *sql.Stmt = MustPrepare(db, "INSERT INTO MyTable(col1, col2) VALUES(?,?)")
var s3 *sql.Stmt = MustPrepare(db, "DELETE FROM MyTable WHERE col1=?")
Do you think, that pattern is usefull, or it cannot work at all.
In go you typicallly initialize a global *DB struct using Open (at least global in your Database Access package). That does not open an actual connection to the DB, but creates a connection pool. Therefore there should be only one instance of it. You can initialize that in init of your package.
See
http://go-database-sql.org/
or
https://www.vividcortex.com/blog/2015/01/14/the-ultimate-guide-to-building-database-driven-apps-with-go/
for a good introductory guide.
Yes it is a good approach. when you go through the go documentation it clearly tells you
It is rare to Close a DB, as the DB handle is meant to be long-lived
and shared between many go routines.
Go maintains its own pool of idle connections. Thus, the Open function should be called just once. It is rarely necessary to close a DB.
As a rule of thumb I don't think its good practice to use database connections this way, you should privatize it and just open/close as you need it :)
But if it works and you like it then nothings wrong with doing it that way.
I am finding the doc for CFStreamCreatePairWithSocketToCFHost confusing:
Specifically, its not clear to me how the function can set the readStream pointer to null on error.
as far as I understand, the pointer is passed by value - so the function can only change the objected pointed to by the pointer.
Right now I can't figure out how to detect connection errors.
Relevant doc snippet:
Creates readable and writable streams connected to a given CFHost object.
void CFStreamCreatePairWithSocketToCFHost (
CFAllocatorRef alloc,
CFHostRef host,
SInt32 port,
CFReadStreamRef *readStream,
CFWriteStreamRef *writeStream
);
readStream
Upon return, contains a CFReadStream object connected to the host host on port port, or NULL if there is a failure during creation. If you pass NULL, the function will not create a readable stream. Ownership follows the Create Rule.
This is my connecting code, it goes all the way to NSLog(#"Connected") even when the server is down.
NSLog(#"Attempting to (re)connect to %#:%d", m_host, m_port);
while(TRUE)
{
CFHostRef host = CFHostCreateWithName(kCFAllocatorDefault, (CFStringRef)m_host);
if (!host)
{
NSLog(#"Error resolving host %#", m_host);
[NSThread sleepForTimeInterval:5.0];
continue;
}
CFStreamCreatePairWithSocketToCFHost(kCFAllocatorDefault, host , m_port, &m_in, &m_out);
CFRelease(host);
if (!m_in)
{
NSLog(#"Error");
}
CFStreamClientContext context = {0, self,nil,nil,nil};
if (CFReadStreamSetClient(m_in, kCFStreamEventHasBytesAvailable | kCFStreamEventErrorOccurred | kCFStreamEventEndEncountered, networkReadEvent, &context))
{
CFReadStreamScheduleWithRunLoop(m_in, CFRunLoopGetCurrent(),kCFRunLoopCommonModes);
}
if (CFWriteStreamSetClient(m_out, kCFStreamEventErrorOccurred | kCFStreamEventEndEncountered, networkWriteEvent, &context))
{
CFWriteStreamScheduleWithRunLoop(m_out, CFRunLoopGetCurrent(),kCFRunLoopCommonModes);
}
BOOL success = CFReadStreamOpen(m_in);
CFErrorRef error = CFReadStreamCopyError(m_in);
if (!success || (error && CFErrorGetCode(error) != 0))
{
NSLog(#"Connect error %s : %d", CFErrorGetDomain(error), CFErrorGetCode(error));
[NSThread sleepForTimeInterval:5.0];
}
else
{
NSLog(#"Connected");
break;
}
}
From the "CFNetwork Programming Guide":
Opening a stream can be a lengthy process, so the CFReadStreamOpen and CFWriteStreamOpen functions avoid blocking by returning TRUE to
indicate that the process of opening the stream has begun. To check
the status of the open, call the functions CFReadStreamGetStatus and
CFWriteStreamGetStatus, which returnkCFStreamStatusOpening if the open
is still in progress, kCFStreamStatusOpen if the open is complete,
orkCFStreamStatusErrorOccurred if the open has completed but failed.
In most cases, it doesn’t matter whether the open is complete because
the CFStream functions that read and write will block until the stream
is open.
Also check out the kCFStreamEventOpenCompleted,
(http://developer.apple.com/library/ios/#documentation/CoreFoundation/Reference/CFStreamConstants/Reference/reference.html)
: a stream event that reports the successful completion of the opening
process. So to conclude, after calling CFReadStreamOpen (or Write),
which will probably succeed, register to listen to the "OpenCompleted"
event to identify a "real" success.
Surely after you call CFStreamCreatePairWithSocketToCFHost() just test readstream to see if it's NULL?
As you're passing in the memory location of the readstream pointer, the function can easily set that to whatever value it chooses (either a reference to a created object, or alternatively NULL).
Edit
I've tried your code, and I agree, it's very confusing. It appears that the CFReadStreamRef is readily created and opened, even for a nonsense host (I literally used "nonsense"). I don't believe this function will return NULL pointers for an unreachable host.
I suppose this makes sense, in as far as until one tries to open the stream, whether it will work or not is unknown.
So, the readStream param is a pointer to the CFReadStreamRef and, as such, can definitely be set to NULL by the function. &foo means "address of foo" and if you have the address you can set the value.
My reading of the documentation for CFStreamCreatePairWithSocketToCFHost is that they will be set to NULL on failure, but that failure is not about connection failure, but other kinds of failure (memory, etc). So not likely you'll get an error there.
Looks to me like the issue is that CFReadStreamOpen can return immediately with true when it can open the stream in the background and so this code is not really opening the stream or testing that it's been opened, merely queuing it for opening). From the documentation for CFReadStreamOpen:
" If the stream can open in the background without blocking, this function always returns true."
So I think you will need to follow the rest of the instructions for CFReadStreamOpen and schedule the stream on a run loop, or perhaps poll (though obviously polling in a tight loop isn't likely to work).
In the documentation for CFReadStreamOpen we see:
Opening a stream causes it to reserve all the system resources it requires. If the stream can open in the background without blocking, this function always returns true.
I suspect that the stream is opening in the background, and thus you are saying "Connected" before it actually opens. You've already scheduled the stream with a runloop, so if you let the run loop run, you'll probably get a callback with the event type set to kCFStreamEventErrorOccurred, and from there you can process the error appropriately.
I'm finding mixed answers to my question out in the web. To elaborate on the question:
Should I instantiate a service client proxy once per asynchronous invocation, or once per Silverlight app?
Should I close the service client proxy explicitly (as I do in my ASP.NET MVC application calling WCF services synchronously)?
I've found plenty of bloggers and forum posters out contradicting each other. Can anyone point to any definitive sources or evidence to answer this once and for all?
I've been using Silverlight with WCF since V2 (working with V4 now), and here's what I've found. In general, it works very well to open one client and just use that one client for all communications. And if you're not using the DuplexHttBinding, it also works fine to do just the opposite, to open a new connection each time and then close it when you're done. And because of how Microsoft has architected the WCF client in Silverlight, you're not going to see much performance difference between keeping one client open all the time vs. creating a new client with each request. (But if you're creating a new client with each request, make darned sure you're closing it as well.)
Now, if you're using the DuplexHttBinding, i.e., if you want to call methods on the client from the server, it's of course important that you don't close the client with each request. That's just common sense. However, what none of the documentation tells you, but which I've found to be absolutely critical, is that if you're using the DuplexHttBinding, you should only ever have one instance of the client open at once. Otherwise, you're going to run into all sorts of nasty timeout problems that are going to be really, really hard to troubleshoot. Your life will be dramatically easier if you just have one connection.
The way that I've enforced this in my own code is to run all my connections through a single static DataConnectionManager class that throws an Assert if I try to open a second connection before closing the first. A few snippets from that class:
private static int clientsOpen;
public static int ClientsOpen
{
get
{
return clientsOpen;
}
set
{
clientsOpen = value;
Debug.Assert(clientsOpen <= 1, "Bad things seem to happen when there's more than one open client.");
}
}
public static RoomServiceClient GetRoomServiceClient()
{
ClientsCreated++;
ClientsOpen++;
Logger.LogDebugMessage("Clients created: {0}; Clients open: {1}", ClientsCreated, ClientsOpen);
return new RoomServiceClient(GetDuplexHttpBinding(), GetDuplexHttpEndpoint());
}
public static void TryClientClose(RoomServiceClient client, bool waitForPendingCalls, Action<Exception> callback)
{
if (client != null && client.State != CommunicationState.Closed)
{
client.CloseCompleted += (sender, e) =>
{
ClientsClosed++;
ClientsOpen--;
Logger.LogDebugMessage("Clients closed: {0}; Clients open: {1}", ClientsClosed, ClientsOpen);
if (e.Error != null)
{
Logger.LogDebugMessage(e.Error.Message);
client.Abort();
}
closingIntentionally = false;
if (callback != null)
{
callback(e.Error);
}
};
closingIntentionally = true;
if (waitForPendingCalls)
{
WaitForPendingCalls(() => client.CloseAsync());
}
else
{
client.CloseAsync();
}
}
else
{
if (callback != null)
{
callback(null);
}
}
}
The annoying part, of course, is if you only have one connection, you need to trap for when that connection closes unintentionally and try to reopen it. And then you need to reinitialize all the callbacks that your different classes were registered to handle. It's not really all that difficult, but it's annoying to make sure it's done right. And of course, automated testing of that part is difficult if not impossible . . .
You should open your client per call and close it immediately after. If you in doubt browse using IE to a SVC file and look at the example they have there.
WCF have configuration settings that tells it how long it should wait for a call to return, my thinking is that when it does not complete in the allowed time the AsyncClose will close it. Therefore call client.AsyncClose().