I have an application that is deployed on tomcat on server A and sends queries to a huge variety of SQL Server databases on an server B.
I am concerned that my application could overload this SQL Server database server and would like some way to preventing it making requests to connect to any database on that server if some arbitrary number of connections were already in existence and unclosed.
I am looking at using connection pooling but am under the impression that this will only pool connections to a specific database on the SQL Server server, I want to control the total of these combined connections that will occur to many different databases (incidentally I can only find out the names of individual db's dynamically as they change day to day). Will connection pooling take care of this for me, are am I looking at this from the wrong perspective?
I have no access to the configuration of the SQL Server server.
Links to tutorials or working examples of your suggested solution are most welcome!
Have you considered using DBCP? See here for more info
You're correct. A database pool will limit connections to either the database, or to all databases, depending on how the pool is configured.
Here are several open source packages that implement database connection pools. One of them probably does what you want.
connection pooling ... this will only pool connections to a specific database on the mssql server
AFAIK, this is correct. Your connection pool will only limit the number of connections to the specific database it is defined for.
want to control the total of these combined connections that will occur to many different databases
I dont think you can control the number of connections to all databases from the pool. You've written you don't have access to change things on the MSSQL server. so creating SYNONYMs to the various databases on MSSQL itself is not an option.
You can write your own class within the application, say a ConnPoolManager, which has an internal counter prior to getting and releasing Connections from any of the pools.
This class should cache all the JNDI lookups to each pool.
For the app to get a connection to ANY pool, it goes through the ConnPoolManager and if the counter shows the maxlimit is not yet crossed, only then does it fetch the connection.
Else it throws some exception for you to try later.
There might be a design pattern for this on the lines of Business Delegate.
Having said that, I think a bigger problem for you will be
incidentally I can only find out the names of individual db's dynamically as they change day to day
since you will be expected to create new or edit the connection pool settings in Tomcat each day? This is a maintenance nightmare in the future.
Related
In traditional webservers you would have a SQL connection pool and persistent connection to the database.
But I am thinking of creating my entire application as Azure Functions.
Will the functions create a new connection the SQL server everytime its called upon?
Azure Functions doesn't currently have SQL as an option for an input or output binding, so you'd need to use the SqlClient classes directly to make your connections and issue your queries.
As long as you follow best practices of disposing your SQL connections (see this for example: C# SQLConnection pooling), you should get pooling by default.
Here's a full example of inserting records into SQL from a function: https://www.codeproject.com/articles/1110663/azure-functions-tutorial-sql-database
Although this is already answered, I believe this answer can provide more information.
If you are not using connection pool then probably you are creating connection every time function is invoked. Creating connection has associated cost, for warmed up instances it is recommended to use connection pool. max number of connection should also be chosen cautiously since there can be couple of parallel functions app running (as per plan).
This is example of connection pool.
There are some features in our LOB application that allow users to define their own queries to retrieve data for reports and listings within the app. The problem that we are encountering is that sometimes these queries they have written a really heavy (and sometimes erroneous) and cause massive load on the server.
Removing these features is out of the question but Im wanting to know if there is a way to create some type of sandbox within SQL server so that the queries that they execute are only allotted a certain amount of resources to execute therefore not giving them the chance to cause any damage to anyone else using the system. Any ideas?
The Resource governor has been mentioned in the comments above already. One other solution I can think of is using SQL Server High Availability Groups.
At the last place I worked had this kind of set up. There is a primary server which takes in all the transactions that write stuff to the database, with a secondary in case the primary fails. Added to this we also had read-only replicas added to the availability group.
The main purpose of this is in the event that your main server goes down you are automatically transferred to another replica. When you connect your application to the database, you connect it to the Availability Group rather than a specific server. Then if a server goes down you are automatically transferred to a secondary server instead. However, it can also be used to optimise application functionality that just needs read-only access by taking load off the primary server.
Any functionality that we knew that it only needed read-only access then we could connect to the availability group and add into the connection string ApplicationIntent=READONLY which means that we're using the read-only replica rather than the primary, leaving the primary for regular transactions. (IIRC, by default the primary will accept any read/write connection, so you have to configure the primary not to accept read-only connections)
Anyway, the kicking off point for reading up about this is here: https://msdn.microsoft.com/en-us/library/ms190202.aspx
The latest Windows 10 1903 upgrade already has inbuilt Sandbox features, where you can run SQL server within it's own sandbox. I don't think SQL Server itself has its own inbuilt sandbox environment, as it would be practically impossible to manage within a normal Windows server that is not using sandbox, if you know what I mean.
I am currently writing an application in VS2010 which will access many different SQL Servers spread out to a couple of servers on our network. This is however a dynamic environment, and the servers might be subject to decommissioning. I have a couple of entity data models which point to custom information-gathering databases in those servers, which will become useless to me when the servers decommission. The problem is that I am worried that if one of these servers decommission, my application would fail because the entity data models won't be able to point to the databases anymore. I cannot go like every 2 weeks to change the source code of the application to meet new server needs, as development time would be wasted.
Are my suspicions right, that my application would fail to work if the data models point to databases which may not exist anymore? Is there a workaround to cater for my needs to "ignore" a connection to a non-existent database?
You will get an exception when you try to do the first thing which connects to the DB.
The exception will note that the underlying provider failed on open, and will have a SqlException as the InnerException giving details of that.
Probably the best thing for you to do is to manually create and open the connection and pass that to the context in the constructor, using this overload.
SQL connection – I planned to create one sql connection object inside my own singleton class (say connectionmanager with Reference counting for ‘SQL connection object’ which is a member of the class) and open/close the connection whenever needed. As I like to give high importance for performance. And my application is a desktop based application accessing a remote database server (SQL server 2008), and will use only one connection string, 50 concurrent desktop users may access db server. Please advice.
As like many articles say if the answer is “Connection pooling is taken care by ado.net” – Does that mean that scope of connection pool is entire life of Application instance? Or is it in the scope of SQLConnection object?
Connection pooling is taken care by ado.net” – Does that mean that scope of connection pool is entire life of Application instance?
Yes. Read MSDN article - SQL Server Connection Pooling (ADO.NET)
Connections are pooled per process, per application domain, per
connection string and when integrated security is used, per Windows
identity. Connection strings must also be an exact match; keywords
supplied in a different order for the same connection will be pooled
separately.
The scope of the connection string. Change the string, a new connection pool is created.
We want to distribute / synchronize data from our Datawarehouse (MS SQL Server) to external customers (also MS SQL Server). The connection has to be secure, because we are dealing with trusted data. Transmission of data from our system to external client system must be via the http/https
In addition it is possible that the clients still run their systems with an older database schema, so already existing tables and columns should be transmitted and non existing ones should be ignored.
Its most likely that we will have large database updates and the updates have to arrive in almost real-time.
And it is definitely necessary that the data is stored in a client side datawarehouse / SQL database.
The whole process should also include good monitoring possibilities in case something goes wrong.
We started to develop our own .NET solution but I thought it should be almost a common problem to exchange data between different systems.
Does anybody know about an existing solution which we can adapt to our scenario?
Any help is appreciated!
The problem is so common that it has a dedicated component in SQL Server: Service Broker. Rather than start your own .Net thing and take care of the many problems (how are you gonna handle down time? Retries? duplicates? out of order delivery? authentication of non-domain joined computers? routing for machines that change names? service upgrades? transactional consistency, rollbacks? are you gonna use dtc?). You can look at the demo I gave to SQL connections to see how you can easily scale SSB to a throughput of well over 1000 msgs/sec (1k payload) on commodity hardware.
the only requirement is that all partitcipants must be at least SQL Server 2005 (no SSB in 2000).
Just use regular SQL connections over a secure VPN or an SSH tunnel. Should be very easy to setup for your networking guys.
For example, you can create a linked server. Then a SQL scheduled job could move the data:
truncate table targetserver.dbname.dbo.tablename
insert into targetserver.dbname.dbo.tablename
select a, b, c
from dbname.dbo.sourcetable
Since the linked server talks to your server over a VPN or SSH tunnel, all data is send encrypted over the internet.