I've been reading through the excellent Developing Multi-tenant Applications for the Cloud, 3rd Edition and am trying to understand how partitioning impacts on query performance in Windows Azure SQL Database.
If you use the shared schema approach and put all tenants records in a single table and separate their data using partitions, is that always going to be slower than a separate schema approach due to the larger number of records in the table, or does the partitioning effectively make each partition act like its own table?
(I appreciate query execution speed is only one of many factors to consider when choosing a multi tenancy strategy, we're not basing our decisions on performance alone.)
The approach that uses different schemas for different tenants has its problems, too. For instance, it is very easy to bloat the plan cache with this approach since each tenant gets its own set of query plans. You may end up with more recompiles (and lower performance) with this approach because of that.
I would recommend to take a look at an approach where you place each tenant in its own database. That provides for great isolation and, in combination with Elastic Database Pools in Azure SQL DB, it actually becomes quite affordable. A good entry point into the documentation is this: https://azure.microsoft.com/en-us/documentation/articles/sql-database-elastic-scale-introduction/.
Related
One of our customer is in manufacturing domain. He has multiple factories across the country. For the quality control, he is using window application deployed independently in all factories (approx 100 in count). Our customer is interested in replacing all the window applications with a single web application. Now the problem is volume of the data will be 100 times bigger and same as the velocity (in case we keep a single database for all the factories). There are lots of reporting use cases in this domain. Looking at the numbers, it looks like SQL will be not be able to handle this much load.
Is it a valid use case to move to NoSQL database?
Can Volume/Velocity alone be a deciding factor to move to NoSQL?
Would we be able to get all those reporting from NoSQL database efficiently?
Any kind of help will be appreciated.
Thanks In Advance
This is a usefull discussion.
In my opinion a well designed MS-SQL server 2012 (or Oracle server, but no experience for me) must be capable of handling 1000 complex transactions per second.
MS-SQL server 2014 with in-memory processing raises even higher expecations.
Consider multi processor, large memories, table partitioning, file mapping, multiple access paths to the SAN or to separate server discs. Use well designed transactions (consider to remove most indexes on transaction tables).
As an extra benefit you keep all functionality of the SQL server. In my opinion most NOSQL solutions are NOSQL because they are deprived of essential SQL functionality.
Switch to NOSQL databases is most usefull when you require functionality outside the transaction domain, e.g. document indexing or network indexing.
I've seen many entries about partitioning tables, but there is not a lot of information on when you should make a partition.
Is there a rule of thumb when you should partition tables in SQL Server.
Thanks
My benchmarks indicate that it depends on the query load.
If the queries you perform ALWAYS contain a filter on the partition field the performance benefit is virtually instant (like 1000 records in the table is already beneficial)
If the queries do NOT always contain a filter on the partition field you really have to benchmark with a good sample of the query load before making the decision.
You also have to account for the partition system you use. if you use "static" partitions there is not much harm in creating them immediately. When you use a "sliding window" system you need to take into account the overhead of creating and merging partitions. (which can take a long time on big tables)
#Filip's post is a great topical guide. When your doing your ontologies and estimating how your application will be used, that is, how your users will interact with the application and how that translates to database access, you should have a good idea of the kind of queries that will be performed and how fast certain tables will grow. If your that confident, then you should immediately partition the tables to defer from any maintenance hapzards.
But if your trying to decide on whether to partition populated tables, or you like to perform partitioning lazily like me, here's a nice little nugget from the PostgreSQL docs:
The exact point at which a table will benefit from partitioning depends on the application, although a rule of thumb is that the size of the table should exceed the physical memory of the database server. [src]
For each account, I have millions of data items (rows in analytics logs), each with 20-50 numeric properties (they can be null too). I need to show them stats which mostly involve queries like SELECT SUM(f1), f2, f3 WHERE f4>f5 GROUP BY f2, f3. The aggregation functions are sometimes more complex than SUM(), and GROUP BY sometimes involves simple functions like ROUND(). The problem is that such queries are built in the user interface and can be run on any combination of those properties (though there are some popular combinations of course).
Once in the database, the data would most likely not be modified, only read. It should be possible to easily add/remove properties – not necessarily realtime in database terms, but it should not require complete table blocks like in MySQL.
What SQL or NoSQL databases would be best to handle these kinds of queries? I was thinking of PostgreSQL or MongoDB, even though in the latter I will most likely have to use MapReduce rather than its Group feature because of its limitations.
Any other advice on performance of such queries? Does this sound possible to do at all, or do I absolutely have to ask users to pre-define which exact queries they want to run?
Any ideas would be much appreciated.
What query performance are you looking for? How often will it be queried?
If you're OK with query performance in the low minutes and have a similarly low query rate, then you can use a relational table with a main table for the data items, and a join table for the properties. Be sure to put a combined index on the second table on the combination (property_type, data_item_id, property_value) to guarantee good query performance. You don't actually need property_value in there, but if you have it then queries can pull their data from the index in a highly efficient manner, which will make joins much, much easier. You can do this with any relational database. I happen to like PostgreSQL, but MySQL can also work. (But less efficiently on complex queries.)
If you follow this strategy then each property you want will require you to add yet another join. But the joins will be fairly efficient.
You can build this kind of application in an RDBMS or in a NoSQL database (Berkeley DB for example, has both a key-value pair API and a SQL API). The key-value pair API is a nice option, since it supports some pretty low level optimizations that may help when looking at how to tune the performance to meet your application needs.
Another option is to look into a columnar data store, but even that kind of product is going to have to retrieve data from multiple columns (which is slow in these kinds of databases) in order to resolve the kinds of queries that you list.
Ultimately the issue here boils down to disk I/O VS cache and data organization. The more data that you can fit into memory, the less I/O you have to perform and I/O is going to be the performance killer. The more compact you can make the data, the more rows will fit in the memory that you have. I would suggest looking into Berkeley DB, especially the key-value pair API. You can then choose to create one or more tables with the properties organized in an manner that optimizes the most frequent kinds of access. Additionally, if you're using the key-value pair API, take a look at the Bulk Get functions -- this allows you to fetch and process whole groups of records at a time.
You may also want to create and maintain some "well known" statistical results (in memory and/or persisted on disk) that allow you to take "shortcuts" when the user is asking for a value that has already been computed.
Good luck in your research.
What you've described - essentially ad hoc aggregate queries on data that does not need to be realtime - is what OLAP solutions are very good at. In addition to other suggestions you've seen, you should look into whether an OLAP solution makes sense for you.
I'm a beginner. How many number of tables are recommended in a SQL Server Express database? Mainly attaining best performance speedwise as an objective. Is it generally recommend to use two databases as compared to one for a single application?
SQL Express databases have a limit of 4GB in size. Within that limit, any number of tables is fair game. The number of tables makes absolutely no impact on performance. The only thing that drives performance of the application vis-a-vis the database is the proper design of the tables, both as logical model and as physical database structure (ie. correct choices of clustered indexes, non-clustered indexes, constraints, defaults, data types etc), and the proper querying and updating of the database ie. queries that can be satisfied (covered, efficiently) by the existing indexes.
Splitting an application database into multiple distinct databases is a bad idea. You are loosing consistency of the recovery unit (you can't backup/restore the two databases in a consistent state) and you need to replicate all the infrastructure around the database twice (security roles and permissions, maintenance activities and procedures etc). Also spliting an application database into distinct databases gaves absolutely no performance advantage.
What you can do to make things speedier:
-break up your databases so that they use multiple files across multiple, fast drives
-federate (not really something you'll do if you're running Express)
-Install memory, memory, memory
-I can't remember Express's limitations and I don't care to look them up, but on the configuration screen where you can assign the number of CPUs to dedicate to SQL, give it as many as you can. You should also be able to set affinity there (if not, then in Task Manager)
Don't run anything you don't need (scheduler, report engine, Server, DHCP Client) if you don't have to
In a recent project the "lead" developer designed a database schema where "larger" tables would be split across two separate databases with a view on the main database which would union the two separate database-tables together. The main database is what the application was driven off of so these tables looked and felt like ordinary tables (except some quirky things around updating). This seemed like a HUGE performance problem. We do see problems with performance around these tables but nothing to make him change his mind about his design. Just wondering what is the best way to do this, or if it is even worth doing?
I don't think that you are really going to gain anything by partitioning the table across multiple databases in a single server. All you have essentially done there is increased the overhead in working with the "table" in the first place by having several instances (i.e. open in two different DBs) of it under a single SQL Server instance.
How large of a dataset do you have? I have a client with a 6 million row table in SQL Server that contains 2 years worth of sales data. They use it transactionally and for reporting without any noticiable speed problems.
Tuning the indexes and choosing the correct clustered index is crucial to performance of course.
If your dataset is really large and you are looking to partition, you will get more bang for your buck partitioning the table across physical servers.
Partitioning is not something to be undertaken lightly as there can be many subtle performance implications.
My first question is are you referring simply to placing larger table objects in separate filegroups (on separate spindles) or are you referring to data partitioning inside of a table object?
I suspect that the situation described is an attempt to have the physical storage of certain large tables on different spindles from the rest of the tables. In this case, adding the extra overhead of separate databases, losing any ability to enforce referential integrity across databases, and the security implications of enabling cross-database ownership chaining does not provide any benefit over using multiple filegroups within a single database. If, as is quite possible, the separate databases you refer to in your question are not even stored on separate spindles but are all stored on the same spindle then you negate even the slight performance benefit you could have gained by physically separating your disk activity and have received absolutely no benefit.
I would suggest instead of using additional databases to hold large tables you look into the Filegroup topic in SQL Server Books Online or for a quick review see this article:
If you are interested in data partitioning (including partitioning into multiple file groups) then I recommend reading articles by Kimberly Tripp, who gave an excellent presentation at the time SQL Server 2005 came out about the improvements available there. A good place to start is this whitepaper
Which version of SQL Server are you using? SQL Server 2005 has partitioned tables, but in 2000 (or 7.0) you needed to use partition views.
Also, what was the reasoning for putting the table partitions in a separate database?
When I've had to partition tables in the past (pre-2005), it's usually by a date column or something similar, with a view over the various partitions. Books Online has a section that talks about how to do this and all of the rules around it. You need to follow the rules to make it work how it's supposed to work.
The key thing to remember is that your partitioning column must be part of the primary key and you want to try to always use that column in any access against the table so that the optimizer can ignore partitions that shouldn't be affected by the query.
Look up "partitioned table" in MSDN and you should be able to find a more complete tutorial for SQL Server 2005 partitioned tables as well as advice on how to set them up for maximum performance.
Are you asking about best practices in terms of database design, or convincing your lead to change his mind? :)
In terms of design... Back in the goode olde days, vertical partitioning was sometimes needed to work around database engine limitations, where the number of columns in a table was a hard limit, like 255 columns. These days the main benefits are purely for performance: putting rarely used columns, or blobs on a separate disk array. But if you're regularly pulling things from both tables it will likely be a loss. It sounds like your lead is suffering from a case of premature optimisation.
In terms of telling your lead is wrong... that requires diplomacy. If he's aware of mutterings of discontent in terms of performance, a benchmark is probably the best way to show the difference.
Create a new physical table somewhere with 'create table t1 as select * from view1' and then run some lengthy batch with the vertically partitioned table and your new table. If it's as bad as you say, the difference should be evident.
But this too may be premature optimisation. Find out what the end-users think of the performance. If the performance is good enough, for some definition of good, then don't fix what ain't broke.
There is a definite benefit for table partitioning (regardless whether it's on same or different filegroups /disks). If the partition column is correctly selected, you'll realize that your queries will hit only the required partition. So imagine if you have 100 million records (I've partitioned tables much bigger than that - about 20+ Billion rows) and if for the most part, more than 70% of your data access is only a certain category or timeline or type of data then it helps to keep the most accessed data in a separate partition. Plus you can align the partition with separate file groups with various type of disks (SATA, Fiber channel, SSDs) so that the most accessed/busy data are on the fastest storage and the least/rarely accessed are virtually on slower disks.
Although, in SQL Server there's limited partitioning ability, unlike Oracle. You can choose only one column for partitioning (even in SQL 2008). So you've to choose a column wisely where that column also is part of most of your frequent queries. For the most part, people find it easy to choose to partition by a date column. However although it seems logical to partition that way, if your queries do not have that column as part of the condition, you won't be gaining sufficient benefits from partitioning (in other words, your query will hit all the partition regardless).
It's much easier to partition for data warehouse/data mining type databases than OLTP as most DW database queries are limited by time period.
That's why these days due to the volume of data being handled by databases, it's wise to design the application in such a way that ever query is limited by some broader group such as time, geographical location or such so that when such columns are chosen for partitioning you'll gain maximum benefits.
I would disagree with the assumption that nothing can be gained by partitioning.
If the partition data is physically and logically aligned, then the potential IO of queries should be dramatically reduced.
For example, We have a table which has the batch field as an INT representing an INT.
If we partition the data by this field and then re-run a query for a particular batch, we should be able to run set statistics io ON before and after partitioning and see a reduction in IO,
If we have a million rows per partition and each partition is written to a separate device. The query should be able to eliminate the nonessential partitions.
I've not done a lot of partitioning on SQL Server, but I do have experience of partitioning on Sybase ASE, and this is known as partition eliminiation. When I have time I'm going to test out the scenario on a SQL Server 2005 machine.