Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I'm execution the same query in two different environments
The first environment has 4GB RAM and an Intel processor 3,09 Ghz.
The second environment has 32GB RAM and an AMD processor 2,20 Ghz.
I'm wondering why the query is taking 6 minutes to be executed in the first environment and many many hours in the second one.
I checked the momory allocated for sql server by exectuting the query below, the two environments have the same value.
SELECT value_in_use
FROM sys.configurations
WHERE name = 'max server memory (MB)'
What could be the reason why the query is taking too much time in the second environment?
PS : The number of rows is the same in both environments (about 2 millions)
Several factors can make the results different. I hesitate to mention them because they are the kind of things people tend to reject, but I have experienced them all and had some confirmed by msft.
First, the person who suggested you look at execution plans is on the right track. That is likely to at least give you a clue as to what's different.
Reasons (assuming size and content of data are identical)
Statistics are different on the two servers, causing different execution plans.
Hardware performance is different. Slower or faster CPUs (different generation for example, even if clock speed is close), slower or faster disk. More cores vs. one core (leads to parallel plans vs. serial plans)
Hardware configuration is different (e.g. perormance difference). One uses a SAN one has direct storage. Sometimes matters a lot. Sometimes makes no difference once data is in cache.
Data storage is physically different. One server has greatly fragmented and sparse data blocks/pages in the table(s) or indexes you care about. One has compact and fast. This can occur due to different scenarios used to load data in the two systems.
SQLServer configuration settings: Limited Memory configuration on one system, for example.
Competing workload on one of the systems.
Missing or different indexes.
Different collation settings resulting in different index statistics and different plans.
Slightly different software version.
Size of data is about the same but content is different, changing query plan.
etc.
The time difference involved suggests a different execution plan is most likely.
If all else (indexes, etc.) is 'the same', sometimes it is worth forcing SQLServer to rerun statistics, then try the query again. Historically SQLServer has automatically updated statistics based on the volume of changes to data in a table. Sometimes this leads to statistics that are temporarily bad, as one table hits the threshold for an update but other tables do not, and the optimizer chooses bad plans. It all seems very improbable, but I once built a product that hit this problem regularly, and was fortunate enough to have a visiting MSFT SQLServer developer help me prove it.
Another that has really hit me is slow san or slow VMWare environment. Sans are often claimed to be fast, but turn out to deliver terrible performance to actual users due to slow network connections or competing workloads. VMWare or other virtualization environment often leads to the same problem, especially in large organizations that don't want to figure out what workload is hitting their clusters. Someone else's workload affects yours. So performance testing loses meaning.
My money is on some factor that changes the plan, like data content, statistics, index configuration.
Related
A basic production level database in Heroku implements a 400Mb cache. I have a production site running 2 dynos and a worker which is pretty heavy on reads and writes. The database is the bottleneck in my app.
A write to the database will invalidate many queries, as searches are performed across the database.
My question is, given the large jump in price between the $9 starter and $50 first level production database, would migrating be likely to give a significant performance improvement?
"Faster" is an odd metric here. This implies something like CPU, but CPU isn't always a huge factor in databases, especially if you're not doing heavy writes. You Basic database has 0mb of cache – every query hits disk. Even a 400mb cache will seem amazing compared to this. Examine your approximate dataset size; a general rule of thumb is for your dataset to fit into cache. Postgres will manage this cache itself, and optimize for the most referenced data.
Ultimately, Heroku Postgres doesn't sell raw performance. The benefits of the Production-tier are multiple, but to name a few: In-memory Cache, Fork/Follow support, 500 available connections, 99.95% expected uptime.
You will definitely see performance boost in upgrading to a Production-tier plan, however it's near impossible to claim it to be "3x faster" or similar, as this is dependent on how you're using the database.
It sure is a steep step, so the question really is how bad is the bottleneck? It will cost you 40 dollar extra, but once your app runs smooth again it could also mean more revenue. Of course you could also consider other hosting services, but personally I like Heroku the best (albeit cheaper options are available). Besides, you are already familiar with Heroku. There is some more information on Heroku devcenter, regarding their different plans:
https://devcenter.heroku.com/articles/heroku-postgres-plans:
Production plans
Non-production applications, or applications with minimal data storage, performance or availability requirements can choose between one of the two starter tier plans, dev and basic, depending on row requirements. However, production applications, or apps that require the features of a production tier database plan, have a variety of plans to choose from. These plans vary primarily by the size of their in-memory data cache.
Cache size
Each production tier plan’s cache size constitutes the total amount of RAM given to Postgres. While a small amount of RAM is used for managing each connection and other tasks, Postgres will take advantage of almost all this RAM for its cache.
Postgres constantly manages the cache of your data: rows you’ve written, indexes you’ve made, and metadata Postgres keeps. When the data needed for a query is entirely in that cache, performance is very fast. Queries made from cached data are often 100-1000x faster than from the full data set.
Well engineered, high performance web applications will have 99% or more of their queries be served from cache.
Conversely, having to fall back to disk is at least an order of magnitude slower. Additionally, columns with large data types (e.g. large text columns) are stored out-of-line via TOAST, and accessing large amounts of TOASTed data can be slow.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
I have developed a website which provides very generic data storage. Currently it works just fine but I am thinking about optimizing the speed.
INSERT/SELECT ratio is hard to predict and changes for different cases but usually SELECT is more often. INSERTs are fast enough. SELECTs are what worries me. There are a lot of LEFT JOINs. E.g. each object can have a image which is stored in separate table (as it can span across multiple objects) and stores additional information about the image as well.
Up to 8 joins are made every select and it can take up to 1 seconds to process - mean value is around 0.3s. There can be multiple of such selects for every request. It has already been optimized multiple times on SQL side and there is not much that can be done there.
Other than buying more powerful machine for DB, what can be done (if anything)?
Django is not a speed demon here as well but we still got some optimizations left there. Switch to PyPy if we must. On DB side I had a few ideas but there they seem to be uncommon - couldn't find any real case scenario.
Use different storage for this part that's faster. We need transactions and we need consistency checks so it may not be preferable.
Searchable cache? Does it make any sense here? E.g. maintain a flat copy of all tables combined in NoSQL or something. Inserts would be more expensive - it needs to update multiple records in NoSQL if some common table changes. Tough to maintain as well.
Is there anything that would make sense or is it just the fastest that can get and just get more RAM, increase cache size in rdbms, get SSD and leave it. Focus on optimizing other parts like pooling database connections as they are expensive as well.
Technologies used: PostgreSQL 9.1 and Django (python).
To summarize. Question is: after optimizing all SQL part - indexes, clustering etc. What can be done to optimize further when static timeout cache for results is not an option (different request arguments, different results anyway).
---EDIT 30-08-2012---
We are already using checking slow queries on a daily basis. This IS our bottleneck. We only order and filter on indexes. Also, sorry for not being clear about this - we don't store actual images in db. Just file paths.
JOINs and ORDER BY are killing our performance here. E.g. one complex query that spits out 20 000 results takes 1800ms (EXPLAIN ANALYZE used). And this assumes that we are not using any kind of filtering based on JOINed tables.
If we skip all the JOINS we are down to 110ms. That's insane... That's why we are thinking of some kind of searchable cache or flat copy NoSQL.
Without ordering we got 60ms which is great but what's with the JOIN performance in PostgreSQL?
Is there some different DB that can do better for us? Preferably free one.
First, although I think that there are times and places to store image files in the database, in general you are going to have extra I/O and memory associated with this sort of operation. If I was looking at optimizing this I would put every image with a path and be able to bulk save these to the fs. This way they are still in your db for backup purposes but you can just pull the relative path out and generate links, thus saving you a bunch of sql queries and reducing overhead. Over a web-based backend you aren't going to be able to get transactions working really well between generating the HTML and retrieving the image anyway since these come in under different HTTP requests.
As for speed, I can't tell if you are looking at total http request time or db time. But the first thing you need to do is break everything apart and look for where most of your time is being spent. This may surprise you. The next thing is to get query plans of those queries which are slow queries:
http://heatware.net/databases/how-to-find-log-slow-queries-postgresql/
Then from there, start using explain analyze to find out what is the problem.
Also in deciding to upgrade hardware you want to have a good idea of where you are currently facing limits. More RAM helps generally (and it is helpful if your db can fit comfortably in RAM), but beyond that it makes no sense to put faster storage in a cpu-bound server or switch to a server with faster cpu's in an I/O bound server. top is your friend there. Similarly depending on the concurrency issues, it might (or might not!) make sense to use a hot standby for your select statements.
But without a lot more information I can't tell you what the best way to go about further optimizing your db is. PostgreSQL is capable of running really fast under the right conditions and scaling very well.
I am seeking a way to find bottlenecks in SQL server and it seems that more than 32GB ram and more than 32 spindels on 8 cores are not enough. Are there any metrics, best practices or HW comparations (i.e. transactions per sec)? Our daily closure takes hours and I want it in minutes or realtime if possible. I was not able to merge more than 12k rows/sec. For now, I had to split the traffic to more than one server, but is it a proper solution for ~50GB database?
Merge is enclosed in SP and keeped as simple as it can be - deduplicate input, insert new rows, update existing rows. I found that the more rows we put into single merge the more rows per sec we get. Application server runs in more threads, and uses all the memory and processor on its dedicated server.
Follow a methodology like Waits and Queues to identify the bottlenecks. That's exactly what is designed for. Once you identified the bottleneck, you can also judge whether is a hardware provisioning and calibration issue (and if so, which hardware is the bottleneck), or if is something else.
The basic idea is to avoid having to do random access to a disk, both reading and writing. Without doing any analysis, a 50 GB database needs at least 50GB of ram. Then you have to make sure indexes are on a separate spindle from the data and the transaction logs, you write as late as possible, and critical tables are split over multiple spindles. Are you doing all that?
I have a problem with a large database I am working with which resides on a single drive - this Database contains around a dozen tables with the two main ones are around 1GB each which cannot be made smaller. My problem is the disk queue for the database drive is around 96% to 100% even when the website that uses the DB is idle. What optimisation could be done or what is the source of the problem the DB on Disk is 16GB in total and almost all the data is required - transactions data, customer information and stock details.
What are the reasons why the disk queue is always high no matter the website traffic?
What can be done to help improve performance on a database this size?
Any suggestions would be appreciated!
The database is an MS SQL 2000 Database running on Windows Server 2003 and as stated 16GB in size (Data File on Disk size).
Thanks
Well, how much memory do you have on the machine? If you can't store the pages in memory, SQL Server is going to have to go to the disk to get it's information. If your memory is low, you might want to consider upgrading it.
Since the database is so big, you might want to consider adding two separate physical drives and then putting the transaction log on one drive and partitioning some of the other tables onto the other drive (you have to do some analysis to see what the best split between tables is).
In doing this, you are allowing IO accesses to occur in parallel, instead of in serial, which should give you some more performance from your DB.
Before buying more disks and shifting things around, you might also update statistics and check your queries - if you are doing lots of table scans and so forth you will be creating unnecessary work for the hardware.
Your database isn't that big after all - I'd first look at tuning your queries. Have you profiled what sort of queries are hitting the database?
If you disk activity is that high while your site is idle, I would look for other processes that might be running that could be affecting it. For example, are you sure there aren't any scheduled backups running? Especially with a large db, these could be running for a long time.
As Mike W pointed out, there is usually a lot you can do with query optimization with existing hardware. Isolate your slow-running queries and find ways to optimize them first. In one of our applications, we spent literally 2 months doing this and managed to improve the performance of the application, and the hardware utilization, dramatically.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 2 years ago.
Improve this question
What optimization techniques do you use on extremely large databases? If our estimations are correct, our application will have billions of records stored in the db (MS SQL Server 2005), mostly logs that will be used for statistics. The data contains numbers (mostly integer) and text (error message texts, URLs) alike.
I am interested in ANY kind of tips, hacks, solutions.
The question is a little big vague, but here are a few tips:
Use appropriate hardware for your databases. I'd opt for 64-bit OS as well.
Have dedicated machines for the DBs. Use fast disks configured for optimal performance. The more disks you can span over, the better the performance.
Optimize the DB for the type of queries that will be performed. What happens more SELECTs or INSERTs?
Does the load happens for the entire day, or for just few hours? Can you postpone some of the things to be run for the night?
Have incremental backups.
If you'll consider Oracle instead of SQL Server, you could use features such as Grid and Table Partitioning, which might boost performance considerably.
Consider having some load-balancing solution between the DB servers.
Pre-design the schemes and tables, so queries will be performed as fast as possible. Consider the appropriate indexes as well.
You're gonna have to be more specific about the way you're going to store those logs. Are they LOBs in the DB? Simple text records?
I don't use it myself but I have read that one can use Hadoop in combination with hbase for distributed storage and distributed analysing of data like logs.
duncan's link has a good set of tips. Here are a few more tips:
If you do not need to query against totally up-to-date data (i.e. if data up to the last hour or close of business yesterday is acceptable), consider building a separate data mart for the analytics. This allows you to optimise this for fast analytic queries.
The SQL Server query optimiser has a star transformation operator. If the query optimiser recongises this type of query it can select what slice of data you want by filtering based on the dimension tables before it touches the fact table. This reduces the amount of I/O needed for the query.
For VLDB applications involving large table scans, consider direct attach storage with as many controllers as possible rather than a SAN. You can get more bandwidth cheaper. However, if your data set is less than (say) 1TB or so it probably won't make a great deal of difference.
A 64-bit server with lots of RAM is good for caching if you have locality of reference in your query accesses. However, a table scan has no locality of reference so once it gets significantly bigger than the RAM on your server extra memory doesn't help so much.
If you partition your fact tables, consider putting each partition on a sepaarate disk array - or at least a separate SAS or SCSI channel if you have SAS arrays with port replication. Note that this will only make a difference if you routinely do queries across multiple partitions.