I have been struggling with a problem that only happens when the database has been idle for a period of time for the data queried. The first query will be extremely slow, on the order of 30 seconds and then related queries will be fast like 0.1 seconds. I am assuming this is related to caching, but I have been unable to find the cause of it.
Changing the mysql variables tmp_table_size, max_heap_table_size to a larger size had no effect except to create the temp tables in memory.
I do not think this is related to the query itself as it is well indexed and after the first slow query, variants of the same query do not show up in the slow query log. I am most interested in trying to determine the cause of this or a way to reset the offending cache so I can troubleshoot the issue.
Pages of the innodb data files get cached in the innodb buffer pool. This is what you'd expect. Reading files is slow, even on good hard drives, especially random reads which is mostly what databases see.
It may be that your first query is doing some kind of table scan which pulls a lot of pages into the buffer pool, then accessing them is fast. Or something similar.
This is what I'd expect.
Ideally, use the same engine for all tables (exceptions: system tables, temporary tables (perhaps) and very small tables or short-lived ones). If you don't do this then they have to fight for ram.
Assuming all your tables are innodb, make the buffer pool use up to 75% of the server's physical ram (assuming you don't run too many other tasks on the machine).
Then you will be able to fit around 12G of your database into ram, so once it's "warmed up", the "most used" 12G of your database will be in ram, where accessing it is nice and fast.
Some users of mysql tend to "warm up" production servers following a restart by sending them queries copied from another machine for a while (these will be replication slaves) until they add them into their production pool. This avoids the extreme slowness seen while the cache is cold. For example, Youtube does this (or at least it used to; Google bought them and they may now use Google-fu)
MySQL Workbench:
The below isn't 100% related to this SO question, but the symptoms are very related and this is the first SO result when searching for "mysql workbench slow" or related terms, so hopefully it's useful for others.
Clear the query history! - following the process at MySql workbench query history ( last executed query / queries ) i.e. create / alter table, select, insert update queries to clear MySQL Workbench's query history really sped up the program for me.
In summary: change the Output pane to History Output, right click on a Date and select Delete All Logs.
The issue I was experiencing was "slow first query" in that it would take a few seconds to load the results even though the duration/fetch were well under 1 second. After clearing my query history, the duration/fetch times stayed the same (well under 1 second, so no DB behavior actually changed), but now the results loaded instantly rather than after a few second delay.
Is anything else running on your mysql server? My thought is that maybe after the first query, your table is still cached in memory. Once it's idle, another process is causing it to be de-cached. Just a guess though.
How much memory do you have any what else is running?
I had an SSIS package that was timing out. The query was very simple, from a single MySQL table, but it sometimes returned a lot of records and would sometimes take a few minutes initially to execute, then only a few milliseconds afterwards if I wanted to query it again. We were stuck with the ADO connection, which meant it would time out after 30 seconds, so about half the databases we were trying to load were failing.
After beating my head against the wall I tried performing an initial query first; very simple and only returning a few rows. Since it was so simple it executed fast and set the table in the cache for faster querying. In the next step of the package I would do the more complex query which returned the large data set that kept timing out. Problem solved - all tables loaded. I may start doing this on a regular basis, the complex queries execute much faster by doing a simple query first.
Ttry and compare the output of "vmstat 1" on the linux command line when running the query after a period of time, vs when you re-run it and get results fast. Specifically check the "bi" column (that's the kb read from disk per second).
You may find the operating system is caching the disk blocks in the fast case (and thus a low "bi" figure), but not in the slow case (and hence a large "bi" figure).
You might also find that vmstat shows high/low cpu usage in either case. If it's low when fast, and disk throughput is also low, then your system may still be returning a cached query, even though you've indicated the relevant config value is set to zero. Perhaps check the output of show engine innodb status and SHOW VARIABLES and confirm.
innodb_buffer_pool_size may also be set high (it should be...), which would cache the blocks even before the OS can return them.
You might also find that "key_buffer" is set high - this would cache the keys in the indexes, which could make your select blindingly fast.
Try check the mysql performance blog site for lots of useful info.
I had issue when MySQL 5.6 was slow on first query after idle period. This was a connection problem, not MySQL instance problem, e.g. if you run MYSQL Query Browser execute "select * from some_queue", leave it alone for a couple of hours, then execute any query, it runs slow, while at the same time processes on server or new instance of Browser will select from same tables instantly.
Adding skip-host-cache, skip-name-resolve to my.ini file solved this problem.
I don't know why is that. Why I tried this: MySQL 5.1 without those options was slowly establishing connections from other networks (e.g. server is 192.168.1.100, 192.168.1.101 connects fast, 192.168.2.100 connects slow), MySQL 5.6 didn't have such problem to start with so we didn't add these to my.ini initially.
UPD: Solved half the cases, actually. Setting wait_timeout to maximum integer fixed the other half. Maybe I even now can remove skip-host-cache, skip-name-resolve and it won't slow down in 100% of the cases
Related
As BigQuery is a shared resource, it is possible that one gets different values running the same code on BigQuery. OK one option that I always use is to turn off caching in Query Settings, Cache preference. This way queries will not be cached. The problem with this setting is that if you refresh the browser or leave it idle, that Cache Preference box will be ticked again.
Anyhow I had a discussion with some developers that are optimizing the code. In a nutshell, they take slow running code, run it 5 times and get an average, then following optimization then run the code again 5 times to get an average value for optimized SQL. Details are not clear to me. However, my preference would be (all in BQ console)
create a user session
turn off sql caching
On BQ console paste the slow running code;
On the same session paste the optimized code
Run the codes (separated by ";")
This will ensure that any systematics like BQ busy/overloaded, slow connection etc will affect "BOTH" SQL piece equally and the systematics will be cancelled out. In my option one only need to run it once as caching is turned off as well. Running 5 times to get an average looks excessive and superfluous?
Appreciate any suggestions/feedback
Thanks
Measuring the time is one way, the other way to see if the query has been optimized is the understanding of the query plan and how slots are used effectively.
I've been with BigQuery more than 6 years, and what you describe was never used by me. In BigQuery actually what matters is reducing the costs, and that can be done iteratively rewriting the query, and using partitioning/clustering/materialized views, caching/temporary tables.
I have a messages table with a few million records in it. My Rails app includes a query on most pages to count the number of unread messages to show the user. This query - and all queries of the messages table - is unchanged and was working fine until yesterday.
Yesterday, I created a new messages column and ran update_all to update the new column on each of my few million messages records. Simple enough, and I've done this plenty of times before, albeit on a smaller number of records.
However, now every query to COUNT or SELECT messages takes 30+ seconds to return. Before the migration and update_all, it was only taking 100ms or so to do the COUNTs or SELECTs.
I've tried a number of things, but the queries remain super slow. I've tried to REINDEX messages and I've tried to VACCUM messages but neither helped significantly.
The rest of the database tables remain working as usual, but this issue with the messages table is keeping everything running at a crawl. Does anyone have any ideas about why this happening, and what else I can try to fix it?
I've tried to REINDEX messages and I've tried to VACCUM messages but neither helped significantly.
You missed the important ANALYZE. Possibly in the form of VACUUM ANALYZE. (Typically autovacuum should take care of this, though!)
Updating all rows creates as many dead tuples in the physical relation. To free up space, it might be the rare occasion for a
VACUUM FULL ANALYZE messages;
Takes an exclusive lock on the table! And since you run counts per user a lot, you might as well use CLUSTER:
CLUSTER messages USING that_index_sorting_by_user_id_and_the_read_flag
For fast counts per user, that appropriate (multicolumn) index is key either way. Depends on undisclosed schema & queries.
Al that said, you shouldn't see such a dramatic drop in performance. Maybe you exhausted a resource. Like, just not enough RAM to keep the bigger indexes cached.
I run heavy query on IBM i. First time it takes a long time, Subsequent times are much faster. It seems to be creating temporary index. How can I remove this index, so I can re-test like the first time?
Use the Visual Explain (VE) tool in the Run SQL Scripts component of ACS to see the differences between runs.
If indeed the issue is a system maintained temporary index (MTI), you can track it down via the schema's tooling in ACS and delete it if you so desire.
However, an MTI only gets deleted by the system when the system reboots (IPL).
So if you seeing differences without rebooting the server, I suspect the differences are caused by psuedo-closing. By default, once the DB see's the same query a few times (3 is the default), instead of hard closing it's cursors, it will psuedo-close them.
Again, VE will show "hard opens" and "pseudo opens".
To get the pseduo closed cursors to hard close, simply disconnect and reconnect.
I have a query that never finishes (takes more that 2 hours) when I run it at 11am, and takes 1 min when I run it at 7pm. So obviously the execution plan changes during the day. What can cause that?
The data does not change. Auto stats are turned off, the stats and indexes get rebuild over night.
I run it in the client's environment, so I can't really tell what else uses the same SQL server. So this is my main concern, I reckon that the server gets so loaded by something else that it can't cope with my query.
At what parameters of the server shall I look to find the cause of the problem?
Where else should I look for a problem? What other factors shall I consider?
Any help will be highly appreciated.
UPDATE: Forgot to mention that I have a whole bunch of other big queries running at the same time (11am) and they all run fine. And actually I have 7 queries that run fine in the evening and don't finish in the morning. They all use the same view which joins quite large tables and that is the only difference between the queries that fail and the queries that don't. So I wonder if SQL server does not have enough memory or something to save the intermediate results of the view execution, and that is why the query never finishes.
So what parameters of the server shall I monitor to find the problem?
UPDATE And unfortunately I don't have the execution plan of the morning run due to the permissions
UPDATE I don't think the the table locks is the cause as I know what's using my database on the server and I can run the same query at say 12pm when nothing else is running from my side (i.e. should be no locks and uncommitted transactions on my tables) and the query takes the same awful amount of time.
Lots of things can impact this. Is this really a query or a stored procedure call? I query is re compiled each call a stored proc uses a cached plan. If the parameters for the stored procedure can provide wildly varying results then you can use the WITH RECOMPILE hint
If you can live with dirty reads I would put:
SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED
at the top of your code. This will allow your query to access data being held in locks by other processes. The risk is that if the data changes your answer may not be 100% correct. If your data is always additive or a point in time value is acceptable then this works really well.
I'd go through the execution plan of the query in the evening and optimise that query even if 1 min is acceptable it may be uploading vast amounts of data which you have capacity for in the evening but are contending for in the morning.
I recently moved a bunch of tables from an existing database into a new database due to the database getting rather large. After doing so I noticed a dramatic decrease in performance of my queries when running against the new database.
The process I took to recreate the new database is this:
Generate Table CREATE scripts using sql servers automatic script
generator.
Run the create table scripts
Insert all data into new database
using INSERT INTO with a select from
the existing database.
Run all the alter scripts to create
the foreign keys and any indexes
Does anyone have any ideas of possible problems with my process, or some key step I'm missing that is causing this performance issue?
Thanks.
first I would an a mimimum make sure that auto create statistics is enabled you can also set auto update statistics to true
after that I would update the stats by running
sp_updatestats
or
UPDATE STATISTICS
Also realize that the first time you hit the queries it will be slower because nothing will be cached in RAM. On the second hit should be much faster
Did you script the indexes from the tables in the original database? Missing indexes could certainly account for poor performance.
Have you tried looking at the execution plans on each server when running these queries - that should allow you to easily see if they're doing something different e.g. table scanning due to a missing index, poor statistics etc.
Are both DBs sat on the same box with their data files on the same drive arrays?
Can you tell what about those queries got slower? New access plans? Same plans but they perform slower? Do they execute slower or are they suspended more? Did all queries got slower or just some? And last but not least, how do you know, ie. what exactly did you measure and how?
Some of the usual suspects could be:
The new storage is much slower (.mdf on slow disk, or on a busy disk)
You changed the data structure during move (ie. some indexes did not get ported)
You changed the data size (ie. compression options) resulting on more pages for the same data
Did anything else change at the same time, new app code or anything the like?
By extending the data size (you do no mention deleting the old tables) you are now trashing the buffer pool (did the page lifetime expectancy decreased in performance counters?)
Look on how you set up the initial size and growth options. If you didn't give it enough space to begin with or if you are growing by 1MB at a time that could be a cause of performance issues.