I'm doing backup of innodb database with mysqldump. It takes 2 min to perform a backup.
Is there any way how to speed it up?
I have machine with 120GB of RAM and I expect that my DB should fit in memory.
Database size on hard drive is around 8 GB:
[user#host:E018 mysql]$ du -hs database
8.3G database
Biggest table has 12054861 records and data size 2991587328.
I have tried to play with innodb_buffer_pool_size but I don't see big performance increase.
If I run mysqldump for first time it takes 2 min 7 sec. If I try it second time it takes around 2 min that is to slow.
I have also tried to archive data to avoid a lot of disk writes:
mysqldump database |pigz > database-dmp.sql.gz that has no influence on performance.
Running mysqldump on different machine from mysql engine does not change anything.
Probably mysql does not cache data to the memory or it sends data to the mysqldump to slow.
Here is configuration that I use:
max_heap_table_size=10G;
innodb_file_per_table=1
innodb_file_format=barracuda
innodb_strict_mode=1
innodb_write_io_threads=4
innodb_read_io_threads=4
innodb_adaptive_hash_index=0
innodb_support_xa=0
innodb_buffer_pool_size=40G
innodb_log_file_size=256M
innodb_log_files_in_group=3
innodb_log_buffer_size=64M
What else can I try to improve mysqldump performance?
Related
We have an aws rds postgres db od type t3.micro.
I am running simple queries on a pretty small table and I get pretty high response time - around 2 seconds per query run.
Query example:
select * from package where id='late-night';
The cpu usage is not high (around 5%)
We tried creating a bigger rds db (t3.meduiom) with the snapshot of the original one and the performance did not improve at all.
Table size 2600 rows
We examined connection with bot external ip and internal ip.
Disk size 20gib
Memory type: ssd
Is there a way to improve performance??
Thanks for the help!
thank you guys in advance.
I am having a 60GB Postgres RDS on aws, and there is databaseA inside this RDS instance, I want to make a duplicate of databaseA called databaseB in the same RDS server.
So basically what I tried is to run CREATE DATABASE databaseB WITH TEMPLATE databaseA OWNER postgres; This single query took 6 hours to complete, which is too slow. I see the max IOPS during the process is 120, not even close to the limit of aws general SSD's limit 10,000 IOPS. I have also tried tunning up work_mem, shared_buffers, effective_cache_size in parameter group, There is no improvements at all.
My last option is to just create two separate RDS instance, but It will be much easier if I can do this in one instance. I'd appreciate any suggestions.
(The instance class is db.m4.xlarge)
As mentioned by Matt; you have two options:
Increase your server size which will give you more IOPS.
Switch to provisioned IOPS
As this is a temporary requirement I will go with 1 because u can upgrade to max. available server --> do database copy --> downgrade db server seamlessly and won't take much time. Switching SSD to provisioned IOPS will take lots of time because it needs to convert your data and hence more downtime. And later again when u will switch back from provisioned iops to SSD again it will take time.
Note that Both 1 & 2 are expensive ( if u really dont need them ) if used for long term; so u can't leave it as is.
I am moving around 10 million data from one table to another in SQL Server 2005. The Purpose of Data transfer is to Offline the old data.
After some time it throws an error Description: "The LOG FILE FOR DATABASE 'tempdb' IS FULL.".
My tempdb and templog is placed in a drive (other than C drive) which has around 200 GB free. Also my tempdb size in database is set to 25 GB.
As per my understanding I will have to increase the size of tempdb from 25 GB to 50 GB and set the log file Auto growth portion to "unrestricted file growth (MB)".
Please let me know other factors and I cannot experiment much as I am working on Production database so can you please let me know if they changes will have some other impact.
Thanks in Advance.
You know the solution. Seems you are just moving part of data to make your queries faster.
I am agree with your solution
As per my understanding I will have to increase the size of tempdb from 25 GB to 50 GB and set the log file Auto growth portion to "unrestricted file growth (MB)".
Go ahead
My guess is that you're trying to move all of the data in a single batch; can you break it up into smaller batches, and commit fewer rows as you insert? Also, as noted in the comments, you may be able to set your destination database to SIMPLE or BULK-INSERT mode.
Why are you using Log file at all? Copy your data (Data and Logfile) then set the mode on SIMPLE and run the transfer again.
We run SQL Server 2005 and have a database that's about 100 GB (the MDF is 100GB and the LDF is 34 GB).
Our maintenance plan takes a full database back up every night. It's set up to
This backup size is usually around 95-100 GB but it all of a sudden grew to 120 GB, then 124 GB then 130 GB then back to 100 GB over 4 consecutive days.
Does anyone know what could cause this? I don't believe we added and then removed that much data in such a short period of time.
If your backup is larger than the MDF, this means you have a lot of log activity recorded too. SQL Server notes data changes that happen during a full backup and does a "mini" log backup to capture this.
I'd say that you need to change Index maintenance and backup timings
I found this script on SQL Authority:
USE MyDatabase
GO
EXEC sp_MSforeachtable #command1=“print ’?' DBCC DBREINDEX (’?', ’ ’, 80)”
GO
EXEC sp_updatestats
GO
It has reduced my insert fail rate from 100% failure down to 50%.
It is far more effective than the reindexing Maintenance Plan.
Should I also be reindexing master and tempdb?
How often? (I write to this database 24 hrs a day)
Any cautions that I should be aware of?
RAID 5 on your NAS? That's your killer.
An INSERT is logged: it writes to the .LDF (log) file. This file is 100% write (well, close enough).
This huge write to read ratio generates a lot of extra writes per disk in RAID 5.
I have an article in work (add later): RAID 5 writes 4 times as much per disk than RAID 10 in 100% write situations.
Solutions
You need to split your data and log files for your database at least.
Edit: Clarified this line:
The log files need go to RAID 1 or RAID 10 drives. It's not so important for data (.MDF) files. Log files are 100% write so benefit from RAID 1 or RAID 10.
There are other potential isues too such as fragmented file system or many Vlog segments (depending on how your database has grown), but I'd say your main issue is RAID 5.
For a 3TB DB, I'd also stuff as much RAM as possible in (32GB if Windows Advanced/Enterprise) and set PAE/AWE etc. This will mitigate some disk issues but only for data caching.
Fill factor 85 or 90 is the usual rule of thumb. If your inserts are wide and not strictly monotonic (eg int IDENTITY column) then you'll have lots of page splits with anything higher.
I'm not the only one who does not like RAID 5: BAARF
Edit again:
Look for "Write-Ahead Logging (WAL) Protocol" in this SQL Server 2000 article. It's still relevant: it explains why the log file is important.
I can't find my article on how RAID 5 suffers compared to RAID 10 under 100% write loads.
Finally, SQL Server does I/O in 64k chunks: so format NTFS with 64k clusters.
This could be anything at all. Is the system CPU bound? IO bound? Is the disk too fragemented? Is the system paging too much? Is the network overloaded?
Have you tuned the indexes? I don't recall if there was an index tuning wizard in 2000, but at the least, you could run the profiler to create a workload that could be used by the SQL Server 2005 index tuning wizard.
Check out your query plans also. Some indexes might not be getting used or the SQL could be wholly inefficient.
What table maintenance do you have?
is all the data in the tables relevant to todays processing?
Can you warehouse off some data?
What is your locking like? Are you locking the whole table?
EDIT:
The SQL profiler shows all interactions with the SQL Server. It should be a DBAs lifeblood.
Thanks for all of the help. I'm not there yet, but getting better.
I can't do much about hardware constraints.
All available RAM is allowed to SQL
Fillfactor is set at 95
Using profiler, an hour's trace offered index tuning with suggested increase of 27% efficiency.
As a result, I doubled the amount of successful INSERTS. Now only 1 out of 4 are failing.
Tracing now and will tune after to see if it gets better.
Don't understand locking yet.
For those who maintain SQL Server as a profession, am I on the right track?