We have an aws rds postgres db od type t3.micro.
I am running simple queries on a pretty small table and I get pretty high response time - around 2 seconds per query run.
Query example:
select * from package where id='late-night';
The cpu usage is not high (around 5%)
We tried creating a bigger rds db (t3.meduiom) with the snapshot of the original one and the performance did not improve at all.
Table size 2600 rows
We examined connection with bot external ip and internal ip.
Disk size 20gib
Memory type: ssd
Is there a way to improve performance??
Thanks for the help!
Related
Thanks in advance!
I am planning to use AWS RDS Postgres for pretty big data (> ~50TB) , but I have couple of questions un-answered
Is 16TB the maximum limit for AWS RDS Postgres instance, if so how do people store > 16TB data.
Is the limit of 16TB for RDS the maximum database size post compression that Postgres can store on AWS.
Also I do not see any option for enabling compression while setting up AWS RDS Postgres DB instance. How to enable compression in AWS RDS Postgres?
I have followed :
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Limits.html
https://blog.2ndquadrant.com/postgresql-maximum-table-size/ (wherein Postgres table can have size greater than 32TB).
https://wiki.postgresql.org/wiki/FAQ#What_is_the_maximum_size_for_a_row.2C_a_table.2C_and_a_database.3F
In addition to RDS for PostgreSQL, which has a 32 TiB limit, you should take a look at Amazon Aurora PostgreSQL, which has a 64 TiB limit. In both cases, the largest single table you can create is 32 TiB, though you can't quite reach that size in RDS for PostgreSQL as some of the space will be taken up by the system catalog.
Full disclosure: I am the product manager for Aurora PostgreSQL at AWS.
As of 2019/02/11, Amazon's docs state that the maximum database size for a Postgres RDS instance is 32TiB.
Additionally, this 32TiB limit appears to be a hard limit (Some AWS limits are considered 'soft' in that a user can request that they be raised)
As others have suggested, one option would be to manage your own database, however this is not likely to be easy given the scale of data we are talking about here.
Staying within the AWS ecosystem, another option would be to store all your data in S3, and use AWS Athena to run queries, although depending on what sort of queries you'll be running, it could get quite expensive.
As for your question about compression, if you are storing data in s3, you can compress it before you upload. You might also find this answer helpful.
RDS limit has been revised to 64 TB.
MariaDB, MySQL, Oracle, and PostgreSQL database instances: 20 GiB–64 TiB
SQL Server for Enterprise, Standard, Web, and Express editions: 20 GiB–16 TiB
thank you guys in advance.
I am having a 60GB Postgres RDS on aws, and there is databaseA inside this RDS instance, I want to make a duplicate of databaseA called databaseB in the same RDS server.
So basically what I tried is to run CREATE DATABASE databaseB WITH TEMPLATE databaseA OWNER postgres; This single query took 6 hours to complete, which is too slow. I see the max IOPS during the process is 120, not even close to the limit of aws general SSD's limit 10,000 IOPS. I have also tried tunning up work_mem, shared_buffers, effective_cache_size in parameter group, There is no improvements at all.
My last option is to just create two separate RDS instance, but It will be much easier if I can do this in one instance. I'd appreciate any suggestions.
(The instance class is db.m4.xlarge)
As mentioned by Matt; you have two options:
Increase your server size which will give you more IOPS.
Switch to provisioned IOPS
As this is a temporary requirement I will go with 1 because u can upgrade to max. available server --> do database copy --> downgrade db server seamlessly and won't take much time. Switching SSD to provisioned IOPS will take lots of time because it needs to convert your data and hence more downtime. And later again when u will switch back from provisioned iops to SSD again it will take time.
Note that Both 1 & 2 are expensive ( if u really dont need them ) if used for long term; so u can't leave it as is.
I tried to use Pyhs2 to communicate with Hive，fetch data and put them in a list(temporary stored in RAM).
But it took a long time to query a table using very simple HQL like 'select fields1,fields2... from table_name', in which data scale is about 7 million rows and less then 20 fields. The whole process costs nearly 90 mins.
My server: CentOS 6.5, 8 cpu units, 32 processors and 32GB RAM
Hadoop cloud: more than 200 machines
Can someone help to solve this problem? Thanks very much
I'm doing backup of innodb database with mysqldump. It takes 2 min to perform a backup.
Is there any way how to speed it up?
I have machine with 120GB of RAM and I expect that my DB should fit in memory.
Database size on hard drive is around 8 GB:
[user#host:E018 mysql]$ du -hs database
8.3G database
Biggest table has 12054861 records and data size 2991587328.
I have tried to play with innodb_buffer_pool_size but I don't see big performance increase.
If I run mysqldump for first time it takes 2 min 7 sec. If I try it second time it takes around 2 min that is to slow.
I have also tried to archive data to avoid a lot of disk writes:
mysqldump database |pigz > database-dmp.sql.gz that has no influence on performance.
Running mysqldump on different machine from mysql engine does not change anything.
Probably mysql does not cache data to the memory or it sends data to the mysqldump to slow.
Here is configuration that I use:
max_heap_table_size=10G;
innodb_file_per_table=1
innodb_file_format=barracuda
innodb_strict_mode=1
innodb_write_io_threads=4
innodb_read_io_threads=4
innodb_adaptive_hash_index=0
innodb_support_xa=0
innodb_buffer_pool_size=40G
innodb_log_file_size=256M
innodb_log_files_in_group=3
innodb_log_buffer_size=64M
What else can I try to improve mysqldump performance?
I am moving around 10 million data from one table to another in SQL Server 2005. The Purpose of Data transfer is to Offline the old data.
After some time it throws an error Description: "The LOG FILE FOR DATABASE 'tempdb' IS FULL.".
My tempdb and templog is placed in a drive (other than C drive) which has around 200 GB free. Also my tempdb size in database is set to 25 GB.
As per my understanding I will have to increase the size of tempdb from 25 GB to 50 GB and set the log file Auto growth portion to "unrestricted file growth (MB)".
Please let me know other factors and I cannot experiment much as I am working on Production database so can you please let me know if they changes will have some other impact.
Thanks in Advance.
You know the solution. Seems you are just moving part of data to make your queries faster.
I am agree with your solution
As per my understanding I will have to increase the size of tempdb from 25 GB to 50 GB and set the log file Auto growth portion to "unrestricted file growth (MB)".
Go ahead
My guess is that you're trying to move all of the data in a single batch; can you break it up into smaller batches, and commit fewer rows as you insert? Also, as noted in the comments, you may be able to set your destination database to SIMPLE or BULK-INSERT mode.
Why are you using Log file at all? Copy your data (Data and Logfile) then set the mode on SIMPLE and run the transfer again.