I am responsible for some RDS instances (MariaDB), all running in the same region (with no multi-az setup) and configured to use the automated backup provided by AWS. According to AWS documentation, the automated backups are stored in S3 but there is no detail on the storage class in the documentation.
Business wants assurances that the backups are stored in multiple AZs (not all storage classes offer this!). I've tried to find out more about RDS backup storage through AWS support, but they've been rather unhelpful, claiming these are "internal details" and they can't tell me anything more than I read in the documentation.
So: is it known whether RDS automated backups are stored in multiple AZs, or should I just use AWS Backup with Cross-Region Backups?
Related
We use Cloud Foundry provided by swisscom for several of our applications. Now we ask ourselves how we can best implement automated backups within cloud foundry for db and s3. Any recommendations? Thanks for the support.
We use this two apps to backup mariadb's and mongodb's:
https://github.com/seinol/db-dumper
https://github.com/seinol/cf-backup-rotation
One app makes dumps and stores them on a s3 storage every hour and the other triggers the backup endpoint for each service according to a backup mechanism(10 backups hourly and then daily up to 20)
Recently, this official backup solution is also available, but we haven't used it yet: https://github.com/swisscom/backman
what is the necessity to provide different storage types in AWS?
in which scenarios will use S3/EBS/RDS in AWS??
Amazon S3 is an object store. It can store any number of objects (files), each up to 5TB. Objects are replicated between Availability Zones (Data Centers) within a region. Highly reliable, high bandwidth, fully managed. Can also serve static websites without a server. A great places for storing backups or files that you want to share.
Amazon Elastic Block Store (EBS) is a virtual disk for Amazon EC2 instances, when a local file system is required.
Amazon RDS is a managed database. It installs and maintains a MySQL, PostgreSQL, Oracle or Microsoft SQL Server database as a managed service.
There are more storage services such as DynamoDB, which is a fully-managed NoSQL database service that provides fast and predictable performance with seamless scalability; DocumentDB that provides MongoDB-compatible storage; Amazon Neptune, which is a graph database; Amazon Redshift, which is a petabyte-scale data warehouse; ElastiCache, which is a fully-managed Redis or Memcached caching service; etc.
The world of IT has created many different storage options, each of which serves a different purpose or sweet-spot.
I would like to put MLbackup directory to cloud. Is there any limitations to pushing Full and incremental backups with journal archiving enabled, to S3 compatible Object storage? Is journal archiving supported in S3 compatible cloud storage? What would happen if I put backup with journal archiving enabled to S3 storage? Will it eventually work or I will get errors?
Also, provide documents link to configure ML to point to cloud storage.
You can backup to S3, but if you want to have Journaling enabled, you will need to have them written to a different location. Journal archiving is not supported on S3.
The default location for Journals are in the backup, but when creating programmatically you can specify a different $journal-archive-path.
Backing Up a Database
The directory you specified can be an operating system mounted directory path, it can be an HDFS path, or it can be an S3 path. For details on using HDFS and S3 storage in MarkLogic, see Disk Storage Considerations in the Query Performance and Tuning Guide.
S3 and MarkLogic
Storage on S3 has an 'eventual consistency' property, meaning that write operations might not be available immediately for reading, but they will be available at some point. Because of this, S3 data directories in MarkLogic have a restriction that MarkLogic does not create Journals on S3. Therefore, MarkLogic recommends that you use S3 only for backups and for read-only forests, otherwise you risk the possibility of data loss. If your forests are read-only, then there is no need to have journals.
I usually use webhosts with cpanel.
I'm trying a larger scale website so I thought I'll give AWS a try.
I've been googling and reading the documentations but still can't find a step by step guide to get my website live.
I've transferred all my files to S3
what do I do with EC2?
How do I create a mySQL database?
Thank you for your help.
David
For setting MySQL database, there are two options :
Setup MySQL in your EC2 instance. This is same like as we do in normal scenario.
Make use of Amazon RDS for database. Launch an Amazon RDS instance and give EC2 security group permissions to RDS security group. Access the newly created RDS instance by making use of endpoint name. Advantage of having RDS instance is you don't have to worry about backups and version upgrades.
Attach an EBS volume to your EC2 instance and store everything in that. EBS volumes can be attached on the fly. Your data persists even if EC2 instance crashes out. We can create snapshots of EBS volume and store them in S3 for backups.
Regards,
Sanket Dangi
I am working on a Java MapReduce app that has to be able to provide an upload service for some pictures from the local machine of the user to an S3 bucket.
The thing is the app must run on an EC2 cluster, so I am not sure how I can refer to the local machine when copying the files. The method copyFromLocalFile(..) needs a path from the local machine which will be the EC2 cluster...
I'm not sure if I stated the problem correctly, can anyone understand what I mean?
Thanks
You might also investigate s3distcp: http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/UsingEMR_s3distcp.html
Apache DistCp is an open-source tool you can use to copy large amounts of data. DistCp uses MapReduce to copy in a distributed manner—sharing the copy, error handling, recovery, and reporting tasks across several servers. S3DistCp is an extension of DistCp that is optimized to work with Amazon Web Services, particularly Amazon Simple Storage Service (Amazon S3). Using S3DistCp, you can efficiently copy large amounts of data from Amazon S3 into HDFS where it can be processed by your Amazon Elastic MapReduce (Amazon EMR) job flow. You can also use S3DistCp to copy data between Amazon S3 buckets or from HDFS to Amazon S3.
You will need to get the files from the userMachine to at least 1 node before you will be able to use them through a MapReduce.
The FileSystem and FileUtil functions refer to paths either on the HDFS or the local disk of one of the nodes in the cluster.
It cannot reference the user's local system. (Maybe if you did some ssh setup... maybe?)