Is there a way to reuse the EBS volumes, that are dynamically provisioned by the AWS EBS CSI driver, after destroying and recreating the EKS cluster? It is not possible to set a static value for the CSIVolumeName tag. So, is there any other way?
No, there is not a direct way due to the reason mentioned in the comment. But there is a solution to this. Solution: Create a Lambda, that uses AWS SDK to fetch the required dynamically provisioned EBS volumes, that are to be reused, and creates PersistentVolumes with claimRef using the tags present in the fetched EBS volumes. In this way, the claimRef will have the name of the previously created PersistentVolumeClaim as its value. And when the PersistentVolumeClaims are recreated after EKS cluster recreation, this Lambda gets executed, the PersistentVolumes get created, the PersistentVolumeClaims are automatically mapped to those PersistentVolumes with their names in the claimRef.
Related
My EMR master node has become full and I need to attach some ESB volumne to it, is there any way to do it without terminating the cluster?
You can add additional EBS volumes & also resize
How to explained here :
https://superuser.com/questions/1409373/how-to-add-an-ebs-volume-by-snapshot-id-to-amazon-emr
https://github.com/qyjohn/AWS_Tutorials/wiki/Grow-EBS-volumes-on-EMR-clusters
I don't think so. This is because you set up Amazon Elastic Block Store (Amazon EBS) volumes and configure mount points when the cluster is launched, so it’s difficult to modify the storage capacity after the cluster is running.
The feasible solutions usually involve adding more nodes to your
cluster, backing up your data to a data lake, and then launching a new
cluster with a higher storage capacity. Or, if the data that occupies
the storage is expendable, removing the excess data is usually the way
to go.
For more details,have a look at: https://aws.amazon.com/blogs/big-data/dynamically-scale-up-storage-on-amazon-emr-clusters/
Is exists any way to get available free space in Ceph cluster from amazon S3 API?
I need to implement automatic deletion of outdated objects from bucket when Ceph cluster has no space to store new objects. I know exists ways to calculate used space in bucket, but its logic data size and I can't compare them to raw size of cluster disks.
From S3 perspective you can't check free space.
Ceph has implemented expiration mechanism, which will delete outdated objects from your object storage.
Check the doc: http://docs.ceph.com/docs/master/radosgw/s3/#features-support
It's my understanding that EC2 snapshots are incremental in nature, so snapshot B contains only the difference between itself and snapshot A. Then if you delete snapshot A, the difference is allocated to snapshot B in Amazon S3 so that you have a complete snapshot. This also leads me to believe that it isn't prohibitively expensive to have daily snapshots A-Z for example, that it in storage cost it is basically the same as one snapshot.
What I really want is to back up my snapshots to a bucket in Amazon S3, so that if an entire EC2 region is having some problems --ahem cough, cough-- the snapshot can be moved into another region and launched as a backup instance in a new region.
However, it seems you can only bundle an instance and then upload a bundled instance to S3, not a snapshot.
The bundle is the entire instance correct? If this is the case then are historical bundled instances significantly more costly in practice than snapshots?
I use an instance store AMI and store my changing data on EBS volumes using the XFS filesystem. This means I can freeze the filesystems, create a snapshot and unfreeze them.
My volumes are 1GB (although mostly empty) and the storage cost is minuscule.
I don't know how an EBS backed AMI would work with this but I can't see why it would be any different. Note, however,that you need to bundle an instance in order to start it. Perhaps you could just snapshot everything as a backup and only bundle them when required.
I currently have an amazon instance (Medium - High CPU) running off the instance store with most of my data and code sitting in /mnt mounted to sda2. The instance is just the way i need it to work. How can I clone this instance and make an exact copy (data and all) to another (preferably cheaper, micro) instance for testing my new code changes? Also what backup suggestions are recommend for this setup?
Thanks
Be careful with instance store, your instance if terminated will restore your data. I suggest you put the important data to an EBS volumes.
Please see my post http://www.capsunlock.net/2009/12/create-ebs-boot-ami.html
It's possible to clone the current instance and make an EBS backed AMI.
I have a local filesystem image that I want to transfer to an Amazon EBS volume and boot as an EC2 micro instance. The instance should have the EBS volume as it's root filesystem - and I will be booting the instance with the Amazon PV-GRUB "kernels".
I have used ec2-bundle-image to create a bundle from the image, and I have used ec2-upload-bundle to upload the bundle to Amazon S3. However, now when I'd like to use ec2-register to register the image for usage, I can't seem to find a way to make the uploaded bundle be the ebs root image. It would seem that it requires an EBS snapshot to make the root device, and I have no idea how I would convert the bundle in to an EBS snapshot.
I do realize, that I could probably do this by starting a "common" instance, attaching an EBS volume to it and then just using 'scp' or something to transfer the image directly to the EBS volume - but is this really the only way? Also, I have no desire to use EBS snapshots as such, I'd rather have none - can I create a micro instance with just the EBS volume as root, without an EBS snapshot?
Did not find any way to do this :(
So, I created a new instance, attached a newly created EBS volume to, attached it to the instance, and transferred the data via ssh.
Then, to be able to boot the volume, I still need to create a snapshot of it and then create an AMI that uses the snapshot - and as a result, I get another EBS volume that is created from the snapshot and is the running instance's root volume.
Now, if I want to minimize expences, I can remove the created snapshot, and the original EBS volume.
NOTE: If the only copy of the EBS volume is the root volume of an instance, it may be deleted when the instance is terminated. This setting can be changed with the command-line tools - or the instance may simple by "stopped" instead of "terminated", and then a snapshot can be generated from the EBS volume. After taking a snapshot, the instance can ofcourse be terminated.
Yes, there is no way to upload an EBS Image via S3, and using a Instance where you attach an additional volume is the best way. If you attach that Volume after the instance is started, it will also not be deleted.
Note, do not worry too much about Volume->snapshot->Volume, as those share the same data blocks (as long as you dont modify them). The storage cost ist not trippled, only 1,1times one Volume. EBS snapshots and image creation is quite handy in that regard. Dont hesitate to use multiple snapshots. The less you "work" in a snapshot the smaller is its block usage later on if you start it as an AMI.