I currently have an amazon instance (Medium - High CPU) running off the instance store with most of my data and code sitting in /mnt mounted to sda2. The instance is just the way i need it to work. How can I clone this instance and make an exact copy (data and all) to another (preferably cheaper, micro) instance for testing my new code changes? Also what backup suggestions are recommend for this setup?
Thanks
Be careful with instance store, your instance if terminated will restore your data. I suggest you put the important data to an EBS volumes.
Please see my post http://www.capsunlock.net/2009/12/create-ebs-boot-ami.html
It's possible to clone the current instance and make an EBS backed AMI.
Related
I've got an Compute Engine instance with 1 GPU attached.
After some time, I increased my quota to have 2 instances available.
Is it possible to attach the second GPU to this existing instance?
Do I have to create the instance again?
Yes, you can add more GPUs to an existing instance via the UI or API. However, to do so, you need to stop the VM first, edit the vm (change the number of gpus for example) and then start the VM.
I can't see the option (when trying to edit a stopped instance) at the moment.
If you choose to create a new instance, consider taking a snapshot of your current disk and then create the new instance using that snapshot - that can make the process easier.
Being relatively new to GCE, but not to other virtualization tools like VmWare or VirtuaBox, I'm not able to find in GCE a concrete way to get a full snapshot of a live machine.
I'm guessing it's my fault or poor knowledge, but really GCE doesn't saves the "system state", or else dumps memory to snapshot?
I'd found many scripts and examples on how to flush buffers to disks before I create the snapshot, but no way to obtain a complete state of the machine, including what the machine itself is running at THAT point.
Let me say that, if this is correct, the GCE snapshot IS NOT a snapshot.
Thanks in advance for your help.
That's a VM image, not a snapshot, and it does not include the contents of RAM or the processor state. A snapshot is a point-in-time copy of a persistent disk.
[link] (http://vcloud.vmware.com/uk/using-vcloud-air/tutorials/working-with-snapshots)
Here's an example of a cloud platform saving true snapshots, portraits of a specific second of a working machine.
Let me add a thought:
I don't know if VCloud is considering a particular state, gains privileged access to disks for a limited time, avoiding contingency, or else does a temporary duplication of the working disk in another volume.
I'm still reading around, trying to get INTO the problem.
BUT... it dumps memory to snapshot.
This is the point, and I'm wondering why this seems to be not possible in GCE.
I signed up for Google Cloud the other day using their free trial promotion. I love it so far. I've got a couple of questions that are probably generic to cloud computing, which I'm new to. I have my test virtual machine up without any issues, using Ubuntu Linux.
My question with cloud concepts are - first:
- How to scale instance. Can you scale from micro to small (also vice versa)?
If scaling isn't done that way, and it's about using instance groups, how do load balancing and instance groups work?
This is the concept I'm most confused with...how would I push an code update if I had 3 instances for the load balancer?
Thanks for your help!
First question: How do you vertically scale an instance? Answer: you must re-create the instance and destroy the old one. You can't just make an existing instance smaller or larger. Luckily, you can script the whole setup. GCE allows you to add a flag called --metadata-from-file. If you are using systemd, I recommend something to the effect of --metadata-from-file user-data=cloud-config.yaml. Since you are using Ubuntu, and Ubuntu's support for systemd is sketchy at best, you probably just want to do something like: --metadata-from-file startup-script=my-startup-script.sh Scripting your deployment will allow you to scale, re-create and document your deployment and is a best practice in cloud computing.
Second question: How do instance groups and load balancing groups work? Answer: Instance groups in GCE are almost always of the "managed" variety. This allows you to create a template that defines how you want your instances to work. Then you can horizontally scale them (i.e. add more or take some away) behind a load balancer. You can even leverage preemptible instances to save you some cash.
Third question: How do I push an update? This depends on how you deploy. But in general I would say:
If you use Docker, push a new image to GCR and have your instances pull it.
If you use CM (like Salt or Ansible) just use those tools normally. They work fine on GCE
If you use startup scripts do something like gcloud compute instances myinstance add-metadata metadata-from-file startup-script=newScript.sh (and restart after)
If everything is contained in a managed instance template, update your template.
When committing a running container with docker commit, is this creating a consistent snapshot of the filesystem?
I'm considering this approach for backing up containers. You would just have to docker commit <container> <container>:<date> and push it to a local registry.
The backup would be incremental, as the commit would just create a new layer.
Also would the big amount of layers hurt io performance of the container drastically? Is there a way to remove intermediate layers at a later point in time?
Edit
By consistent I mean that every application that is designed to survive a power-loss should be able to recover from this snapshots. Basically this means that no file must change after the snapshot is started.
Meanwhile I found out that docker supports multiple storage drivers (aufs, devicemapper, btrfs) now. Unfortunately there is hardly any documentation about the differences between them and the options they support.
I guess consistency is what you define it to be.
In terms of flattening and the downsides of stacking too many AUFS layers see:
https://github.com/dotcloud/docker/issues/332
docker flatten is linked there.
I am in a similar situation. I am thinking about not using a dedicated data volume container instead committing regularly to have some kind of incremental backup. Beside the incremental backup the big benefit is for a team developing approach. As newcomer you can simply docker pull a database image already containing all the data you need to run, debug and develop.
So what I do right now is to pause before commit:
docker pause happy_feynman; docker commit happy_feynman odev:`date +%s`
As far as I can tell I have no problems right now. But this is a developing machine so no I have no experience on heavy load servers.
I have a local filesystem image that I want to transfer to an Amazon EBS volume and boot as an EC2 micro instance. The instance should have the EBS volume as it's root filesystem - and I will be booting the instance with the Amazon PV-GRUB "kernels".
I have used ec2-bundle-image to create a bundle from the image, and I have used ec2-upload-bundle to upload the bundle to Amazon S3. However, now when I'd like to use ec2-register to register the image for usage, I can't seem to find a way to make the uploaded bundle be the ebs root image. It would seem that it requires an EBS snapshot to make the root device, and I have no idea how I would convert the bundle in to an EBS snapshot.
I do realize, that I could probably do this by starting a "common" instance, attaching an EBS volume to it and then just using 'scp' or something to transfer the image directly to the EBS volume - but is this really the only way? Also, I have no desire to use EBS snapshots as such, I'd rather have none - can I create a micro instance with just the EBS volume as root, without an EBS snapshot?
Did not find any way to do this :(
So, I created a new instance, attached a newly created EBS volume to, attached it to the instance, and transferred the data via ssh.
Then, to be able to boot the volume, I still need to create a snapshot of it and then create an AMI that uses the snapshot - and as a result, I get another EBS volume that is created from the snapshot and is the running instance's root volume.
Now, if I want to minimize expences, I can remove the created snapshot, and the original EBS volume.
NOTE: If the only copy of the EBS volume is the root volume of an instance, it may be deleted when the instance is terminated. This setting can be changed with the command-line tools - or the instance may simple by "stopped" instead of "terminated", and then a snapshot can be generated from the EBS volume. After taking a snapshot, the instance can ofcourse be terminated.
Yes, there is no way to upload an EBS Image via S3, and using a Instance where you attach an additional volume is the best way. If you attach that Volume after the instance is started, it will also not be deleted.
Note, do not worry too much about Volume->snapshot->Volume, as those share the same data blocks (as long as you dont modify them). The storage cost ist not trippled, only 1,1times one Volume. EBS snapshots and image creation is quite handy in that regard. Dont hesitate to use multiple snapshots. The less you "work" in a snapshot the smaller is its block usage later on if you start it as an AMI.