When committing a running container with docker commit, is this creating a consistent snapshot of the filesystem?
I'm considering this approach for backing up containers. You would just have to docker commit <container> <container>:<date> and push it to a local registry.
The backup would be incremental, as the commit would just create a new layer.
Also would the big amount of layers hurt io performance of the container drastically? Is there a way to remove intermediate layers at a later point in time?
Edit
By consistent I mean that every application that is designed to survive a power-loss should be able to recover from this snapshots. Basically this means that no file must change after the snapshot is started.
Meanwhile I found out that docker supports multiple storage drivers (aufs, devicemapper, btrfs) now. Unfortunately there is hardly any documentation about the differences between them and the options they support.
I guess consistency is what you define it to be.
In terms of flattening and the downsides of stacking too many AUFS layers see:
https://github.com/dotcloud/docker/issues/332
docker flatten is linked there.
I am in a similar situation. I am thinking about not using a dedicated data volume container instead committing regularly to have some kind of incremental backup. Beside the incremental backup the big benefit is for a team developing approach. As newcomer you can simply docker pull a database image already containing all the data you need to run, debug and develop.
So what I do right now is to pause before commit:
docker pause happy_feynman; docker commit happy_feynman odev:`date +%s`
As far as I can tell I have no problems right now. But this is a developing machine so no I have no experience on heavy load servers.
Related
I have a Ceph cluster managed by Rook with a single RGW store over it. We are trying to figure out the best backup strategy for this store. We are considering the following options: using rclone to backup object via an S3 interface, using s3fs-fuse (haven’t tested it yet but s3fs-fuse is known to be not reliable enough), and using NFS-Ganesha to reexport the RGW store as an NFS share.
We are going to have quite a lot of RGW users and quite a lot of buckets, so all three solutions do not scale well for us. Another possibility is to perform snapshots of RADOS pools backing the RGW store and to backup these snapshots, but the RTO will be much higher in that case. Another problem with snapshots is that it does not seem possible to perform them consistently across all RGW-backing pools. We never delete objects from the RGW store, so this problem does not seem to be that big if we start snapshotting from the metadata pool - all the data it refers to will remain in place even if we create a snapshot on the data pool a bit later. It won’t be super consistent but it should not be broken either. It’s not entirely clear how to restore single objects in a timely manner using this snapshotting scheme (to be honest, it’s not entirely clear how to restore using this scheme at all), but it seems to be worth trying.
What other options do we have? Am I missing something?
We're planning to implement Ceph in 2021.
We don't expect a large number of users and buckets, initially.
While waiting for https://tracker.ceph.com/projects/ceph/wiki/Rgw_-_Snapshots, I successfully tested this solution to address the Object Store protection by taking advantage of multisite configuration + sync policy (https://docs.ceph.com/en/latest/radosgw/multisite-sync-policy/) in the "Octopus" version.
Assuming you have all zones in the Prod site Zone Sync'd to the DRS,
create a Zone in the DRS, e.g. "backupZone", not Zone Sync'd from
or to any of the other Prod or DRS zones;
the endpoints for this backupZone are in 2 or more DRS cluster
nodes;
using (https://rclone.org/s3/) write a bash script: for each the
"bucket"s in the DRS zones, create a version enabled "bucket"-p in the backupZone
and schedule sync, e.g. twice a day, from "bucket" to "bucket"-p;
protect the access to the backupZone endpoints so that no ordinary
user (or integration) can access them, only accessible from the other nodes in the
cluster (obviously) and the server running the rclone-based script;
when there is a failure, just recover all the objects from the *-p
buckets, once again using rclone, to the original buckets or to
filesystem.
This protects from the following failures:
Infra:
Bucket or pool failure;
Object pervasive corruption;
Loss of a site
Human error:
Deletion of versions or objects;
Removal of buckets
Elimination of entire Pools
Notes:
Only the latest version of each object is sync'd to the protected
(*-p) bucket, but if the script runs several times you have the
latest states of the objects through time;
when an object is deleted in the prod bucket, rnode just flags the
object with the DeleteMarker upon sync
this does not scale!! As the number of buckets increases, the time to
sync becomes untenable
Being relatively new to GCE, but not to other virtualization tools like VmWare or VirtuaBox, I'm not able to find in GCE a concrete way to get a full snapshot of a live machine.
I'm guessing it's my fault or poor knowledge, but really GCE doesn't saves the "system state", or else dumps memory to snapshot?
I'd found many scripts and examples on how to flush buffers to disks before I create the snapshot, but no way to obtain a complete state of the machine, including what the machine itself is running at THAT point.
Let me say that, if this is correct, the GCE snapshot IS NOT a snapshot.
Thanks in advance for your help.
That's a VM image, not a snapshot, and it does not include the contents of RAM or the processor state. A snapshot is a point-in-time copy of a persistent disk.
[link] (http://vcloud.vmware.com/uk/using-vcloud-air/tutorials/working-with-snapshots)
Here's an example of a cloud platform saving true snapshots, portraits of a specific second of a working machine.
Let me add a thought:
I don't know if VCloud is considering a particular state, gains privileged access to disks for a limited time, avoiding contingency, or else does a temporary duplication of the working disk in another volume.
I'm still reading around, trying to get INTO the problem.
BUT... it dumps memory to snapshot.
This is the point, and I'm wondering why this seems to be not possible in GCE.
I've deployed a single micro-instance redis on compute engine using the (very convenient) click-to-deploy feature.
I would now like to update this configuration to have a couple of instances, so that I can benchmark how this increases performance.
Is it possible to modify the config while it's running?
The other option would be to add a whole new redis deployment, bleed traffic onto that over time and eventually shut down the old one. Not only does this sound like a pain in the butt, but, I also can't see any way in the web UI to click-to-deploy multiple clusters.
I've got my learners license with all this, so would also appreciate any general 'good-to-knows'.
I'm on the Google Cloud team working on this feature and wanted to chime in. Sorry no one replied to this for so long.
We are working on some of the features you describe that would surely make the service more useful and powerful. Stay tuned on that.
I admit that there really is not a good solution for modifying an existing deployment to date, unless you launch a new cluster and migrate your data over / redirect reads and writes to the new cluster. This is a limitation we are working to fix.
As a workaround for creating two deployments using Click to Deploy with Redis, you could create a separate project.
Also, if you wanted to migrate to your own template using the Deployment Manager API https://cloud.google.com/deployment-manager/overview, keep in mind Deployment Manager does not have this limitation, and you can create multiple deployments from the same template in the same project.
Chris
I am reading about docker and I am trying to understand whether or not this is something I should learn to use.
From what I read best practices states that you should have one process per container. Now, this mean that I need one container for JBoss, one for database, one for file storage, build server, ...
Now would I manually have to start each of these containers? Or are there some kind of dependencies you can set up?
What about the order and requirements that one process in a container can have? JBoss needs the database to be started before it starts etc?
Is this handled?
one process per container
This advice is valid if you want to follow a microservices architecture. microservices have advantages but also drawbacks. Depending on your situation you might find it more convenient to have a container running multiple processes.
Running multiple containers on one single host
If you want to start multiple containers together on one single docker host, the easiest way is to use fig. The fig configuration file is very easy to understand as its syntax mimics docker commands. This video gives you a nice presentation of fig (by one of fig authors Aanand Prasad)
Note that tools such as fig AFAIK won't be able to wait for a first container to start and finish initializing before starting another container depending on the first one. The way to handle this is to have the 2nd container implement some kind of test and loop until the dependency is ready, then start its process. This can be achieved by different means (wrapper script, straight in your application code, ...)
Running multiple processes in one container
As a docker container will stop as soon as no process is running in the foreground, there are different techniques you can use (supervisor, running a first process as a daemon and a last one in the foreground, using phusion/baseimage, ...)
I am using redis version 2.8.3. I want to build a redis cluster. But in this cluster there should be multiple master. This means I need multiple nodes that has write access and applying ability to all other nodes.
I could build a cluster with a master and multiple slaves. I just configured slaves redis.conf files and added that ;
slaveof myMasterIp myMasterPort
Thats all. Than I try to write something into db via master. It is replicated to all slaves and I really like it.
But when I try to write via a slave, it told me that slaves have no right to write. After that I just set read-only status of slave in redis.conf file to false. Hence, I could write something into db.
But I realize that, it is not replicated to my master replication so it is not replicated to all other slave neigther.
This means I could'not build an active-active cluster.
I tried to find something whether redis has active-active cluster capability. But I could not find exact answer about it.
Is it available to build active-active cluster with redis?
If it is, How can I do it ?
Thank you!
Redis v2.8.3 does not support multi-master setups. The real question, however, is why do you want to set one up? Put differently, what challenge/problem are you trying to solve?
It looks like the challenge you're trying to solve is how to reduce the network load (more on that below) by eliminating over-the-net reads. Since Redis isn't multi-master (yet), the only way to do it is by setting up each app server with a master and a slave (to the other master) - i.e. grand total of 4 Redis instances (and twice the RAM).
The simple scenario is when each app updates only a mutually-exclusive subset of the database's keys. In that scenario this kind of setup may actually be beneficial (at least in the short term). If, however, both apps can touch all keys or if even just one key is "shared" for writes between the apps, then you'll need to bake locking/conflict resolution/etc... logic into your apps to consolidate local master and slave differences (and that may be a bit of an overkill). In either case, however, you'll end up with too many (i.e. more than 1) Redises, which means more admin effort at the very least.
Also note that by colocating app and database on the same server you're setting yourself for near-certain scalability failure. What will happen when you need more compute resources for your apps or Redis? How will you add yet another app server to the mix?
Which brings me back to the actual problem you are trying to solve - network load. Why exactly is that an issue? Are your apps so throughput-heavy or is the network so thin that you are willing to go to such lengths? Or maybe latency is the issue that you want to resolve? Be the case as it may be, I recommended that you consider a time-proven design instead, namely separating Redis from the apps and putting it on its own resources. True, network will hit you in the face and you'll have to work around/with it (which is what everybody else does). On the other hand, you'll have more flexibility and control over your much simpler setup and that, in my book, is a huge gain.
Redis Enterprise has had this feature for quite a while, but if you are looking for an open source solution KeyDB is a fork with Active Active support (called Active Replica).
Setting it up is just a little more work than standard replication:
Both servers must have "active-replica yes" in their respective configuration files
On server B execute the command "replicaof [A address] [A port]"
Server B will drop its database and load server A's dataset
On server A execute the command "replicaof [B address] [B port]"
Server A will drop its database and load server B's dataset (including the data it just transferred in the prior step)
Both servers will now propagate writes to each other. You can test this by writing to a key on Server A and ensuring it is visible on B and vice versa.
https://github.com/JohnSully/KeyDB/wiki/KeyDB-(Redis-Fork):-Active-Replica-Support