I want to export metrics data, alerts, servicemonitors from one prometheus to another prometheus.
How snapshot backup and restore works as in what all things are backed up (only
metrics data (or) servicemonitors/jobs, alerts, metrics is everything
backed up)?
How to restore the snapshot in new prometheus?
How to verify if the metrics are imported to new prometheus?
Also is there a way to take backup of alert rules,
servicemonitors/jobs and restore it to new prometheus?
Note: Prometheus is deployed using Kube-prometheus-stack/prometheus-operator
Found few posts which suggests to use "snapshot" or "pointing to old PV" methods to backup the prometheus data.
I deleted the previous folder and added my new snapshot folder as
suggested in these links -
https://suraj.io/post/how-to-backup-and-restore-prometheus/
https://devopstales.github.io/kubernetes/backup-and-retore-prometheus/
But couldn't able to see the restored data in new prometheus.
Tried Pointing my Persistent volume to old PV as mentioned here
-https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack/
In this method also i couldn't able to see metrics,
servicemonitors/jobs, alerts getting backed up!
Related
I have data on several machines that I want to backup in away that I can restore to certain points in time.
From what I read Snapshot Replication achieves this (as opossed to back-up that clobbers previous results).
The main motivation is that if the data files are ransacked, and encoded, then if I just back-up I can end up in a state where the backed up files are also encrypted.
One way to do this is by using 2 Synology NAS machines where I can have:
rsync processes to back-up files from multiple machines into a NAS1
apply Snapshot Replication from NAS1 to NAS2
In this way, if the data is hijacked at certain point, I can restore the data to the last good state by restoring NAS2 to previous point in time.
I would like to know if:
Snapshot Replication is the way to go, or there are other solutions?
are there other ways to achieve Snapshot Replication, e.g. with single NAS?
I have an older Synology 2-Bay NAS DS213j.
Assuming that I buy a second, newer, NAS (e.g. DS220j), are the 2 NAS machines expected to work together?
Thanks
I found out that Hyper Backup can save snapshots in time, so I'm using it instead of Snapshot Replication
We are using MarkLogic 9 version. We have developed API above MarkLogic
We are doing daily full backup for last 7 days.
Now we are upgrading our MarkLogic server to 10 & as a part of disaster recovery if due to some reason upgrade fails and we need to restore backup from yesterday
I want to understand
How restore process works, do restore for each stand and then serve from remaining ones?
Whether API requests will be served during restore process ?
If API requests will be served then which data will be used to serve that request ?
Do we need to go for downtime as a part of restore process ?
If we go for incremental backup & then restore, will there be any difference to above points?
The Restore will restore the entire backup to a temporary directory. Once all of the stands/forests have been copied to disk it will swap the current forests/stands with the restored ones.
API requests will still be serviced by the active stands, with the existing data. It will not use the restored data until after the restore completes.
There will be a very small amount of downtime as the existing forests shutdown, and the restored forests start up. This is usually just a few seconds, but it does depend on how big the forests are.
There is no difference in the behavior if you are restoring a full + incremental/s or only a full backup.
Be aware that you will need enough disk space for both the current data and the restored data, as they will coexist for a period of time.
Backup and Restore Transactions
Phases of Backup or Restore Operation
Currently I have two buckets in the same region. (50TB)
I am trying to backup all the contents from one to the other through the sync command. But the sync command copies it doesn't really backup
However after the sync finished I realized that the versionIDs in the objects in the new bucket are not the same as in the old... I use the version ID for some features in my applications. So sync is not really an option for me.
Is there an alternative to sync that can backup an entire bucket with its metadata etc... intact ? namely the version Id ?
If you don't have a region limitation, you can use cross-region replication to backup the bucket.
https://docs.aws.amazon.com/AmazonS3/latest/dev/crr.html
How do you export and import data in Prometheus? How do you make sure the data is backed up if the instance gets down?
It does not seem that there is a such feature yet, how do you do then?
Since Prometheus version 2.1 it is possible to ask the server for a snapshot. The documentation provides more details - https://web.archive.org/web/20200101000000/https://prometheus.io/docs/prometheus/2.1/querying/api/#snapshot
Once a snapshot is created, it can be copied somewhere for safe keeping and if required a new server can be created using this snapshot as its database.
The documentation website constantly changes all the URLs, this links to fairly recent documentation on this -
https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-admin-apis
There is no export and especially no import feature for Prometheus.
If you need to keep data collected by prometheus for some reason, consider using the remote write interface to write it somewhere suitable for archival, such as InfluxDB (configured as a time-series database).
Prometheus isn't a long term storage: if the database is lost, the user is expected to shrug, mumble "oh well", and restart Prometheus.
credits and many thanks to amorken from IRC #prometheus.
There is an option to enable Prometheus data replication to remote storage backend. Later the data collected from multiple Prometheus instances could be backed up in one place on the remote storage backend. See, for example, how VictoriaMetrics remote storage can save time and network bandwidth when creating backups to S3 or GCS with vmbackup utility.
How to have 3 dgraphs backup saved automatically in dgraph server and not on ITL server . By default backup of dgidx output gets saved on ITL server . I want it to be saved on dgraph server ie MDEX host. Please help.
I don't believe there to be an Out-of-the-Box option for backing up the deployed dgidx output on the target server. Have you gone through the documentation? I would also question whether it is a good idea. Consider you are deploying and 2 of the 3 servers have gone through successfully but the third one fails. You now need to roll back only two of the machines. Your central EAC will not know which ones to rollback and which ones to keep. However, by keeping it all at a central point (ie. on the ITL server) in the event of a rollback you will always push the same backup out to all three servers.
Assuming that you are trying to speed up the deployment of very large indices (Endeca copies the entire dgidx output to each MDEX), you can always look at the performance tuning guide.
You should be able to do this in any number of ways:
In any baseline update, dgidx_output is automatically copied to each
dgraph server. You should be add a copy or archive job as a
pre-shutdown task for your dgraph.
You could also create a custom copy job that would for each dgraph
server that would run at the end or beginning of a baseline update.
Or it could be offline from your baseline update entirely.
To a point radimpe makes, making copies on the dgraph servers is not that hard, but rather, it's the rollback process you need to really consider. You need to set that up and ensure it uses whatever backup copies you've made, whether local to the ITL machine or on the dgraph servers.
Also know that dgidx_output will not include any partial update information added since the index was created. Partial update info only be available in the dgraph_input on the dgraph servers. Accordingly, if you incorporate partial updates, you should archive the dgraph input and make that available for any rollback job.
You can create a DGRAPH post startup task and assign it in the graph definitions. It will will be executed on each MDEX startup
<dgraph id="Dgraph01" host-id="LiveMDEXHost01" port="10000" pre-shutdown-script="DgraphPreShutdownScript" post-startup-script="DgraphPostStartupScript">
<script id="DgraphPostStartupScript">
<bean-shell-script>
<![CDATA[
...code to backup here
]]>
</bean-shell-script>
</script>