what is the right way to back-up a substrate basepath? - amazon-s3

Question

can anyone point me at best-practice guidance for substrate basepath backup and restore? specifically, i'm wondering about these things:
it's my experience that backing up the basepath while the substrate service is running, results in a corrupt database that cannot be used in a restore. is that expected? it's not all that surprising if so.
syncing while the node service is running produces errors and warnings (in the sync process) like this:
warning: Skipping file /var/lib/substrate/polkadot/chains/ksmcc3/db/full/642785.sst. File does not exist.
and
upload failed: ../../var/lib/substrate/polkadot/chains/ksmcc3/db/full/642836.sst to s3://calamari-kusama-smoothie-calamari-systems/var/lib/substrate/polkadot/chains/ksmcc3/db/full/642836.sst [Errno 2] No such file or directory: '/var/lib/substrate/polkadot/chains/ksmcc3/db/full/642836.sst'
a basepath restored from a backup taken while the node service was running, always errors (for me) like so:
Error: Service(Client(Backend("Invalid argument: Column families not opened: col11, col10, col9, col8, col7, col6, col5, col4, col3, col2, col1, col0")))
or:
Error: Service(Client(Backend("IO error: No such file or directory: While opening a file for sequentially reading: /var/lib/substrate/chains/calamari/db/full/MANIFEST-009085: No such file or directory")))
is rsyncing the basepath the right way to go about doing backups? for ec2 nodes, i currently sync to an s3 bucket like so (the exclusions omit node key files from the backup):
para=calamari
relay=ksmcc3
bucket=${para}-${relay}-${HOSTNAME//./-}
basepath=/var/lib/substrate
sudo systemctl stop ${para}.service
/usr/bin/aws s3 sync ${basepath} s3://${bucket}${basepath} \
--exclude "chains/${para}/network/*" \
--exclude "polkadot/chains/${relay}/network/*"
which files in the basepath are useful for a restore and should be backed up? as well as the block numbered *.sst and *.log files, this is what i see (on a recently started, still syncing node):
$ ls -hl --ignore *.sst --ignore *.log /var/lib/substrate/{polkadot/chains/${relay}/db/{full,full/parachains/db},chains/${para}/db/full/}
/var/lib/substrate/chains/calamari/db/full/:
total 23M
-rw-r--r-- 1 root root 16 Nov 15 17:13 CURRENT
-rw-r--r-- 1 root root 37 Nov 15 16:50 IDENTITY
-rw-r--r-- 1 root root 0 Nov 15 16:50 LOCK
-rw-r--r-- 1 root root 21M Nov 16 06:33 LOG
-rw-r--r-- 1 root root 1.5M Nov 16 06:23 MANIFEST-002019
-rw-r--r-- 1 root root 48K Nov 15 17:13 OPTIONS-001991
-rw-r--r-- 1 root root 48K Nov 15 17:13 OPTIONS-002022
-rw-r--r-- 1 root root 1 Nov 15 17:13 db_version
/var/lib/substrate/polkadot/chains/ksmcc3/db/full:
total 1.4G
-rw-r--r-- 1 root root 16 Nov 15 17:13 CURRENT
-rw-r--r-- 1 root root 37 Nov 15 16:50 IDENTITY
-rw-r--r-- 1 root root 0 Nov 15 16:50 LOCK
-rw-r--r-- 1 root root 1.4G Nov 16 06:35 LOG
-rw-r--r-- 1 root root 74M Nov 16 06:35 MANIFEST-008095
-rw-r--r-- 1 root root 48K Nov 15 17:13 OPTIONS-007788
-rw-r--r-- 1 root root 48K Nov 15 17:13 OPTIONS-008098
-rw-r--r-- 1 root root 1 Nov 15 17:13 db_version
drwxr-xr-x 3 root root 4.0K Nov 15 16:50 parachains
drwxr-xr-x 2 root root 4.0K Nov 15 17:13 pvf-artifacts
/var/lib/substrate/polkadot/chains/ksmcc3/db/full/parachains/db:
total 2.1M
-rw-r--r-- 1 root root 16 Nov 15 17:13 CURRENT
-rw-r--r-- 1 root root 37 Nov 15 16:50 IDENTITY
-rw-r--r-- 1 root root 0 Nov 15 16:50 LOCK
-rw-r--r-- 1 root root 2.1M Nov 16 06:33 LOG
-rw-r--r-- 1 root root 437 Nov 15 17:13 MANIFEST-000033
-rw-r--r-- 1 root root 24K Nov 15 17:13 OPTIONS-000031
-rw-r--r-- 1 root root 24K Nov 15 17:13 OPTIONS-000036
-rw-r--r-- 1 root root 1 Nov 15 17:13 parachain_db_version

Answer 1

Here are so pro tips around doing any kind of DB snapshots:
Always stop the node before backing up! Failing to do this will usually result in a corrupted DB.
Always exclude {basepath}/keystore and {basepath}/network from your backups or only ever explicitly backup {basepath}/
The entire db folder should be backed up. This can be rsynced directly but it can also be tar'd or compressed then backed up.