====Cold ZFS Snapshots with Syncoid Replication for QEMU/KVM Virtual Guests====
This will setup a system to replicate ZFS datasets from a source server to a destination server with the primary goal of backing up QEMU/KVM virtual guest disk images. Additionally there will be a "cold_backup" snapshot taken of the virtual guest when it is in an offline state.

====Requirements====
Two or more [[tech_documents:virtualization:virtualization_host_centos8|CentOS 8 Virtualization Hosts]] with [[tech_documents:linux:zfs_centos8|ZFS on CentOS 8]], a routine on the source server to make cold snapshots of virtual guest [[tech_documents:virtualization:cold_backups_zfs|Cold Backup Script from ZFS Snapshots]] and [[tech_documents:linux:zfs_sanoid_syncoid|Sanoid/Syncoid]] installed. Note: sanoid on the destination/backup server should be configured to cleanup snapshots only, not create them.

The backup/destination virtualization host shouldn't be used for production virtual guests, its primary role should be a backup destination of production virtual guests and an emergency backup virtualization host. No Sanoid snapshots should be made of the datasets on the backup/destination virtualization host.

====SSH Keys====
Create ssh keys for passwordless login. The user on the source (the server you are going to backup) must have permissions to read the zfs dataset, use zfs allow to set proper permissions, here we will be using the sudo user as created in the Centos 8 Virtualization Server document so we will give the wheel group permissions. Since the Syncoid script will be run locally as root (on the destination or backup server) we are going to create the key pair under the root user profile.

==ZFS Permissions==
This is to be run on the source server, not the backup/destination server
<code>
sudo zfs allow -g wheel compression,clone,create,destroy,hold,promote,receive,rollback,send,snapshot,mount,mountpoint vg_images
</code>

On the backup/destination server create destination dataset for each server to be backed up
<code>
sudo zfs create vg_images/backups
sudo zfs create vg_images/backups/vhsrv01_vg_images
</code>


==Create Key Pair as Root==
<code>
ssh-keygen -C "root@<backup_server_name>" -f /root/.ssh/root@<backup_server_name>-<remote_user>@<remote_server_ip> -t ed25519
</code>
Example:
<code>
ssh-keygen -C "root@172.18.18.234" -f ~/.ssh/root@172.18.18.234-pladmin@172.18.18.236 -t ed25519
</code>

Repeat this for each additional host that you are going to backup.

==Copy Public Key== 
This will copy the public key to the user profile on the remote server that is to be backed up, this user profile should belong to the user that has permissions to read the ZFS dataset that is being backed up.
<code>
ssh-copy-id -i /root/.ssh/root@<backup_server_name>-<remote_user>@<remote_server_ip>.pub <remote_user>@<remote_server_ip>
</code>
Example:
<code>
ssh-copy-id -i ~/.ssh/root@172.18.18.234-pladmin@172.18.18.236.pub pladmin@172.18.18.236
</code>

==Test Key Login==
If it works correctly then there should be no password prompt.
<code>
ssh -i /root/.ssh/root@<backup_server_name>-<remote_user>@<remote_server_ip> <remote_user>@<remote_server_ip>
</code>
Example:
<code>
ssh -i /root/.ssh/root@172.18.18.234-pladmin@172.18.18.236 pladmin@172.18.18.236
</code>

====Syncoid Script====

==Create Syncoid Script==
If you are going to backup/replicate more than 1 host then create a separate script for each host with unique names.
<code>
mkdir /root/scripts
vim /root/scripts/syncoid_cold_backups.sh
</code>

Add the following and modify variables for your environment (add --compress=gzip if backing up over WAN only or if you don't use disk encryption at the guest level)
<file bash syncoid_with_static_snapshot_support.sh>
#!/bin/bash -vx
#
# This script is used to for syncoid replication from a remote host to a local
# host with the added feature of destroying existing snapshots. This is needed
# if you create snapshots with the same name over and over as snapshots can't
# overwritten, they must be destroyed and recreated. We make a specific
# named cold_backup when the virtual guests are powered off and it is this
# snapshot that we are creating this script for.
# https://stackoverflow.com/questions/1602378/how-to-calculate-a-hash-for-a-string-url-in-bash-for-wget-caching
# https://stackoverflow.com/questions/21208736/removing-on-linux-md5sum
#
# Variables
#
# This is the root of the dataset you are backing up, the source. Note, we are
# only backing up the child datasets, not the root.
ZFS_SOURCE_DATASET_ROOT="vhsrv02_vg_images"

# Name of ZFS snapshot that doesn't change, e.g. cold_backup
STATIC_ZFS_SNAPSHOT="cold_backup"

# This is the root of the dataset where the backup will go, the destination.
# Be sure to create the destination datasets beforehand!
ZFS_DESTINATION_DATASET_ROOT="vhsrv05_vg_backups/vhsrv02_vg_images"

# The SSH private key (full path). Note, the user on the remote host must have
# read access to the datasets you are backing up.
SSH_PRIVATE_KEY="/root/.ssh/root@VHSRV05-vastermin@VHSRV02"

# The remote user used for backups @ the remote host that will be backed up.
REMOTE_USER_AND_HOST="vastermin@172.18.18.172"

# End of Variables

#
# List snapshots of destination and source, remove all paths from and including @ to root of path
# Create MD5 of snapshot lists and compare if they are equal, if not then delete the destinstation static zfs snapshot
#
STATIC_SNAPSHOT_LIST_DESTINATION=`zfs list -r -t snapshot -o name,creation -s creation $ZFS_DESTINATION_DATASET_ROOT | grep ${STATIC_ZFS_SNAPSHOT}`
	echo Destination list: $STATIC_SNAPSHOT_LIST_DESTINATION

STATIC_SNAPSHOT_LIST_DESTINATION="${STATIC_SNAPSHOT_LIST_DESTINATION##*@}"
	echo Destination short list: $STATIC_SNAPSHOT_LIST_DESTINATION

SNAPSHOT_MD5_DESTINATION=`/bin/echo $STATIC_SNAPSHOT_LIST_DESTINATION | /usr/bin/md5sum | cut -d' ' -f1`

STATIC_SNAPSHOT_LIST_SOURCE=`ssh -i $SSH_PRIVATE_KEY $REMOTE_USER_AND_HOST "/sbin/zfs list -r -t snapshot -o name,creation -s creation $ZFS_SOURCE_DATASET_ROOT | grep $STATIC_ZFS_SNAPSHOT"`
	echo Source list: $STATIC_SNAPSHOT_LIST_SOURCE

STATIC_SNAPSHOT_LIST_SOURCE="${STATIC_SNAPSHOT_LIST_SOURCE##*@}"
	echo Source short list: $STATIC_SNAPSHOT_LIST_SOURCE

SNAPSHOT_MD5_SOURCE=`/bin/echo $STATIC_SNAPSHOT_LIST_SOURCE | /usr/bin/md5sum | cut -d' ' -f1`

if [[ "$SNAPSHOT_MD5_DESTINATION" != "$SNAPSHOT_MD5_SOURCE" ]]; then
        zfs list -r -H -o name -t snapshot ${ZFS_DESTINATION_DATASET_ROOT} | grep ${STATIC_ZFS_SNAPSHOT} | xargs -n1 zfs destroy
        echo Destination and Source $STATIC_ZFS_SNAPSHOT snapshots are NOT the same
        echo MD5 of Destination $SNAPSHOT_MD5_DESTINATION
        echo MD5 of Source $SNAPSHOT_MD5_SOURCE
fi

/usr/local/sbin/syncoid --recursive --skip-parent --dumpsnaps --no-privilege-elevation --sshkey=${SSH_PRIVATE_KEY} "${REMOTE_USER_AND_HOST}:${ZFS_SOURCE_DATASET_ROOT}" ${ZFS_DESTINATION_DATASET_ROOT}
</file>

Set the file permissions
<code>
chmod 770 /root/scripts/syncoid_cold_backups.sh
</code>

==Add Crontab Entry==
The frequency should be equal less than the frequency of the Sanoid snapshot policy of the source server. Add a new line for each host to be backed up.
<code>
crontab -e
</code>
Add the following for every hour on the 15th minute, though try to schedule just after the snapshots are made on the source server
<code bash>
#m(0-59) h(0-23) dom(1-31) m(1-12) dow(0 is Sunday) command
15 * * * * /root/scripts/syncoid_cold_backups.sh
</code>


The sanoid on destination server configured to remove old snapshots.





Note: auto rollback on syncoid will rollback the newest snapshot made, if this is the cold_backup or the regular sanoid backups it doesn't matter, it will rollback the newest snapshot. You can manually rollback a snapshot on the destination server if for instance you wanted to rollback to the last cold_backup.

The --no-rollback option broke our backup method since as a part of the method the @cold_backup snapshot needed to be deleted beforehand otherwise it failed, the --no-rollback option causes syncoid to fail if the destination dataset has been modified.

Note: for the first few months check the snapshot usage for both syncoid and sanoid (autosnap). I've had instances where syncoid didn't delete the previous syncoid snapshot, this is likely a permissions issue.

Note: disabling the swap file on Windows hosts greatly reduces the zfs snapshot size, no issues so far on a Server 2012R2 domain controller, WSUS server, Windows 7/10 hosts.