If 2 SSD + 2 HDD then mirror /boot / and swap on SSD, leave rest blank for ZFS
If 2 SSD/HDD then mirror /boot / and swap on SSD/HDD, leave rest blank for ZFS
If 4 SSD/HDD then mirror /boot / and swap on 2 of SSD/HDD and leave rest blank for ZFS

For systems with 4 drives or more you can choose between using all disks for the OS and virtual guests or leave a pair of disks empty that will be configured as a mirrored backup destination later. If you don't have another network location to backup your guests to then a local mirror is highly recommended.

When paritioning, /boot should be XFS 2GB on a standard R1 array, if EFI (600MB or so) is used then put in on the same R1 array, swap and / should be on an LVM volume group which is R1, / should be 40GB + the amount of the RAM the system has, swap should be 4GB if swap won't be used or if swap will be used then same size as the amount of system RAM.

Note: if using a software R1, install grub2 on both drives that will participate in the R1 array or the R1 part of the R10 array. (grub2-install /dev/sda and grub2-install /dev/sdb or whatever device name you have; also make sure both drives are in the BIOS boot order list if possible).

The reason for the adding the amount of RAM in GB to your / partition is that the virtual guests will save their RAM states here on suspend.

Optionally use LUKS2 to encrypt the swap and / partitions, we will later use NBDE to auto-mount the encrypted partitions. Use LVM to create R1 array and set the encryption at the LVM level.

Setup network as you see fit, DHCP to start with is fine, this must be working and enabled for NTP to be configured. Ideally use 1 NIC for access/management bridge and 1+ NICS for use as network bridge for virtual guests, or use bonding and VLANs and bridges (configure the bridge NICs later).

Select your timezone, enable NTP

Minimal

Set root password, uncheck “lock root account” and check “Allow root SSH login with password” then start the install…

Update OS and install basic utils

dnf update
dnf install vim tar

useradd example_user && passwd example_user
usermod -aG wheel example_user
reboot

Login using sudo user

sudo rm /etc/ssh/sshd_config.d/01-permitrootlogin.conf
sudo vim /etc/ssh/sshd_config.d/03-ssh_options.conf

then set

PermitRootLogin no

Do this if you're going to rsync large files over SSH, otherwise the connection will close while rsync is verifying data and not transmitting data over SSH

ClientAliveInterval 60
ClientAliveCountMax 10

sudo systemctl restart sshd

Read more about ZFS to fully realize all of it's utility ZFS on CentOS 8
Note: if you have Secure Boot enabled you may not be able to load the ZFS module. (you'll get the error “odprobe: ERROR: could not insert 'zfs': Key was rejected by service”)

!!!ZFS stops loading between point releases, read the ZFS article listed above for a better understanding and to plan updates for point releases!!!

sudo dnf install epel-release wget
sudo dnf install https://zfsonlinux.org/epel/zfs-release-2-2$(rpm --eval "%{dist}").noarch.rpm

Install:

sudo dnf install zfs

Limit the amount of RAM it uses:

sudo vim /etc/modprobe.d/zfs.conf

Add: (This will use 2GB of your system RAM, adjust as needed.)

# Min/Max 2048 MB Limit
options zfs zfs_arc_max=2147483648
options zfs zfs_arc_min=2147483648

Load ZFS module

sudo /sbin/modprobe zfs

Do this on the drives that have the /boot / and swap paritions already. Read through the whole ZFS section though if you have 4 or more drives as your layout might be different from this 2 drive example.

Get your storage device info, here we'll assume you have 2 drives, sda and sdb

sudo lsblk

sudo parted -a optimal /dev/sda
print free

Take the Start and End Values of your Free Space and use them to create the parition

mkpart primary 72.0GB 500GB
print
quit

Repeat for each disk that will have free space added to the zpool. Run

sudo lsblk

to get a list of the partitions that will be used in the zpool. If you created a boot / and swap partition then the ZFS partition should be sda4 and sdb4.

Optionally but recommended (since zpools get dropped on disk shuffles when using device names only): After you make your partitions, get the Drive IDs and use these to create the ZFS pools.

cd /dev/disk/by-id
ls

Should give you something like:

ata-Samsung_SSD_860_EVO_1TB_S3Z8NY0M742137W
ata-Samsung_SSD_860_EVO_1TB_S3Z8NY0M742137W-part1
ata-Samsung_SSD_860_EVO_1TB_S3Z8NY0M742137W-part2
ata-Samsung_SSD_860_EVO_1TB_S3Z8NY0M742137W-part3
ata-Samsung_SSD_860_EVO_1TB_S3Z8NY0M742137W-part4
ata-Samsung_SSD_860_EVO_1TB_S3Z8NY0M742149J
ata-Samsung_SSD_860_EVO_1TB_S3Z8NY0M742149J-part1
ata-Samsung_SSD_860_EVO_1TB_S3Z8NY0M742149J-part2
ata-Samsung_SSD_860_EVO_1TB_S3Z8NY0M742149J-part3
ata-Samsung_SSD_860_EVO_1TB_S3Z8NY0M742149J-part4
nvme-Samsung_SSD_980_PRO_2TB_S6B0NL0T938556N
nvme-Samsung_SSD_980_PRO_2TB_S6B0NL0T938608D

In the above example we have 2 1TB Samsung SSDs with 4 paritions each, the 4th will be used for ZFS and the 2 Samsung NVME with no paritions, both full drives will be dedicated to ZFS

Enable ZFS encryption
https://serverfault.com/questions/972496/can-i-encrypt-a-whole-pool-with-zfsol-0-8-1

Create your 256bit keyfile
https://unix.stackexchange.com/questions/19013/how-to-use-keyfiles-for-zfs-whole-disk-encryption

sudo mkdir /root/.keys
sudo openssl rand -out /root/.keys/zfs_vhsrv01_vg_images_256.key 32
sudo chown -R root:root /root/.keys

Create the zpool (here were are using SSDs and mirroring them, mounting them at /var/lib/libvirt/images and giving the zpool a name of vhsrv01_vg_images where vhsrv01 is the name of the virtualization host) Note: Use ashift=13 on any Samsung SSD 850 era and newer, use ashift=12 on hard drives.

sudo zpool create -f -o ashift=13 -o feature@encryption=enabled -O encryption=on -O keylocation=file:///root/.keys/zfs_vhsrv01_vg_images_256.key -O keyformat=raw -m /var/lib/libvirt/images vhsrv01_vg_images mirror ata-Samsung_SSD_860_EVO_1TB_S3Z8NY0M742137W-part4 ata-Samsung_SSD_860_EVO_1TB_S3Z8NY0M742149J-part4

To load mount the encrypted ZFS pool automatically:

sudo vim /etc/systemd/system/zfs-load-key@.service

Add

[Unit]
Description=Import key for ZFS pool
Documentation=man:zfs(8)
DefaultDependencies=no
After=systemd-udev-settle.service
After=zfs-import.target
After=systemd-remount-fs.service
Before=zfs-mount.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/sbin/zfs load-key -r %i

[Install]
WantedBy=zfs.target

Enable it:

sudo systemctl enable zfs-load-key@vhsrv01_vg_images

Or the equivalent of RAID10 with 4 HDDs

sudo zpool create -f -o ashift=12 -m /var/lib/libvirt/images vhsrv01_vg_images mirror /dev/sda4 /dev/sdb4 mirror /dev/sdc4 /dev/sdd4

Here vhsrv01_vg_images is the pool name

sudo zfs set xattr=sa vhsrv01_vg_images
sudo zfs set acltype=posixacl vhsrv01_vg_images
sudo zfs set compression=lz4 vhsrv01_vg_images
sudo zfs set atime=off vhsrv01_vg_images
sudo zfs set relatime=off vhsrv01_vg_images

If you are using SSDs please enable TRIM
To run trim:

sudo zpool trim vhsrv01_vg_images

To check trim status:

sudo zpool status -t vhsrv01_vg_images

To make it automatic

sudo zpool set autotrim=on vhsrv01_vg_images

Note: autotrim seems like it may not work, consider setting a cron command instead…

This will verify the integrity of the data on the drives and repair any errors.

You should do this weekly if you are using cheap drives like I am:

sudo vim /etc/crontab

Add: (this will scrub every Sunday at 2am, be sure it isn't schedules with other disk intensive activities. If you have another zpool on a separate set of drives you can schedule those at the same time)

0 2 * * 0 root /usr/sbin/zpool scrub vhsrv01_vg_images

If you have 4 or more drives and aren't using them all in a R10 array then use 2 of them for backups of the virtual guest disk images. Your /boot / and swap should already be on the larger/slower HDDs if going this route and the other 2 drives should be SSDs dedicated to /var/lib/libvirt/images.

In this case you'll have sda4 and sdb4 mounted at /vg_backups with a zpool name of vhsrv01_vg_backups

sudo zpool create -f -o ashift=12 -m /vg_backups vhsrv01_vg_backups mirror /dev/sda4 /dev/sdb4
sudo zfs set xattr=sa vhsrv01_vg_backups
sudo zfs set acltype=posixacl vhsrv01_vg_backups
sudo zfs set compression=lz4 vhsrv01_vg_backups
sudo zfs set atime=off vhsrv01_vg_backups
sudo zfs set relatime=off vhsrv01_vg_backups
sudo zpool set autotrim=on vhsrv01_vg_backups

And will use sdc and sdd for /var/lib/libvirt/images

sudo zpool create -f -o ashift=13 -m /var/lib/libvirt/images vhsrv01_vg_images mirror /dev/sdc /dev/sdd
sudo zfs set xattr=sa vhsrv01_vg_images
sudo zfs set acltype=posixacl vhsrv01_vg_images
sudo zfs set compression=lz4 vhsrv01_vg_images
sudo zfs set atime=off vhsrv01_vg_images
sudo zfs set relatime=off vhsrv01_vg_images
sudo zpool set autotrim=on vhsrv01_vg_images

Omit the autotrim=on line if the drive is a HDD.

Add the scrub cron job for both in /etc/crontab

0 2 * * 0 root /usr/sbin/zpool scrub vhsrv01_vg_images
0 2 * * 0 root /usr/sbin/zpool scrub vhsrv01_vg_backups

Give permission to your wheel group (which your sudo user is a part of) to the ZFS datasets/pools

sudo zfs allow -g wheel compression,clone,create,destroy,hold,promote,receive,rollback,send,snapshot,mount,mountpoint vhsrv01_vg_images
sudo zfs allow -g wheel compression,clone,create,destroy,hold,promote,receive,rollback,send,snapshot,mount,mountpoint vhsrv01_vg_backups

If you want to “zfs send” from another host to this host using a sudoer then you'll need add the following permissions to the visudo file otherwise it will prompt you for a password helper and won't work.

sudo visudo

Add the following (omit the @syncoid lines if you aren't using sanoid/syncoid

%wheel ALL=(ALL) NOPASSWD:/sbin/zfs get *
%wheel ALL=(ALL) NOPASSWD:/sbin/zfs snapshot *
%wheel ALL=(ALL) NOPASSWD:/sbin/zfs receive *
%wheel ALL=(ALL) NOPASSWD:/sbin/zfs rollback *@syncoid*
%wheel ALL=(ALL) NOPASSWD:/sbin/zfs destroy *@syncoid*

Create datasets in the ZFS pool for each virtual guest or virtual guest image if you want very granular snapshot control.

This will also create the directory /var/lib/libvirt/images/vm_guest_name

sudo zfs create vhsrv01_vg_images/vm_guest_name

Create RAW images via qemu-img instead of the Virt-gui as it defaults to falloc allocation which will take forever.

sudo qemu-img create -f raw /var/lib/libvirt/images/vm_guest_name/VM_GUEST_NAME.img 50G -o preallocation=off

When using the Virt-Manager to create a virtual guest, to select the disk image you just created via qemu-img, use “browse local → /var/lib/lib/images/vm_guest_name/VM_GUEST_NAME.img” to select the disk image as it won't be listed in the default storage pool.

This is should be used if you like it or aren't installing Gnome Session.

sudo dnf install cockpit cockpit-machines cockpit-pcp virt-install qemu-kvm libvirt tuned
sudo systemctl enable cockpit.socket
sudo systemctl start cockpit.socket
sudo systemctl enable pmlogger
sudo systemctl start pmlogger
sudo firewall-cmd --add-service=cockpit --permanent
sudo firewall-cmd --reload

Access the cockput via https://ip.add.re.ss:9090/ or remove the firewall rule and tunnel through SSH. Follow the example of the tunnel by the VNC server below, just use a web browser to open localhost:9090 after the tunnel is established.

If you want a custom install more than cockpit can give you create the guest via the cli:

sudo virt-install \
--name SERVER2022 \
--os-type=windows \
--os-variant=win2k22 \
--ram=4096 \
--vcpus=4 \
--cpu host-passthrough,cache.mode=passthrough,topology.sockets=1,topology.cores=4,topology.threads=1 \
--disk bus=scsi,cache='none',io='native',discard='unmap',detect_zeroes='unmap',path=/var/lib/libvirt/images/server2022/SERVER2022.img \
--controller type=scsi,model=virtio-scsi \
--disk /var/lib/libvirt/images/virtio-win.iso,device=cdrom,bus=sata \
--cdrom=/var/lib/libvirt/images/Server2022.iso \
--network bridge=virbr0,model=virtio \
--graphics vnc \
--video qxl \
--input tablet \
--boot cdrom,hd,menu=on \
--controller type=virtio-serial \
--channel unix,mode=bind,path=/var/lib/libvirt/qemu/guest01.agent,target_type=virtio,name=org.qemu.guest_agent.0 \
--controller type=usb \
--sound default \
--autostart \
--memballoon virtio

To get OS variants:

sudo osinfo-query os

Common variants: linux2020, win10, win2k22, win2k12r2

This is optional if you're using Cockpit, the recommendation being use Cockpit only to save resources.

Install basic Gnome + Utilities:

sudo dnf install gnome-classic-session gnome-terminal nautilus-open-terminal control-center liberation-mono-fonts vim tar gnome-disk-utility gnome-system-monitor firefox

If you want Gnome to load on reboot run the commands below (though you don’t need to if you are only going to use VNC for remote management)

sudo unlink /etc/systemd/system/default.target
sudo ln -sf /lib/systemd/system/graphical.target /etc/systemd/system/default.target

Install Virtualization packages (virt-manager for a nice GUI for your virtual clients, skip it though if you're not using Gnome GUI)

  
sudo dnf install libvirt virt-manager virt-top tuned

https://computingforgeeks.com/use-virt-manager-as-non-root-user/

sudo usermod -aG libvirt example_user

Edit libvirtd.conf

sudo vim /etc/libvirt/libvirtd.conf

Uncomment/edit the following:

unix_sock_group = "libvirt"
unix_sock_ro_perms = "0770"
unix_sock_rw_perms = "0770"

You may also want to give the libvirt group r/w access to the /var/lib/libvirt/images directory so that you can scp remotely to and from using your sudo user since root ssh is disabled. If you are going to use Sanoid/Syncoid this will be needed and if you copy a guest image from another host run these commands again to update it's permission; maybe even put this in a daily script…

sudo chown -R root:libvirt /var/lib/libvirt/images
sudo chmod -R 771 /var/lib/libvirt/images
sudo chmod -R g+s /var/lib/libvirt/images

And if you've got a /vg_backups mount point:

sudo chown -R root:libvirt /vg_backups
sudo chmod -R 771 /vg_backups
sudo chmod -R g+s /vg_backups

Restart libvirtd

sudo systemctl restart libvirtd

Sometimes on boot, parallel startup of guests has caused some guests to have high CPU usage, even when idle, rebooting these again fixes it. Having guests start up in series instead of parallel seems to fix this. Also, if we initiate a shutdown without shutting down guests first we want guests to shutdown gracefully before the host shuts down. https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/sect-shutting_down_rebooting_and_force_shutdown_of_a_guest_virtual_machine-manipulating_the_libvirt_guests_configuration_settings

!!!!!!!Is empty in Rocky 9 by default, test it afterwards!!!!!
sudo vim /etc/sysconfig/libvirt-guests

Set or uncomment the following variables:

#Boot guests that were on when the host was shutdown, AUTOSTART guests will still be booted regardless
ON_BOOT=start
#Delay 10 seconds between starting each guest
START_DELAY=10
#Shutdown guests on host shutdown
ON_SHUTDOWN=shutdown
#Shutdown up to 20 guests at a time
PARALLEL_SHUTDOWN=20
#Give all guests a total of 300 seconds to shutdown
SHUTDOWN_TIMEOUT=300

Enable and start the libvirt-guest service. Note that restart the service will shutdown guests.

sudo systemctl enable libvirt-guests
sudo systemctl start libvirt-guests

sudo dnf install chrony
sudo systemctl enable chronyd
sudo systemctl start chronyd

If you used SSD drives on MDADM (linux software RAID) then enable FSTRIM service for cleanup

sudo systemctl enable fstrim.timer
sudo systemctl start fstrim.timer

Check status of timer by showing systemd timers:

systemctl list-timers

Check trim support by:

sudo lsblk --discard

Stop writing a timestamp every time a file is accessed Edit the /etc/fstab file and replace all the defaults strings by defaults,noatime.

sudo vim /etc/fstab

For example:

/dev/mapper/rhel-root   /         xfs     defaults        1 1

becomes:

/dev/mapper/rhel-root   /         xfs     defaults,noatime        1 1

OBSOLETE??? If using LVM enable Trim function by editing /etc/lvm/lvm.conf and change

issue_discards = 0

to

issue_discards = 1

OBSOLETE END???

Other notes for using SSDs: https://www.certdepot.net/rhel7-extend-life-ssd/

Set the proper performance profile via tuned-adm:

sudo systemctl enable tuned
sudo systemctl start tuned
sudo tuned-adm profile virtual-host

then check to make sure:

sudo tuned-adm list

This should adjust the swappiness, change to the deadline scheduler and other things.

By default swappiness is set to 10 with the virtual-host profile, if you really want to try to avoid using RAM set it to 1, though make sure you have enough RAM for all of your guests. Avoiding swaps if your swap file is on a SSD is good, otherwise the default of 10 should be fine for spinning disks. You might want to set your virtual guests that run linux the same so they avoid swapping if posssible.

sudo vim /etc/sysctl.conf

Add the following:

vm.swappiness = 1

I'm not sure of the cause but this is the fix, research and determine if a better option exists. This is needed only for some architectures.

sudo vim /etc/modprobe.d/kvm.conf

Add the line:

options kvm ignore_msrs=1

Do this only if you've installed the Gnome GUI

Install a VNC server so you can quickly manage the VMs remotely:

sudo dnf install tigervnc-server

Edit the VNC user file and add your user/port

sudo vim /etc/tigervnc/vncserver.users

Example (this runs a server on port 5908 for user vadmin)

:8=vadmin

Edit the VNC config file and add gnome as your session manager

sudo vim /etc/tigervnc/vncserver-config-defaults

Add the following

session=gnome

Set VNC password, set a password different from your sudo user password, say no to view only password.

vncpasswd

Connect on port 5908 using your VNC viewer (preferably TigerVNC)

Note: If you connect via TigerVNC viewer it will show that your connection is insecure. This is because the certificates used aren't trusted, however TLS encryption should be active, you can verify this by pressing F8 when using TigerVNC viewer and checking connection info.

Allow VNC server access, or skip the firewall rules and tunnel through port 22 instead for more security.

sudo firewall-cmd --permanent --zone=public --add-port=5908/tcp
sudo firewall-cmd --reload
sudo systemctl daemon-reload
sudo cp /lib/systemd/system/vncserver@.service /etc/systemd/system/vncserver@:8.service
sudo systemctl enable vncserver@:8.service
sudo systemctl start vncserver@:8.service

https://www.techrepublic.com/article/how-to-connect-to-vnc-using-ssh/
If doing the SSH tunnel instead on your local computer (not the virtualization host)

ssh -L 5908:localhost:5908 user@REMOTE_IP

If using putty, go to the SSH → tunnel area, put in REMOTE_IP:5908 in the destination and 5908 in the Source port then click on Add. Go back to the session then connect like normal via SSH.

Open up Tiger VNC viewer on your local computer and connect by:

localhost:5908

https://idroot.us/install-fail2ban-centos-8/ https://www.digitalocean.com/community/tutorials/how-to-protect-an-apache-server-with-fail2ban-on-ubuntu-14-04

sudo dnf install fail2ban

sudo vim /etc/fail2ban/jail.d/sshd.local

Add the following:

[sshd]
enabled = true
port    = ssh
logpath = %(sshd_log)s
backend = %(sshd_backend)s

[selinux-ssh]
enabled  = true
port     = ssh
logpath  = %(auditd_log)s

sudo systemctl start fail2ban
sudo systemctl enable fail2ban
sudo fail2ban-client status sshd

*Note: on older Dell servers DSU isn't supported anymore, definitely not R710/R410/R210 generation*

If you are using a Dell server it is recommended that you download and install Dells OMSA and use Dell System Update to update things. Dell: http://linux.dell.com/repo/hardware/dsu/ Set up the Dell OpenManage Repository at like this:

*Note: OMSA is EOL as of 2024 and support will be ended in 2027*

curl -O https://linux.dell.com/repo/hardware/dsu/bootstrap.cgi
sudo bash bootstrap.cgi
sudo dnf install yum-utils
sudo dnf config-manager --set-enabled crb
sudo dnf install srvadmin-all dell-system-update tar

run dsu to update firmware/bios/etc

sudo dsu

If you have a host that has no Internet connection you can download the Dell Server Update Utility in ISO format and use that to update your offline server. First download the ISO from Dell for your server (Systems Management→Dell Server Update Utility) then mount and run:

sudo mkdir /mnt/disc
sudo mount -o loop SUU_930-x64-LIN-74.ISO /mnt/disc
cd /mnt/disc
sudo ./suu --import-public-key (1st time)
sudo ./suu (all other times)
sudo ./suu --compliance (to check without doing anything)

sudo ./suu --import-public-key
Reading the catalog ...
Getting System Inventory ...
Determining Applicable Updates ...
Installing iDRAC-with-Lifecycle-Controller_Firmware_X89ND_LN64_7.10.50.10_A00
Installed successfully
Installing Diagnostics_Application_PHPW6_LN64_4303A19_4303.19
Installed successfully
Installing Drivers-for-OS-Deployment_Application_KC3R4_LN_24.05.04_A00
Installed successfully
Installing Systems-Management_Application_7Y30Y_LN64_5.3.1.0_A00
Installed successfully
Installing Network_Firmware_RXW7G_LN64_22.91.5_01
Installed successfully
Installing Network_Firmware_234W1_LN_23.0.8_A00_01
Installed successfully
Please restart the system for successful update(s) to take effect
Progress report is available at:/usr/libexec/dell_dup/SUU_STATUS.json
Exiting DSU!

Note: The ISO may not have the latest versions available. An ISO downloaded on Nov 1st 2024 didn't have a BIOS update that was released on Sept 24 2024. Check Dell's site for the most current updates and download manually if needed.

Note: you can login to OMSA from the local computer at: https://localhost:1311
Login as root otherwise you won't be able to change things.

Network Bridge for virtual clients. It is recommended to have a dedicated (or multiple dedicated) bridge(s) for your clients(if more than 1 bridge, separate bridges need to be vlan’ed or go to separate networks or you’ll create a loop), lag groups for better throughput is good too. Also, use a separate network card for management and file transfers that won’t interfere with bridged network traffic:

Use a consistent bridge name across hosts to move VMs is easy, remember case sensitive to! Recommend naming them BR0, BR1, BR2, BRX. Please try to have consistency across hosts so if you have two bridges on 1 host, have 2 bridges on all others configured the same way, connected to the same switched network.

To find the HWADDR do this:

ip link

Record both the MAC addresses and the interface name.

Since this is RHEL 9, they don't like network-scripts any more so they've made something much less appealing in /etc/NetworkManager/system-connections/, go in there and delete everything… except the interface you're using to connect via SSH (if you're doing so). I really like the virtual clipboard in idrac 9.

In the /etc/sysconfig/network-scripts directory it is necessary to create 2 config files. The first (ifcfg-eth0) (or ifcfg-em1 or em0 or eth0 etc) defines your physical network interface, and says that it will be part of a bridge:

 
sudo vim /etc/sysconfig/network-scripts/ifcfg-eno1

Configure as so:

DEVICE=eno1
HWADDR=00:16:76:D6:C9:45 (Use you actual HWADDR, or mac address here)
ONBOOT=yes
BRIDGE=br0

The second config file (ifcfg-br0) defines the bridge device:

 
sudo vim /etc/sysconfig/network-scripts/ifcfg-br0

Configure as so:

DEVICE=br0
TYPE=Bridge
BOOTPROTO=none
ONBOOT=yes
DELAY=2

WARNING: The line TYPE=Bridge is case-sensitive - it must have uppercase 'B' and lower case 'ridge'

Also, if you have only 1 Ethernet adapter you will want to give the Bridge device an IP on your LAN for management, see static IP example below. After changing this restart networking (or simply reboot) .

nmcli connection reload && systemctl restart NetworkManager

Sometimes even this won't reload devices so if you don't want to reboot you can disconnect and reconnect the devices

nmcli device disconnect eno1
nmcli device connect eno1

Example of ifcfg-br0 for static IP:

DEVICE=br0
TYPE=Bridge
BOOTPROTO=none
ONBOOT=yes
DELAY=2
IPADDR=10.222.190.249
NETWORK=10.222.190.0
NETMASK=255.255.255.0
GATEWAY=10.222.190.250
DNS1=208.67.220.220
DNS2=208.67.222.222

Create datasets in the ZFS pool (we use separate datasets per vm image for snapshot purposes)

sudo zfs create vhsrv01_vg_images/vm_guest_name

If using datasets per VM then create the dataset first then the VM in the dataset, moving it afterward is like moving between real partitions and will take a while. Also, create RAW images via qemu-img instead of virt-gui as it defaults to falloc allocation which will take forever, use this instead:

sudo qemu-img create -f raw VIRTUAL-GUEST-NAME.img 50G -o preallocation=off

This is used if you are going to copy virtual guests between hosts

Install NFS packages and enable services

sudo dnf install nfs-utils libnfsidmap
sudo systemctl enable rpcbind
sudo systemctl enable nfs-server
sudo systemctl start rpcbind
sudo systemctl start nfs-server
sudo systemctl start rpc-statd
sudo systemctl start nfs-idmapd

make a directory called VG_BACKUPS in /var/lib/libvirt/images

mkdir /var/lib/libvirt/images/VG_BACKUPS

We have to modify “/etc/exports“ file to make an entry of directory “/var/lib/libvirt/images” that you want to share .

sudo vim /etc/exports

Example of exports file

/var/lib/libvirt/images 172.21.21.0/24(rw,sync,no_root_squash,fsid=<somenumber>)

/var/lib/libvirt/images: this is the shared directory
172.21.21.0/24: this is the subnet that we want to allow access to the NFS share
rw: read/write permission to shared folder
sync: all changes to the according filesystem are immediately flushed to disk; the respective write operations are being waited for
no_root_squash: By default, any file request made by user root on the client machine is treated as by user nobody on the server.(Exactly which UID the request is mapped to depends on the UID of user “nobody” on the server, not the client.) If no_root_squash is selected, then root on the client machine will have the same level of access to the files on the system as root on the server.
fsid=somenumber gives the mount a unique id so that mounts are more easily managed by hosts. I recommend using the first and last octets of the host static IP as the “somenumber”

Export the the NFS share

sudo exportfs -r

We need to configure firewall on NFS server to allow client servers to access NFS shares. To do that, run the following commands on the NFS server.

sudo firewall-cmd --permanent --zone public --add-service mountd
sudo firewall-cmd --permanent --zone public --add-service rpc-bind
sudo firewall-cmd --permanent --zone public --add-service nfs
sudo firewall-cmd --reload

sudo dnf install nfs-utils libnfsidmap
sudo systemctl enable rpcbind
sudo systemctl enable nfs-server
sudo systemctl start rpcbind
sudo systemctl start nfs-server
sudo systemctl start rpc-statd
sudo systemctl start nfs-idmapd

Set SELinux options

sudo setsebool -P nfs_export_all_rw 1
sudo setsebool -P virt_use_nfs=on

Create mount points for NFS shares

sudo mkdir /mnt/ VHSRV02/VG_IMAGES

(where VHSRV02 is the remote computer name, make one for each mount you will have). Client FSTAB entry to mount NFS share:

172.18.18.24:/var/lib/libvirt/images     /mnt/VHSRV02/VG_IMAGES         nfs4    noauto,nofail,x-systemd.automount,_netdev,x-systemd.device-timeout=14,proto=tcp,rsize=131072,wsize=131072 0 0
192.168.21.14:/VG_BACKUPS     /mnt/VHSRV02/VG_BACKUPS               nfs4    noauto,nofail,x-systemd.automount,_netdev,x-systemd.device-timeout=14,wsize=131072,rsize=131072 0 0

If power is lost or server isn't shutdown cleanly then the VNC server service might not restart on boot and manually restarting the VNC service fails. Normally the fix is to delete the lock files in the /tmp folder.

https://access.redhat.com/discussions/1149233 Example:

[root@xxx ~]# ls /tmp/.X
.X0-lock   .X1-lock   .X11-unix/ .X2-lock
[root@xxx ~]# rm -Rf /tmp/.X0-lock
[root@xxx ~]# rm -Rf /tmp/.X1-lock
[root@xxx ~]# rm -Rf /tmp/.X11-unix
[root@xxx ~]# rm -Rf /tmp/.X2-lock

And when connecting be sure you're connecting via port 5908 if you followed the setup according to this document, so… ip.add.r.ess:5908 (otherwise it defaults to 5900).

If your virtual host becomes unplugged from a network switch then all network interfaces (bonds, bridges, vlans and vnets) will go down. On plugging it back in the bonds, bridges and vlans will come back up automatically but the vnets won't. This means your virtual guests won't have network access until your shut them down then back on. Using

ip link setup vnet<x> up 

seems like it brings the interface up and the guest can ping out but devices on the other side of the vnet interface can't seem to get in. Still working on an automated way to fix this. Nice IP command cheatsheet from Redhat: https://access.redhat.com/sites/default/files/attachments/rh_ip_command_cheatsheet_1214_jcs_print.pdf

I am still unclear if the guests cleanly shutdown when the host is issued a shutdown -r now, there is a config file

/etc/sysconfig/libvirt-guests

where options can be set on what to do but I haven't tested them. Here is a link to some info from Redhat: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/sect-shutting_down_rebooting_and_force_shutdown_of_a_guest_virtual_machine-manipulating_the_libvirt_guests_configuration_settings

Also, if you shutdown your virtual guests to do dnf updates on the host, if any of the guests are set to autoboot at startup then will automatically start after an update to libvirt is installed. They will also do this if you restart libvirtd.