Virtual pfsense HA cluster deployed on KVM
Environment
KVM, pfsense
Description
My homelab setup has changed recently and as part of this i wanted to setup virtual firewalls. This was to keep the number of always on hardware low and i was sure the resource cost would be low for the KVMs on my 2 current physical servers.
Solution
This post shows the steps taken to get pfsense configured with HA clustering on KVM. KVM is required as we are running FreeBSD on linux. At a high level 2 kvm instances created with 3 virtual interfaces. 1 interface for the WAN, 1 for the LAN and 1 for the HA communication named SYNC. CARP virtual interfaces created for WAN/LAN with associated FW and NAT rules. Node exporter then configured and installed for metrics. OpenVPN configured for VPN of LAN outbound traffic.
Steps
Provision KVMs
2 KVM with following resources.
- CPU: 1
- RAM: 512MB
- STORAGE: 12GB
- NIC: 3 virtio for WAN/LAN/SYNC. WAN on bridge interface attached to router. LAN on bridge interface to network switch. SYNC i am using interface with crosover cable to other physical host running the BACKUP KVM instance.
Install pfsense
Download pfsense community edition (v2.4 at this time) AMD64 CDROM ISO from HERE
Install is pretty straightforward using the ISO. I didnt use zfs (although my kvm hosts do have zfs storage for snapshotting and other hot greatness). You then have to configure interfaces so you need to have your IPs and subnets planned out. I also enabled SSH to configure node_exporter remotely through the LAN interface and any troubleshooting i might need.
There is a full document for virtualised pfsense here this is still relevant so have a read. There is one very important part here that you have to do.
To disable hardware checksum offload, navigate under System > Advanced and select Networking tab. Under Networking Interfaces section check the Disable hardware checksum offload and click save. Reboot will be required after this step.
If you fail to do this virtual hosts on the same guest hardware will not be able to communicate with other networks i.e. no internet access. So make sure you disable it or tcpdump will show packet checksum errors.
Configure HA
With the MASTER and BACKUP pfsense instances installed and configured on KVM with SSH enabled. Ensure you can connect to the webmin on both as admin user and that the same password is setup for admin on both. Also ensure all interfaces are enabled and configured as this state is not replicated from the MASTER to the BACKUP via pfsense HA cluster.
There is a full document for HA installing here
Following this document briefly make sure you have:
- Setup interfaces for WAN LAN SYNC
- Add FW rules for SYNC interfaces
- Setup HA on MASTER only to replicate to BACKUP (you cant replicate both ways)
- Setup 2xCARP address for WAN and LAN
- setup manual NAT outbound. Edit all rules to use the CARP VIP on the WAN interface.
- Point LAN hosts to LAN CARP VIP as gateway.
At this point you could power off MASTER and test failover of the CARP virtual interfaces to the BACKUP.
Install and Configure Node Exporter for prometheus
I use prometheus for metrics. In order for prometheus to collect metrics it needs to scrape the agent node exporter. Installation isnt so bad but you need to enable FreeBSD repos on pfsense which it is based on.
SSH to each host and edit the files for packages, you need to set enabled from ‘no’ to ‘yes’
“enabled: yes”
/etc/pkg/FreeBSD.conf
/usr/local/etc/pkg/repos/FreeBSD.conf
/usr/local/etc/pkg/repos/pfSense.conf
Install node exporter with the following at the command line
pkg install node_exporterNext enable the service either one of the bellow will do
echo "node_exporter_enable="YES"" >> /etc/rc.confsysrc node_exporter_enable=YESStart the service
service start node_exporterAdd firewall rule for 9100 on LAN interface in pfsense for MASTER this should replicate to BACKUP confirm this.
Confirm node exporter is online by opening the agent page. It will be on the LAN IP of the host port 9100 e.g. if your LAN was 192.168.1.0/24 and you set MASTER to 1 BACKUP to 2
http://192.168.1.1:9100/metrics http://192.168.1.2:9100/metrics
Your prometheus yaml config should have entry similar to this:
- job_name: 'pfsense1'
static_configs:
- targets: ['192.168.1.1:9100']
- job_name: 'pfsense2'
static_configs:
- targets: ['192.168.1.2:9100']
Grafana dashboard for prometheus data source
I use grafana to create dashboards to visualise prometheus data sources. This dashboard is updated version of one i found on grafana website. Link below to download and import.
Once imported there are dashboard variables for instance and port which depend upon the job_name in the prometheus yaml config beginning with pfsense (obviously you can change this).
VPN
Setup documentation is availble from providers 2 examples here:
Remember to set NAT Outbound rules to use translation address ‘interface address’ and not the CARP IP Address for WAN. https://ipleak.net/ should then show your VPN provider IP address and not your routers ISP assigned address.
Conclusion
This setup archives a low cost and highly available firewall working very well with the light resources i assigned to the KVMs. This is backed up by the metrics visualised in the grafana dashboard collected by node exporter.
You can pass devices through the firewall without routing over the VPN. At home you might do this for your TV for apps like netflix. You could also setup 2 subnets for services and personal devices this way your homelab goes out over the ISP but your laptop and phone are out over the VPN, which tested 80% the speed of my ISP line for a single personal device).