Tuesday, February 10, 2015

Running pfSense in Proxmox/KVM with PCI Passthrough

Below is how I was able to get pfSense 2.2 running under Proxmox 3.3 with PCI passthrough for two Intel NICs. My first attempts were trying to utilize VirtIO and e1000 network devices but the performance was abysmal. With PCI passthrough I was able to achieve native throughput in my environment.
I am assuming that you have Proxmox running and a pfSense virtual machine already created.

Configure the Proxmox Test Repository

The first thing we need to do is enable the Proxmox test repository so that we may install the 3.10 kernel.
echo 'deb wheezy pvetest' >> /etc/apt/sources.list

Install the 3.10 Kernel

apt-get install pve-kernel-3.10.0-6

Edit Grub Configuration

We need to pass a kernel flag to enable IOMMU. In my case I am using an AMD processor and added amd_iommu=on to the following line in /etc/default/grub. If you are using an Intel processor you would add intel_iommu=on.
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on"
Update the Grub configuration
Reboot the server. By default the 3.10 kernel should be selected.

Identify Your NICs

Identify the PCI devices you want to passthrough to your virtual machine.
In my case I was looking for my Intel NICs.
lspci | grep Intel

03:00.0 Ethernet controller: Intel Corporation 82541PI Gigabit Ethernet Controller (rev 05)
04:00.0 Ethernet controller: Intel Corporation 82541PI Gigabit Ethernet Controller (rev 05)
You will need to note their addresses.

Edit the Virtual Machine Configuration

Below is an example of my working configuration. You can find these configurations in /etc/pve/qemu-server/. The file you are looking for will correspond with the virtual machine ID. In my case 100.conf.
boot: cdn
bootdisk: ide0
cores: 2
cpu: qemu32
hostpci0: 03:00.0,pcie=1,driver=vfio
hostpci1: 04:00.0,pcie=1,driver=vfio
ide0: local:100/vm-100-disk-1.qcow2,format=qcow2,size=16G
ide2: local:iso/pfSense-LiveCD-2.2-RELEASE-i386.iso,media=cdrom,size=206916K
machine: pc-q35-2.0
memory: 1024
name: pfSense
onboot: 1
ostype: other
smbios1: uuid=0f590e3e-88a0-4084-8a6f-f5a2380a01fa
sockets: 2
tablet: 0
Notice that I added the hostpci0, hostpci1, and machine options. The hostpciX options identify which PCI devices we want to passthrough. As we found above I was looking for my NICs at 03:00.0 and 04:00.0. The machine must be set to pc-q35-2.0 for PCI passthrough to work with FreeBSD from my experience.


With those options set you should be able to boot your pfSense virtual machine and see your PCI devices natively.