Virtualised pfSense on Proxmox with Open vSwitch

In my recent post about my networking setup I mentioned that my firewall is a virtualised pfSense system running on a Proxmox host. In the comments to that post I was also asked if I was making use of Open vSwitch. Since the answer is that I use Open vSwitch in my pfSense/Proxmox setup, I thought I’d write up my setup for those that are interested.

I’ve actually been meaning to write this up for a long time. I’ve had this setup running since shortly after we moved into this house. On the one hand this means that the setup is pretty battle tested. All of the inter-VLAN and Internet bound traffic on my network runs through this and it’s been running pretty flawlessly for nearly two years.

On the other hand, given the length of time that has elapsed since I set this up and the writing of this post it means that this will be more like archeological exploration than documentation! I’m unlikely to remember every detail or the issues I encountered along the way. As such this post will pretty much document the state of the setup as I can extract it from the running system! Basically, you should only use this post as a rough guide and go away and do your own research. I’ll apologise for this incompleteness in advance. If you try this please let me know of anything I’ve missed and I’ll update the post with extra details.

How’s this going to work?

The basic premise of this whole thing is a Proxmox host with two physical NICs. One of these is the LAN port on which the host will have it’s internal IP. The second is the WAN port, which is assigned directly to the pfSense VM. In my case this is complicated by networking setup required by our Fibre connections here in NZ. These require a connection to the Fibre ONT on VLAN10 over which a PPPoE session to the ISP is established.

Since the WAN interface is directly assigned to the VM, this is all handled internally to pfSense. This means that that the host machine is not exposed to the external network. [OK, for the purists among you, this isn’t strictly true. The host will be exposed at lower levels of the network stack to allow it to forward packets through to the VM. However, since it doesn’t have an IP address on that interface it won’t be accessible from the Internet. I’m sure someone out there will tell me why this is all kinds of horrible.]

On the LAN side we create an Open vSwitch switch and add the LAN interface as a VLAN trunk on it. Another (virtual) trunk interface goes into the pfSense VM and becomes it’s LAN interface. This is analogous to just having another physical switch between the host and the VM. The purpose of this extra complexity is that it allows us to connect other VMs on the host into the vSwitch. These can be in on multiple different VLANs if required.

Hopefully the diagram below makes this somewhat clearer:

pfsense proxmox open vswitch
Sometimes a picture really is worth a thousand words

The Proxmox Host

The Proxmox host itself is a Dual Ethernet Haswell based mini-computer from AliExpress. I’ve been really happy with this as a platfrom aside from the fact that I would have spec’d it with more than 4GB of RAM if I’d been intending to run Proxmox initially. I also added an extra 120GB SSD drive on the internal SATA port for VM storage.

I started out with this host running pfSense natively, which also worked fine. One thing I did find is that when I switched over to Proxmox (Linux based) from pfSense (FreeBSD based) it ran much cooler. I guess that’s just down the the Linux kernel’s better hardware support.

This host is still running Proxmox 5.4 since I haven’t had time to upgrade it to 6.0 yet. This system is pretty much as close to “production” as it gets for me, since the Internet is used all the time!

Proxmox Network Setup

Proxmox enumerates the two NICs as ens1 (LAN) and enp1s0 (WAN). With the WAN port, I created a simple Linux Bridge vmbr1 to allow it to be added to the pfSense VM.

On the LAN side, I created an “OVS Bridge” port and added an “OVS IntPort” named admin which will be the primary interface to the host machine. As such, this interface is assigned a static IP and is assigned to the VLAN that we want the host to be on.

pfsense proxmox open vswitch
The network setup in Proxmox

I have to give kudos here to the Proxmox developers. They’ve made the Open vSwitch setup here pretty much trivial! For what I would consider advanced functionality it’s just as easy as configuring any other network.

A note should also be given here as to what’s going to happen when you configure this. By design Proxmox doesn’t apply any networking changes until you reboot. This is pretty useful to prevent yourself getting locked out. If you are connected directly on the LAN interface (with a static IP) you should make sure that everything is correct before rebooting. After the reboot, reconfigure your local interface to the VLAN you chose in the setup and a static IP. You should then be able to access the Proxmox web interface again.

Setting Up pfSense

The pfSense installation was fairly standard. The only change I ended up making was to change the default CPU type to enable AES-NI instructions. This took a little bit of experimentation and looking up the capabilities of various processors, but I finally settled on the “Westmere” processor.

pfsense proxmox open vswitch
I selected a “Westmere” processor in order to make AES-NI instructions available to the VM

After setting this architecture in the VM settings, rebooting pfSense shows both the correct CPU architecture and that AES-NI is available. It seems that this is probably less important than it was when I set up the system, since Netgate have now decided that AES-NI will not be required for pfSense 2.5.0.

pfsense proxmox open vswitch
AES-NI successfully enabled

One other thing is that you should disable hardware checksum offloading to work with the virtio drivers, as per the official documentation. Before you do this the network will be very sluggish.

Once the pfSense installation was complete I restored from a backup of my previous setup. This made the task of setting up my interfaces significantly easier. However, I’ll go through the networking aspects anyway for those who may be setting up a new system.

pfSense Networking

Luckily for us the pfSense tool to assign interfaces allows us to also set up the VLANs. This is useful to set up a minimal configuration to get you access to the web interface. Basically you want to set up the VLAN for your main LAN segment. Then you can set up the pfSense LAN interface on this VLAN with a static IP. If you’re using a fibre connection similar to mine you can skip the WAN setup for now. Once the “Assign Interfaces” wizard is complete you should have access to the Web Configurator.

The next step was to setup my WAN connection. I first added a VLAN with tag 10 on the vtnet0 device which is the device that corresponds to the physical WAN bridge as enumerated by pfSense. I added a corresponding interface for this and then added a PPPoE interface using the details provided by my ISP. This is then assigned to the WAN interface via the “Interface Assignments” page.

In terms of setting up the local networks, you can pretty much set up whatever VLANs you would like at this point. Take a look at my previous post for inspiration.

Conclusion

As stated earlier, I’ve found this setup to be very stable in production and it’s even made my hardware run cooler. Having my firewall virtualised has also had several other benefits for me. Firstly, I can backup and snapshot the firewall VM at will. I no longer need to worry about an update or bad configuration hosing my firewall. I just snapshot before doing anything major and roll back if anything goes wrong.

The second major benefit is that I can run extra VMs and containers on the host machine, which I couldn’t when it was a dedicated firewall. I’ve used this to implement my small DMZ for Internet facing services. This has the added benefit that DMZ traffic only transits the vSwitch internal to the host and doesn’t have to be shuttled back and forth over the physical network infrastructure. This is much faster, since the virtualised interfaces should (in theory) be 10GBps. However, this is somewhat irrelevant when the upstream Fibre connection is only 100Mbps.

As always, I’m keen to receive feedback and constructive criticism of this setup. Please feel free to get in contact via the feedback channels.

9 responses to “Virtualised pfSense on Proxmox with Open vSwitch”

  1. João Avatar
    João

    OK! You just didn’t show how you did the proxmox network. Hehehe

  2. Rob Connolly Avatar
    Rob Connolly

    There is a whole section on the Proxmox network setup! It’s all done via the GUI and the provided screenshot shows my setup! The GUI is pretty easy to use, so I’m hoping my readers can deal without me going through it step by step.

  3. K. Callis Avatar
    K. Callis

    I had run across your posting awhile ago, and had bookmarked it for later perusal. Initially, I had thought I would be something in line with what you were doing hardware wise, but since then I made a change.

    I have decided to move proxmox to a laptop so that when I am out in the field, I can have my homelab right next to me. Like nearly all laptop, I only have a single interface on the laptop.

    So what do I need to do to setup pfSense under Proxmox? I thought I would just create three virtual interfaces (WAN, LAN, OPT) which would reflect my old hardware setup, and let pfSense deal with the various VLANs that I use attached to my OPT interface. I assume that I am going to have to also create VLAN entries on the OVS side as well??? Or do I have to do that.

    Draft Network would be the following:

    DSL Gateway
    |
    |
    |
    Cisco SG-300
    |
    |
    |
    Proxmox Server
    |
    |
    eth0
    |
    |
    |
    OVS
    |
    |
    |
    br1(WAN) br2(LAN) br3(OPT)
    |
    |
    |
    pfSense

  4. KC Callis Avatar
    KC Callis

    What would be the trick with a single port setup. I created VLANs in the networking section on PM 6.1. I have created VLANs under pfSense, and I have created VLAN memberships on my Cisco. I also created (at least I think I have created) an OPT interface (a VLAN to my LAN port as well). But I still can’t seem to send traffic to my Cisco router.

    Any pointers?

  5. KC Callis Avatar
    KC Callis

    I installed ovs on my Proxmox 6.1 host. I am trying to understand what do you mean my a trunk to the pfSense VM as well as a trunk to the interface. I have a single interface, and I have defined all of the VLANs in the ovs configuration. I have a chicken before the egg issue. When I spin up pfSense and add the WAN VLAN, which is connected to my L3 switch, I am able to get a WAN IP address on the pfSense VM. But from the PM host, it has no knowledge of the VM and more importantly, I don’t know how to make the PM host make use of the VLAN02 (my WAN connection to get internet access?

  6. K.C. Callis Avatar
    K.C. Callis

    I have used the same configuration and having no joy. I am working with two NICs, with the linux bridge connected directly to my DSL gateway, and the OVS connected to my Cisco SG-300 switch. I can console into my vm, and the vm is able to get an IP address for the WAN interface (as well as ping out though the WAN interface).

    Regardless, I am not able to access the pfSense web interface (nor can I even ping the LAN interface from neither the proxmox host or my laptop attached to the VLAN that the pfsense trunk is connected).

    This has been plaguing me for weeks and I am not sure where I am going wrong. As I stated earlier, I have configured everything exactly as you described. Any pointers would be greatly appreciated!

  7. sussudio Avatar
    sussudio

    Hello,

    I think it’s one of the best articles about PFSense. I can’t figure out the difference between VirtIO and PCI passthrough, now it’s clearer. Correct me if I’m wrong:

    – VirtIO is better than full virtualization and slightly below PCI passthrough. It’s better suited for a virtual infrastructure and the host system has direct access.
    – PCI passthrough brings native performance and allows not to have a dedicated router/firewall, everything is consolidated. On the other hand, Proxmox has no access and it is impossible to create vswitches or bridges for our VMs. It is more adapted to an infrastructure of physical machines.

    Do I understand everything?

  8. Kensh Avatar
    Kensh

    HI,
    Thank you for this guide. I just want to ask how do you do this step:

    “reconfigure your local interface to the VLAN you chose in the setup and a static IP. You should then be able to access the Proxmox web interface again.”

    any guide for this? sorry just new to linux/proxmox

  9. Joachim Avatar
    Joachim

    Hi Rob,
    I’m about to try a similar config with a dmz on a proxmox host using pfsense. Thanks for sharing this as it shows me that my plans aren’t that complicated as I feared they might be.

    Greetings
    Joachim

Leave a Reply

Your email address will not be published. Required fields are marked *

Bad Behavior has blocked 270 access attempts in the last 7 days.