Earlier this month, Cybera launched a Virtual Firewall Service (VFS) for Alberta’s public sector institutions. In this first of a two-part blog series, I will describe the history of Cybera’s VFS and what the future holds. Part Two will then provide a deep-dive into the technologies that the VFS uses, and how Cybera members can extend its functionality using other services, like our Rapid Access Cloud.
Introducing the Virtual Firewall Service
Cybera’s VFS is a single, per-user virtualized Palo-Alto Networks VM-Series next-generation firewall, running in our Rapid Access Cloud. The service uses OpenStack Neutron virtual networks to connect the virtual firewall to Cybera’s network. There is quite a bit to unpack in that statement, so let's break it down:
single, per-user virtualized
Each user of the VFS has a separate and secure environment with their own dedicated virtual networks, virtual machine and storage. The only shared resource is the hardware running underneath everything. There aren’t many layers of abstraction between the hardware and the virtual firewall, meaning this service is at the 'less-cloudy' end of the ‘X as a service' spectrum. We will explore this more in the technological deep-dive in Part Two.
Palo-Alto Networks VM Series next-generation firewall
Each virtual firewall is simply a virtual machine 'guest' running on a 'host'. Virtual machines run their own operating system, like Windows or Linux, separate from the host. For the VFS, we chose to use Palo-Alto Networks’ VM-Series product, as it had ready-to-use images for our cloud. In this regard, Palo-Alto Networks Operating System (PANOS) is the guest operating system providing all the next-generation firewall functionality.
running in Cybera's Rapid Access Cloud
The VFS runs in Cybera’s Alberta-hosted, award-winning Rapid Access Cloud. We will talk more about how and why the VFS was set up in this cloud environment below.
uses OpenStack Neutron virtual networks to connect [to] Cybera's network
Members connect to the Rapid Access Cloud and their virtual firewall directly via CyberaNet using extended VLANs and VPNs. This creates a complete environment for the member, with separate and secure virtual networks.
Highly-resilient, not highly-available
When we were evaluating firewall solutions for the production environment, we knew a balance had to be reached between cost, performance, and stability. Typically, one expects stability and performance to go up as the cost rises, and conversely, stability and performance to decline as costs decrease. The VFS was designed to keep costs down, but not give up the stability and performance required by our members.
In discussions with our pilot users, we found that downtime was less of a concern for them; if a firewall could be brought back to a working condition with traffic flowing in mere minutes, some disruption to the users was excusable. This model works very well in a virtual environment — as long as the process for launching a fully-working and configured firewall could be automated and streamlined — such that we could get the turnover time to less than 10 minutes. This is referred to as high-resiliency, which means that there is no attempt to avoid downtime as one would with high-availability, but instead the focus is on reducing downtime when outages occur.
In contrast, a high-availability configuration has multiple firewalls running concurrently, so when any one firewall fails, the remaining firewalls continue to run and handle the traffic. This means that there is zero downtime. However, those additional firewalls need to be licensed and they also consume resources, which leads to higher costs.
Virtual Firewall Service Control Panel
To facilitate the rapid recovery of a firewall and reduce the downtime, we wanted to create a tool that would make the process of launching and recovering a firewall as simple as possible given the levels of complexity involved in managing virtual resources. And, further to our goal of reducing costs, we have given our users the tools needed to manage their own firewall, relieving Cybera of much of the need to provide extra support.
These tools are already in use in the Rapid Access Cloud, but we did need to develop some custom tools for the VFS to streamline the process and codify what we felt were best-practices for the service. The primary tool we developed is a simple control panel that uses the PANOS XML API (eXtensible Markup Language Application Program Interface) to help users ensure that the current configuration of their firewall, referred to as the running-config, can be preserved and re-applied in case they need to relaunch a new firewall. In this way, the virtual firewall an be seen as a virtual appliance that can be 'thrown out' if it breaks and 'replaced' with a working firewall with the click of a button (using a simple browser-based interface).
Firewall-as-a-Service; Past as prologue
Now that you know about the current incarnation of the VFS, I would like to give you a little history of the project and how we arrived at the platform as it is.
Since February 2014, Cybera has been working with a handful of Alberta school divisions on a K-12 Shared Firewall Pilot Project. The goal of this networking pilot was to help participants develop their expertise in implementing and operating a single shared firewall appliance, in order to aggregate internet gateway services. The firewall device is hosted by Cybera, while all operations and management is undertaken by the participants.
The pilot demonstrated that a shared firewall service model is beneficial, achievable, and can generate significant cost-savings for K-12 school divisions, but by 2016, it was reaching its maximum participant capacity. With interest in the K-12 Shared Firewall Pilot growing, it had become clear that there was a demand for a scalable shared firewall service.
The “Firewall as a Service” (FwaaS) Pilot Project was launched in 2017 to help Alberta K-12 schools centralize their network security at Cybera's network perimeter through virtual firewalls hosted on Cybera's Rapid Access Cloud. Its goal was to provide significant benefits in terms of reduced costs — with respect to hardware, firewall hosting and management — and simplified network architectures for K-12 school divisions, while also providing scalability, security, and a longer-term sustainability model compared to the K-12 Shared Firewall Pilot Project.
FWaaS leveraged Cybera's networking and cloud development, administrative, and operational expertise. It also acted as a platform to develop and share Cybera's experiences with network function virtualization with the K-12 school divisions.
Not only was this an interesting technical challenge, it also required coming up with a self-service model that could work in our existing Rapid Access Cloud, while still providing a simple but powerful interface that would allow users to be responsible for configuring and managing their own firewall.
Proof-of-Concept to Pilot
When we began FWaaS pilot began, nothing like this had been implemented in Alberta before, so we decided to go with the simplest thing to implement first: a virtual machine that can do port filtering. We built a proof-of-concept model using Ubuntu Linux and the Linux kernel firewall, and simply forwarded the traffic from one port on the virtual machine, through the kernel for filtering, and out another port. Those ports connected to OpenStack Neutron virtual networks, which in turn connected to CyberaNet via VLANs. This is a fairly simple model that allows multi-tenancy in logically separated environments, with the tenant retaining full ownership of their resources in the Rapid Access Cloud.
The Ubuntu Linux proof-of-concept was, however, just that. We knew users were looking for more than simple TCP port filtering, and from on-going discussions with possible participants in the program (given that many were migrating from the existing Shared Firewall), there was already familiarity with Palo Alto and its operating systems. Palo Alto has an OpenStack compatible image, and plenty of documentation on getting the Palo Alto VM-Series firewall up and running in an OpenStack environment, so we moved forward with it as the next-generation firewall solution. However, we quickly encountered a problem.
Performance woes
Our proof-of-concept was doing only simple TCP port filtering, but PANOS provides much more functionality and, as such, we found there to be performance limits in our current configuration. As traffic would transit the virtual firewall, we saw that our 1 Gb link was pushing less than 400 Mbps of traffic, whereas we would have expected near full usage of the link. We tried tweaking Neutron by moving away from OpenVSwitch in our implementation to just linuxbridge, meaning there would be less virtualization overhead. This improved things a little, but still not enough to give us the performance we needed. The last trick to try was something called Single Route I/O Virtualization, or SRIOV. This would move some of the burden of network virtualization off the host kernel, using the network card itself to virtualize the network ports. Implementing SRIOV required us to upgrade the OpenStack cloud to a newer release, including an upgrade to the Ubuntu Linux release. After we updated to Ubuntu 16.04 and OpenStack Mitaka, SRIOV worked, and our network performance improved from the abysmal sub 400 Mbps to over 900 Mbps, right where we expected it to be!
Pilot to Production
With the performance where we wanted it, we next turned our focus on making our solution easy to deploy (and redeploy) by people without a background in cloud development or DevOps techniques. Typically, handling virtual resources in a cloud requires the ability to manage and configure tools, otherwise users need manual processes to handle the launching of the virtual machine and associated resources, such as networks. Many tools for deploying virtual resources do exist, such as Terraform, however, we wanted to use existing OpenStack technologies to accomplish the task of deployment, so we looked to OpenStack Heat.
Palo Alto had already created Heat templates for launching their VM-Series firewall in OpenStack, so we just needed a way to give the users a simple interface. We were able to tie everything together using OpenStack APIs (heat, swift, nova, neutron) using Python in the control panel, reducing management of the firewall to four key functions: launch, backup, restore and upgrade. These functions are for the life-cycle of the virtual machine, all other firewall functionality is handled from the firewall itself, typically via the web interface.
The future of Cybera’s VFS
Palo Alto is not the only option for virtual firewalls, and the VFS will likely expand to include as many different virtual firewall technologies as we can possibly support. Our next steps include looking at other proprietary solutions that can ‘drop-in’ easily into the existing platform, as well as non-proprietary solutions for users to choose, particularly ones that do not require licenses to operate. However, the open-source solutions will have no additional support beyond the basic community support provided by open-source projects. We are also looking at supporting a wider selection of proprietary solutions; after all, there have been newer entrants to the virtual firewall appliance arena, and Cybera always tries to keep the Rapid Access Cloud neutral with respect to the operating systems it can support.