Creating a secure BMS system for your customers: Part 6, Edgerouter

With the previous pages defining the network and setting up the VPN server we are now ready to create clients to connect to it. Here I have chosen the Ubiquiti Edgerouter X. This is a very capable network router running a derivative of the Vyatta router OS. The ERX enjoys a strange combination of being low cost with a MSRP of $49 and very well supported with ongoing software updates including feature additions and security improvements. It also gives you four ports to use for your BMS equipment. If for some reason you needed a more powerful router or SFP support these instructions should work for anything in Ubiquities Edgerouter lineup but I think you will find the ERX is ideal for what we need.

First of all you need to get the ERX set up with a basic configuration. I’m not going to repeat this here, just follow the instructions on these links;

  1. This quick start guide(PDF), follow this to get to the GUI of the router, go no further.
  2. Obtain the latest firmware and apply it through the GUI.
  3. SSH into the router and check the bootloader for the boot as switch problem. If it returns the old md5 sum patch the bootloader. Unfortunately I’m still finding the old bootloader in ERX purchased mid 2018 and have yet to find one patched out of the box.

With the new ERX up to date we can start out with the basic startup wizzard in the GUI. You need to specify your LAN subnet for the building BMS network this router, here I’m going to consider this ERX BMS building 1, which is VPN client1 which used the network 10.105.101.0 with mask 255.255.255.0. Once the ERX reboots you will need to move your network cable to a LAN port you specified, change your computer IP and reconnect to the GUI of the ERX. Connect to the WAN port to any network you have available that connects to the internet and verify you can reach the internet.

Now for the VPN part. This is an example config file broken down with explanations;

# This file compatible with topology SUBNET
# Intended for Ubiquiti ERX or similar
# Edit port setting for what server you want to connect to below. 

# VPN type
client

# Connection setup
proto tcp-client
remote vpn.example.com 20005
nobind

We are telling the client to connect to the server with TCP as discussed earlier. I’m also using a DNS entry instead of a IP address, which will make things easier in the future if the data center IP ever changes.

The port number looks a little odd because the way I structured our network I chose to have a different VPN server for each customer and each one has its own port to connect to to reach the VPN server it needs.

# Encryption 
cipher AES-256-CBC
tls-client
remote-cert-eku "TLS Web Server Authentication"
# -Ubiquiti EdgeRouter
ca /config/auth/ca.crt
cert /config/auth/client1.crt
key /config/auth/client1.key

# Authentication
tls-auth /config/auth/ta.key 1

This is letting the client know what encryption protocol to use and where the keys are.

# Compression "compress" new standard, (not supported in edgerouter??)
# lz4 best performance, lzo backwards compatibility
;compress lz4
comp-lzo

# Network setup
dev tun

I’ve tried the newer compress and lz4 methods on the edgerouter with no success, may try again in the future.

Network is a virtual point to point IP link with routing controlled by the OpenVPN config and CCD files that get remotely pushed to the client upon connection.

# Logging level
verb 3

Same level of logging as the server, seems adequate for troubleshooting and doesn’t fill things up too bad.

The complete config file:

# This file compatible with topology SUBNET
# Intended for Ubiquiti ERX or similar
# Edit port setting for what server you want to connect to below. 

# VPN type
client

# Connection setup
proto tcp-client
remote vpn.example.com 20005
nobind

# Encryption 
cipher AES-256-CBC
tls-client
remote-cert-eku "TLS Web Server Authentication"
# -Ubiquiti EdgeRouter
ca /config/auth/ca.crt
cert /config/auth/client1.crt
key /config/auth/client1.key

# Authentication
tls-auth /config/auth/ta.key 1

# Compression "compress" new standard, (not supported in edgerouter??)
# lz4 best performance, lzo backwards compatibility
;compress lz4
comp-lzo

# Network setup
dev tun

# Logging level
verb 3

Now with this file complete we need to take all the files listed below and place them in /config/auth/ in the edgerouter. On windows use something like WinSCP. Don’t copy anything extra from the key server, just what is necessary;

  • client1-1.ovpn -This is the file above, I name it the same as the client key to keep things straight
  • ca.crt -Root CA certificate
  • client1-1.crt -Client1 certificate
  • client1-1.key -Client1 secret key
  • ta.key -TLS authentication secret key

With the files in place we are ready to SSH into the router and do some final configuration.

ubnt@Devops:~$ configure
[edit]
ubnt@Devops# set system offload hwnat enable
[edit]
ubnt@Devops# commit
[edit]
ubnt@Devops# save
Saving configuration to '/config/config.boot'…
ubnt@Devops# exit

This example turns on hardware NAT firewall offloading. The ERX has hardware to do this but not all edgerouter do, I’m aware the ER Lite doesn’t.

When you first SSH into the edgerouter you are at a general $ prompt, we need to shift over to the configuration mode designated with # prompt. From there you set a configuration command. Committing that command sets that command active in the running (in memory) configuration, this is your time to test things and if something went drastically wrong and you lost access you are a reboot away from reloading the last known good config saved. Save takes the current running config in memory and saves it to flash making it permanent.

To start the openvpn connection you need to get back to the configure prompt and issue the following command, pay attention to the file name you copied to the router earlier:

set interfaces openvpn vtun0 config-file /config/auth/client#.ovpn

Hopefully you are able to commit this change and save it. Any errors I’ve seen here are generally pretty helpful and easy mistakes. Once back at the general $ prompt do this command to see how the OpenVPN connection is doing at your client side;

show interfaces openvpn detail

This will show the active tunnels with some traffic statistics. If you see nothing it means the tunnel never came up. This command searches the edgerouter log file for anything tagged openvpn;

egrep openvpn /var/log/messages

OR check in the VPN server C:\Program Files\OpenVPN\log\

IF the VPN tunnel came up you should be able to reach these things from the server; the 10.8.x.x virtual VPN address (ping and edgerouter GUI here), and the 10.105.101.1 edgerouter LAN address (ping and edgerouter GUI here).

Once you have added a second or third client you will be able to reach the virtual IP and LAN addresses of other sites from whatever site you are connected to. Most importantly anything sitting on the LAN of one of these sites will be able to reach any other device sitting on the LAN side of another edgerouter.

If at this point things aren’t working well google is your friend here. Try copying errors straight from the log file and paste it into google. Believe me someone has run into that problem before, nothing here is cutting edge.