A storage solution

My plans to play with a canary this weekend were spoiled while doing some maintenance so unfortunately I’ve had to put that off. I generally go in once a month and “update all the things” in the network and it’s pretty much uneventful but I’ve been hit with issues enough to know why Read Only Friday exists at work.

I’ve technically had two Synology disk stations serving up my network shares for everything up until now. Technically because my DS1515+ was hit by the Intel C2000 CPU random death bug. While mine didn’t fail it did hang on boot once making me do a  search and finding my serial number was right in the middle of the batch that was expected to fail, just a matter of time really. Synology did an in warranty pre replacement and according to my serial number I got a unit several years older than what I sent in. It worked and appeared new but I definitely had mixed feelings about that.

Thursday I signed in my DS418 which up until that time had been fantastic. It had emailed me a couple days ago informing me of an update available as it had done many times before. Unfortunately this time when I hit update and reboot it didn’t come back. It came back to a crashed array claiming both drive 1 and 2 failed at the same time with write errors. As I only had single drive fault tolerance the array was now considered lost with all data. Both drives still show fine in SMART though. Call me crazy but I kinda think it unlikely both drives failed at the same time, but I am sure going to find out. Synology has an open support case and hopefully will look into it Monday when they get back to work.

Enter player 2; FreeNAS

For a while now I’ve been reading about FreeNAS and thinking it and ZFS might be a nice thing to check out some day. I’m a growing fan of open source and this project in particular seems to have gained excellent traction. I especially like the flexibility of the platform that frees me from any one particular hardware vendor and isn’t a big black box.

FreeNAS build and planning

Use case, this is a general file share dump. Plenty of movies played back with a Dune network player, no transcoding. The most stress this will see is when something gets written to it and scheduled rsync tasks.

Intel Celeron G3930, has ECC
Single 16GB DDR4 ECC, from supermicro approved list.
Seasonic S12II 620W PSU
SAS 9207-8I based on recommendations seen on FreeNAS forums
Boot from USB via Sandisk ultra fit 64gb
Recycling my decade old Yeong Yang cube server case with a couple new bay adapters. Should hold up to 18 3.5″ drives.

Main pool (in above machine)
Existing four 8TB Seagate Ironwolf will be the main pool that everyone reads/writes to. Drives are approaching a year old. I’m thinking about making this 4×1 ZFS1 but am toying with 2×2 ZFS1. Calculator says that’s either 14TB or 20TB useable but I’m not sure which way I should go with it, seeing mixed advice here. I’m at about 8.5TB in use but slowly growing. There are a couple automation tasks downloading files to this at any time and three computer using it as a backup, this will be on 24×7.
These drives are currently in the DS418 discussed above. If I have to RMA two drives so be it, but I’ve mentally written off my data here.

Complete Backup Pool (in above machine)
I was thinking about skipping this but realized I have eight existing 2TB Seagate LP(PDF) laying around so why not use them? This is the reason I added the LSI card. These drives are 7+ years old, some are already RMA replacements but they will all go through spinrite before going in the pool. I would like this pool to sleep when not being used (noise and power concerns) and have read that to accomplish that these need to be the ones attached to the motherboard though ideally I’d prefer my main pool attached to the southbridge ports. I figure I’ll set a rsync task from the main pool to this one every week or so, otherwise they should be off. Should be 11.8TB useable space in a 8×1 ZFS1 pool.

Critical cold backup (attached to above machine)
USB3 Seagate 8TB external. This will hold the irreplaceable data from above main pool, but on a longer backup schedule maybe doing a rsync task every month or so just in case of malware or cryptolocker etc. I will have to remember to attach it and run a rsync task, hope I can do that. Will be looking for another one on sale over black friday, this one is full and stopped doing weekly backups last month which puts me in a double bad position with the crashed DS418. Currently holding a second best bet of my data, but a little old.

Off site backup
Synology DS1515+ with SHR1 across three 2TB Seagate LP (also ~7 years old) and two 8TB shucked WD Easystores (6 months old) that yielded Red drives getting me about 12TB SHR1 formatted space. The DS itself is back from RMA about 6 months ago. This is (was) a nightly backup target from above DS418 over VPN. This currently has the best copy of my data, so looks like the above machine will go for a drive when it’s built as downloading 9TB over a cable modem upload speed isn’t doable.

Blew a ton of time researching this weekend but hopefully a worthy addition to the homelab.

Passive Defense

I’ve been following Thinkst for a while and I love to hear Haroon Meer get interviewed as he has some amazing viewpoints. You can catch him quite a bit on Risky.biz which is worth a listen.

I think I would like to deploy one of their canary devices but can’t swing the price at the moment for a home lab but I’m trying to convince work, unfortunately to be truly effective we would need a dozen or two. Been thinking about the OpenCanary product and seems like now would be the time to get started on that as they just pushed some updates.

I’ve also deployed quite a few of their free tokens in sensitive places. Luckily so far I’ve only caught myself blundering around which is good! They also have a blurb explaining what that’s all about.

Basically if you are reading this blog these are some basic things that you might want to deploy to make sure you are the only one inside your network.

A BMS VPN solution

These articles make some assumptions and hopefully these are familiar to you;

  1. You are a BMS vendor or service BMS systems for your customers.
  2. You have BMS equipment on customer networks that you do not control.

Point 2 is the especially difficult part that you are likely already very familiar with. Generally many BMS vendors stick the network architecture and management of these IT issues on the customer. We can help them with requirements but dealing with BMS is never going to be a priority for IT, or maybe your customer doesn’t have an IT person. You are likely also aware of the classic conflict between facilities management and IT, facilities wants to put a bunch of esoteric device on the network and IT wants to do everything they can to keep these suspicious devices with open protocols off of the network. Frankly BMS vendors have a justifiably bad reputation for IT security issues which is going to be an attitude you will have to turn around. The only way I know to do that is show that you have a thorough understanding of the vulnerabilities your devices have and demonstrate how you are mitigating that risk.

I don’t claim ownership of any of this information, these are things that are freely available in many places. The goal here is to put together a system that works with all the necessary information in one place.

A note and an apology:

I have done a lot of browsing and reading on OpenVPN and I see so many support question posts saying “x stopped working but OpenVPN says it is connected”. Most OpenVPN articles don’t really cover the infrastructure around the tunnel. Without understanding what’s going on in the network you won’t ever be completely successful, and that’s why this collection of articles are a little long.

Part 1; Planning

Part 2; Network Layout

Part 3; OpenVPN Server

Part 4; Crypto Keys

Part 5; Client Configuration Directory

Part 6; Ubiquiti Edgerouter X

Part 7; Security

Part 8; BACnet BBMD/FDR