Thanks to a lucky sequence of events elsewhere, I inherited a Ubiquiti Cloud Gateway Ultra router.

My Netgate 1100 running pfSense was not underperforming and didn’t need to be replaced, but I took the opportunity to replace the only major piece of network gear left in the house that wasn’t Ubiquiti.

I have a moderately sophisticated home LAN, beyond what most of the people I know need, but not as over the top as some others. I currently have one internet service provider, Starlink, though for a while I had an LTE provider as well. There is a fiber provider building out in our area and we have signed up. They have installed our ONT to a fiber stub that runs the the curb and all over our neighborhood, there are runs along the street between stubs. Still waiting for the rest of it to be completed somewhere. Whenever that does happen, we will have fiber and Starlink for long enough for me to trust the fiber.

I have the router, a switch and two WiFi access points in the house, a switch and a WiFi access point in the workshop and a wireless bridge connecting them. The main complications to this otherwise fairly straight forward deployment are: 1) My ISP is currently Starlink and to avoid all the trees around the house, the Starlink dish is physically installed at the workshop, requiring the use of an isolated VLAN to backhaul Starlink across a wireless bridge to the house where the router and all the other central gear is and 2) I have a moderate number of IoT devices, particularly inexpensive home automation devices, that should be somewhat isolated from the rest of the network.

I had set up a Unifi Network Controller running as a Docker container on my Synology NAS to administer the Ubiquiti switches and APs. No more CPU than this task takes, this has hardly been a load on the NAS at all. However, the Cloud Gateway Ultra can take over this task as well.

I read that there are ways to back up the running configurations for all of my devices and restore them to the new controller. It’s not particularly complicated, but I elected to take the opportunity to clean slate my configurations and reset each of these devices and configure them anew. This may have cost me more time than was necessary, but it definitely made me understand and address specific elements of the configurations.

I started by locating an ‘as built’ drawing that I made after adding the Ubiquiti switches and APs.

I used this drawing and consulted the existing configurations to verify the port numbers and VLANs, particularly dealing with the Starlink backhaul.

The flow of the VLANs are better illustrated here.

The thing to realize is that VLAN 50 allows data to and from Starlink to connect directly between port 7 of Flying Dog switch and port 15 of the Hippy Hollow switch without being available to any other ports on either switch. The router then processes it as a WAN source and distributes it out it’s LAN port, connected to port 16 of Hippy Hollow switch. All other ports on both switches have all VLANs except VLAN 50 available, so nothing can connect directly to Starlink, bypassing the router.

In any case, because I was going to deploy these devices with factory resets, establishing this VLAN backhaul added a twist. I had already brought the Gateway up in the house and changed it’s network IP from the default 192.168.1.0/24 to my existing 172.29.0.0/24. I then took the Gateway over to the workshop and connected it directly to Starlink and a random switchport and importantly, I plugged the workshop AP in to one of the Gateway’s switchports. This let it come up with the Gateway easily reachable from my phone. When things settled down, I reset the switch, adopted it, configured the VLAN on port 7 and the trunk on port 8 (which has the bridge between the workshop and house) then I moved the Gateway back into the house to reconfigure that switch.

In the house, I had the advantage of a laptop in the network cabinet, so I didn’t have to necessarily worry about the AP immediately.

I was able to quickly get the VLAN backhaul for Starlink up and going. Then came everything else.

I understand there is some method of resetting APs over the ethernet cable, but I needed a ladder to reach only two of the four and adoption to the new controller went smoothly.

There were two mildly troublesome parts to all the wireless stuff. First, it took so long for me to get all the APs reset that all the open DHCP scopes assigned IPs that were previously fixed and I couldn’t conveniently reassign them to the addresses they once had. There was kind of a plan there at one time. I just sighed and left the things that needed to be fixed where they landed, mostly cameras and printers. Second, the whole idea of having a separate IoT subnet is that devices on that network can reach the internet, but not your other local networks. A simple checkbox enables this isolation, but if something *does* need access, if for example if your Home Assistant server is on the main network and a bunch of your wireless home automation devices are on the IoT network, then this simple network isolation checkbox is not the solution; you need a couple of appropriate firewall rules instead. That is why all of my WiFi Home Assistant devices were grayed out. 🙂

To get all these devices up and running, I elected to remove the network isolation checkbox and work on setting proper firewall rules later.

As is often the case, a couple of months have passed between the previous paragraph and this one. In the interim, Ubiquiti has released Zone Firewall for my router, so I need to figure that out. It’s not expected to be difficult, but I haven’t even looked at it yet 🙂

I have four APs, three in the house and one in the workshop. Two U6 Lite APs were purchased together, one in the house and one in the workshop. I needed to add one in the house later and managed to secure a used AC Pro for free, which is a significantly older unit. By itself, that didn’t matter much, but it would probably be better if both of the units in the house were at least similar in features, such as WiFi 6, so I swapped the older one to the workshop. I also have an AC Mesh AP for some outdoor connectivity.

Also, that AC Pro had complained about the wiring ever since it was installed, claiming that it was Fast Ethernet instead of Gigabit. Not surprisingly, the U6 Lite didn’t like the wiring either. I shuffled the attic wiring around so that my USW Flex in the attic now powers one of the AC Mesh and one of the U6 Lites instead of the AC Mesh and a camera. The wiring that was on the troubled AP is now going to that camera, which is only Fast Ethernet anyway. I started by just running all three on the USW Flex, but it ran too close to the max PoE power budget for the switch and kept dropping the newly added AP.

There is one more VLAN thing I’d like to solve. I’d like for all the cameras to be in either the IoT VLAN or maybe their own VLAN. It is pretty trivial to move the WiFi cameras, but for some reason, I can’t seem to get the wired camera to be happy in anything except the default VLAN. I can definitely make the switch port appear to be in the expected VLAN, but then the camera just stops communicating. It seems to refuse to get a new IP from the DHCP in the new network. Maybe the zone firewall rules will make that easier to understand and manage. 🙂

I have mentioned the WiFi camera at our front gate before. It has come up in a discussion about the power system for it and in a mysterious DHCP flooding issue that was discovered when I began looking at router logs.

As it would turn out, the DHCP issue was because the camera would not usually hear the router. The conversation would basically be:

Camera: Hey, can I have this IP?

Router: Sure, sounds good to me.

Camera: Hey, can I have this IP?

Router: Sure, sounds good to me.

Camera: Hey, can I have this IP?

Router: Sure, sounds good to me.

Happily, the camera would default to the last IP it had and since the data was largely one way from camera to NAS, I didn’t really miss anything. Neither did it occur to me that the DHCP messages were a symptom of this one-sided communication, though it seems obvious to me now.

Once this occurred to me, I took some steps to address the problem. When I installed the ceiling mounted APs in the house, I had the wall mounted APs laying around. This model provides passthrough PoE, so I took it into the attic at the garage end of the house, closest to the gate camera. The driveway camera is PoE, so I reconfigured the cabling so that the cable powered the AP and the AP powered the camera. Then I reconnected the gate camera to WiFi and the attic AP was now a better signal than whichever one it had connected to before and the DHCP issue got better.

It did not go completely away, but now instead of DHCP requests several times per minute, all day, every day, it reduced to bursty, frequently everyday. I don’t know what made it work better sometimes than others, but it was definitely better.

I still wanted it to work correctly. I ordered an actual outdoor AP. I was going to quickly set it up for testing before permanently installing it and headed into the attic with it. I suddenly relized that the new AP doesn’t do the PoE passthrough as would be needed to have both the AP and the wired camera without running another CAT5 cable.

For the test, I connected the AP and even with it in the attic, the gate camera made only one DHCP request for the entire 30 minutes it was connected, which was definitely an improvement. I went ahead and drilled the hole and mounted the AP outside.

I had decided disconnect the wired camera for a day or two while I waited for the final piece of equipment to arrive, a USW-Flex switch. The USW-Flex is a 5 port switch that is both powered by PoE and provides PoE, within a reasonable power budget. As luck (and shipping) would have it, I had it the next day. It went in like a breeze.

The nails it is nestled between just happen to already be there and work perfectly to hold it in position where can see it lit up from the attic stairs.

The ports are limited to 25W max each and 46W total, at a temperature maxing out at 131F. The power ratings are reduced at higher temperatures. The high temperature range tops out at 149F and that range derates the maximum PoE power available to 25W. It seems obvious that Ubiquity expected these units to be deployed in attics where someone would use it like have to keep from running another CAT5 cable.

The AP and the camera each draw less that 5W each. I have two ports available for adding cameras as well. Assuming I stay with similar cameras, that seems likely.

Oh, and the gate camera does one DHCP renewal per hour, which is as it should be. And the frame rate increased.

Here is the home network as it currently sits, with Ubiquiti switches and APs:

That little box labeled “WiFi Devices” accounts for 50+ devices now. I have partially converted the things that should be on the IoT VLAN over, but I just haven’t done all of them yet.

I was responding to a post on a forum about pfSense and Ubiquiti and how they get along (very well, thank you) and it occurred to me that I left part of the story hanging.

When last we left our heroes, I had kinda iffy WiFi coverage in the house with the new gear, compared to the old gear, but I had all the desired VLANs working. The solution, or at least the obvious solution, was to further finance the purchase of more Bentleys for Robert Pera by upgrading my APs.

As you may (or may not) recall, one problem I had was that stationary devices in the house would suddenly feel the need to roam and managed to connect to the AP out in workshop. The workshop was so distant that shortly they would roam off it and the only other option was back to the AP in the house again. Vicious cycle ensues, cat and dogs, etc.

Well, the new APs arrived. I initially connected and tested by laying them about on desktops, but quickly got serious and ran wire for the main one to be deployed in the kitchen ceiling. The whole WiFi thing in the house got much better.

It is not without the occasional mystery, however. I have an apparently not terribly smart WiFi device that is the intermediate display panel for my Accurite weather station. The suite of weather sensors outside communicates via a 433MHz signal to the display panel, which in turn connects to my LAN via WiFi and kicks the data out to WeatherUnderground. This WiFi device is literally 15 feet line of sight, with no obstacles, from the kitchen AP. As I type this, the display panel is within reach of my left arm and the AP is two steps away. Consequently, this stupid display panel will connect only to the hallway AP on the other side of the house and ONLY that AP. I have tried every conceivable way to reset the display panel and the APs. The only combination that will work is if the hallway AP is online and accepting the Accurite panel. Nothing seems to make it forget the hallway AP and connect to the much closer kitchen AP.

My new WiFi is argubly worse than my old WiFi, but I think there is a reason.

I recently completed a pretty major [up|cross]grade of my home LAN. There were two main goals. First is to add some additional wireless VLANs so that I can help isolate my growing network of IoT devices and thus protect them and the rest of my devices from them. Second, and this is admittedly not a super important thing, it would also be nice if I didn’t have to connect to a different SSID between the house and workshop.

As detailed elsewhere, that was a journey of discovery, ending with the eventual replacement of two TP-Link APs and two Cisco switches with new Ubiquiti devices. Between the interopterability and easy of configuration, it was a good, though not particularly cheap, move.

Or so I thought.

A week or so later, it appears that I have a lot of connectivity issues. Once I really noticed it and started looking into it, I think I know what’s happening and at least a good bit of the solution, the full spectrum of which of course involves helping someone at Ubiquiti buy another Bentley.

I put the new APs exactly where the old ones were, but the old ones were perhaps a little more sophisticated, RF-wise. The TP-Link APs had an array of four large antennas and, sadly, physics matters. The UniFi APs look like oversized light switches and simply cannot have the same antenna performance. However, the one in the house seemed to work just fine until I had a compelling reason to move it. I wont go into those details, but it had to move from about the center of the house to one side of the house. As I am typing this post, the UniFi controller reports my WiFi experience as “Good”, rather than excellent, along with everything else here on this, the opposite side of the house from the AP.

The real issue comes about when, for reasons I have yet to determine, the experience gets enough worse that the connection is lost and the device, most noticeable to me when it is my laptop, sees the other AP, all the way out in the workshop, but with a marginally ok signal, at least compared to the one it just dropped and it roams to that AP. Now that weak signal results in a substantial slowdown and maybe it drops that connection, too. Then it sees the original AP and connects to it again. This vicious cycle continues.

Once I figured out that this seems to be what is happening, I set the workshop AP to lower power. This makes is a less viable alternative to reconnect to and at least stops the toggling. It does not, however, address the actual problem, which is that the new Ubiquiti APs I chose simply don’t have the coverage that the TP-link devices have.

So, I ordered a couple more APs, ceiling mounts this time. I think I can justify a unit in the hallway between the bedrooms and the livingroom and another in the kitchen. If those two provide dense enough coverage in the house, I can deploy the other wall mount in the workshop, to have two out there as well. If needed, however, I can find a place for a third in the house. One has seemed to be enough for the workshop thus far. Everything WiFi out there is within about 20 feet of the AP.

Update: The power setting on the workshop AP has definitely reduced the number of roaming events in the logs. [facepalm] Now they just disconnect instead. [/facepalm]

You load 16 ports, what do you get?
Another day older and deeper in debt

The 16 port PoE switch arrived today. With the exception of one adoption hiccup, it works really well and finally, all the VLANs are doing (mostly) what they are supposed to do. Now that the layer two works, I do still need to write firewall rules for them.

I configured DHCP to statically set the IP I wanted to use, so the switch came up where I wanted it to and I clicked Adopt and away it went. About 30 minutes later, it was obvious that it wasn’t going to complete that process, so I did a factory reset and had the controller forget it and this time, it completed the adoption process without further drama.

The only configuration I really needed was to make that one port where Starlink pops out on this end into a native VLAN 50. All the other ports would be just trunk ports. I did label a few ports.

I moved four cables over, the link to the workshop, the Starlink/WAN to the router, the LAN to the router and the laptop I was working from. The router had noted the loss of ping from Starlink and has switched to OneSource, so I had to jump in and force it to switch back, but otherwise everything worked perfectly and I moved the rest of the cables over and powered down the Cisco SG200.

I almost forget to test connecting my phone to the IoT WiFi, but I did and it worked, as expected. This is, afterall, one of the reasons I began this whole exercise. Victory dance!

I think the only thing left now is really more of an irritation than an issue.

When I moved the wiring over, I moved most of them over jack for jack, so port 4 happened to have the NAS connected to it and the controller (presumably) discovered it and helpfully labeled it. I decided to move it to port 10 to free up a PoE port, but it apparently is finished discovering and from what I can find thus far, it does not appear to be an end user editable field. That seems very unlikely but, so far, that is what I see so I am stuck with an empty port labeled with something that has been moved to another port which remains unlabeled.

Then again, it’s 2AM.

Act I: Somewhere deep in my first catchup post in this blog, I related that I had acquired a couple of nice TPLink wireless routers with the intent to mesh them together so that the house and workshop would be on the same SSID, only to discover that this particular router apparently cannot be a mesh client in that situation. Further, none of the mesh clients in that product line support ethernet backhaul, which is pretty much required for this specific situation.

Act II: Enter home automation and a sudden propagation of WiFi IoT devices. I am not in any real danger of running out of IPs in my main LAN. It would also be trivial to just expand it if I were; it is all RFC1918 address space, with the exception of dealing with the work VPN and the RFC1918 subnets routed there. However, best practices would have the IoT stuff separated from the rest of the network in case they get somehow compromised on the internet and many of the home automation devices need no internet access at all.

With these two related goals in mind, WiFi roaming and multiple WiFi networks, I shopped around and decided that a couple of Ubiquiti APs would serve this need. Between Ubiquiti being affected by chip shortages like everyone else and a ceiling mount not being my first choice for various reasons, I got two in-wall APs as the best compromise that was also not too expensive.

While they were enroute, I knew that I would need the controller console software running somewhere. Ubiquiti marketing really makes it sound like one of their hardware hosts is the only option, one of the Dream Machine or Cloud Key devices, but they have downloadable software that can run on a variety of OS platforms. You don’t really need the software except for initial setup unless you want to track statistics, so it can be run as needed on, for example, a Windows PC.

Another option is to run it as a Docker container in your NAS.

At one time, not that many years ago, I was an anti-VM snob. I was a hardware purist. I have pretty much gotten over that. There is almost no computer that is busy enough to justify doing only one task, especially one server task. Even modest hardware spends most of it’s time waiting for something to do. My Synology DS220+ is arguably pretty modest hardware and I have it recording six IP cameras, hosting my home automation leviathan and now its also the controller for my small collection of Ubiquiti networking components. Oh yeah, it’s still a NAS, too, backing up two computers. And it’s barely clicking over most of the time.

I digress.

There was really only one issue I had bringing them up on this controller. The APs need DHCP option 43 in order to find this controller and that took a bit of research, both in how to format the info and how to implement that in pfSense. It’s not difficult, but it did take some Googling to find the details.

The ’01:04′ tells the AP that the following info is the controller IP, then the next 4 octets are just the IP of the controller in hexadecimal format, 172.29.0.250 in this case.

The related DHCP issue wasn’t super critical, but I wanted to assign static IPs for the APs. I neglected to set the static mapping before I plugged them in, so between the option 43 stuff and changing IPs, I had to have the controller ‘forget’ them, then factory reset them a couple of times to get things settled in.

Interestingly, I had not noticed until I pasted this image that the WorkshopAP, currently in the same room with the HouseAP while we wait for a PoE switch to arrive for deployment in the workshop, for some reason connected via WiFi instead of the ethernet by which it is powered. I will look into that, but I don’t expect that to be a long term issue; that AP will be elsewhere soon.

Meanwhile, I have created my VLANs, subnets and WiFi networks for the redesign. At the time I wrote that sentence, I needed only to update the VLAN configuration in the switch for everything to get talkin’. Little did I know…

As mentioned in the Starlink saga, VLANs are not a proprietary thing, but as I discovered, not all engineering teams approach them the same way or to the same degree. To recap that experience, the TPLink switches would let me segment ports within a switch, essentially dividing it up into smaller switches, but they did not pass that VLAN tagging information on, so no connected switch was aware of any of those VLANs. I had a somewhat more sophisticated Cisco switch on hand, one that as a bonus also provided PoE for my cameras, so I found it a playmate to put in the workshop and I was able to do the VLAN task I needed in order to backhaul Starlink from the workshop to the house while keeping it isolated from the general LAN.

In this diagram, the blue lines are VLAN50, the Starlink backhaul and the green lines are VLAN1, the main LAN. The switches are able to trunk both VLANs over a connection, so whether it was a piece of wire or a couple of wireless devices, the VLAN tagging was preserved.

With the new APs, the task is similar. I want to be able to run multiple isolated WiFi networks, so it would seem to be the same task, but I could not wrench the Cisco SG200 to do the same thing between ports on the same switch after days of trying. On the other hand, I also learned that one weird trick about VLANs and pfSense on the Netgate 1100 appliance.

The path has not been a smooth one, either. During the last several days, I have lost access to the switch multiple times and to the router once. With the switch, sometimes a power cycle was in order, sometimes just moving to a port I had not accidentally lobotomized. For the router, I was able to connect to it’s serial port and restore a previously saved working configuration.

One important and telling troubleshooting step was to monitor DHCP requests with Wireshark. In that process, I could definitely see the discover messages from the AP hitting the switch port it was connected to, but they were not coming back out the switch port to the router. This largely cemented the issue as being the switch not really trunking VLANs as expected.

The solution, as is often the case, has turned out to not be free. I knew I would need a PoE switch switch for the AP in the workshop and since (I guess?) I am kinda slowly moving to the Ubiquiti ecosystem, I ordered an 8 port switch to put out there. I started doing all this VLAN stuff while it was enroute. Somewhere along the way I wondered, since all the UniFi YouTube videos depict this as a largely click and go process, if the new switch would handle it as expected.

Yes. Yes, it does.

When the switch arrived, I first did all the adoption stuff and got it stable. Leaving it with all ports set to the “All” port profile, which is the default and expected to carry all VLANs, I plugged the USW-Lite-8-PoE switch in line between my Cisco switch and the LAN port on the pfSense router, with one of the APs also on the Ubiquiti switch. The VLAN 1 WiFi came up immediately, but then, it had been working all along. I found one thing (the weird VLAN tagging thing for the internal switch in the Netgate 1100 appliance mentioned above) was not quite right due to all the desperate experimentation I had been doing, but once that was corrected, I was able to connect to the IoT WiFi with my phone and get an IP address. It was glorious. I may have peed a little.

I removed the Ubiquiti switch and put the AP back on the Cisco, just to verify that it would fail again, that I had not just finally gotten all the configurations correct, and it definitely stopped working. So, lesson learned: for all it’s other strengths, the SG200-26P just doesn’t handle port to port VLAN trunking, at least not as the AP needs it to deploy multiple WiFi networks.

I then ordered a USW-Lite-16-PoE to replace the SG200-26P. It should be here in a couple of days.

Meanwhile, I configured a port on the USW-Lite-8-PoE for untagged VLAN50 and moved it out to the workshop. It plugged in and Starlink came up working perfectly. The Cisco again handles this sort of VLAN over a trunk just fine, and it doesn’t care that it’s partner switch is not a Cisco switch.

Importantly, the AP out there came up on the main VLAN and I can roam on that same WiFi network between the house and workshop; I no longer have to swap between HippyHollow and FlyingDog.

For nostalgic reasons, I will miss FlyingDog. It was a specifically chosen name for the network. I know! I will rename the AP!! Actually, I will rename both APs to what their networks used to be called. I are silly.

Not surprisingly, though, the IoT WiFi doesn’t work from out there, further evidence that the Cisco switch just isn’t passing those VLANs to the router. I am reasonably confident that when the USW-Lite-16-PoE is in place, the various alternate WiFi networks will work from everywhere.

Of course, then I have to start moving all the IoT devices to that network…

There was really nothing wrong with my old NAS. It was largely underlutilized, other than I had hit the limit on how many cameras it could record in Surveillance Station because Synology is smart enough not to oversubscribe the hardware’s capabilities and limit the DS120j to 5 cameras. That was really the main thing I wanted to upgrade for and even that wasn’t a big rush.

Still, I had planned to upgrade once I paid off my Amazon credit card and since I had done that…

I did some comparison shopping and finally decided that I would get the most bang for my buck with the DS220+. I wanted the ability to add a 2nd drive and the Intel Celeron in the DS220+ with only 2 cores running at 2.0 to 2.9 GHz could still outperform the DS218/DS218play and had an expansion memory slot. That feature alone would turn out to be handy for me.

Well, it would be once I got the right memory module. I ordered the right stuff, DDR4. The labelling on the packaging was correct. The module in the package was a DDR3 SODIMM, different pinout. I processed the return to Amazon thinking I just ordered the wrong thing, before I looked at the packaging carefully. No matter, it’s sorted out now and the DS220+ is no maxed out at 6GB.

It was not clear ahead of time whether I could swap the physical drive from my DS120j into the DS220+ and have it work correctly. I seem to recall that once I had the hardware in hand, it became obvious that it could be done, but that the performance would suffer because of the differences in the newer and older file systems. I chose what was probably the best way, though it would turn out to be very slow. I set up the new unit fresh, then did a backup of the old unit’s files, using a share on the new unit as the destination. Then I could restore the old unit’s files from that now local source.

It took a surprisingly and irritatingly long time for that backup to complete. Then the local restoral wasn’t a whole lot faster. The whole process was most of 48 hours. On the other hand, it was also completely trouble free.

When everything was restored and settled down, I was able to move my camera license pack from the old NAS to the new NAS and although I didn’t have one immediately available, it did not complain about it when I attempted to add one more camera. Mission accomplished!

I have currently set up the old NAS as a backup target for the new NAS and set up a schedule. It just runs. I am not backing up the surveillance footage simply because that is just huge and pretty perishable anyway.

Soon, however, I would find an even better reason why I am glad I went for the more powerful CPU and full boat of RAM.