Argus Panoptes

Ok, Argus Panoptes may be a little over the top since I currently only have five cameras instead of 100.

In the time we have lived out here in our largely rural property, I have had a few network cameras. My first foray into IP cameras was with a pair of DLink DCS-932 cameras, which if you follow that link, you will learn as I did that I can now only find it on D-Link’s non-US websites.

This camera served me fairly well. Its a wifi camera that can utilize, but does not require, a “cloud” presence to work. It outputs in fairly decent for it’s time 640×480 VGA, had auto night vision and built in IR illumination. It worked by a built in web server and could operate with either an ActiveX or Java applet.

I had one in the barn overlooking the stalls and one by the front door.

I still have them and so far as I know, they still work. I might add them to the mix someday.

More recently, I had an issue with a FedEx delivery. I was expecting a shipment that required a signature. I left the gate open and put a sign on the door that I work from home and to please ring the bell. Long story short, they “made” three delivery “attempts” and were about to return my goods, having not once rang my doorbell. My desk is within site of the front door and the dogs would not have slept through a doorbell.

I had to call and get a bit nasty and even then, the driver was literally walking away from my door 10 seconds after ringing the bell.

We hates them. They used to be the premier delivery service, but now I generally prefer even USPS.

That whole experience encouraged me to get a doorbell camera.

As mentioned in other blog posts, I am not exactly the best guy to try to sell on The Cloud. I appreciate the concept and my company is quite dependent on cloud services, not the least of which is RingCentral telephones, which is my particular gig within the company.

What I like about the cloud is great; it lets you export high powered data processing and storage to what the cloud really is, just someone else’s servers.

What I hate about many cloud dependent devices is that without a fast and reliable internet connection, it’s completely worthless crap that gets between you and a physical thing that you bought and are holding in your hand.

Thus was the state of affairs at our house. Our DSL was mostly worthless. That is to say, it was better than dialup. My first foray into home automation was with a Nexia z-Wave controller. I discovered it’s absolute reliance on an internet connection only once I had it installed. It worked, but the delay was unworkable. The worst case scenario was my wife hearing something while I was working out of town and hitting the button to turn on the outside lights, which, thanks to crappy DSL, would promptly turn on 20-30 seconds later, if at all. That was not gonna work.

So, I discovered Vera, which has cloud features, which are good, but it does not depend on an internet connection to work. That whole thing is documented elsewhere.

Those last few paragraphs are just to explain that I will not install something as critical as a security camera that depends on a internet connection to even work. I’m looking at you, Ring.

My shopping lead me in pretty short order to the Amcrest AD110, available from Amazon. Of course.

The installation of the doorbell camera was neat. I ordered a wedge thingy with it that pointed it more directly into the fourier that is our front entry.

The first issue I faced is that the AD110 is powered from the doorbell circuit and it requires at least 16V. It took some failures and some Googling to resolve it. Our existing doorbell transformer was only 10V, so a quick trip to the hardware store for a replacement fixed that issue.

Eventually, things settled in just fine and the next really important delivery, via FedEx of course, went better. I can’t actually claim that the camera had anything to do with it. At least the $1400 guitar my wife won from Sweetwater didn’t go back three times before they let us have it. Plus, I have Toni on camera walking out the front door right past that huge box without seeing it….

Flushed with this success, I ordered a couple more Amcrest cameras, a surface mount and a bullet mount. Both are PoE powered and have SD card slots for local recording. The surface mount ended up looking at the driveway and the bullet is in the barn looking at horse stalls.

I had some 32GB SD cards laying around and, not surprisingly, it doesn’t take long to fill this card up, especially if you don’t want to trust all recording to be triggered by motion alarms. The camera has parameters for when to start recording over older material. 32GB worked out to about 3 days. Since 256GB is 8 times the storage of 32GB, you’d think you would get 24-ish days, but instead, it only showed about two weeks.

I noticed that the camera had an option to store to Network Attached Storage and I had been thinking of building or getting one anyway, so that was all the excuse I needed.

For all the advantages of building a NAS from a Raspberry Pi or repurposing a PC or laptop, et cetera, I have reached an age and station wherein I just don’t wanna mess with stuff like that. I expired my cutting edge tech card a couple of decades ago. I like cool tech, but I just want it to work. Consequently, I shopped and decided on what is probably quite literally the lowest spec NAS that Synology currently sells, a DS120j. It is a 1 bay unit with an ARM processor. The ARM processor is the brains in a cubic buttload of smartphones, though it is not the only one. They are what most techies would call “markably powerful, within their limitations.” Good, cheap, but not the best. In any case, my DS120j, as ordered, came with a 2TB drive.

I’ll have to give it to them, though, it was a flawlessly easy setup. The NAS and the drive came in separate packaging, so the hardest thing was installing the drive into the cabinet. It took, maybe, 5 minutes. They are designed to go together, after all.

Before I even had much chance to set up a shared folder for the cameras, I noticed that one of Synology’s many available apps for the NAS is called Surveillance Station and it is basically a Network Video Recorder that is just included with a Synology NAS. It has WAY more features than the cameras alone and it bundles all your various cameras into a single interface. One throat to choke, so to speak.

One of the coolest things the NAS can do is give me access to data and more specifically, viewing the cameras, from outside of the house. This can be done with either a cloud redirect service Synology maintains or with port forwarding done in my own router. The DSCam app on my phone lets me check in remotely at pretty much any time. Plus, I just added every MP3, OGG and Apple music file I could find to the audio hosting feature it can also do.

Between the cameras and all the other stuff I have running on my NAS, such as backing up home directories on our various PCs, I was not necessarily pushing the 2TB storage, but in looking to the future, I elected to upgrade the space to 6TB anyway.

Anyway, this is night and day views of my five current cameras, four different models of cameras by Amcrest.

That magic number five turns out to be kind of an issue.

I quickly discovered that Synology Surveillance Station requires a license for each camera attached to it. The DS120j (and all models, I think) comes with two included licenses. Out of the box, you can only attach two cameras. I’m not a big fan of the license model; if you own a thing, you shouldn’t need a note from your Mom saying it’s ok to use it. I understand it, however, and I can choose to not put cameras on Surveillance Station or I can choose to buy licenses. I elected to purchase a four-pack of licenses because I could easily foresee six cameras in my system.

My next discovery was that the DS120j is limited to five cameras. It turns out that Synology, wisely, limits the number of cameras a NAS can monitor based on the capabilities of the host device hardware. The number is buried in the specs somewhere, but each model Synology NAS has a maximum number of cameras it can handle and mine can handle five. Consequently, I bought six licenses, one of which is not doing me any good.

Upgrade options are plentiful, though. Even the arguably least effective but cheapest upgrade is to the DS118, another single bay NAS, but it has significantly more processing power and it is rated for 12 IP cameras, $180 on Amazon as I write this. A more sensible upgrade would be for a two-bay system, where I could put the old 2TB together with the 6TB or upgrade for even more storage. The DS218 will do 25 cameras and the DS220 will do 20, for $250 or $300 respectively. More cameras and $50 less makes more sense to me.

In practical terms, I have captured a variety events, some planned, some not.

The very first thing I caught with the doorbell camera, which was the first camera as well, was escaped horses. I was expecting another delivery, so when the motion alarm went off, I hurried to the door and nobody was there. I went out side to see if I could catch them and what I caught instead was our two horses in the driveway. πŸ™‚

We live in the country, so we have wildlife. A family of raccoons frequents our barn. I’ve caught them on occasion just going out there, so I did expect to see some in the barn camera and was not disappointed. I did find it disappointing that they consider my camera to be climbing hold, even after I tried to cover it with something very uncomfortable to step on. Note the old D-Link camera post. That has been removed since this pic was taken. It was yet another climbing aid.

I had purchased a camera to put by the gate, but had not yet deployed it, so I set it up temporarily in the barn walkway, pointed at a live trap.

I eventually decided I would want a permanent camera for outside the barn doors and finally deployed the gate camera at the gate.

The challenge was power. There is a gate opener out there with a small solar panel to charge it. Experience has shown that it is adequate for keeping the gate battery charged and I considered just connecting the camera to that battery. The opener pulls such a tiny bit of current unless the operators are actually moving the gate, but the last thing I want to do is have a camera pulling current 24/7 and run down the battery just in time to lock my wife in or out.

The gate camera has a USB power cord and I have ordered a 12V to USB adapter dongle to connect directly to a battery. Due to the nature of the USB power, I could feel reasonably confident it should draw 10W or less. The new solar panel is larger, a 25W panel, probably more than is needed. I also found a charge controller that has USB outputs, so I don’t even need the power dongle anymore. I already had a couple of battery boxes and elected to rob a lawn tractor battery from a mower and replace it later.

I mounted the camera, connected it to my Wi-Fi network and reconfigured it as needed.

The gate controller is other side of the board. Yeah, the fence needs painting. I painted the gate back in October, though πŸ™‚

The camera catches any delivery vehicle, trash pickup, our own comings and goings, etc. I have the motion detection set to catch motion directly in front of the mailbox and, due to the close angle, I can get an alarm when the gate itself opens or closes.

I was not surprised to find that the Wi-Fi signal from the gate to the house is ok, but not really solid. My intent was to deploy an old Belkin Wi-Fi extender from years past. I have seen it recently, but I could not find it. I presume it is in a box that I didn’t open. Instead of continuing the search, I ordered a BrosTrend AC1200 to extend the house WiFi into the garage, which is closer to the gate. This increased signal strength somewhat. I think better positioning of the extender to expose it more to the garage window would bring it up even more.

The last camera I have on the NAS for now is another of the same model as the gate camera, mounted outside the barn with a view of the water trough and stall doors.

Sharp eyes may notice a dove nesting in the rafter, behind the minutes digits (43) of clock display.

One reason I wanted a camera out there is just to monitor the water trough, largely because one of the horses has done mischief with the equipment. That is why there is a steel guard covering the float valve. With the camera in place, it didn’t take long to catch some hijinks.

As I suspected, it’s Bonk. While you can’t see it in this still view, there is a green plastic tub that normally covers the water circulating pump. He had knocked it off earlier, then picked it up and moved it closer to the trough. The splashing he’s doing above was enough to put a couple of inches of water in the bottom of that tub. I found it entertaining.

One funny side effect of the cameras switching to night mode and turning on their IR illumination is that one camera can see evidence of another. The driveway camera can see the gate camera so well that I thought I had left the floodlight feature activated, visible here as a bright slash at the top center.

In this one, the light spilling from the stall doors is actually the illuminator on the stall camera.

Ignorance Is Bliss

One of the many jobs a Synology NAS can do is to serve as a syslog server. Most network gear can support sending it’s logs to a syslog server and in doing so, you can have one place to go to review logs on all those devices.

On the other hand, what you don’t know can’t hurt you, right? Right?!

Seriously, the issues I have discovered in just over 12 hours of using Log Center are arguably not actually serious, but they are bothersome just because now I know.

I configured Log Center to collect logs from my pfSense router/firewall, two Cisco switches and, just for the entertainment value, one of my IP cameras.

Activity from the switches is very light. I verified that it would log a port being disconnected and reconnected and other than an hourly DHCP refresh from each switch, they have been pretty quiet.

The camera is also pretty quiet. It mostly shows login and logout activity from both my laptop and the NAS while playing with some settings and silence since then.

The router, on the other hand, is quite chatty. It also has more granular control over what gets sent to syslog.

Note that I have unchecked Firewall Events. Before doing that, the log was just stupid busy. The firewall blocks a LOT of traffic. I do need to analyze that traffic at some point. Some of the blocked traffic is internal.

The thing that bothers me but probably shouldn’t is the number of DHCP requests from stuff that is obviously online and operating.

Does my camera out by the gate really need to refresh it’s IP every 4 seconds ALL DAY? The camera alone accounts for almost 71% (32,552 out of 46,198) of the log events between midnight and a bit after 11AM when I pulled the log to look at it. Why does a camera that is online and operating have to do that?

I will figure it out….

I Re-re-renumbered My Home LAN

As part of deploying Starlink as my primary internet provider, I had to renumber my LAN. Starlink uses as its default client network and that is not alterable, so far as I can tell.

pfSense didn’t know when I started configuring it, so it didn’t argue, but also didn’t work. I finally took a guess that the same IP on both sides of the router could confuse it and changed mine to and my stuff started working!

Well, it did until the first time I connected to my work VPN, Cisco AnyConnect. At first, I didn’t specifically notice that the VPN was the trigger. I was working away at something and wanted to check on my camera system to see if the mail had arrived, but I couldn’t get to my NAS. I didn’t panic or anything, but later (without realizing that the VPN was disconnected, I was able to reach the NAS and cameras just fine. Some time later still, I connected the VPN then tried to refresh the camera display and it finally dawned on me that with the VPN connected, I couldn’t get to the NAS. I tried to verify the issue by connecting to something else, one of the switches I think, and: no luck. Disconnect VPN, get right in. Reconnect VPN, get nowhere. Well, except for my router. I suppose the VPN needs the default gateway.

AnyConnect info includes a list of secured routes. It turns out there are secured routes cover most, though thankfully not all, of RFC-1918 address space. All of is either in use or allocated internally. Quite a few legacy locations use various ranges within, especially in the bottom half or so. Finally, there are only a few used within

I dashed off a note to the network team exlaining my issue and asking for assistance in finding a subnet I could use without interference, but that note as yet has not been answered. I used my Linux jumpbox, which is a VM in one of our data centers, to investigatively Nmap parts of this space, looking for some vacant space. Running Nmap from home is limited to only the secured routes provided through AnyConnect; running it from the data center should also detect systems that are not specifically accounted for in the VPN.

First run through, it looked like a large block near the top of the space should be open. I chose as being easy to remember and easy to type.

Even though I was just testing, I was still going to have to update the DHCP reservations I have because the devices I need to be able to reach all have reserved IPs. The process is to edit the LAN IP address, but don’t apply the changes yet, edit all of the DHCP reservations, then apply the changes under both menus.

As soon as you apply those changes, the IP you are using to connect to the router will no longer be valid, so you need to refresh your IP address then log in to the router again.

And it still interferes with the VPN.

Just for the entertainment value, I tried again, but this time, since the gateway for Starlink was already connected, pfSense wouldn’t let me use it.

I then went to my next alternative, the range. There are a couple in use, but they are at the low end of the range. Ages ago, we had a WAN provider that had assigned the entire range to our company within their network. At that time, about half of our branches were numbered as 172.29.X.0/24 or 172.30.X.0/24, where X was the branch number and 29 or 30 revealed whether that location connected via VPN over internet or a point to point T1. It worked well for us. It was fun to realize that my muscle memory could still type really fast, so….

I renumbered my LAN again to and FINALLY, all my stuff coexists with the work VPN!


I was not fully aware of Starlink as an internet service for a while. I must have started paying attention on or about January 28, 2021, which is the date I received confirmation for signing up for emails about the service.

On February 19th, 2021, I received the email that it was available to order. The email came in around 7PM and it was 10ish before I saw it and jumped on to pay my $99 deposit. In a mere 34,387,200 seconds, it arrived.

Starlink wisely did not email reminders that their dates were sometimes slipping. You have to log in to your account to see the status of your order, which rarely changed. Mine was originally something like “mid to late 2021” then “late 2021” then “first quarter 2022”. In December 2021, it finally said “March 2022”. Since they were willing to narrow the window down to a specific month, I then had the confidence to step up my preparation efforts. The chronology on these various efforts is a bit muddled. I can specify what date I ordered some piece of gear, but for this story, what I did and tried to do are more important than exact dates or times they were tried. It will be like a movie that skips around in the timeline, but it still has a beginning, a plot and an ending. My Starlink kit arrived March 24th, 2022 and it’s working pretty well in my network right now as I write this. This post is the story of making it work.

As detailed elsewhere, I had switched my TP-Link Wi-Fi router for a pfSense router and reconfigured the TP-Link to be an access port. The pfSense router was specifically the Netgate 1100 appliance preloaded with pfSense+.

This appliance has three ports, labeled WAN, LAN and OPT. Most commonly, OPT is used for either a 2nd LAN, like a DMZ, or for a 2nd WAN connection, which was my intent.

I didn’t really start documenting the network until I had pfSense in place, so there aren’t a whole lot of differences between this diagram and the diagram for the network as built.

Even as I look at this now, I see a couple of minor typos in text, but this is a pretty good representation of what was in place around October 2021. I don’t remember for sure, but I think this diagram was inspired by the installation of the TP-Link switch in the house. Before this, it was just whatever little 8 port switch I had, maybe a Belkin.

What is not illustrated well here is the number of Wi-Fi devices in the house. Just as I am writing this, there are 25 Wi-Fi devices that I can list, plus a couple of my wife’s devices that aren’t here right now.

Following r/Starlink and other research indicates that obstructions to the dish are a big issue. Also, though I didn’t know it at the time, the dish was likely to orient itself pointing north. From the house, north is pretty much where all the trees in the back yard are, but there are no such obstructions north of the workshop.

The easiest solution I could think of to have the dish at the workshop but connected to the router in the house via VLAN. One port on the switch at one end untagged in a unique VLAN and one port on the switch on the other end untagged in the same VLAN should be all that was needed.

What follows is a completely non-exhaustive description of what a VLAN is. I could be wrong, but this is what I think I know about them.

A VLAN is a group of ports on a switch that are separated by an identifying tag in the ethernet frame. Terminology can vary between manufacturers, but most commonly, the physical port VLAN membership is “untagged”, meaning the frames going in and out of that port do not themselves have a VLAN tag. Internally, the switch tags those frames with the VLAN number. No matter what the device is or what it knows about VLANs, everything on that port is in the same VLAN. Most often, this is also the “default” VLAN, which is usually but not always VLAN 1.

More sophisticated switches will have a VLAN mode for each port, usually “access”, meaning that port provides access to it’s untagged VLAN number, and “trunk”, meaning the switch will pay attention to the tags in frames going to and from that port. A trunk port can allow all VLANs or just a list of VLANs in or out of the port.

The nice thing about a trunk port is that if you have two or more switches connected by trunk ports in both switches and you have everything configured correctly, ports in both switches configured for untagged VLAN 2, for example, will pass traffic between themselves, yet they will be isolated from the other ports on the switch. It is a simple implementation of this characteristic that I am using to backhaul my Starlink dish over the wireless link between the house and workshop, without anything else in the network seeing, or more importantly responding, to those packets.

Lesson one is that even if a switch advertises VLANs, that doesn’t necessarily mean it will do all this. It took a good bit of experimentation to determine that my TP-Link switches, even if a port is configured for tagged VLANs, that doesn’t mean it is actually a trunk port.

In order to connect Starlink to an existing network, it needs an ethernet port. The previous version of dishy, or more precisely, the previous model Starlink power brick, had an ethernet port that could be used for that. The newer rectangular dish comes with a one box power supply/router that does not have an ethernet port. To plug in, you need an ethernet adapter that is cheap enough that I wish they would have just built it in and charged me $20 more. When I completed the dishy order, I also ordered one of these adapters, but it is not expected to ship until mid April. [ed: shipped April 13]

There are all kinds of dishy hacks all over the internet, but for me, waiting for the official adapter is less painful than risking non-warranty damage to the equipment. What I have done in the mean time is use a Wi-Fi extender that also has an ethernet port. I ordered a BrosTrend AC1200 extender, configured it to extend the Starlink Wi-Fi, but I largely ignore that and use the ethernet. It supposedly runs up to 850-something Mbps in 5GHz mode. The bad part is the that ethernet port is only a 100 Mbps port, so that will cap the maximum speed available from this setup to some fraction of 100 Mbps.

When I took Starlink out of the box, it was very simple to set up and use. Knowing it was temporary wiring anyway, I opened the window into the bedroom where the equipment is, fed the dishy cord through and connected everything up. I then moved the dish as far to teh west as the cord would reach, while still keeping away from trees to the north.

It was about where the dot is.

I powered it up. There was a little interaction with the Starlink app, mostly setting the SSID and password and not much else. After some time, maybe 5 minutes, probably less, it connected well enough to run a speed test. The first one from my phone was not terribly impressive.

It would turn out that speeds on my cell phone are never as good as on a PC. I presume there is a reason that I have not seen as yet. All the tips tend to be for making sure you have a strong signal. <shrug>

At this point, I had not yet received the Wi-Fi extender, so for a little while, my only option was to connect to the Starlink router for high speed or connect to my existing equipment for anything else. It was only for a day or so.

When it did arrive, testing it was easy and setting it up was easy, too. When you power it up, it provides an admin network to connect to and a webserver for configuration. Basically, tell it what network you want it to extend, provide credentials and it’s done.

I tried for a while to connect it directly to the OPT port of the pfSense, but I couldn’t quite figure out how to configure it. I did find that I could just plug it into the WAN port and the router would just get an IP from Starlink and work. It didn’t care that it was a different subnet than OneSource was. I began to suspect, however, that the issue may have been related to teh fact that Starlink comes up with the same 192.168.1.X network that I am using on the LAN side. As much as I hated doing so, it kind of made sense to renumber my network, since Starlink doesn’t give you any control of it’s subnet.

Sadly, that didn’t do it, either. Furthermore, it would soon come to light that the new IP range, 192.168.2.X, conflicts with routes on my work laptop when I connect to the company VPN. If VPN is connected, I can get to my router, but nothing else on the LAN. The biggest effect is that I can’t get to Gnaz to view the cameras and I can’t remote to my non-work laptop to when Cisco Umbrella blocks a website I want to get to. All that means I will need to renumber again, this time avoiding all the routes in the AnyConnect. I think it the LAN and WAN subnets should be different, if for no other reason that clarity, so I won’t be going back to 192.168.1.X.

Somewhere in here, I had broken something on the router that I couldn’t recover and ended up recording all the static DHCP mappings manually and factory resetting pfSense. This time, I configured it with Starlink on WAN and OneSource on OPT and I guess I had cleared something that was keeping it from working or I had a better YouTube video to follow or something, because I got it to work and even switch to OneSource on Starlink fail. Even now, I’m not totally happy with the failback. If the gateway goes offline due to failed pings to its monitor IP and comes back on its own, it will switch back, and I’ve seen that happen a few times naturally. However, if I unplug either provider’s cable to simulate an outage, it doesn’t want to switch back to it when it comes back up. I have to manually set it to down and then back up.

With that part of the puzzle working, I could move on to local testing of the VLAN setup. At this point, the both TP-Link switches were in the same cabinet with Starlink and really everything else. I had Starlink Wi-Fi to the AC-1200, the AC-1200 plugged into port 8 of the workshop switch, a cable between port 7 on each switch to simulate the link to the workshop and the WAN port of the router connected to port 8 of the house switch.

I started with the simplest configuration I could think of and both switches configured the same. Ports 1-6 were untagged on the default VLAN 1. Port 8 was untagged VLAN 2. Port 7 was to be the trunk port, so I first tried untagged VLAN 1, tagged VLAN 2, which should have allowed 7 to carry both VLANs. In short, I tried all four combinations of tagged and untaged VLANs 1 & 2 on port 7.

The two would not pass traffic between port 8 workshop and port 8 house.

I Googled a bunch and finally found a thread where someone explained it well. Those tagged and untagged determine how the local switch treats the ports and VLANs. The packets coming out of the switch do not carry the tagging information at all, so it can’t pass it to another switch. In short, those are not trunk ports.

As I had mentioned a few times in the network blog, I have a Cisco SG200-26P. This switch has 26 ports, 12 of which can provide up to 100W of PoE and it can definitely trunk VLANs as I expected. I just needed to get it a playmate that could the same. I ended up ordering a SG250-08, an eight port switch with essentially the same set of features, except no PoE and, of course, fewer ports.

Even that was not an instant success. The two switches are different enough that the dialog for same setting may not look the same between them. In the 8 port, the port types are “access” and “trunk”, period. In the 26 port, there are “General”, “Access”, “Trunk” and “Customer”. Choosing Access grays out all the default VLAN, etc. General looked like what I probably needed for all but the trunk ports. Customer had a note beside it that made me wonder if it was some kind of auto VLAN, putting each port on a VLAN of its own, like if you were using it in a hotel or multitenant situation.

Long story short, it still didnt quite work, but it was getting close. The MAC address of the AC1200 plugged in to the 8 port began showing up in the MAC address table of the 26 port, but it still didn’t pass any traffic. More Googling lead to a recommendation that all ports be Trunk, but just control what VLANs are available for each trunk. I reconfigured all the previously “access” ports to “trunk” and made sure my tagged and untagged stuff was configured as I thought it should be and it suddenly started working! I was hitting Starlink from the house and it was traversing both switches to get to the WAN port of the router.

It was definitely too late to start moving stuff to the roof, so I would leave it like that to run overnight and through the next day.

The plan was a simple one. Move Starlink, the AC1200 and the 8 port switch to the workshop, with the dish on the roof. Though it was kind of slow going, that all worked out well.

At first I grumbled about having to get on the roof to do this, but it suddenly occurred to me that it just needs to be off the ground. There is no particular advantage in this location between the peak of the roof and where it is shown in this picture. There are no obstacles to clear with a few more feet. I ran the wire into the barn, where the eaves are screened, but not actually sealed and immediately into the loft/attic area above the workshop, which is roughly the left half of this picture. It’s dusty and dirty up there, but I got the job done. I also ran the wire that feeds one of the cameras through teh same path. It has been temporarily run for months. πŸ™‚

Once in place, I find it interesting that the dish orients itself in a more northeasterly direction.

It was a quick thing to change out the old switch in the workshop for the Cisco and connect the AC1200 and Starlink. I waited for the satellite to come up, verified I could connect directly to it and get data then tried from the Wi-Fi out there. Connecting to Wi-Fi, but no internet. Hmmmm.. Oh, yeah, I still need to make the connections in the house. Hooked it up and everything came up!

I plugged the one hanging cord into the switchport it was supposed to go into. It didn’t come up, even after giving it about 5 minutes. I manually down/up’d the WAN interface in pfSense and it came up!

Again, it was too late in the evening to rewire everything in the cabinet. At this point, the Cisco switch had the two ports involving the backhaul and a couple of other ports in use, but the rest of the stuff was still plugged in to the TP-Link switch.

The next afternoon, I took everything out of the cabinet, cleaned it up and recabled with some color coding. It was getting old tracing black wires connected to mostly black equipment inside a black cabinet. When I finished, it looked like this:

Pretty quickly, I noticed that I was connecting to the Wi-Fi, but not getting internet. I had a link light and all the wiring appeared fine. I dug in the switch configs and found that a feature called Smartport showed a different status for the port the AP was plugged into. I moved the AP to 18 and was able to get internet via Wi-Fi on my phone. I tried the laptop and it would connect to Wi-Fi, but no internet. I looked at Smartport and now the port I moved the AP to looked like it’s previous port.

I chased the Smartport stuff in the switch and online for quite a while and got nowhere. I found that if I put the WAN connection back through the TP-Link and also put the AP on the the TP-Link that everyone worked as it should. Again, I had killed pretty much the whole evening troubleshooting this one issue.

The next day, I mentioned it to a coworker. Networking is not his specialty, but in describing the problem to him, something occurred to me. When I plugged everything in, I could get one complete valid connection, then nothing else could get internet. It tickled the back of my mind that this is an intentional behavior sometimes, to prevent users from adding rogue switches or APs to company switches. It took me a a little more Googling to remember that is called Port Security. I found it under Security, not interfaces or Smartport or VLANs. Sure enough, the ports the APs had been on were locked, configured to allow 1 MAC address. I cleared them, set them to allow 50 MACs and normalized all the wiring. The AP works perfectly now.

As it turns out, the Smartport thing could have been a clue. When port security locked a port, Smartport responded to that action; it never was Smartport doing it.

All that has been operating untouched for 24 hours, pretty much as I am typing this. For now, I’ma call it good. Here is the updated drawing.

The next thing should be another simple change when the ethernet adapter arrives. I don’t know what the effect of disabling Starlink’s router will be, other than freeing up an AC1200…

The Obligatory First Catchup Post

As is often the case, when I (or most people, I suspect) start a blog on a subject, it’s something they were already doing and suddenly decided that blogging about it might be fun, or even helpful. What follows is the story thus far….

My home networks since the inception of the common use of post-dialup internet have generally been the minimalist one or two hardwired PCs connected to whatever switch/router combo the internet provider offered. That was plenty at the time. When we moved to our current house in a largely rural neighborhood in late 2010, it coincided with both a desire to execute some home automation and the proliferation of more smart devices, more phones, more laptops, more tablets, more TVs.

The only internet providers available at the time were DSL from the ILEC CenturyLink and the usual suspects for conventional satellite internet, like HughesNet and Viasat. There may be others <shrug>. CenturyLink DSL was adequate for general internet and was also reasonably stable, though there were a few outages during the years we had it.

When I started this post, I had intended to tell the whole story of the network and home automation, because there is a fair bit of crossover between them, but now it occurs to me that one massive blog entry is going to be hard to absorb and enjoy and it’s already too long, so I am going to to split them into two related categories, Home Automation, which I started a freakin’ decade ago and I suppose this one will just be about the network in general.

What follows is roughly in chronological order, but not necessarily strictly so.

The router that came with our DSL service was a decent one, I suppose. I think the first one was a Westell and at some point, they replaced it with a ZyXel. Eventually, I wanted more control, so I bought a Belkin router to put behind it, or maybe the Belkin had a phone jack for DSL? Purchased from BestBuy, if I remember right. The Belkin had 5GHz Wi-Fi, which helped with some devices. The factory Wi-Fi password for this router was a decent one, easy enough to remember, so I’ve kept it a couple of routers later. I also eventually got a Belkin Wi-Fi extender. All the network gear is in a bedroom at the corner of the house and sometimes the signal was a bit weak in the kitchen at the opposite corner of the house, but importantly, where we spend much of our time.

The Wi-Fi extender was not enough to provide decent coverage in the workshop. The shop is roughly 100 feet from the house and it is even near the corner bedroom where the Wi-Fi is located, but the signal was still weak.

With the exception of these two BestBuy purchases, from here on out when I mention ordering something, it is pretty safe to assume I probably ordered it via Amazon. If I didn’t and its important, I’ll comment to the effect.

Anyway, based on some experiences with point to point links between buildings at work, in late 2013, I got a set of EnGenius ENH500 outdoor access bridges. This was an extremely simple and robust system that lasted nine years, through power failures, thunderstorms, ice in winter and Texas sun in summer. They were just never the problem. Since BestBuy was handy, I also got a little Belkin 8 port switch and that was plenty to support a PC and some radio gear in the shop.

Cheaping out on Wi-Fi for the workshop *was* a problem. I had a surplus Cisco WAP that seemed to be a good unit. It had 5GHz Wi-Fi, three antennas for signal diversity and a good reputation otherwise. Mine however, had a tendency to go to sleep. Rebooting it would bring it back, but it would just freeze and stop working again later. I basically dealt with it because the only thing I really needed Wi-Fi for was my phone when I was out there.

If I recall correctly, the CenturyLink DSL was spec’d out at 7.5Mbps download and 2.5Mbps upload. It never met those exactly, but for most of the time we had it, it was pretty solid for 5M down and 2M up. I had little need to work from home then and when I did, it was almost always consoling into a Linux server or some switches and routers, so it was almost all text.

Eventually, however, DSL service began to degrade. I could turn in a ticket and the tech would reset something or replace some card in the DSLAM and get it kinda back up, but after a couple more years, it became a case where dialup internet would have been only a little slower and probably more reliable. I think CenturyLink just didn’t want to invest in upgrading the infrastructure for a handful of houses in a rural neighborhood. There were no new customers to be had.

Terrestrial wireless internet began to look appealing. I don’t remember if we had more than one provider come out, but we did eventually get wireless internet from OneSource in November of 2017.

The installation was what I would classify as marginal. The LTE modem was put on the roof of the house, but at the minimum viable height, just barely shooting over an outbuilding next to the house. Even so, it worked great, 10M/2M as promised and pretty reliable and troublefree except when the growing season was in full tilt and the trees were in full foliage.

One particularly verdant season, it was affected more than usual, but coincidentally, they were upgrading equipment on the tower, which was to require changing the equipment at the house.

The tech that came out put the new modem on a tall roof mount at the peak of the roof, where it gained 8 or 10 feet of altitude. The new equipment was also good for 25M/5M, so I signed up for the upgrade.

I also befriended that tech, which is good not only because he’s a cool guy to hang out with, but it also allowed us both to benefit from my willingness to experiment and help him by testing stuff. It also helps that I’m not Joe User when it comes to this stuff and can converse with him intelligently about issues. And blacksmithing. And firearms.

Somewhere in here, a coworker who had just changed out his whole home network to Ubiquiti gave me a Cisco SG200-26P that he no longer needed. It would sit unused for a while, but would one day become very important. πŸ™‚

OneSource has rocked along since then pretty solid. The old equipment on their tower could only go as high as 10Mbps, but the tower antenna was basically pointed south southeast and we are pretty much due southeast of the tower, so we enjoyed a strong signal, even though the antenna on our end was barely over the roof of the craft shack. The new equipment has higher capacity, but the antennas are pointed due north, south and east, putting us in a comparatively weak crossing of the south and east antennas. Our modem will connect to either. I forget which is which, but connected to one, we get the full 25Mbps, but it’s not as stable and reliable, dropping out occasionally. On the other, we get 10-15Mbps, but it’s more solid. I tend to stay on the solid side because….

Now I work from home almost exclusively. Long story short, my company moved headquarters from near downtown Fort Worth, which is a nice 35-45 minutes drive to Addison, which is a solid hour to 90 minutes in the morning and longer to get home if you don’t leave either well before 4PM or after 6PM. We had negotiated being able to work from home for a number of days, and I often used my “work from home” days to work from the old Fort Worth office on certain days because I had pistol matches in or near Fort Worth, so that was even closer than working from home.

Then came the unspecified virus of unknown origin. <cue dramatic orchestra hit>

Suddenly, working from home was pretty much the only option and 2M upload that wasn’t always 2M was just barely enough. I could join conference calls, but I was the only one without a camera on.

Oh, and then I broke my shoulder and had total shoulder joint replacement surgery, but the virus was a bigger issue.

During this time, my company was in the process of changing to RingCentral telephones and since my gig with the company is the phones, that Cisco SG200 switch mentioned above came in handy as 12 of its 26 port provide PoE and it has a 100W power budget. I could connect all the phones I needed for conducting labs and configuration testing and all the stuff one might otherwise normally do at the office.

When I first heard about Starlink (I am, among other things, a bit of a space nerd) I signed up to be notified when service was going to be in my area. Shortly, I received notice that they had a Beta program wherein one could pay a $99 deposit and end up on a list of possible beta testers in an area. I did that in February of 2021 and tracked the news, but otherwise went on with my life.

In a move to improve Wi-Fi coverage in the house, in June of 2021, I decided to upgrade the old Belkin router. It had no external antennas, so I chose a TP-Link unit with an array of them, hoping to maximize RF flux in the house. I also put the freshly liberated Belkin router out in the workshop as a simple access point and finally had reliable Wi-Fi out there.

I was pretty impressed with the features and benefits of the TP-Link router, including their mesh products. Combined with this router, I could backhaul ethernet to the workshop and have another TP-Link access point out there and they could be the same SSID! No more having to connect to the workshop Wi-Fi out there and reconnect to the house back here. So, by July, I ordered a 2nd TP-Link router of the same model.

Well, it turns out that the router product apparently can’t do the ethernet backhaul *as an access point*. It can be the controller and any of several models of mesh capable access points could backhaul to it over ethernet and use the same SSID, but not two of this model back to back. Poo. Rather than return it, I still deployed it as an access point, but now I renamed the networks HippyHollow for the house and FlyingDog for the workshop, though I still kept the password that my family already knows.

October got busy, partly because Starlink’s delivery estimate narrowed down somewhat to “Late 2021”, so I started tweaking my infrastructure to accommodate what I hoped would be delivery soon. Having studied some of the requirements for Starlink, I knew that the same trees in our back yard that caused us to install the OneSource modem on the far west end of the house would never work with Starlink’s dishy, so I formulated a plan to install the dish, once received, on the roof of the workshop because there are no trees over there. I would then use a VLAN to backhaul it over the point to point bridge to the house and plug it into the router. To that end, I ordered a couple of TP-Link switches with VLAN features. For a $20-30 switch, these are pretty nice units.

Around this time, I saw a Network Chuck video extoling the benefits of the pfSense firewall/router. It occurred to me that having a router with multiple WAN ports could allow me to, at least for a time, keep and use OneSource even after Starlink is in place and my TP-Link router had only one WAN port. So, early in October, I ordered a Netgate 1100 appliance with pfSense+ preinstalled. Sadly, it would be mid November before I could get it deployed, coincidentally while my wife was away on a cruise. That way, if I got stuck on something and killed our internet, she wouldn’t have to suffer, and thus, neither would I. πŸ™‚

The wizard in pfSense works pretty well and getting the router up was easy. It was also easy to break the config enough that it stopped all traffic. It was then also easy to reset it to factory and set it all back up again, having learned my lesson. πŸ™‚ I reconfigured the TP-Link router as a simple access point, like the one in the workshop and kept it in place.

It was around here that I had ordered something that required a signature on delivery. The FedEx guy kept coming to the door while I was home and apparently knocking with a feather so that not even the dogs could hear it and leaving notices that they were about to return my thing to the sender. I called in and probably completely irritated someone that day, but eventually we met up at my door and I got my stuff. The whole experience inspired me to install a doorbell camera, partly to be able to collect evidence, but also because when there is an obvious camera, people tend to act differently, in this case, hopefully try harder to complete a delivery.

Cameras are largely considered to be in the home automation segment, but most of the story of mine is pretty network centric, so I have overlapping parts of the story in both blogs.

Because I knew I would still be upload limited for a while, I wanted to avoid anything that was dependent on the cloud (you know, what we used to call servers, but are now just someone else’s servers), so Ring was right out. I dug around and found that Amcrest had decent products and while <buzzword> cloud </buzzword> (and don’t even get me started on the people who now call conventional servers the “private cloud”) was an option, it was not required; they could record to a local SD card and they could be administered locally. I ordered an Amcrest AD110 doorbell camera and two other cameras suitable for watching our driveway and watching the horses in the barn.

They were pretty easy to connect. The barn camera would have to wait a while, but the doorbell connects to the existing doorbell wiring for power and has some kind of terminator adapter thingy that connects across the existing chime in order to operate it. The camera appeared to come up mostly, but would never successfully connect to the network. A bit of Googling revealed that it requires at least a 16V doorbell transformer in order for Wi-Fi to operate correctly and sure enough, mine was only 10V. Quick trip to Lowes and a quick trip to the attic to swap that out and it came up beautifully. Literally a couple of days later, I am working at my desk and the doorbell motion detector goes of. It turns out to be that the horses got out!

If I am being completely honest, the AD110 does require the largely cloud-based app for configuration of the camera, but it pretty seemlessly integrates, so it’s hard to tell what might be local and what might be cloud. I should test for that.

The driveway camera was more involved because I needed to run an ethernet cable to where I wanted to place it. Happily, most of our attic is pretty open and the wire was easy to run. The camera runs on PoE, so again glad to have that Cisco SG200 switch.

I didn’t have PoE in the workshop/barn, so I had to order a suitable injector. Again the camera came up easily and I have grown to appreciate Amcrest products.

I found that the 32G SD cards that I had readily available for the cameras’ local recording would fill up pretty fast, so I ordered some 256G cards from Amazon. They arrived and I’m not sure what the issue was. They were SanDisk class 10 devices, but they would randomly unmount and recording would stop, sometimes requiring a reboot to restore functionality. [ed: now I know that the write speed of SD cards is important with video] I put the 32G cards back in, but at even pretty modest settings, they would fill and auto overwrite in just a few days, particularly as each alarm event is recorded at a higher rate and I was still getting a lot of video motion detection alarms as I tweaking all that.

I had noticed that one of the recording options was to send to a Network Attached Storage device. I thought that might be a good solution, plus we didn’t really have any appreciably valid data backup strategies for our laptops. I did some shopping and YouTubing and decided that a single drive Synology NAS would probably suit me well, so I ordered a DS120j with a 2TB disk.

Almost immediately upon getting it set up and learning about its features, I found that Synology has a Network Video Recorder app for their NAS called Surveillance Station. That turns out to be way better than just using the NAS to store the camera’s recording files. It is a fully featured NVR and video is stored locally.

However, nothing is perfect. I discovered that Surveillance Station is license bound. The NAS comes with licenses for two cameras. When you have two cameras and go through the procedure to add a third, you are given an option to continue after selecting which of your existing cameras to disable or to cancel. The licenses are not particularly cheap, either. However, I decided that I liked the advantages enough to pay it, and it was still cheaper than a dedicated NVR. So, thinking about where I wanted to put cameras, I ordered a 4-pack of licenses for a little less than $200.

Then is when I learned about the other imperfection. The essentially minimal hardware NAS I had would support a maximum of five cameras. The system resources are the limitation and Synology, wisely I suppose, will not let you over subscribe the hardware. So now I had five usable licenses and one unusable $50 license. Sigh. The next higher rated hardware is a two drive unit that can support 12 cameras and the next one after that does 25. If I feel the need enough, I can migrate my existing disk to it

I know that I can migrate a disk because I have done it once. By then, I had four cameras connected and I was also taking up some file space with several hundred uncompressed photos for astrophotography.

I had fears that I would outgrow my 2TB disk. By December, I ordered a 6TB replacement disk and a 2TB USB 3 external SSD and performed a complete backup using one of the Synology provided tools. It was a very slow process; I left it overnight because I got tired of waiting for it to finish. [ed: yes, the write speed of the 2TB USB drive was a big factor in that slow process] Synology treated the new 6TB disk like it was a new out-of-the-box NAS. Once it completed all its formatting gyrations, I plugged in my backup drive and started the restore process. It took another several hours, but worked flawlessly. The only loss was the footage that would have been live between in end of the backup and the end of the restoral.

It suddenly occurs to me that it was probably slow on backup because Surveillance Station was hitting the drive with 4 new live streams the whole time it was trying to back up the disk. Perhaps I will remember that and take it offline first next time.

Network Chuck strikes again. In my hope to further avoid cloud services for things in my house, I think the benefits of reverse proxy are attractive. In this particular instance, Network Chuck deploys it in the form of a free Kemp load balancer.

I should have made notes setting this up because getting the hardware in place to run a Proxmox hypervisor upon which the Kemp VM can be deployed had it’s own story. I had a fanless PC that I once tried to use as a Windows jump box for ham radio applications out in the workshop. I never could align all the sleep features such that it would not turn off after a couple of hours. Proxmox of course has none of those kinds of features. I think I upgraded RAM but I definitely added a 120GB SSD. BTW, I did try to load the free ESXi hypervisor, but there was something it didn’t like about the machine, but Proxmox loaded and ran just fine.

I did not finish getting the Kemp stuff up and running. Rather than strictly follow the examples in the video where Chuck shows us how to set up a domain name with FreeNom, I wanted to use an underutilized name I already have with a different registrar. I had some trouble getting it to do what was expected and I have not yet finished the process. I will revisit at some point. [ed: plus, Starlink’s use of CGNAT pretty much breaks this particular application of reverse proxy, at least not without outside help.]

More cameras: The driveway camera can *just* see the front gate and I wanted to put a camera out there by the gate for better coverage. My phone has a usable signal from the house Wi-Fi , but being 170ft from the house means power could be an issue.

I ordered a camera that sounded promising, an outdoor Wi-Fi connected camera with a USB power cord and a 12V to USB adapter dongle, the intent being to connect it to the solar charged battery that is already out there for the gate opener. Unfortunately, it would be late March before I got around to setting it up. In the mean time, it also occurred to me that the gate draws nearly nothing unless the operators are actually moving the gates, but the camera is going to draw something 24/7, though due to the nature of the USB connector, I could feel confident it should be 10W or less. As the gate is a kind of security and safety thing for which it would suck to have a dead battery, I decided I would just add a separate battery, solar panel and charge controller for the camera.

The panel is a 25W panel, I’m sure bigger than is needed. I also found a charge controller that has USB outputs, so I don’t even need the power dongle anymore. I’ll repurpose those to our kayaks or something. I already had a couple of battery boxes and elected to rob a lawn tractor battery from a mower and replace it later. [ed: as time has passed, this has been a reasonably reliable setup, but several days of cloudy conditions is enough for the battery to run down enough to lose power to the camera. I am not sure if I need to address that with more PV panel, more battery or both.]

I was not completely surprised to find that the Wi-Fi signal from the gate to the house is ok, but not really solid. My intent was to deploy the old Belkin Wi-Fi extender from years past. I have seen it recently, but I could not find it. I presume it is in a box that I didn’t open. Instead of continuing the search, I ordered a BrosTrend AC1200 to extend the house WiFi into the garage, which is closer to the gate. Another AC1200 will figure prominently into the Starlink story.

The EnGenius WDS bridge between the house and workshop/barn had been in solid service for nearly 10 years when something started getting flaky on one or the other. The received signal on the workshop end dropped significantly, -100 dBm, down from -30 or so. It’s unclear whether that one had a receive problem or the other end had a transmit problem; they would look the same if you just look at signal strength. The unit on the workshop end is fairly near a light that is on all the time. During the spring and summer, there is an absolute bounty of bugs, spiders and geckos feasting on one another, so I thought perhaps some bugs had gotten inside the unit, as I’m sure anyone might suspect given the conditions.

Unfortunately for the story, the unit is well made and the electronics are very well protected from the ingress of dust, let alone critters. It was very clean inside.

Since my plans for Starlink would depend on a reliable link between the house and workshop, I started shopping. My original plan was to stick with what works, another pair of ENH500s, or since it’s been 10 years, whatever may have replaced them in EnGenius’ lineup. They do have a newer model, but they were about $200 a pair, which is not super expensive, but I kept shopping anyway. There were several units that were sold singly for less than $100 a pair but I had just a little instinctive reservation about them; hard to explain. However, I also found that Ubiquiti has units that were physically similar, identical perhaps, to the suspect units. Ubiquiti has a good reputation, so I went for a pair of AirMAX LiteBeam ACs, $140 delivered. They arrived quickly, were easy to mount and easy to configure. With default settings, they were good for 300-ish Mbps, which should be plenty, but I tweaked with settings and got quite a bit more potential data rate out of them.

I was in California for a work telephone project when I got the email: Starlink was ready to ship! [ed: ironically, I was in Hawthorne that day, where Starlink would ship from] I quickly confirmed my order, forked over the balance and also ordered the ethernet adapter that is required to plug it in. The older version had an ethernet jack, but for the new one, it’s an add-on accessory. There is much grumbling about that on r/Starlink.

Since the whole Starlink installation is a week long experience of its own, and as I am writing this, it I have *just* completed it, I’ll continue that story in a post of its own.