Azbilt
Here is the home network as it currently sits, with Ubiquiti switches and APs:
That little box labeled “WiFi Devices” accounts for 50+ devices now. I have partially converted the things that should be on the IoT VLAN over, but I just haven’t done all of them yet.
Saucer Attack!
I was responding to a post on a forum about pfSense and Ubiquiti and how they get along (very well, thank you) and it occurred to me that I left part of the story hanging.
When last we left our heroes, I had kinda iffy WiFi coverage in the house with the new gear, compared to the old gear, but I had all the desired VLANs working. The solution, or at least the obvious solution, was to further finance the purchase of more Bentleys for Robert Pera by upgrading my APs.
As you may (or may not) recall, one problem I had was that stationary devices in the house would suddenly feel the need to roam and managed to connect to the AP out in workshop. The workshop was so distant that shortly they would roam off it and the only other option was back to the AP in the house again. Vicious cycle ensues, cat and dogs, etc.
Well, the new APs arrived. I initially connected and tested by laying them about on desktops, but quickly got serious and ran wire for the main one to be deployed in the kitchen ceiling. The whole WiFi thing in the house got much better.
It is not without the occasional mystery, however. I have an apparently not terribly smart WiFi device that is the intermediate display panel for my Accurite weather station. The suite of weather sensors outside communicates via a 433MHz signal to the display panel, which in turn connects to my LAN via WiFi and kicks the data out to WeatherUnderground. This WiFi device is literally 15 feet line of sight, with no obstacles, from the kitchen AP. As I type this, the display panel is within reach of my left arm and the AP is two steps away. Consequently, this stupid display panel will connect only to the hallway AP on the other side of the house and ONLY that AP. I have tried every conceivable way to reset the display panel and the APs. The only combination that will work is if the hallway AP is online and accepting the Accurite panel. Nothing seems to make it forget the hallway AP and connect to the much closer kitchen AP.
Loving Both Of You Is Breaking All The Rules
Ok, maybe that’s a bit much.
I had a couple of really busy months adding and changing stuff in Home Assistant. A lot of it was adding something, breaking something else, fixing that, adding something new, breaking something else, ad nauseum.
However, at some point, intentionally or not, I ended up kinda leaving it alone for a while, mostly at or after Christmas. I tweaked some dashboards and adjusted a timeout here or there, but really I just didn’t do a lot with it for a while.
My wife had knee surgery. To keep the herd of buffalo that we call Dachshunds from trampling her knee and so that her completely unnatural sleep “schedule” would have the smallest impact possible, she set up shop in the spare bedroom. I put a Tasmota bulb in the lamp and moved one of the Amazon Echo devices in there so that she could control said lamp without necessarily having to get up. The Echo is also handy as an intercom when I am working elsewhere in the house and she needs something.
Trouble is, Alexa would work fine controlling the lamp (or any devices) for a couple of days, then she’d claim there was no such device. Or she would ding like she had turned it on or off, but it didn’t do anything. That would go on for a day or so, then she’d be back to controlling devices again.
I made sure my wife had the Home Assistant app on her phone and a dashboard that had all the general stuff she cares about controlling so whenever Alexa got stubborn, at least she could still control the lamp and the thermostat.
Importantly, Home Assistant always controlled the devices. Alexa, not so much. But Alexa always worked for stuff that didn’t involve Home Assistant. Clue numero uno.
I Googled every combination of ‘home assistant alexa unreliable’ and similar keywords and it seemed like all hits were things about troubleshooting an initial connection between Alexa and Home Assistant. Nothing about Alexa’s moodiness.
During the time I wasn’t doing a lot of work on Home Assistant, I didn’t have cause to connect remotely, via the Nabu Casa cloud, but a couple of days ago, I did. I immediately noticed that a bunch of stuff on my dashboard was grayed out, like the devices were offline. I had notifications and there were 9 (!) updates pending, including an OS update. An OS update was very recently done and I wondered what went so wrong that they had to update the OS again, although that kinda made senseas to why there would also be a bunch of other updates, responding to that one.
I decided that I was definitely not going to do all those updates remotely and I would just wait until I got home to start working on it.
I got home, logged in. All devices look fine, no updates pending.
WTFO?
I opened a new browser tab and logged in via Nabu Casa and I had missing devices and pending updates. I checked the logs from both logins and they both had current entries, but not the same entries. Clue Numero Dos. Then I noticed that a change I had made to my dashboard only the night before was not there on the dashboard via remote. Clue Numero Tres!
I looked for a while for some feature or button that would synchronize the two, but no luck. I cried ‘uncle’ and submitted a ticket to Nabu Casa about it.
Meanwhile, I was chatting with a friend who is also running Home Assistant, the friend who had in fact got me started down this road. He suggested that maybe I had two Docker containers running. Well, he didn’t remember that I had abandoned running Home Assistant as a Docker container originally because there had been so many weird hoops to jump through just to add HACS and secondly because running it instead as a VM meant that I can recover from egregious errors by restoring a snapshot backup. 🙂
In any case, he is not an idiot and neither am I, so I followed his advice and verified that, no, I had no rogue VMs or Docker instances running.
About that time, I got a note back from Nabu Casa. The tech also suggested that, especially from the clue with the logs having entries that were in the same time frame but different, that there must be another instance of Home Assistant running. He detailed some sometimes people spin up a VM or a Docker to try it out, then install to hardware and forget about the Docker and leave it running. He suggested checking the Network menu on both to see what the IP addresses were.
I didn’t have to do that because as soon as I saw the word ‘hardware’ in his note, it reminded me.
Early in December, before Christmas, before my wife’s knee surgery, one of the things I was looking into was moving Home Assistant off the VM and onto a fanless PC I have. The idea was to escape a perceived issue with the USB dongles for Zwave and Zigbee, having recently spent an embarassingly long time getting them working again after what should have been, at most, a mild interruption. In any case, I was able to successfully bring up Home Assistant on this hardware device and, after a fashion, restore a current backup to it. I verified that all my configurations, dashboards, etc. had all moved over. I just didn’t want to commit to moving the USB devices for fear that there would be some issue with the PC then for some unknown reason, I would not be able to bring them back up on the VM. Besides, the snapshot backups are a pretty good reason to stay on the VM for now. So, I shut down the PC and didn’t think anything of it.
Well, spurred by the tech’s phrasing, I verified that the fanless PC was up and that was indeed the instance of Home Assistant that Nabu Casa we connecting to!
I looked into the logs deeply, particularly for anything to do with rebooting and sure enough, there it was. December 13, we lost powere at the house for several hours. The NAS and Home Assistant hosted there showed the reboot on the 13th, as did the Home Assistant running on the fanless PC. The little PC did what I’m sure it was configure to do, boot upon restoral of power. Since it probably booted up faster than the NAS and VM, it was first in line to authenticate to Nabu Casa.
Both systems were probably syncing with Alexa, so whichever had sync’d last had her attention. When she couldn’t find a device by a name I know was there was probably when she was connected to the PC and the names on it hadn’t been changed. Since I never logged into that one, none of those nine updates had been applied yet. The dashboards on *that* system had not been editted.
It’s unplugged now.
And Alexa is going on four days of continuous cooperation.
Update: 9 days and Alexa has been reliable. For this, anyway.
Gate Battery Revisited
Hard work often pays off after time, but laziness always pays off now.
I received the replacement Renogy solar charge controller. I actually ordered three of them, with visions of them doing other stuff. However, with the current camera connection to the battery working just fine, I am not in a huge hurry to change it and that laziness has paid off because the gate camera has been just fine.
Oh, I will change it out, just not while it’s still cold outside. We will have a warm day soon, I’m sure. It is winter in Texas, afterall.
I think I will put the “broken” controller on the Kawasaki Mule, to keep it’s battery charged even if we don’t use it all the time.
Flat Fields Matter
For a couple of months, High Point Scientific had my new tripod on backorder. It is apparently quite popular, being solid yet inexpensive. I was pretty excited to get the notice that it was shipping.
As it is winter and Orion is quite prominent in the night sky, I thought it would be nice to capture the Orion nebula. Because of it’s location, basically formed around Orion’s dagger, it should be easy to find, at least compared to many, maybe most, deep sky objects.
For my first try, I had my CLS filter in place. We have a big sodium light that is basically in the same direction as Orion when I am set up in what is arguably a very handy place, in my driveway, just outside the garage. This filter, however, is pretty dark and it made it more difficult to find anything. At some point, I decided that maybe I was pointed the right direction and that I just couldn’t see the nebulosity in my test shots, so I set the thing loose taking 180 x 30 second subs. I spent some of the capture time in the house doing things that needed doing and some of it waiting in the car with the heater on, which was kinda novel.
I had started a little later than planned, plus all that attempting to find a nebula push my capture kind late. The exact place I had set up was inadvertently planned for my capture plan. This was where the camera was pointed at the end of 180 frames.
I captured a really pretty field of stars and I had only missed the nebula by this much:
As luck would have it, a couple nights later was clear and a Friday, so I set up again. This time I removed the CLS filter, hoping it would make it easier to find the nebula. I am not sure whether or not it made a real difference, but I did find it!
It was very exciting not only to see the nebula show up on the viewscreen, but also to be able to frame it so perfectly.
I captured another set of 180 x 30 second subs, about 30 darks, flats and bias. I set up a little bit out in the yard to keep from catching the house if capture went long. I also set up a heater and for the most part, sat with the equipment for most of the capture time.
I also ran a small test of two other bits of equipment, a dew heater for the astrograph and my Bluetti power station. While the power station was not purchased specifically for astrophotography (power loss during winter was the big thing), using it for possible dark site travel was a consideration.
This was the first time I had even powered up the dew heater. According to the Bluetti display, on high, it draws 6 watts. I could hardly even tell it was warm against the aluminum dew sheild of the Redcat. I suppose that all it has to do is keep it just warm enough to discourage condensation. Shrug. Whether or not it was succesful would come up soon enough.
We had plans for early Saturday afternoon, so I decided to stay up and do at least a preliminary stack of the capture. I scrolled fairly quickly through the subs and discovered a couple of where I presume I had bumped the tripod and excluded those subs. The final stack came out… ummm… odd.
This is after a bit of stretching in GIMP. There are two anomalies about this image. The most obvious to me is that the bottom 1/3 or so seems blurred, out of focus. I had run through checking the subs pretty quickly, but certainly none were way out of focus. It also strikes me as odd to only be out of focus on the bottom of the image. The top and middle seem to in sharp focus.
The other thing, and this was harder to notice because of the blurring, but there is a definite linear gradient from top to bottom.
It was too late and I was too tired to do much about it just then, so I hit it again the next morning. One of my more careful trips through the subs, I noticed that several towards the end of the capture seemed to have soft focus, so I excluded them and it was essentially unchanged. I reviewed my flats and noticed that they had a linear gradient to them and thought, oh, that makes sense, so I restacked again without flats: no change. It occurred to me later that I may have unchecked all the flat captures, but maybe not the flat master that the previous stack process created.
I posted a png of the stretched blurry image on the Nebula Photos Patreon community page, with some details about the capture. Nico took an interest and a few private emails later, I had much more carefully tried stacking without the flats. I cleaned all of the .info files out of the lights folder, moved the exclude lights to another folder, as well as moving the master flat to another folder. When I stacked this time, it was 131 lights, zero flats and the presumably good dark master and bias master. The image came out great and with a couple of stretches and a crop:
My favorite astro image thus far!
To further verify that the flats were the issue, I kept all the rest of the conditons the same and added back the flats and I got this different image. It may seem to be ok, but upon closer examination, it is still very wrong.
It is hard to tell at full size, but the bottom half of the picture, getting worse as it goes lower, the stars split into three divergent images of red, green and blue.
I will be the first to admit that I do not understand the inner workings of Deep Space Stacker and how it uses the calibration files, but it now seems obvious that if there is an issue with those files, it can damage your final image in probably unpredictable ways.
Some of the discussion with Nico was about my flat capture process and I am going to rework how I am doing that. For this session, flats were captured by holding a USB tracing pad up to the end of the dew shield and adjusting exposure until it was just a little underexposed, which turned out to be 1/2500″. This is probably way too fast and catching a pattern of sensor noise as well as PWM flicker and shutter artifacts from the brightness control of the panel itself.
There are several ways to address this and I will report on what works well for me.
The Gate Keeper
I have mentioned that I have a camera mounted by our front gate, powered by a dedicated solar panel, charger and battery.
I chose this Renogy Wanderer 10A solar charge controller specifically because it has USB power outlets built in, elegantly feeding the USB power cord to the camera.
The battery box lid, designed for indoor use apparently, had air vents on the top and allowed rain inside. The first controller died a wet death. I covered those air vents. It’s not as if the rest of the lid fits with a hermetic seal, so the box is still well ventilated, but now it is at least rain tight.
Wintertime is tough on solar stuff. So many overcast days result in less efficient charging. The camera draws 100-200mA;. The granularity of the charge controller’s output meter is 0.1A and it’s usually says 0.1A, but sometimes it’s 0.2A. Maybe when it is darker and the IR illuminator is on?
A couple of overcast days is enough for the lawn tractor battery to fall behind and drop power, especially after dark. The tractor battery is not intended for deep cycle use, although the gate controller is only on it’s 2nd battery in about 10 years. I think the constant drain of the camera, even at only 200mA, is probably the issue.
I have considered changing the battery to an actual deep cycle battery. WalMart carries an inexpensive ($80-ish) marine battery in the 24 size class. Real specifications for that specific battery were elusive, but I found several 24 size marine batteries from other sources. If they mentioned it at all, the amp-hour rating was 75-80 AH, so that seems likely for the EverStart at WalMart.
A more useful number for power service is the watt hour, literally how many watts for how long. It’s really handiest when comparing power systems that might not be running the same voltage. Conversion of AH to WH is easy enough. Voltage times amperage is wattage. For example, an off-grid solar system running 12V with 100AH of battery can be compared directly to a 48V system with 25AH of battery because they would both provide 1200 watt hours of power.
12V x 100AH == 48V x 25AH
In the case of the EverStart, 80AH x 12V = 960WH. A 12 volt battery is rarely actually 12 volts. Lead-acid batteries like to charge at 2.25 or so volts per cell, so a 12V battery’s ideal charging voltage is 13.8V. The open circuit voltage for a fully charged battery is about 2.1 volts per cell, so 12.6 is typical. Plug that number into the formula and a 12.6 volt 80AH battery should be good for 1008 watt hours.
Once you have a watt hour rating, figuring out how long a charged battery should last is pretty straight forward. In the above example, under ideal conditions, you should be able to draw one watt for 1008 hours, 2 watts for 504 hours, 50 watts for 20+ hours, and so on.
In practice, lead acid batteries can be permanently damaged if discharged too deeply. Deep cycle batteries are optimized to limit this damage, but you can’t eliminate it. It’s just how the chemsitry in there works. It is good practice to limit discharging a lead acid battery to no more than 50% charge, so derating that watt hour rating by half would be wise. One watt for 504 hours, 2 watts for 252 hours, etc.
Assuming 200mA for the camera and the specified 10mA for the charge controller itself, an 80AH battery should be able to run the camera for 192 hours to the 50% charge level, or about 8 days. That shoudl be plenty.
Then again, lead acid batteries need maintenance, especially during the summer months. All the cool kids are using lithium iron phosphate batteries in their solar systems, mostly because they enjoy certain efficiencies, but also because they are essentially zero maintenance, so I started looking into that.
LiFePO4 batteries can be charged and discharged faster than lead acid, are much lighter and because of smart onboard electronics, don’t really need a super smart charger. They are not particular cheap, however.
I found that 20AH LiFePO4 batteries run similar in price to 80AH lead acid batteries. 20AH works out to 96 hours and these batteries protect themselves at about 75% discharge, so 72 hours is likely.
So, roughly the same price as the big lead acid batter and only 3 days without a charge, but zero maintenance and an arguably longer service life. I decided it was worth trying one out at the very least.
Via Amazon, I chose at Vestwoods 20AH battery. The Renogy charge controller has a lithium mode. Part of setting that mode involves setting the charge voltage. The labelling on the battery indicates that it is a 12.8 volt battery, so I took that as the best guess.
It was delivered at 7:24 on December 30, ironically about a foot from where it would be installed. Here’s Amazon’s delivery photo. The battery box nearest to it is where it goes and the camera above it is what it powers.
It’s about the size of a large motorcycle battery.
I connected it almost immediately, by flashlight.
According to the recordings from that camera, it ran from 9:05PM on December 30 until 12:37 AM New Years Day.
The immediate presumption was that changing the battery type caused an issue. I didn’t have a lot of free time to troubleshoot, so it took me a few days to determine that what had actually happened was that the power output from the charge controller had failed. All indications are that it is on, but there is no power on either the USB jacks or the screw terminals.
Before I found the Renogy charge controller, I had purchased a 12V to USB adapter, assuming I would be using something like that connected directly to a battery. Using that, I was able to at least get the camera operational again, pending the arrival of a replacement charge controller.
Tendrils of the Power Event
After I finally beat the dongles into submission, it took a little while to discover that many of my devices did not seem to be operating, but in reality, they were just not responding to verbal commands via Alexa.
Complicating things is that I was also dealing with some WiFi trauma while settling in the new Ubiquiti gear. I was suffering a lot of disconnections and I could not be 100% sure that the WiFi devices that I was trying to control may not have just been switching on and off like a crazy monkey. Short version on that seems to have been overzealous roaming defaults with only two fairly distant APs, but that is a story for that blog.
Once the WiFi was settled down, if not solved, it was obvious that Alexa could definitely see all the devices. I could change a device name and it would be detected, but I still could not control any devices with verbal commands. There were a couple of responses but by far, the most common was that there was no <device> in my profile. Sometimes, I would get that the device was not responding. What I would never get is a device that was controlled.
The timing for this failure is abysmal. My wife just had knee surgery and in preparation for that, I made sure the lamp in the bedroom where she would be staying (she must avoid the dogs for a while) had a smart bulb in it that she could control verbally, except that verbal control wasn’t happening.
After trying no shortage of useless stuff and random restarts of various services, I was getting to be on the right track and had tried removing the skill from Alexa. In Googling about that, I stumbled across a forum posting where someone suggested removing one file and restarting home assistant. I chose instead to rename that one file, just in case. Well, praise Jack Bauer because it worked. All the Alexa stuff works again!
Well, I had to re-re-rename some things back to normal, basically clean up the messes I made troubleshooting. I’ve had to do that a lot lately.
The file in question is /config/.storage/cloud. My old one was 3 times the size of the new one that the system created automatically to replace the missing one. It had several blocks that seemed to be associated with Google devices (of which I have zero, so far as I know) but generally, it was just a bigger file with more stuff in it. I’m sure there was redundancy or perhaps one scrambled line in there.
Now that I can tell Alexa to turn on the Christmas tree and she does, of freakin’ COURSE the rotating base for the tree would stop passing power through to the lights. It is always something.
All That Glitters
My new WiFi is argubly worse than my old WiFi, but I think there is a reason.
I recently completed a pretty major [up|cross]grade of my home LAN. There were two main goals. First is to add some additional wireless VLANs so that I can help isolate my growing network of IoT devices and thus protect them and the rest of my devices from them. Second, and this is admittedly not a super important thing, it would also be nice if I didn’t have to connect to a different SSID between the house and workshop.
As detailed elsewhere, that was a journey of discovery, ending with the eventual replacement of two TP-Link APs and two Cisco switches with new Ubiquiti devices. Between the interopterability and easy of configuration, it was a good, though not particularly cheap, move.
Or so I thought.
A week or so later, it appears that I have a lot of connectivity issues. Once I really noticed it and started looking into it, I think I know what’s happening and at least a good bit of the solution, the full spectrum of which of course involves helping someone at Ubiquiti buy another Bentley.
I put the new APs exactly where the old ones were, but the old ones were perhaps a little more sophisticated, RF-wise. The TP-Link APs had an array of four large antennas and, sadly, physics matters. The UniFi APs look like oversized light switches and simply cannot have the same antenna performance. However, the one in the house seemed to work just fine until I had a compelling reason to move it. I wont go into those details, but it had to move from about the center of the house to one side of the house. As I am typing this post, the UniFi controller reports my WiFi experience as “Good”, rather than excellent, along with everything else here on this, the opposite side of the house from the AP.
The real issue comes about when, for reasons I have yet to determine, the experience gets enough worse that the connection is lost and the device, most noticeable to me when it is my laptop, sees the other AP, all the way out in the workshop, but with a marginally ok signal, at least compared to the one it just dropped and it roams to that AP. Now that weak signal results in a substantial slowdown and maybe it drops that connection, too. Then it sees the original AP and connects to it again. This vicious cycle continues.
Once I figured out that this seems to be what is happening, I set the workshop AP to lower power. This makes is a less viable alternative to reconnect to and at least stops the toggling. It does not, however, address the actual problem, which is that the new Ubiquiti APs I chose simply don’t have the coverage that the TP-link devices have.
So, I ordered a couple more APs, ceiling mounts this time. I think I can justify a unit in the hallway between the bedrooms and the livingroom and another in the kitchen. If those two provide dense enough coverage in the house, I can deploy the other wall mount in the workshop, to have two out there as well. If needed, however, I can find a place for a third in the house. One has seemed to be enough for the workshop thus far. Everything WiFi out there is within about 20 feet of the AP.
Update: The power setting on the workshop AP has definitely reduced the number of roaming events in the logs. [facepalm] Now they just disconnect instead. [/facepalm]
Bungle in the Dongle
With apologies to Ian Anderson.
Full disclosure. I am still not entirely sure exactly which of the myriad steps finally worked or what order of steps I finally accidentally hit, but the Zigbee and Zwave dongles just came up working again. Furthermore, it’s actually been a few days since that happened, so my memory of it is imperfect as well.
The all critical final step, however, is emblazened in my retinas forever. After doing this SEVERAL times already, removing and adding back the USB devices at the virtual machine level in the Synology NAS, with at least some number of minutes/hours/decades delay seems to let something reset enough that the device is not only detected (which it has been all along) but also works.
Of course, the damage I had done troubleshooting needed to be undone. Some devices had lost their names and almost worked. They had reverted to the default names given when the devices are first interviewed, generally some variation of their brand and model number.
Most of these were pretty easy to figure out, either by being relatively unique (at that point, I had deployed only one Third Reality motion sensor) or by what automation was still associated with the device. Curiously, those automations didn’t work until the name of the device was adjusted. For the afore mentioned motion detector, I had set up a fairly sophisticated timer automation as suggested by Smart Home Junkie and parts of the automation needed to adjusted with new device names as well.
I have all but one or two of those names resolved now, mostly because I just haven’t walked to the doors or plugs in question and verified which was which. Soon.
This power event did inspire me to look into moving Home Assistant over to dedicated hardware. Whether that is really justified depends on whether the advantages of dedicated hardware outweigh the advantages of hosting it as a VM and that really comes down to how it handles USB hardware under unusual conditions like an unplanned power loss.
Raspberry Pi hardware is still very hard to obtain, but I have an Celeron N4100 based fanless PC that is more than adequate to the task. I had a false start (not all the installation methods install identical versions), but did manage to get a suitable Home Assistant image on it and was able to restore a backup of my current configuration. For personal reasons beyond merely being the week of Christmas, I don’t want to tear into the actual move of all these devices to that hardware just this moment. After the holidays, I will revisit it.