The Heating Project

Having gone through three Wellington winters in our home, Lisa and I really didn’t want to make it a fourth this year. Whilst our property is fortunate enough to get a lot of sun during the day, it still can’t sustain a comfortable temperature in the middle of winter when we have weeks with day highs of 10c and lows of 5c. In the worst case, it’s gotten right down to 2c outside.

We insulated as much as possible shortly after we moved in, both the roof and floor, but unfortunately with house being very low to the ground, there were sections of the floor that simply couldn’t be insulated, so we ended up with approximately 60% coverage. We aren’t sure about the walls either, some of the ones we’ve had to cut open to do work have revealed insulation but we don’t know for sure if it is fully present in all walls of the house or not.

Insulation upgrade in progress

Most New Zealand homes are also only single glazed and ours is no exception, which means even with the roof (100%), floor (~ 60%) and walls (??%) insulated, we still have massive bay windows leaking heat like crazy. So regardless of the natural sun we get, a high capacity heat source is still critical to maintain a proper temperature at the property.

Which is also precisely what we lacked – when we moved in the house had a rusted wood fireplace in dubious condition and an unflued gas heater which are generally just a bad idea all-round. Neither of these were safe options, so we needed to replace them with something better.

We considered a few different options:

  1. A new wood fire would offer a nice ambience, but access to the property is a challenge and we’d go through most of the firewood we’ve ended up collecting from dropping overgrown trees. So long term, we’d be buying expensive firewood and having to shift it onto the property – plus we’d lose a good amount of space storing a winter’s worth of firewood.
  2. A gas fireplace was considered due to us already having existing piped gas. A modern flued gas fireplace is efficient and doesn’t exhaust into the living space, plus offers all the classic ambience of a fireplace. The only downside is that the heat would be focused in our lounge or dining rooms and wouldn’t propagate nicely through the rest of the house. There are some models that feature ducting that can exhaust some of the heat to other parts of the house (eg 2 or 3 other rooms) but our size of property wouldn’t be able to get full coverage meaning we’d need to looking at needing to install two fireplaces or additional heating to fully cover the property.
  3. We seriously considered a gas boiler and European style radiators emitting radiant heat in every room. This would result in us losing some wall space, although we would have been able to put them under the windows. Ultimately this option ended up not being possible, as our existing gas pipe is too small in diameter and the cost of running a new pipe from the street to the house is extremely uneconomical.

The gas pipe limitation also impacted our ability to drive other high consumption gas appliances, for example feeding large fireplaces could be tricky and we’d probably be unable to run more than a single instantaneous hot water system. Probably for the better anyway given that it’s a fossil fuel, but it’s so much more cost effective than electricity on a per kWh basis in NZ that it’s a compelling option if you have gas piped to your property.

Given the above issues, the only remaining option that made any sense was electric heat pumps. Heat pumps are generally 300-400% efficient, meaning although they use super expensive electricity, you get very good heat output compared to running traditional electric heaters.

What most NZders will think of when you say “heat pump”

These are becoming quite popular now in NZ as an affordable upgrade (a fully installed basic unit might cost as little as $2-3k NZD) for existing properties – the purchase of a mini-split system such as the above allows for easy retrofit installation and you’ll often see these added to lounges to ensure a reliable heat source in the main living space.

Their weakness is that you don’t get the heat spreading from the primary room out into other parts of the house, such as bedrooms. And they’re not cheap enough that you would install one in every room to get the coverage, but even if you did, you’d then have to deal with managing them all in a sensible fashion so they don’t end up fighting each other when maintaining temperatures.

 

To solve this problem, we went for a less common (in NZ residential homes) option of putting in a central ducted heat pump. These units use the same essential technology (using a refrigerant to shift heat from the outside unit through pipes to an inside unit), but rather than having an on-wall unit, we have a large unit in the attic, with ducting pushing air into every room of the house, to be recirculated back via a hallway intake.

Due to the amount of specialist skills and tools required for this project, plus a lot of time climbing around the attic which I just love doing, we instead contracted Wasabi Air to perform the complete installation for us. They did a really good job, so pretty happy with that.

Diagram from Mitsubishi’s website illustrating how a ducted heat pump setup looks.

We went with a Mitsubishi PEAD 140 unit based on Wasabi’s recommendation as an appropriately sized unit. This unit is capable of 16 kW heat output or 13 kW cooling output and can consume up to 4 kW of electricity to operate at peak.

The installation is pretty full on, it took 3 days for a team to get everything installed and set up. I took a selection of photos during the install since I’m a weirdo who finds installing things like this interesting.

Firstly the exterior unit needed installation. Whilst modern heat pumps are “quiet” they’re still audible when working hard so you want them away from bedrooms – thankfully we had a convenient space between two bay windows where it could sit below the floor level of the house. The only slight downside of this location, is that I’ve noticed the deck has the tendency to slightly amplify the vibration of the unit at times, whereas if we had gotten it mounted on a concrete pad elsewhere it may have been less of an issue.

Exterior unit. This thing is *big* and shovels a significant amount of air through itself when operating.

Vacuum pumping the refrigerant lines ready for the gas to go in.

Because the exterior unit needs to run copper piping for the refrigerant, power and controls up into the roof of the house, we needed to go from under the floor to the attic at some point. Rather than an ugly boxed conduit on the outside of the house, we had it installed through the middle of the house.

As mentioned earlier, our fireplace was wrecked, so I knocked it out entirely a few months back (post coming once I finally finish the project) which has left a messy hole an accessible space from the floor to the attic that was perfect for linking the internal and external units.

Whilst the piping is copper, it has a surprising degree of flexibility so the team was able to feed it through and bend it varying degrees, whereas I had assumed it would all have to be right angles and welds. The piping is wrapped in a insulation tube which does a pretty good job at avoiding heat loss – even when running on full, the insulation is just mildly warm to the touch.

Piping and cabling going in.

One of those big electrical cables had to go all the way back to the switch board to feed the unit with electricity. Unfortunately our switchboard is totally full, so the electrician ended up having to add a couple additional breakers off it to the side.

It’s not wonderful but it solved the problem – given future plans for the property (electrical hot water, induction cooking, car charging), I think we’ll end up doubling our switchboard and making it all tidy as a future project.

Not going to win any prettiness awards, but it’s safe and compliant.

Of course the exterior unit is just half the fun – we also needed the interior unit installed and all the ducting connected. We had what is considered a slim unit, but it’s still a very sizeable item to try and lift up into a roof space!

Indoor unit before going into the roof and having ducting connected.

Impromptu skylight

As there’s no way to get the unit into the roof via existing access ways, the team cut a massive hole in the hallway roof. They then swung the unit up into the roof with a team of guys who do a lot more physical activity than me and backfilled the space with the intake vents.

Intake vents going into the hallway to nicely fill the big hole that got made earlier.

Once fitted in the roof, it takes up even more space for all the ducting to be connected. A large amount of our spacious attic is now consumed with this unit and associated ducting. I’m just glad we’d already done insulation and most of the cabling work we’ll want to do in there for the foreseeable future as it’s a lot more difficult to navigate now.

Fully installed in the attic space

Despite being above the bedroom, the unit is really quiet when operating and we can barely hear it. I took some video of the installation to give an idea of it’s low level of background noise even when inside the attic.

We decided to install outlets into every single bedroom (x4), one in the lounge and one in the dining room for a total of x6 outlets. The hallway features the intake vents for recirculating the air.

Note that there are two key downsides to the ducted heat pump option vs radiators here:

  1. The first is that you can’t directly heat damp areas like the bathrooms or kitchen, as pumping air into those areas will force moisture out into other parts of the house. Radiators have the upper hand here, since you can get appliances such as towel rail radiators to directly provide warmth. That being said, our bathroom gets enough residual heat from being in the middle of the house and when we renovate the second one I plan to look at putting in underfloor heating as it’s on a concrete pad, so it’s not a big issue for us.
  2. The second is that you don’t get per-room control. Whilst it’s technically possible to install dampeners (manual or electronic), when you close off one vent you can unbalance the system a bit and make other rooms too breezy or noisy. So it’s recommended to just heat the whole place and enjoy a fully warm house, rather than trying to create hot + cold rooms.

Ducting getting prepped pre-installation

The above photo showing the ducting being prepared shows the built-in dampeners used to balance the system when it’s installed. These are used to restrict/open the flow of air to each outlet to handle the fact that outlets have different length ducting runs and sizes. It also makes it possible to do things such as setting the heat output in the bedrooms to be a bit lower than that of the living spaces by keeping their volume of air output a little lower.

Once installed, the outlets are unobtrusive in the corner of each room.

I had some concerns around how the vents would handle our high ceilings and we do get some degree of heat blanketing when the system first starts heating the house (warm head, cold toes), but once it brings up the whole house to temperature the high ceilings aren’t a major issue.

That being said, it does mean there’s a lot more air to heat than a property with lower ceiling heights and certainly on the coldest days we’ve found the system takes too long to bring the house house up to a comfortable temperature if we’ve let the house get very cold. The solution is pretty simple, we just leave the house on a lower temperature when away from home to ensure it’s only a few degrees off our comfortable temperature when we come home (typically found 20c when awake and 18c when sleeping is our preferred climate).

It works well – for the first time we’ve actually had a comfortable time in winter, rather than huddling for warmth in only a couple rooms with the rest of the house closed off trying to conserve heat.

 

Whilst the heat pump ensures the house can now always be at a comfortable temperature, it won’t prevent moisture build up. It can reduce window condensation on account of the warmth and even be put into a drying mode, but it’s ultimately not a ventilation system and doesn’t cycle out moist stale air if you fail to manually ventilate your house.

Our house wasn’t too bad with moisture (not compared to many other NZ properties) on account of the kitchen and bathrooms all featuring extractor fans which ensured the bulk of moisture produced in the house is vented out, but the simple act of living in the house is enough to create moisture and having weeping windows in the morning was still not uncommon for us, especially during winter when opening windows isn’t very compelling.

Given the fact that we have long and cold winters, we decided to invest a bit more and had Mitsubishi’s ventilation product (Lossnay) installed as part of this project. This unit integrates with the ducted heat pump and regularly cycles out stale air and sucks in fresh air, recovering most of the energy in the process so you’re not constantly discarding the energy you just spent heating (or cooling) the house.

Diagram from Mitsubishi’s website illustrating how a Lossnay connects to the ducted heat pump system.

Lossnay unit operating away quietly in the maze of ducting.

Because of the layout of the house, the two Lossnay vents ended up having to go through the roof – the egress near the existing bathroom fan vent and the intake a number of meters away on the other side to prevent just recirculating the same air, so we now have a few more mushrooms to add to the other unsightly appearance of our roof. Summer 2018-2019 has a painting project on my list to tidy it all up with a fresh coat of paint.

Lossnay vent (R) alongside existing bathroom vent (L)

The end result is that we have a completely moisture free property and I haven’t opened the windows once all June, since the system is constantly ensuring we have fresh, dry and warm air in the property. Even on the coldest days, the windows are now completely dry which is just fantastic.

 

To control the combined system, there’s a central thermostat and control interface installed underneath the return intake. It’s pretty basic, capable of setting modes, fan speeds, temperatures and timers, but does the fundamentals nicely enough.

Nice modern look but it’s not all that intelligent.

Ultimately I wanted proper IP control of the system to allow a range of integrations rather than the manual drudgery of having to touch actual buttons with my actual hands.

Surprisingly for 2018 this isn’t standard for all heat pumps like this, some vendors seem to lack options entirely and whilst Mitsubishi have their “WiFi Control” product, it’s still sold as an extra ~$250 add-on, rather than being natively built into the control interface as standard feature.

Essentially it works by connecting a small embedded computer to the heat pump via it’s onboard serial port  (You can see the little white box on one of the earlier photos of the attic install with the trailing cable going to the unit’s control circuit). It then bidirectionally syncs with Mitsubishi’s cloud services, which then allows you to control it from anywhere via your phone app. The experience is well polished and Mitsubishi is even doing HTTPS for their connections which was a (positive) surprise. Plus the WiFi module was designed/built by a local New Zealand company on behalf of Mitsubishi, so that’s kinda cool.

WiFi module + app

Note that I mentioned serial before – this isn’t too tricky to interface with and it is possible to build your own module – I’m not great with hardware and didn’t want the hassle and warranty risks, but if that’s your kind of thing, take a look at this blog post on Nicegear.

Overall it’s a pretty decent Internet of Things product, the only issue I had with it is that it relies entirely on their cloud service. And whilst Mitsubishi’s cloud app is *probably* going to stick around for a while, I wanted local control to allow me to link it to Home Assistant to make it part of a holistic smart house, not just an independent appliance. Getting it into Home Assistant also makes it possible to then control via Apple HomeKit, something not possible with the off-the-shelf product.

I had a bit of fun figuring out how I could interface with it locally (see my talk at the inargural Wellington Home Automation Hackers if interested) but ultimately I discovered that the latest version of the IP module (MAC-568IF-E) supported a Japanese standard called ECHONET Lite. This standard allows for completely local network control of various smart home appliances, including heat pumps, which allowed me to then write a bridge application between Home Assistant (using MQTT) and the heat pump (using ECHONET Lite).

The source code for this bridge application is available at https://github.com/jethrocarr/echonetlite-hvac-mqtt-service including a Docker image for easy installation.

The only limitation is that I have not found a way to control the Lossnay via ECHONET Lite – given that the official Mitsubishi app also lacks control, I suspect they simply haven’t developed functionality into the WiFI control module to expose the options for the Lossnay, it’s probably not a massively popular addon

End result is that I now have the ability to control the mode and temperature via HomeKit from my iPhone/iPad – and since HomeKit links with the wider Apple Ecosystem and the fact we have an Apple TV, it’s possible for both Lisa and myself to control the heat pump (and all the other linked stuff) remotely from anywhere without a VPN needed. It also makes things super trivial to delegate and revoke access to guests when visiting.

Heating controls in HomeKit!

The other main drive to integrate locally is that I want to be able to setup smarter rules inside Home Assistant. Some of the stuff I’m planning is:

  1. Automatically set the temperature when we are away from home vs at home. We already have occupancy data from a range of metrics that we can use so this won’t be too tricky.
  2. Take exterior temperature into account, eg if the day is going to heat up there could be hours where we can shutdown the system entirely for power savings.
  3. Being able to read temperatures from sensors around the house rather than the averaged thermostat reading near the return intake and take into consideration when setting temperatures – eg “I always want the bedrooms at 18c, even if it means the hallway and lounge end up slightly warmer than they should otherwise”.
  4. Automatically shut down the system if someone leaves the front/back door open for more than a minute.
  5. A smarter night setback mode. The current behaviour of the system is to always run the fans whether actively heating or not, but this can sometimes be annoying at night when the breeze can be felt more noticeably. I’d like to program in logic to automatically shutdown the aircon when the temp is above 18, but if it drops below 18, heat back up to 20 then switch off again – thus only running the fans when actively heating but still ensuring the house stays warm. So no breezes other than when heating. The system has this with “night setback” but it just turns on if it gets below a threshold, it doesn’t then turn off again once it’s at a comfortable level.

 

So aside from the remaining automation work, project complete! We are super happy with this solution, it’s completely changed the liveability of the property during winter and it’s an upgrade I would absolutely make to any other property that I ever buy in future or recommend to anyone else living in our climate here in NZ.

It’s not a cheap solution – expect to spend $15-20k NZD for a 3-4 bedroom home, although it will vary considerably based on sizing and installer effort needed. Like most home upgrades, labour is a huge component of the ultimate bill. Our final bill came in around $22k, but this also included the Lossnay ventilation unit.

There’s also the electrical cost to operate. This one is super subjective since it depends on your usage patterns, house, etc. Our first winter month bill at ~$450 NZD seemed super high, but when we compared with previous years, it was only 30% more and instead of having two rooms barely liveable, we got an entire house that was comfortable. In fact it’s efficient enough that we’re likely to still come in just on or under the 8,000 kWh yearly low user consumption, given that our hot water is currently gas. Once we go full electric with our hot water, we will probably surpass that and need to go to the standard plan (cheaper per kWh rate, higher line charges).

I’m expecting to get 15 years lifespan on the system, maybe longer. At 15 years, it’s essentially $1,460 per year deprecation for the asset, which isn’t too bad for the level of comfort it provides. If we get longer, or can refurbish to extend beyond that period, even better. It’s also worth noting that even if the mechanical components need replacing at their end of life, we have all the ducting, piping, electrical etc already installed which would be reusable, making a replacement project at it’s EOL considerably cheaper than a new installation given how much of the cost is the labour doing ducting runs, cutting holes, etc.

Whilst there are many other reputable brands out there, we went with the Mitsubishi offering for two main reasons.

  1. It had an IP module (“WiFi control”) available. If you’re buying a unit, make sure to research this closely, not every vendor has one and you don’t want to get burnt with a system that’s extremely difficult to get integrated into your wider smart home.
  2. The integration with the ventilation product ensured that we could get a single vendor solution for the complete system, rather than trying to mix one vendor’s ventilation system with another vendor’s heat pump system and having to figure out which one was at fault if they didn’t cooperate nicely. This may depend on who your installer is as well, ours only did the one ventilation product, so kinda made the choice easy.

Having seen the effort and complexity that went into this installation, I’m glad I got the pros to do it. It would have taken ages to do by myself over weekends, etc and there’s a number of specialist steps (eg filling the refrigerant lines with gas) which are non-trivial plus just the experience of knowing how to size and setup the ducting runs appropriately for the property. So I’d recommend staying away from a DIY project on this one, despite the temptation given the costs involved. A good friend went through a project to do most of the work on his own install himself and it ultimately didn’t save as much as expected and created a number of headaches along the way.

Posted in Uncategorized | Tagged , , , , , , , , , | Leave a comment

Wellington Home Automation Hackers

I’m slowly setting up more and more Internet-of-Things stuff in my house and because of my interest in this space I’ve started a Wellington Home Automation Hackers meetup that will meet monthly! If you’re in Wellington and are interested in this sort of thing, please do join the group and come along!

I’m aiming to have 2 different presenters lined up every month to ensure a range of topics, including a mix of presentations suitable for less technical beginners as well as more technical gurus (and everything in-between).

I kicked off the first month by talking about my adventures so far with Home Assistant, using it to integrate with my air conditioning, house alarm and how I’m using it in conjunction with Apple Homekit. If you missed it but this sounds fun, check it out on YouTube below!

Posted in Uncategorized | Tagged , , , , | 2 Comments

Firewall rules for HomeKit with HomeAssistant

I’ve recently been playing with the popular open source home automation software Home Assistant. One of the nice features of this platform is that it can export most of the devices it manages as HomeKit devices for easy use from iOS devices.

HomeKit isn’t perfect, it’s a generic management platform so it’ll never be as good at doing thing X compared to a native app from vendor X – it just can’t have all the same parameters and configurability.

Despite this, there’s some compelling features for a household that’s fully on the Apple ecosystem:

  1. It puts all the assorted IoT “stuff” that we have into a single interface. This interface is available on my iPhone, iPad and Watch.
  2. It makes it easy to share to others who probably aren’t so technical they’re running a VPN to your house thanks to the built in tunnelling via Apple TV or Apple HomePod.
  3. The protocol has been opened up by Apple, so that you can now write and use uncertified devices using libraries such as HAP-Python or HAP-NodeJs. This is how it’s now possible for Home Assistant to expose various devices connected via other means to the HomeKit network.

The only thing that’s a bit annoying, is that if you get your firewall rulesets wrong it can be tricky to debug.

I had opened up TCP port 51827 (used by HomeKit) and was able to pair my device successfully, but then had weird issues where the accessories would go into “No Response” state for prolonged periods and only occasionally update with the latest information.

Steve says you’re holding it wrong

The trick to finding this was to do some packet dumping. I ran a packet dump for all traffic from my phone to the server running the Home Assistant app to see what was coming across the wire and could see a pile of mDNS requests that weren’t being answered.

Wireshark never lies

mDNS is a tricky protocol – essentially it’s DNS, but instead of going to a name server for resolution, devices using mDNS send out a multicast packet to the network and wait to see who replies with the answer. Devices implementing mDNS need to listen to these packets and respond where appropriate. It’s most commonly implemented as Bonjour (Apple) and Avahi (Linux).

This means that we need to setup a firewall rule for UDP port 5353 to allow HomeKit clients to find the HomeKit accessory (in this case, Home Assistant). Without it, you get the “No Response” problem when lookups fail.

Why did it work at all without it? Not 100% sure, but I think HAP-Python might occasionally send out it’s own multicast messages advertising itself to iOS devices which allows them to find it for a period of time, but when the TTL expires and they try to re-resolve for connected accessories it’s nowhere to be found.

So the complete set of iptables rules you probably want (or something like them) is:

# mDNS
iptables -A INPUT -p udp -m multiport --dports 5353 -j ACCEPT

# Homekit Protocol
iptables -A INPUT -p tcp -m multiport --dports 51827 -j ACCEPT

# Home Assistant interface
iptables -A INPUT -p tcp -m multiport --dports 8123 -j ACCEPT
Posted in Uncategorized | Tagged , , , | 1 Comment

The bathroom fan debacle

I completed this project a while back and had the images saved up for a blog post – somehow almost a year has gone by since then in a blink of an eye. But anyway, enjoy this delayed update about the exciting world of bathroom fans!

If our house had one deficiency, it would be that the layout of the property has resulted in a rather small and interior only bathroom. Since this bathroom has no direct access to any outside walls for easy ventilation via windows, it instead had a pretty chunky fan already installed when we purchased the property to extract all the shower moisture and other undesirable elements.

By my estimation, the fan would be a good 20 years old and whilst it was doing a good job of extracting air from the bathroom it had two major issues:

  1. Whilst it extracted air successfully from the bathroom, it didn’t send it anywhere useful – instead it was dumping all the moisture directly into the attic space making the attic (and by extension, the whole house) damp.
  2. The bathroom roof features a large skylight which is great for making the room feel light and more spacious than it actually is, but it also acts as a giant shower dome, with steam going up into the skylight space and condensing as the fan is not at the highest point of the roof.

The second issue above really started causing significant issues, particularly since the moisture was damaging the paint and plasterboard and with a small bathroom that makes it almost impossible to extend a ladder and reach the super high ceilings, mould was becoming an issue due to inability to access to tackle the moisture.

 

The situation required remedial work and one day the old fan decided to make the decision easy with the fan cover getting brittle enough that it suddenly fell out of the roof into the bath one evening without warning with a loud crash.

Unscheduled rapid disassembly

 

For this replacement project, I started with tackling the ventilation problem to make sure the air would actually get extracted out of the house, rather than into the attic (side note: pretty sure venting bathrooms into attics is now forbidden under the building codes for any new installs).

To do this, I brought a 150mm Simx/Manrose Thru Roof Cowl Kit – these can be found at consumer hardware stores and it’s a kit that consists of a plastic tube, the metal cowl ontop and a suitable rubber mounting boot and assorted hardware.

Going through the roof was the only option on this property. Not only is the bathroom in the middle of the house, even if I ran a long ducting run to the nearest walls, there’s no soffits or other tidy location to install the vent without damaging the character of the property.

Who cares about the iPhone, *this* is the unboxing you’ve been holding out for.

Installing this was… fun. I purchased my new most-favourite-thing-ever, a reciprocating saw (also called a sabre saw) in order to cut a hole in the roof. This tool has gone on to work hard in a large number of other projects and I consider it an extremely good investment given it’s versatility.

Cut all the things!

One of the quirks of our property being approximately 100 years old is that the roof is sarked with timbers which makes it extremely solid – a proper house, from a more refined age. They stopped building houses like this a long time ago, I think the whole non-sustainable forestry part become a slight issue…

In this situation the solid build both helped and hindered – trying to cut through corrugated iron is a lot easier when there’s not 20mm of native hardwood underneath it as the saw has the habit of picking up the iron and vibrating it like crazy whilst trying to cut the timbers.

But the upside is that it meant I could screw the roof vent directly into the timber and be assured of a tight and long-lasting fit, whereas if I had only floating iron sheets it would have been a lot tricker to get a really tight fit… If this had been the case like in a more modern property, I’d have probably brought some plywood sheets and run them across the rafters to provide a solid surface for screwing the vent into for a similar effect.

You can get an idea of how solid the house is from this photo inside the attic. In this photo the vent has been installed and ducting attached.

Once I cut the roof hole, I sealed the vent kit rubber boot thoroughly with silicone, with a layer between the boot and the iron sheets, as well as around the edge of the boot and the plastic tube.

The rubber boot has a metal strip allowing it to be bent to fit the corrugated iron snugly. Once the screws went in, it’s a very tight seal and the silicone makes 100% sure it’s sealed.

Note the diagonal placement – this is intentional since it ensures you don’t end up with a pool of water collecting at the top of the boot, which could happen if you placed it square.

The new vent next to the bathroom skylight. You can see the interior intake through the skylight.

You’ll notice that I’ve cut the hole overlapping two separate sheets. This… isn’t ideal, I’d much rather have cut into a single sheet (easier to seal, less to go wrong as the sheets expand and contract) but I was trying to utilise an existing hole that was already cut in the sarking timbers for what must have been a small vent (maybe an overflow pipe?) in the past to minimise the amount of cutting needed.

This placement also caused another small annoyance for me, which is that the vent is not quite 100% straight and you can notice it sometimes. It’s not an issue for water ingress in the rain, but it annoys me not being 100% perfect. That being said, I’ve had trades install other vents in the property (future project post coming!) and they didn’t get it properly straight either, so I feel somewhat vindicated.

Slightly wonky vent will annoy me everyday forever more…. but if I’ve learnt anything re DIY, don’t fuck with anything that ain’t broke.

 

To connect the fan to the vent, I brought some insulated ducting. It’s important to use insulated ducting for this, rather than the cheaper uninsulated stuff, since bathroom air is warm and moist – if the ducting is not insulated, you are likely to suffer condensation in the duct as the air cools when it transits through the cooler attic temperatures. By keeping it as warm as possible on it’s way out you can avoid this.

I was worried about some condensation occurring in the vent tube itself – I can’t insulate outside of the roof after all – but this fortunately hasn’t been a problem. Additionally there was some concern that the roof cowl wouldn’t be enough to keep out rain in Wellington winds, but this also hasn’t been an issue – it could be a different situation in more exposed areas of the city and there are cowl fittings that are more suited for unfriendly wind conditions if this was the case.

 

Next I had to sort out a new fan. I was tempted to keep the existing fan given it’s strong air throughput and the motor still running fine, but I couldn’t easily find a replacement front for it, and the back was completely exposed so there was no easy way to fit the ducting to the back of the fan.

I ended up buying a 250mm “high pressure” fan, but unfortunately this didn’t work out well for me. Despite being described as high pressure, I can confirm that these consumer-available bathroom fans are absolutely useless and not worth buying if you want anything more than a faint breeze.

I had it in place for 1 week, during which time we not only quickly noticed it struggling to remove the steam from the room, but that it was also slowing filling up with water that was condensing in the fan as it struggled the clear the room.

First iteration. Note the side exiting fan which required twisted ducting – not ideal. You can also see the hole in the sarking is larger than needed – that’s because there was a pre-existing hole that I took advantage of which was longer than I needed.

Unfortunately given how much of a complete failure this fan had been, I had to remove it and find an alternative.

 

The solution was a 150mm pro-series Simix inline fan capable of delivering 166l/s (597m3/hr) air throughput. For comparison, the previous attempted fan was maybe 69l/s (250m3/hr) at best and even that seems dubious given how poorly it performed.

Nimbus is a big fan of this model.

I couldn’t find these at any consumer hardware store and ended up taking advantage of a friend with an electrical trade account who was able to place an order with the supplier for me.

The one key difference with this solution vs my previous attempt is that the fan is inline, which means the bathroom needed a vent installed, with ducting from the intake vent to the fan, then ducting to the outlet vent. This does have some noise advantages since you can place the fan motor further away from the intake.

Second iteration. Mounting it on framing timber is a little overkill but I had framing timber and not plywood handy. Because of the solid roof, I found it easier to mount to the underside of the roof, rather than building a platform on the attic floor.

This solution worked much better – the fan is able to extract a considerable amount of air and whilst a bit noisy due to high RPM and small diameter, it does a good job of clearing the bathroom throughout the shower and not letting it build up too much moisture.

When researching this project, it was suggested  that you shouldn’t be clever and mix duct sizes (eg a 200mm fan into a 150mm vent), so I kept the same spec throughout – if I had gone for a larger roof vent and duct in the beginning, I might have gone with something larger to get more throughput but also larger fans tend to be quieter (as a general rule).

The other big positive of this approach is that I was able to solve the skylight condensing problem by putting the new intake vent directly into the side of the skylight wall to rapidly extract out the air.

This is working extremely well – whilst we have the existing damage from past moisture build up which will require remedial work (complete repaint, maybe some new plastering), since putting in this new vent we’ve had very little moisture build up since the fan keeps the air moving in the skylight preventing condensation. And since heat rises, all the steam from the shower naturally gravitates towards the vent anyway.

This photo illustrates the issue with the placement of the old vs new fan – the old one did nothing for all the stream that wafted up into the skylight space, vs the new one keeping that space clear.

I found it really hard to find an intake vent that wasn’t horribly ugly and plasticky, so ended up paying a bit of a premium for an aluminium model. It ended up being a right pain to install so maybe I should have gone for a cheap nasty plastic 150mm that would have been simple to fit, but I think it was worth it and looks good (once I fix all the paintwork anyway). I even managed to get it dead centre which given I was cutting it out from inside the attic due to inability to get ladder high enough in the bathroom was a pretty good outcome.

 

Anyway despite the challenges, I’m pretty happy with this setup now. It’s working reliably, I’ve checked the ducting a few times to make sure there’s been no moisture build up or water ingress from outside (both good!) so I’m expecting this solution to last for a long time.

I still need to fix the plasterboard and paint job in the bathroom. It’s kind of stuck pending access to a more flexible ladder/indoor scaffold type system just due to the height of the bathroom roof and very limited placement points for ladders. Still it annoys me daily so it’ll get fixed sooner or later when I get really sick of it looking so bad.

Rough cost for the project was around $500-600 NZD in parts – the most expensive bits being the fan motor, and then the roof vent kit – a wall vent solution would be a fair bit cheaper.

If I was doing this project again from scratch, I’d probably have done a few things differently.

  • I’d almost certainly have considered getting the bigger model and going for a 200mm fan able to shift 272l/s (980m3/hr) of air. The current model is good, but I’d almost have enjoyed the bathroom being like a vacuum cleaner for maximum dryness. And the larger fan size should be a bit quieter.
  • I utilised the existing hardwired appliance circuit as a straight replacement of one fan for another, but it would have been good to get a timer fan installed, to keep it running for a given time period following the fan being switched off. This is something I might end up getting an electrician to install for me in future anyway, but I might have been able to save some money getting a model of fan with the timer circuit build in, vs having to now buy a timer module for the circuit.
  • I don’t love the ducting install. I ended up with two 90 degree bends which were unavoidable due to the original hole being intended for a fan directly below the hole, but I’d have preferred an almost straight run to ensure minimal workload for the fan (maybe some noise reduction too?). This could have been easily accomplished by installing the fan further up the roof line and running the duct straight from the interior vent, through the fan, then up and out the roof. But it wasn’t worth sealing one hole and cutting another to fix.
  • If I ever build a house, I’m making sure the bathrooms have at least one exterior wall to make ventilation so much simpler!
  • I put in all the vent bolts (hex head) using an automotive socket set by hand. This works totally fine but just takes ages, so an impact driver would have been a nice addition to the tool kit. That being said, I’ve since done other hex head screws using a socket adaptor drill bit which allows me to use the cordless drill to drive hex head screws, although admittedly lacking the high torque of a proper impact driver.
Posted in Uncategorized | Tagged , , , , | 2 Comments

Deep Dive into ECS

I spent a fair bit of time in 2017 re-architecting the carnival.io platform onto Amazon ECS, including working to handle some tricky autoscaling challenges brought on by the nature of the sudden high-load spikes experienced when we deliver push messages to customers.

I’ve now summed up these learnings into a deep dive talk on the Amazon ECS architecture that I presented at the Wellington AWS Users Group on February 12th 2018.

This talk explains what container orchestration is, some key fundamentals about ECS, how we’ve tackled CI/CD with ECS and going into details around some of the unique autoscaling challenges caused by millions of cellphones sending home telemetry all at once.

This talk is technical, but includes content appropriate for both beginners wanting to know how ECS functions and experts wanting to see just what can be accomplished with the platform.

 

Posted in Uncategorized | Tagged , , , , , , , | Leave a comment

Puppet Autosigning & Cloud Recommendations

I was over in Sydney this week attending linux.conf.au 2018 and made a short presentation at the Sysadmin miniconf regarding deploying Puppet in cloud environments.

The majority of this talk covers the Puppet autosigning process which is a big potential security headache if misconfigured. If you’re deploying Puppet (or even some other config management system) into the cloud, I recommend checking this one out (~15mins) and making sure your own setup doesn’t have any issues.

 

Posted in Uncategorized | Tagged , , , , , , | Leave a comment

Firebase FCM upstream with Swift on iOS

I’ve been learning a bit of Swift lately in order to write an iOS app for my alarm system. I’m not very good at it yet, but figured I’d write some notes to help anyone else playing with the murky world of Firebase Cloud Messaging/FCM and iOS.

One of the key parts of the design is that I wanted the alarm app and the alarm server to communicate directly with each other without needing public facing endpoints, rather than the conventional design when the app interacts via an HTTP API.

The intention of this design is that it means I can dump all the alarm software onto a small embedded computer and as long as that computer has outbound internet access, it just works™️. No headaches about discovering the endpoint of the service and much more simplified security as there’s no public-facing web server.

Given I need to deliver push notifications to the app, I implemented Google Firebase Cloud Messaging (FCM) – formerly GCM – for push delivery to both iOS and Android apps.

Whilst FCM is commonly used for pushing to devices, it also supports pushing messages back upstream to the server from the device. In order to do this, the server must be implemented as an XMPP server and the FCM SDK be embedded into the app.

The server was reasonably straight forwards, I’ve written a small Java daemon that uses a reference XMPP client implementation and wraps some additional logic to work with HowAlarming.

The client side was a bit more tricky. Google has some docs covering how to implement upstream messaging in the iOS app, but I had a few issues to solve that weren’t clearly detailed there.

 

Handling failure of FCM upstream message delivery

Firstly, it’s important to have some logic in place to handle/report back if a message can not be sent upstream – otherwise you have no way to tell if it’s worked. To do this in swift, I added a notification observer for .MessagingSendError which is thrown by the FCM SDK if it’s unable to send upstream.

class AppDelegate: UIResponder, UIApplicationDelegate, MessagingDelegate {

 func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplicationLaunchOptionsKey: Any]?) -> Bool {
   ...
   // Trigger if we fail to send a message upstream for any reason.
   NotificationCenter.default.addObserver(self, selector: #selector(onMessagingUpstreamFailure(_:)), name: .MessagingSendError, object: nil)
   ...
 }

 @objc
 func onMessagingUpstreamFailure(_ notification: Notification) {
   // FCM tends not to give us any kind of useful message here, but
   // at least we now know it failed for when we start debugging it.
   print("A failure occurred when attempting to send a message upstream via FCM")
 }
}

Unfortunately I’m yet to see a useful error code back from FCM in response to any failures to send message upstream – seem to just get back a 501 error to anything that has gone wrong which isn’t overly helpful… especially since in web programming land, any 5xx series error implies it’s the remote server’s fault rather than the client’s.

 

Getting the GCM Sender ID

In order to send messages upstream, you need the GCM Sender ID. This is available in the GoogleService-Info.plist file that is included in the app build, but I couldn’t figure out a way to extract this easily from the FCM SDK. There probably is a better/nice way of doing this, but the following hack works:

// Here we are extracting out the GCM SENDER ID from the Google
// plist file. There used to be an easy way to get this with GCM, but
// it's non-obvious with FCM so here's a hacky approach instead.
if let path = Bundle.main.path(forResource: "GoogleService-Info", ofType: "plist") {
  let dictRoot = NSDictionary(contentsOfFile: path)
  if let dict = dictRoot {
    if let gcmSenderId = dict["GCM_SENDER_ID"] as? String {
       self.gcmSenderId = gcmSenderId // make available on AppDelegate to whole app
    }
  }
}

And yes, although we’re all about FCM now, this part hasn’t been rebranded from the old GCM product, so enjoy having yet another acronym in your app.

 

Ensuring the FCM direct channel is established

Finally the biggest cause I had for upstream message delivery failing, is that I was often trying to send an upstream message before FCM had finished establishing the direct channel.

This happens for you automatically by the SDK whenever the app is loaded into foreground, provided that you have shouldEstablishDirectChannel set to true. This can take up to several seconds after application launch for it to actually complete – which means if you try to send upstream too early, the connection isn’t ready, and your send fails with an obscure 501 error.

The best solution I found was to use an observer to listen to .MessagingConnectionStateChanged which is triggered whenever the FCM direct channel connects or disconnects. By listening to this notification, you know when FCM is ready and capable of delivering upstream messages.

An additional bonus of this observer, is that when it indicates the FCM direct channel is established, by that time the FCM token for the device is available to your app to use if needed.

So my approach is to:

  1. Setup FCM with shouldEstablishDirectChannel set to true (otherwise you won’t be going upstream at all!).
  2. Setup an observer on .MessagingConnectionStateChanged
  3. When triggered, use Messaging.messaging().isDirectChannelEstablished to see if we have a connection ready for us to use.
  4. If so, pull the FCM token (device token) and the GCM Sender ID and retain in AppDelegate for other parts of the app to use at any point.
  5. Dispatch the message to upstream with whatever you want in messageData.

My implementation looks a bit like this:

class AppDelegate: UIResponder, UIApplicationDelegate, MessagingDelegate {

 func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplicationLaunchOptionsKey: Any]?) -> Bool {
  ...
  // Configure FCM and other Firebase APIs with a single call.
  FirebaseApp.configure()

  // Setup FCM messaging
  Messaging.messaging().delegate = self
  Messaging.messaging().shouldEstablishDirectChannel = true

  // Trigger when FCM establishes it's direct connection. We want to know this to avoid race conditions where we
  // try to post upstream messages before the direct connection is ready... which kind of sucks.
  NotificationCenter.default.addObserver(self, selector: #selector(onMessagingDirectChannelStateChanged(_:)), name: .MessagingConnectionStateChanged, object: nil)
  ...
 }

 @objc
 func onMessagingDirectChannelStateChanged(_ notification: Notification) {
  // This is our own function listen for the direct connection to be established.
  print("Is FCM Direct Channel Established: \(Messaging.messaging().isDirectChannelEstablished)")

  if (Messaging.messaging().isDirectChannelEstablished) {
   // Set the FCM token. Given that a direct channel has been established, it kind of implies that this
   // must be available to us..
   if self.registrationToken == nil {
    if let fcmToken = Messaging.messaging().fcmToken {
     self.registrationToken = fcmToken
     print("Firebase registration token: \(fcmToken)")
    }
   }

   // Here we are extracting out the GCM SENDER ID from the Google PList file. There used to be an easy way
   // to get this with GCM, but it's non-obvious with FCM so we're just going to read the plist file.
   if let path = Bundle.main.path(forResource: "GoogleService-Info", ofType: "plist") {
    let dictRoot = NSDictionary(contentsOfFile: path)
     if let dict = dictRoot {
      if let gcmSenderId = dict["GCM_SENDER_ID"] as? String {
       self.gcmSenderID = gcmSenderId
     }
    }
   }

  // Send an upstream message
  let messageId = ProcessInfo().globallyUniqueString
  let messageData: [String: String] = [
   "registration_token": self.registrationToken!, // In my use case, I want to know which device sent us the message
   "marco": "polo"
  ]
  let messageTo: String = self.gcmSenderID! + "@gcm.googleapis.com"
  let ttl: Int64 = 0 // Seconds. 0 means "do immediately or throw away"

  print("Sending message to FCM server: \(messageTo)")

  Messaging.messaging().sendMessage(messageData, to: messageTo, withMessageID: messageId, timeToLive: ttl)
  }
 }

 ...
}

For a full FCM downstream and upstream implementation example, you can take a look at the HowAlarming iOS app source code on Github and if you need a server reference, take a look at the HowAlarming GCM server in Java.

 

Learnings

It’s been an interesting exercise – I wouldn’t particularly recommend this architecture for anyone building real world apps, the main headaches I ran into were:

  1. FCM SDK just seems a bit buggy. I had a lot of trouble with the GCM SDK and the move to FCM did improve stuff a bit, but there’s still a number of issues that occur from time to time. For example: occasionally a FCM Direct Channel isn’t established for no clear reason until the app is terminated and restarted.
  2. Needing to do things like making sure FCM Direct Channel is ready before sending upstream messages should probably be handled transparently by the SDK rather than by the app developer.
  3. I have still yet to get background code execution on notifications working properly. I get the push notification without a problem, but seem to be unable to trigger my app to execute code even with content-available == 1 . Maybe a bug in my code, or FCM might be complicating the mix in some way, vs using pure APNS. Probably my code.
  4. It’s tricky using FCM messages alone to populate the app data, occasionally have issues such as messages arriving out of order, not arriving at all, or occasionally ending up with duplicates. This requires the app code to process, sort and re-populate the table view controller which isn’t a lot of fun. I suspect it would be a lot easier to simply re-populate the view controller on load from an HTTP endpoint and simply use FCM messages to trigger refreshes of the data if the user taps on a notification.

So my view for other projects in future would be to use FCM purely for server->app message delivery (ie: “tell the user there’s a reason to open the app”) and then rely entirely on a classic app client and HTTP API model for all further interactions back to the server.

Posted in Uncategorized | Tagged , , , , , , , , , , | Leave a comment

MongoDB document depth headache

We ran into a weird problem recently where we were unable to sync a replica set running MongoDB 3.4 when adding new members to the replica set.

The sync would begin, but at some point during the sync it would always fail with:

[replication-0] collection clone for 'database.collection' failed due to Overflow:
While cloning collection 'database.collection' there was an error
'While querying collection 'database.collection' there was an error 
'BSONObj exceeded maximum nested object depth: 200''

(For extra annoyance the sync would continue with syncing all the other databases and collections on the replica set, before then only realising it had actually failed earlier at the very end of the sync and then restarting the sync from the beginning again).

 

The error means that one or more documents has a max depth over 200. This could be a chain of objects, or a chain of arrays in a document – a mistake that isn’t too tricky to cause with a buggy loop or ORM.

But how is it possible that this document could be in the database in the first case? Surely it should have been refused at time of insert? Well the nested document limit size and enforcement has changed at various times in past versions and a long-lived database such as ours from early MongoDB 2.x days may have had these bad documents inserted before the max depth limit was enforced and only now when we try to use the document do the limits become a problem.

In our case the document was old, but didn’t have any issues syncing back on Mongo 3.0 but now failed with Mongo 3.4.

Finding the document is tricky – the replication process helpfully does not log the document ID, so you can’t go and purge it from the collection to resolve the issue.

With input from my skilled colleagues with better Mongo skills than I, we figured out three queries that allowed us to identify the bad documents.

1. This query finds any documents that have a long chain of nested objects inside them.

db.collection.find({ $where: function() { return tojsononeline(this).indexOf("} } } } } } } } }") != -1 } })

2. This query finds any documents that have a long chain of nested arrays. This was the specific issue in our case and this query successfully identified all the bad documents.

db.collection.find({ $where: function() { return tojsononeline(this).indexOf("] ] ] ] ] ] ]") != -1 } })

3. And if you get really stuck, you can find any bad document (for whatever reason) by reading the document and then re-writing it back out to another collection. This ensures the document gets all the limits applied at write time and will identify their ID, regardless of the specific reason for them being refused.

db.collection.find({}).forEach(function(d) { print(d["_id"]); db.new_collection.insert(d) });

Note that all of these queries tend to be performance impacting since you’re asking your database to read every single document. And the last one, copying collections, could take considerable time and space to complete.

I recommend restoring the replica set to a test system and performing the operation there where you know it’s not going to impact production if you have any data of notable size.

Once you find your bad document, you can display it with:

db.collection.find({ _id: ObjectId("54492129902178d6f600004f") });

And delete it entirely (assuming nothing important in it!) with:

db.collection.deleteOne({ _id: ObjectId("54492129902178d6f600004f") });
Posted in Uncategorized | Tagged , , | 2 Comments

MacOS High Sierra unable to free disk space

I recently ran out of disk space on my iMac. After migrating a considerable amount of undesirable data to either the file server or /dev/null, I found that despite my efforts, the amount of free disk space had not increased.

I was worried it was an issue with the new APFS file system introduced to all SSD-using Macs as of High Sierra, but in this case it turns out the issue is that Time Machine retains local snapshots on disk, in addition to the full backup history that is retained on the network time machine device.

Apple state that they automatically remove local snapshots when disk space is low, but their definition of low is apparently only 5GB of free space remaining – not really much free working space in 2017 when you might want scratch space of 22GB for 1 hour of 4k 30FPS footage.

On older MacOS releases, it was possible to disable the local snapshot feature entirely, this doesn’t seem to be the case with High Sierra – but it does appear to be possible to force an immediate purge of local snapshots with the following command:

sudo tmutil thinLocalSnapshots / 10000000000 4

For example;

Back into the time vortex with you, filthy snapshots!

Note that this snapshot usage is not visible as a distinct item in the Disk Utility or Storage Management application.

In my case, all the snapshots appeared to be within the last 24 hours, so if I hadn’t urgently needed the disk space, I suspect the local snapshots would have flushed themselves after a 24 hour period restoring considerable disk space.

The fact this isn’t an opt-in user-accessible feature is a shame. It adds convenience for a user of not having to get physical access to the backup drive or time capsule-like-thingy in order to restore data, but any users of systems with SSD-only storage are likely to be a bit precious about how every GB is used and there’s almost no transparency about how much space is being consumed. Especially annoying when you urgently need more space and are stuck wondering why nothing is freeing up room…

Posted in Uncategorized | Tagged , , , , , | 3 Comments

Access Route53 private zones cross account

Using Route53 private zones can be a great way to maintain a private internal zone for your server infrastructure. However sometimes you may need to share this zone with another VPC in the same or in another AWS account.

The first situation is easy – a Route53 zone can be associated with any number of VPCs within a single AWS account using the AWS console.

The second is more tricky but is doable by creating a VPC association authorization request in the account with the zone, then accepting it from the other account.

# Run against the account with the zone to be shared.
aws route53 \
create-vpc-association-authorization \
--hosted-zone-id abc123 \
--vpc VPCRegion=us-east-1,VPCId=vpc-xyz123 

# Run against the account that needs access to the private zone.
aws route53 \
associate-vpc-with-hosted-zone \
--hosted-zone-id abc123 \
--vpc VPCRegion=us-east-1,VPCId=vpc-xyz123 \
--comment "Example Internal DNS Zone"

# List authori(z|s)ations once done
aws route53 \
list-vpc-association-authorizations \
--hosted-zone-id abc123

This doesn’t even require VPC peering since it works behind the scenes, with the associated zone now being resolvable using the default VPC DNS server on each zone that has been associated.

Note that the one catch is that this does not help you if you’re linking to a non-AWS VPC environment, such as an on-prem data centre via IPSec VPN or Direct Connect. Even though you can route to the VPC and systems inside it, the AWS DNS resolver for the VPC will refuse requests from IP space outside of the VPC itself.

So the only option is have an EC2 instance acting as a DNS forwarder inside the VPC, which is reachable from the linked data centre and yet since it’s in the VPC, can use the resolver.

Posted in Uncategorized | Tagged , , , , | 3 Comments