Monthly Archives: July 2015

Puppet modules

I’m in the middle of doing a migration of my personal server infrastructure from a 2006-era colocation server onto modern cloud hosting providers.

As part of this migration, I’m rebuilding everything properly using Puppet (use it heavily at work so it’s a good fit here) with the intention of being able to complete server builds without requiring any manual effort.

Along the way I’m finding gaps where the available modules don’t quite cut it or nobody seems to have done it before, so I’ve been writing a few modules and putting them up on GitHub for others to benefit/suffer from.

 

puppet-hostname

https://github.com/jethrocarr/puppet-hostname

Trying to do anything consistently with host naming is always fun, since every organisation or individual has their own special naming scheme and approach to dealing with the issue of naming things.

I decided to take a different approach. Essentially every cloud provider will give you a source of information that could be used to name your instance whether it’s the AWS Instance ID, or a VPS provider passing through the name you gave the machine at creation. Given I want to treat my instances like cattle, an automatic soulless generated name is perfect!

Where they fall down, is that they don’t tend to setup the FQDN properly. I’ve seen a number of solution to this including user data setup scripts, but I’m trying to avoid putting anything in user data that isn’t 100% critical and sticking to my Pupistry bootstrap so I wanted to set my FQDN via Puppet itself.

(It’s even possible to set the hostname itself if desired, you can use logic such as tags or other values passed in as facts to define what role a machine has and then generate/set a hostname entirely within Puppet).

Hence puppet-hostname provides a handy way to easily set FQDN (optionally including the hostname itself) and then trigger reloads on name-dependent services such as syslog.

None of this is revolutionary, but it’s nice getting it into a proper structure instead of relying on yet-another-bunch-of-userdata that’s specific to my systems. The next step is to look into having it execute functions to do DNS changes on providers like Route53 so there’s no longer any need for user data scripts being run to set DNS records at startup.

 

puppet-rirs

https://github.com/jethrocarr/puppet-rirs

There are various parts of my website that I want to be publicly reachable, such as the WordPress login/admin sections, but at the same time I also don’t want them accessible by any muppet with a bot to try and break their way in.

I could put up a portal of some kind, but this then breaks stuff like apps that want to talk with those endpoints since they can’t handle the authentication steps. What I can do, is setup a GeoIP rule that restricts access to the sections to the countries I’m actually in, which is generally just NZ or AU, to dramatically reduce the amount of noise and attempts people send my way, especially given most of the attacks come from more questionable countries or service providers.

I started doing this with mod_geoip2, but it’s honestly a buggy POS and it really doesn’t work properly if you have both IPv4 and IPv6 connections (one or another is OK). Plus it doesn’t help me for applications that support IP ACLs, but don’t offer a specific GeoIP plugin.

So instead of using GeoIP, I’ve written a custom Puppet function that pulls down the IP assignment lists from the various Regional Internet Registries and generate IP/CIDR lists for both IPv4 and IPv6 on a per-country basis.

I then use those lists to populate configurations like Apache, but it’s also quite possible to use it for other purposes such as iptables firewalling since the generated lists can be turned into Puppet resources. To keep performance sane, I cache the processed output for 24 hours and merge any continuous assignment blocks.

Basically, it’s GeoIP for Puppet with support for anything Puppet can configure. :-)

 

puppet-digitalocean

https://github.com/jethrocarr/puppet-digitalocean

Provides a fact which exposes details from the Digital Ocean instance API about the instance – similar to how you get values automatically about Amazon EC2 systems.

 

puppet-initfact

https://github.com/jethrocarr/puppet-initfact

The great thing about the open source world is how we can never agree so we end up with a proliferation of tools doing the same job. Even init systems are not immune, with anything tha intends to run on the major Linux distributions needing to support systemd, Upstart and SysVinit at least for the next few years.

Unfortunately the way that I see most Puppet module authors “deal” with this is that they simply write an init config/file that suits their distribution of choice and conveniently forget the other distributions. The number of times I’ve come across Puppet modules that claim support for Red Hat and Amazon Linux but only ship an Upstart file…. >:-(

Part of the issue is that it’s a pain to even figure out what distribution should be using what type of init configuration. So to solve this, I’ve written a custom Fact called “initsystem” which exposes the primary/best init system on the specific system it’s running on.

It operates in two modes – there is a curated list for specific known systems and then fallback to automatic detection where we don’t have a specific curated result handy.

It supports (or should) all major Linux distributions & derivatives plus FreeBSD and MacOS. Pull requests for others welcome, could do with more BSD support plus maybe even support for Windows if you’re feeling brave.

 

puppet-yas3fs

https://github.com/pcfens/puppet-yas3fs/commit/27af462f1ce2fe0610012a508236062e65017b5f

Not my module, but I recently submitted a PR to it (subsequently merged) which introduces support for a number of different distributions via use of my initfact module so it should now run on most distributions rather than just Ubuntu.

If you’re not familiar with yas3fs, it’s a FUSE driver that turns S3+SNS+SQS into a shared filesystem between multiple servers. Ideal for dealing with legacy applications that demand state on disk, but don’t require high I/O performance, I’m in the process of doing a proof-of-concept with it and it looks like it should work OK for low activity sites such as WordPress, although with no locking I’d advise against putting MySQL on it anytime soon :-)

 

These modules can all be found on GitHub, as well as the Puppet Forge. Hopefully someone other than myself finds them useful. :-)

10 months in

It’s been almost 10 months since Lisa and I brought our current house and moved in. Things are going well, having our own place and not paying a landlord is a fantastic and freeing feeling, but home ownership certainly isn’t a free ride and the amount of work it generates is quite incredible.

So what’s been happening around Carr Manor since we moved in?

Home sweet home

Can’t beat Wellington on a good day!

Generally the house is in good shape, most of my time has been spent in the grounds of the estate clearing paths, overgrown vegetation and various other missions. However we have had a couple smaller issues with the house itself.

 

The most serious one is that part of the iron roof in the house was leaking due to what looks like a number of different patch jobs combined with a nice unhealthy dose of rust.

Hmm cracks in the roof that let water in == bad right?

Hmm cracks in the roof that let water in == bad right?

The outside doesn't look a whole lot better.

The outside doesn’t look a whole lot better.

The “proper” fix is that this section of roof needs replacing at some point as it’s technically well-past EOL, but roof replacement is expensive and a PITA, so I’ve fixed the issue by stripped off as much rust as I could and then re-sealing the roof using Mineral Brush-On Underbody Seal.

Incase you’re wondering, yes, the same stuff that you can use on cars. It’s basically liquid tar, completely waterproof and ever so wonderful at sealing leaky roofs. I liberally applied a few cans over flashings, patches and the iron itself getting a nice thick seal.

Repair!

Repair!

The same stuff did wonders on the rusted shed roof flashing as well.

The same stuff did wonders on the rusted shed roof flashing as well.

Up next I need to complete a repaint of both sheds and the house roof. I’m probably going to do a small job in whatever colour I have lying around for the worst part of the roof and then go over the whole roof again at a later stage when we decide on a colour for the full repaint.

 

The other issue we had was that one of the window hinges had rusted out leaving us with a window that wouldn’t open/close properly.

So rusty :-/

I’m not expert, but I don’t think hinges are supposed to look like this….

This was a tricky one to fix – the hinge and the screws were so rusted out I couldn’t even remove them, in the end I removed the window simply by tearing the hinge apart when I pulled on it leaving a shower of rust and more disturbingly, cockroaches that had been living amongst the bubbled rust.

This left me with two parts of metal hinge stuck in the wall and on the window frame held in by screws that would no longer turn – or in some cases, even lacked heads entirely.

To get them out, I put a very small drill bit into the electric drill and drilled out the screw right down the middle of it. It’s pretty straightforwards once you get it going, but it was a bit tricky to get started – I ended up using the smallest bit I had to make a pilot hole/groove in the screw head, and then upsized the bit to drill in through the screw. Once done, the metal remains tend to just fall out and come out with a little prodding.

I’ve since replaced it with a shiny new hinge and stainless steel screws which should last a lot longer than their predecessors.

Shiny new

Shiny new hardware

 

Painting has been an “interesting” learning experience, I’ve found it the hardest skill to pickup since it’s just so time consuming and you have to take such extreme care to avoid dripping any paint on other surfaces.

One of my earliest painting jobs was doing the lower gate. This gate spends a lot of time in the shade and even in spring was feeling damp and waterlogged and generally wasn’t looking that sharp – especially the fact the bolt was a pile of rust barely holding together.

The rustic delight of unfinished timber.

I’m sure unfinished timber looks great when it’s first built, but the moss dirt and damp doesn’t lead to it aging well.

It's like new!

Much sharper!

Things like the gate take time and need care, but it’s nothing compared to the absolute frustration of painting window frames where a few mm to the wrong side or a stray bristle leads to paint being smeared across the glass.

I did the french doors initially as the paint had peeled and was starting to expose the timber to the elements, some of the putty had even fallen out and needed replacing.

Probably the most frustrating thing I've ever had to do.

Applying painter’s tape to this is one of the most frustrating things I’ve ever had to do :-/

Because I was painting around glass, I applied painter’s tape the whole thing before hand. It took hours, incredibly frustrating and I feel that the end result wasn’t particularly great.

I’ve since found that I can get a pretty tidy result using a sash/trim brush and taking extreme care not to bump the glass, but it is tricky and mistakes do happen. I’m figuring with enough practice I’ll get better at windows… and I have plenty of practice waiting for me with a full house paint job pending. Of course I could pay someone to do it, but at $15k+ for a re-paint, I’m pretty keen to see if I can tackle it myself….

 

The shed works haven’t proceeded much – I had the noble goal of completely repairing it over summer, but that time just varnished sorting out various other bits and pieces.

On the plus side, thanks to help from one of my colleagues, the shed has been dug out from it’s previously buried state and the rot and damage exposed – next step is to tear off the rotten weatherboards and doors and replace them with new ones, before repainting the whole shed.

Dug out shed

A small 1meter retaining wall would have been more than enough to protect the shed, but instead the earth has ended up piled around it causing it to rot and collapse.

 

I also had help from dad and toppled the mid-size trees that were in-between the shed and the path. Not only were they blocking out light, but they were also going to be a clear issue to shed and path integrity in the future as they got bigger.

Much tidier!

Much tidier! Just need to fix the shed itself now…

I’m still really keen to get this shed fixed so intend to make a start on measuring and sourcing the timber soon(ish) and maybe taking a few days off work to line up a block of time to really attack and fix it up.

 

A more pressing issue has been our pathways. We have two long 30-40meter concrete paths, a long ramped one (around 20-30 degree slope) up to the upper street and carpad and another zig-zag path with a mix of ramps and steps heading down to the lower street where the bus stop is.

Both paths are not in the best condition. The lower one requires a complete replacement, it’s probably around 80 years old and the non-reinforced concrete has cracked and shifted all over the place.

The upper one is more structurally intact, but has it’s own share of issues. The first most serious issue is that the steeper upmost end gets incredibly slippery in winter. It seems that although the concrete has been brush-finished whenever it rains, any grip it had just vanishes and it basically becomes a slide.

Jethro vs Autumn

Jethro vs Autumn

Naturally slipping to a broken/leg/face/life isn’t ideal and we’ve been looking at options to fix it. We could convert the steepest bit from a ramp to steps, but steps have their own safety issues and we aren’t keen the lose the ramp as it’s the best way for getting large/heavy items to/from the house.

So a couple months ago I put down some Resene Non-slip Deck & Path which is a tough non-slippery paint product that basically includes a whole heap of sand which turns the smooth concrete path into something more like fine sandpaper.

We weren’t too sure about how good it would be, so we put down a 0.5l strip to test it out on the worst most part of the path.

A/B Testing IRL

A/B Testing IRL

It doesn’t feel that different to brushed concrete in the dry, but in the wet the difference is night & day and you really do feel a bit more attached to the path. We’ll still need to invest in a decent handrail and fence, but this goes a long way towards an elegant fix.

I’ve since brought another 10l and painted the upper portion, essentially all the “good” concrete we have. I thought that it might be too dark but actually it looks very sharp and once we put a new fence up (maybe white picket?) it will look very clean and tidy.

Slick new path!

Old concrete, as good as new! :-)

The other ~30meters down to the house isn’t in such good shape, the surface is quite uneven in places and it’s missing chunks. We have a project to do to repair or replace the rest of it, once done the intention will be to paint the rest of the path in the same colour and it should look and feel great.

 

All this work requires a fair few tools, I’ve finally clean up the dining room where they had been accumulating and they’re now living properly in the shed.

Shed

Shed

One of the most interesting lessons I’ve had so far is that buying decent tools is often far cheaper than hiring tradies to do something for you – generally tools are cheap, even decent ones, but labour is incredibly expensive.

CHAIN SAW

Why yes, that is a hardwood lamppost that I’m chainsawing.

The same thing applies to parts, it’s generally cheaper to just buy a new replacement of something than it is to fix it – I’m used to this from the IT world, but didn’t expect it from IRL.

In our cases, we had a shower mixer that decided to start letting a constant small stream of water through rather than shutting off properly.

Jethro vs Shower

Jethro vs Shower

Taking it apart and even removing it from the wall entirely isn’t too tricky, but I found after removing it all that the issue wasn’t anything trivial like needing a new o-ring and had to call out the plumbers.

Plumbers took it out, look at and it and are all “yeah that needs a new part”, so I ended up paying for the part + the labour – I’d have been better off just buying the whole new part myself and fitting it rather than trying to fix it.

 

Never underestimate the amount of waste you produce moving into a new place. I filled a skip with 1/3 concrete rubble, 1/3 polystyrene and 1/3 misc waste and there’s still another skip worth of debris around the property, possibly more once I tear all the rotten timber out of the shed.

Polystyrene is my number one enemy right now, almost everything we had shipped to the house when we moved in came with some and it’s crumbly and completely non-recyclable for good measure >:-(.

Where did all this junk come from?

Where did all this junk come from?

 

 

Finally on the inside of the house things haven’t progressed much. Lisa has been working on the interior decor and accessories whilst I’ve done exciting things like overseeing the installation of insulation and fixing the loo in the laundry. :-/

Warming sheep fluff!

Warming sheep fluff!

I hate plumbing!

I hate plumbing!

I also had a whole bunch of fun with the locks – when we moved in I had the locksmith change the tumblers, but we’ve since found the locks were pretty worn out and the tail pieces inside started failing, so I had to buy whole new locks and fit them.

Turns out, whole new locks is way cheaper than getting the locksmith out to change the tumblers. If you’re moving into an older place, I’d recommend consider just getting new locks instead since the old ones probably aren’t much good either.

The only downside is that the sizing was slightly different, so I had to do some “creative woodwork” using a drill bit as a file (I didn’t have a file…. or the right size drill bit. A bit dodgy, but worked out OK).

It's not just the IT world where the lack of standards means a bit of hackery to make stuff function.

It’s not just the IT world where the lack of standards means a bit of hackery to make stuff function.

Tidy job at the end of the day!

Tidy job at the end of the day!

 

A lot of this work has been annoying in that it’s not directly visible as an improvement, but it’s all been important stuff that needed doing. I’m hoping to spend the next few months getting stuck into some of the bigger improvements like fixing the paths, sheds, etc which will be a lot more visible.

Until then, need to make more evenings to just sit back, relax and enjoy having our own place – feels like I’ve been just far too busy lately.

Beer time

Beer time