10 months in

It’s been almost 10 months since Lisa and I brought our current house and moved in. Things are going well, having our own place and not paying a landlord is a fantastic and freeing feeling, but home ownership certainly isn’t a free ride and the amount of work it generates is quite incredible.

So what’s been happening around Carr Manor since we moved in?

Home sweet home

Can’t beat Wellington on a good day!

Generally the house is in good shape, most of my time has been spent in the grounds of the estate clearing paths, overgrown vegetation and various other missions. However we have had a couple smaller issues with the house itself.

 

The most serious one is that part of the iron roof in the house was leaking due to what looks like a number of different patch jobs combined with a nice unhealthy dose of rust.

Hmm cracks in the roof that let water in == bad right?

Hmm cracks in the roof that let water in == bad right?

The outside doesn't look a whole lot better.

The outside doesn’t look a whole lot better.

The “proper” fix is that this section of roof needs replacing at some point as it’s technically well-past EOL, but roof replacement is expensive and a PITA, so I’ve fixed the issue by stripped off as much rust as I could and then re-sealing the roof using Mineral Brush-On Underbody Seal.

Incase you’re wondering, yes, the same stuff that you can use on cars. It’s basically liquid tar, completely waterproof and ever so wonderful at sealing leaky roofs. I liberally applied a few cans over flashings, patches and the iron itself getting a nice thick seal.

Repair!

Repair!

The same stuff did wonders on the rusted shed roof flashing as well.

The same stuff did wonders on the rusted shed roof flashing as well.

Up next I need to complete a repaint of both sheds and the house roof. I’m probably going to do a small job in whatever colour I have lying around for the worst part of the roof and then go over the whole roof again at a later stage when we decide on a colour for the full repaint.

 

The other issue we had was that one of the window hinges had rusted out leaving us with a window that wouldn’t open/close properly.

So rusty :-/

I’m not expert, but I don’t think hinges are supposed to look like this….

This was a tricky one to fix – the hinge and the screws were so rusted out I couldn’t even remove them, in the end I removed the window simply by tearing the hinge apart when I pulled on it leaving a shower of rust and more disturbingly, cockroaches that had been living amongst the bubbled rust.

This left me with two parts of metal hinge stuck in the wall and on the window frame held in by screws that would no longer turn – or in some cases, even lacked heads entirely.

To get them out, I put a very small drill bit into the electric drill and drilled out the screw right down the middle of it. It’s pretty straightforwards once you get it going, but it was a bit tricky to get started – I ended up using the smallest bit I had to make a pilot hole/groove in the screw head, and then upsized the bit to drill in through the screw. Once done, the metal remains tend to just fall out and come out with a little prodding.

I’ve since replaced it with a shiny new hinge and stainless steel screws which should last a lot longer than their predecessors.

Shiny new

Shiny new hardware

 

Painting has been an “interesting” learning experience, I’ve found it the hardest skill to pickup since it’s just so time consuming and you have to take such extreme care to avoid dripping any paint on other surfaces.

One of my earliest painting jobs was doing the lower gate. This gate spends a lot of time in the shade and even in spring was feeling damp and waterlogged and generally wasn’t looking that sharp – especially the fact the bolt was a pile of rust barely holding together.

The rustic delight of unfinished timber.

I’m sure unfinished timber looks great when it’s first built, but the moss dirt and damp doesn’t lead to it aging well.

It's like new!

Much sharper!

Things like the gate take time and need care, but it’s nothing compared to the absolute frustration of painting window frames where a few mm to the wrong side or a stray bristle leads to paint being smeared across the glass.

I did the french doors initially as the paint had peeled and was starting to expose the timber to the elements, some of the putty had even fallen out and needed replacing.

Probably the most frustrating thing I've ever had to do.

Applying painter’s tape to this is one of the most frustrating things I’ve ever had to do :-/

Because I was painting around glass, I applied painter’s tape the whole thing before hand. It took hours, incredibly frustrating and I feel that the end result wasn’t particularly great.

I’ve since found that I can get a pretty tidy result using a sash/trim brush and taking extreme care not to bump the glass, but it is tricky and mistakes do happen. I’m figuring with enough practice I’ll get better at windows… and I have plenty of practice waiting for me with a full house paint job pending. Of course I could pay someone to do it, but at $15k+ for a re-paint, I’m pretty keen to see if I can tackle it myself….

 

The shed works haven’t proceeded much – I had the noble goal of completely repairing it over summer, but that time just varnished sorting out various other bits and pieces.

On the plus side, thanks to help from one of my colleagues, the shed has been dug out from it’s previously buried state and the rot and damage exposed – next step is to tear off the rotten weatherboards and doors and replace them with new ones, before repainting the whole shed.

Dug out shed

A small 1meter retaining wall would have been more than enough to protect the shed, but instead the earth has ended up piled around it causing it to rot and collapse.

 

I also had help from dad and toppled the mid-size trees that were in-between the shed and the path. Not only were they blocking out light, but they were also going to be a clear issue to shed and path integrity in the future as they got bigger.

Much tidier!

Much tidier! Just need to fix the shed itself now…

I’m still really keen to get this shed fixed so intend to make a start on measuring and sourcing the timber soon(ish) and maybe taking a few days off work to line up a block of time to really attack and fix it up.

 

A more pressing issue has been our pathways. We have two long 30-40meter concrete paths, a long ramped one (around 20-30 degree slope) up to the upper street and carpad and another zig-zag path with a mix of ramps and steps heading down to the lower street where the bus stop is.

Both paths are not in the best condition. The lower one requires a complete replacement, it’s probably around 80 years old and the non-reinforced concrete has cracked and shifted all over the place.

The upper one is more structurally intact, but has it’s own share of issues. The first most serious issue is that the steeper upmost end gets incredibly slippery in winter. It seems that although the concrete has been brush-finished whenever it rains, any grip it had just vanishes and it basically becomes a slide.

Jethro vs Autumn

Jethro vs Autumn

Naturally slipping to a broken/leg/face/life isn’t ideal and we’ve been looking at options to fix it. We could convert the steepest bit from a ramp to steps, but steps have their own safety issues and we aren’t keen the lose the ramp as it’s the best way for getting large/heavy items to/from the house.

So a couple months ago I put down some Resene Non-slip Deck & Path which is a tough non-slippery paint product that basically includes a whole heap of sand which turns the smooth concrete path into something more like fine sandpaper.

We weren’t too sure about how good it would be, so we put down a 0.5l strip to test it out on the worst most part of the path.

A/B Testing IRL

A/B Testing IRL

It doesn’t feel that different to brushed concrete in the dry, but in the wet the difference is night & day and you really do feel a bit more attached to the path. We’ll still need to invest in a decent handrail and fence, but this goes a long way towards an elegant fix.

I’ve since brought another 10l and painted the upper portion, essentially all the “good” concrete we have. I thought that it might be too dark but actually it looks very sharp and once we put a new fence up (maybe white picket?) it will look very clean and tidy.

Slick new path!

Old concrete, as good as new! :-)

The other ~30meters down to the house isn’t in such good shape, the surface is quite uneven in places and it’s missing chunks. We have a project to do to repair or replace the rest of it, once done the intention will be to paint the rest of the path in the same colour and it should look and feel great.

 

All this work requires a fair few tools, I’ve finally clean up the dining room where they had been accumulating and they’re now living properly in the shed.

Shed

Shed

One of the most interesting lessons I’ve had so far is that buying decent tools is often far cheaper than hiring tradies to do something for you – generally tools are cheap, even decent ones, but labour is incredibly expensive.

CHAIN SAW

Why yes, that is a hardwood lamppost that I’m chainsawing.

The same thing applies to parts, it’s generally cheaper to just buy a new replacement of something than it is to fix it – I’m used to this from the IT world, but didn’t expect it from IRL.

In our cases, we had a shower mixer that decided to start letting a constant small stream of water through rather than shutting off properly.

Jethro vs Shower

Jethro vs Shower

Taking it apart and even removing it from the wall entirely isn’t too tricky, but I found after removing it all that the issue wasn’t anything trivial like needing a new o-ring and had to call out the plumbers.

Plumbers took it out, look at and it and are all “yeah that needs a new part”, so I ended up paying for the part + the labour – I’d have been better off just buying the whole new part myself and fitting it rather than trying to fix it.

 

Never underestimate the amount of waste you produce moving into a new place. I filled a skip with 1/3 concrete rubble, 1/3 polystyrene and 1/3 misc waste and there’s still another skip worth of debris around the property, possibly more once I tear all the rotten timber out of the shed.

Polystyrene is my number one enemy right now, almost everything we had shipped to the house when we moved in came with some and it’s crumbly and completely non-recyclable for good measure >:-(.

Where did all this junk come from?

Where did all this junk come from?

 

 

Finally on the inside of the house things haven’t progressed much. Lisa has been working on the interior decor and accessories whilst I’ve done exciting things like overseeing the installation of insulation and fixing the loo in the laundry. :-/

Warming sheep fluff!

Warming sheep fluff!

I hate plumbing!

I hate plumbing!

I also had a whole bunch of fun with the locks – when we moved in I had the locksmith change the tumblers, but we’ve since found the locks were pretty worn out and the tail pieces inside started failing, so I had to buy whole new locks and fit them.

Turns out, whole new locks is way cheaper than getting the locksmith out to change the tumblers. If you’re moving into an older place, I’d recommend consider just getting new locks instead since the old ones probably aren’t much good either.

The only downside is that the sizing was slightly different, so I had to do some “creative woodwork” using a drill bit as a file (I didn’t have a file…. or the right size drill bit. A bit dodgy, but worked out OK).

It's not just the IT world where the lack of standards means a bit of hackery to make stuff function.

It’s not just the IT world where the lack of standards means a bit of hackery to make stuff function.

Tidy job at the end of the day!

Tidy job at the end of the day!

 

A lot of this work has been annoying in that it’s not directly visible as an improvement, but it’s all been important stuff that needed doing. I’m hoping to spend the next few months getting stuck into some of the bigger improvements like fixing the paths, sheds, etc which will be a lot more visible.

Until then, need to make more evenings to just sit back, relax and enjoy having our own place – feels like I’ve been just far too busy lately.

Beer time

Beer time

Tagged , , , , , , , , , , | 2 Comments

Baking images with Packer & Pupistry

One of the common issues when building modern infrastructure-as-code style systems is that whilst automation is great, it also has a habit of failing at the worst possible time. There’s nothing quite like the fun of trying to autoscale only to find that a newer version of a package breaks compatibility or the repository mirror or Puppet master has gone offline breaking the whole carefully tuned process.

Naturally this is an issue. And whilst I’ve seen some organisations simply ignore the issue and place trust in their repos and configuration management servers, I’m also too pessimistic about technology to trust numerous components for any mission critical applications.

Fortunately there is a solution – we can bake a machine image that has all the applications and configuration pre-applied, so that autoscaling has no third party dependencies (or as close to no dependencies as we can get).

Baking has negative connotations of the bad old days when engineers would assemble custom machine images by hand and then copy them to build new systems, but it doesn’t have to be that way. We can still respect infrastructure-as-code principals and use modern tools like Puppet and Packer to reliably build consistent images as needed.

These images could be as simple as a base AMI image for Amazon AWS which includes the stock OS image plus your Puppet setup. Or they could be as complex as a fully configured and provisioned application server ready-to-go at the first boot.

To make baking images easier, I’ve added support for generating Packer templates pre-loaded with bootstrap data into Pupistry, making it quick and easy to get started. Here’s how you can use it:

Assumptions/prerequisites:

  • You’ve already got Pupistry setup and functional (No? Read the tutorial here)
  • You’ve installed the third party Packer utility.
  • You have an Amazon AWS account for doing the AMI build. Note that Packer isn’t exclusive to Amazon, so you can also use the same technique with other providers including Digital Ocean and OpenStack – but you’ll have to write your own template.

First we can list what Packer templates are available with Pupistry. If the OS/platform of your choice isn’t included, it’s not particularly hard to add it – these are mostly intended to provide a good starting point for customising your own.

pupistry packer

Screen Shot 2015-05-31 at 23.57.20

We can select a template with –template NAME and also pass the resulting output to a file with –file NAME.  The following will build an Amazon Linux template pre-loaded with Pupistry and the default manifest applied:

pupistry packer --template aws_amazon-any --file output.json

Screen Shot 2015-06-01 at 00.00.01

The generated template is a JSON file that includes various instructions to Packer on how to build the image, as well as the bootstrap data that can also be generated independently with pupistry bootstrap. Various variables can be tweaked, we can export out the variables available and see their default settings with:

packer inspect output.json

Screen Shot 2015-06-01 at 00.02.00

You can see here that we must set a VPC ID and Subnet ID – this is because they differ per AWS account and need to be provided. (Side note: technically you can do EC2 classic with Packer and avoid this, but the VPC instance types like t2 are cheaper to run… and we like cheap :-).

The AWS Region and AWS AMI values are interlinked. If you choose to build for a different region, eg us-west-1, you will need to lookup the appropriate AMI ID for that region and change both the aws_ami and aws_region variables when you bake your image. For some reason Amazon chose to make their AMI IDs specific to a particular region which really does make life a bit more difficult than it really needs to be. :-(

The hostname is worth noting. By default we set it to “packer” so you can target your manifests to handle it specifically, but you could make this anything you wanted such as a particular machine or application type. When using the sample puppet repo that ships by default with Pupistry, we have defined specific configuration to run on the Packer built images:

Screen Shot 2015-06-01 at 00.09.08

Assuming we are happy with the defaults, we just have to set the VPC and Subnet IDs to launch the current image in ap-southeast-2.

packer build \
 -var 'aws_vpc_id=vpc-example' \
 -var 'aws_subnet_id=subnet-example' \
 output.json

As soon as we kick off, we can see that Packer has built a machine in our AWS account to use for the image generation process.

Screen Shot 2015-06-01 at 00.13.53

 

It can take up to a minute for the machine to become available via SSH. Once this happens, Packer opens a connection and starts to feed in the bootstrap commands that have been added into the template by Pupistry.

Screen Shot 2015-06-01 at 00.14.23

This process can take a number of minutes – remember you’re having to install all the various OS updates and then packages and dependencies needed to run Puppet and of course Pupistry itself.

Once all the dependencies are done, Pupistry will run and provision the machine with your Puppet manifests and then return the ID of the AMI that has been generated:

Screen Shot 2015-06-01 at 00.31.57

 

We can see that Packer has now terminated our temporary machine:

Screen Shot 2015-06-01 at 00.22.28

And given us a shiny new AMI in return:

Screen Shot 2015-06-01 at 00.34.14

 

We can now use that AMI to launch a new machine and check out what Pupistry did. For convenience, there is a launch button on the AMI page that will build a new machine for the selected AMI, however you can also take the AMI ID and use it in CloudFormation, from the API or from the usual instance creation screen.

Connecting to the newly spun up instance using our fresh AMI, we can see that it has had the Pupistry rules for the packer node applied and we can also set that the daemon is configured and running in the background.

Except that it took less than 1 minute, rather than needing 5+ minutes to do all the usual updates and dependency installation. And there was no risk of a broken repository or package preventing the launch of our machine. If it was an application server, we could have preloaded it and thrown it right into an ELB within 1 minute after it starting up – that’s ideal for autoscaling!

Screen Shot 2015-06-01 at 00.38.28

Packer supports a number of different options and different providers, so don’t be afraid to pull it down and experiment. You can even write your own custom providers if needed.

Sure you could always just write a script that does all the same things as Packer for your cloud provider of choice, but Packer provides a solid framework for doing this stuff in a reliable and reproducible way saving you time and keeping complexity down.

Tagged , , , , , , , | Leave a comment

Easy Lockscreen MacOS

Whilst MacOS is a pretty polished experience, there’s some really simple things that are stupidly hard sometimes such as getting the keybindings to work right for real keyboards or in this case, getting the screen to be lockable without sleeping the computer.

No matter what configuration I set in power management, the only MacOS keyboard combination that does anything for me (Command + Option + Eject/Power/F12) will not only put up the lock screen, but also immediately sleeps the computer, much to the dismay of any background network connections or audio.

One of the issues with MacOS is that for any issue there are several dubious software vendors offering you an app that “fixes” the issue with quality ranging from some excellent utilities all the way to outright dodgy Android/Windows-style crapware addons.

None of these look particularly good. Who the hell wants Android-style swipe unlock on a Mac??

None of these look particularly good. Who the hell wants Android-style swipe unlock on a Mac??

Naturally I’m not keen for some crappy third party app to do something as key as locking my workstation so went looking for the underlying way the screen gets locked. From my trawling I found that the following command executed as a normal user will trigger a sleep of the display, but not the whole machine:

pmset displaysleepnow

Turns out getting MacOS to execute some line of shell is disturbingly easy by using the Automator tool (Available in Applications -> Utilities) and creating a new Service.

Screen Shot 2015-05-26 at 23.47.59

Then add the Run Shell Script action from the Library of actions like below:

Screen Shot 2015-05-26 at 23.47.00

Save it with a logical name like “Lock Screen”. It gets saved into ~/Library/Services/ so in theory should be possible to easily copy it to other machines.

Once saved, your new service will become available to you in System Preferences -> Keyboard -> Shortcuts and will offer you the ability to set a keyboard shortcut.

Screen Shot 2015-05-26 at 23.50.37

And magic, it works. Command + Shift + L is a lot easier in my books than hot corners or clicking stupid menu items. Sadly you don’t have full flexibility of any key, but you should be able to get something that works for you.

 

For reference, here are my other settings windows. First the power management (Energy Saver) settings. I select “Prevent computer from sleeping automatically” to avoid any surprises when sleeping.

Screen Shot 2015-05-27 at 00.14.29

And secondly, your Security & Privacy settings should require a password after sleep/screen saver:

Screen Shot 2015-05-27 at 00.12.07

 

Tested on MacOS 10.10 Yosemite with pretty much a stock OS installation on an iMac 5k – I wouldn’t expect any variation by hardware, but YMMV (Your Mileage May Vary).

Tagged , , , , , | 3 Comments

Setting up and using Pupistry

As mentioned in my previous post, I’ve been working on an application called Pupistry to help make masterless Puppet deployments a lot easier.

If you’re new to Pupistry, AWS, Git and Puppet, I’ve put together this short walk through on how to set up the S3 bucket (and IAM users), the Pupistry application, the Git repo for your Puppet code and building your first server using Pupistry’s bootstrap feature.

If you’re already an established power user of AWS, Git and Puppet, this might still be useful to flick through to see how Pupistry fits into the ecosystem, but a lot of this will be standard stuff for you. More technical details can be found on the application README.

Note that this guide is for Linux or MacOS users. I have no idea how you do this stuff on a Windows machine that lacks a standard unix shell.

 

1. Installation

Firstly we  need to install Pupistry on your computer. As a Ruby application, Pupistry is packaged as a handy Ruby gem and can be installed in the usual fashion.

sudo gem install pupistry
pupistry setup

01-installThe gem installs the application and any dependencies. We run `pupistry setup` in order to generate a template configuration file, but we will still need to edit it with specific settings. We’ll come back to that.

You’ll also need Puppet available on your computer to build the Pupistry artifacts. Install via the OS package manager, or with:

sudo gem install puppet

 

2. Setting up AWS S3 bucket & IAM accounts

We need to use an S3 bucket and IAM accounts with Pupistry. The S3 bucket is essentially a cloud-based object store/file server and the IAM accounts are logins that have tight permissions controls.

It’s a common mistake for new AWS users to use the root IAM account details for everything they do, but given that the IAM details will be present on all your servers, you probably want to have specialised accounts purely for Pupistry.

Firstly, make sure you have a functional installation of  the AWS CLI (the modern python one, not the outdated Java one). Amazon have detailed documentation on how to set it up for various platforms, refer to that for information.

Now you need to create:

  1. An S3 bucket. S3 buckets are like domain names -they have a global namespace across *all* AWS accounts. That means someone might already have a bucket name that you want to use, so you’ll need to choose something unique… and hope.
  2. An IAM account for read-only access which will be used by the servers running Pupistry.
  3. An IAM account for read-write access for your workstation to make changes.

To save you doing this all manually, Pupistry includes a CloudFormation template, which is basically a defined set of instructions for AWS to execute to build infrastructure, in our case, it will do all the above steps for you. :-)

Because of the need for a globally unique name, please replace “unique” with something unique to you.

wget https://raw.githubusercontent.com/jethrocarr/pupistry/master/resources/aws/cfn_pupistry_bucket_and_iam.template

aws cloudformation create-stack \
--capabilities CAPABILITY_IAM \
--template-body file://cfn_pupistry_bucket_and_iam.template \
--stack-name pupistry-resources-unique

Once the create-stack command is issued, you can poll the status of the stack, you need it to be in “CREATE_COMPLETE” state before you can continue.

aws cloudformation describe-stacks --query "Stacks[*].StackStatus" --stack-name pupistry-resources-unique

02-s3-setup-init

 

If something goes wrong and your stack status is an error eg “ROLLBACK”, the most likely cause is that you chose a non-unique bucket name. If you want easy debugging, login to the AWS web console and look at the event details of your stack. Once you determine and address the problem, you’ll need to delete & re-create the stack again.

04-s3-aws-cfn-gui

AWS’s web UI can make debugging CFN a bit easier to read than the CLI tools thanks to colour coding and it not all being in horrible JSON.

 

Once you have a CREATE_COMPLETE stack, you can then get the stack outputs, which tell you what has been built. These outputs we then pretty much copy & paste into pupistry’s configuration file.

aws cloudformation describe-stacks --query "Stacks[*].Outputs[*]" --stack-name pupistry-resources-unique

03-s3-setup-explain

Incase you’re wondering – yes, I have changed the above keys & secrets since doing this demo!! Never share your access and secret keys and it’s best to avoid committing them to any repo, even if private.

Save the output, you’ll need the details shortly when we configure Pupistry.

 

3. Setup your Puppetcode git repository

Optional: You can skip this step if you simply want to try Pupistry using the sample repo, but you’ll have to come back and do this step if you want to make changes to the example manifests.

We use the r10k workflow with Pupistry, which means you’ll need at least one Git repository called the Control Repo.

You’ll probably end up adding many more Git repositories as you grow your Puppet manifests, more information about how the r10rk workflow functions can be found here.

To make life easy, there is a sample repo to go with Pupistry that is a ready-to-go Control Repo for r10k, complete with Puppetfile defining what additional modules to pull in, a manifests/site.pp defining a basic example system and base Hiera configuration.

You can use any Git service, however for this walkthrough, we’ll use Bitbucket since it’s free to setup any number of private repos as their pricing model is on the number of people in a team and is free for under 5 people.

Github’s model of charging per-repo makes the r10k puppet workflow prohibitively expensive, since we need heaps of tiny repos, rather than a few large repos. Which is a shame, since Github has some nice features.

Head to https://bitbucket.org/ and create an account if you don’t already have one. We can use their handy import feature to make a copy of the sample public repo.

Select “Create Repository” and then click the “Import” in the top right corner of the window.

05-bitbucket-create

Now you just need to select “GitHub” as a source with the URL of https://github.com/jethrocarr/pupistry-samplepuppet.git and select a name for your new repo:

06-bitbucket-import

Once the import completes, it will look a bit like this:

07-bitbucket-done

The only computers that need to be able to access this repository is your workstation. The servers themselves never use any of the Git repos, since Pupistry packages up everything it needs into the artifact files.

Finally, if you’re new to Bitbucket, you probably want to import their key into your known hosts file, so Pupistry doesn’t throw errors trying to check out the repo:

ssh-keyscan bitbucket.org >> ~/.ssh/known_hosts

 

4. Configuring Pupistry

At this point we have the AWS S3 bucket, IAM accounts and the Git repo for our control repo in Bitbucket. We can now write the Pupistry configuration file and get started with the tool!

Open ~/.pupistry/settings.yaml with your preferred text editor:

vim ~/.pupistry/settings.yaml

09-config-edit

There are three main sections to configure in the file:

  1. General – We need to define the S3 bucket here. (For our walk though, we are leaving GPG signing disabled, it’s not mandatory and GPG is beyond the scope for this walkthrough):10-config-general
  2. Agent – These settings impact the servers that will be running Pupistry, but you need to set them on your workstation since Pupistry will test them for you and pre-seed the bootstrap data with the settings:11-config-agent
  3. Build – The settings that are used on your workstation to generate artifacts. If you create your own repository in Bitbucket, you need to change the puppetcode variable to the location of your data. If you skipped that step, just leave it on the default sample repo for testing purposes.12-config-use-bitbucket

Make sure you set BOTH the agent and the build section access_key_id and secret_access_key using the output from the CloudFormation build in step 2.

 

5. Building an artifact with Pupistry

Now we have our AWS resources, our control repository and our configuration – we can finally use Pupistry and build some servers!

pupistry build

13-pupistry-build

Since this our first artifact, there won’t be much use to running diff, however as part of diff Pupistry will verify your agent AWS credentials are correct, so it’s worth doing.

pupistry diff

14-pupistry-diff

We can now push our newly built artifact to S3 with:

pupistry push

15-pupistry-push

In regards to the GPG warning – Pupistry interacts with AWS via secure transport and the S3 bucket can only be accessed via authorised accounts, however the weakness is that if someone does manage to break into your account (because you stuck your AWS IAM credentials all over a blog post like a muppet), an attacker could replace the artifacts with malicious ones and exploit your servers.

If you do enable GPG, this becomes impossible, since only signed artifacts will be downloaded and executed by your servers – an invalid artifact will be refused. So it’s a good added security benefit and doesn’t require any special setup other than getting GPG onto your workstation and setting the ID of the private key in the Pupistry configuration file.

We’ve now got a valid artifact. The next step is building our first server with Pupistry!

 

6. Building a server with Pupistry

Because having to install Pupistry and configure it on every server you ever want to build wouldn’t be a lot of fun manually, Pupistry automates this for you and can generate bootstrap scripts for most popular platforms.

These scripts can be run as part of user data on most cloud providers including AWS and Digital Ocean, as well as cut & paste into the root shell of any running server, whether physical, virtual or cloud-based.

The bootstrap process works by:

  1. Using the default OS tools to download and install Pupistry
  2. Write Pupistry’s configuration file and optionally install the GPG public key to verify against.
  3. Runs Pupistry.
  4. Pupistry then pulls down the latest artifact and executes the Puppetcode.
  5. In the case of the sample repo, the Puppetcode includes the puppet-pupistry module. This modules does some clever stuff like setting up a pluginsync equalivent for master-less Puppet and installs a system service for the Pupistry daemon to keep it running in the background – just like the normal Puppet agent! This companion module is strongly recommended for all users.

You can get a list of supported platforms for bootstrap mode with:

pupistry bootstrap

Once you decide which one you’d like to install, you can do:

pupistry bootstrap --template NAME

16-pupistry-bootstrap

Pupistry cleverly fills in all the IAM account details and seeds the configuration file based on the settings defined on your workstation. If you want to change behaviours like disabling the daemon, change it in your build host config file and it will be reflected in the bootstrap file.

 

To test Pupistry you can use any server you want, but this walkthrough shows an example using Digital Ocean which is a very low cost cloud compute provider with a slick interface and much easier learning curve than AWS. You can sign up and use them here, shamelessly clicking on my referrer link so my hosting bill gets paid – but also get yourself $10 credit in the process. Sweetas bru!

Once you have setup/logged into your DigitalOcean account, you need to create a new droplet (their terminology for a VM – AWS uses “EC2 Instance”). It can be named anything you want and any size you want, although this walkthrough is tight and suggests the cheapest example :-)

18-digitalocean-create-droplet-1

 

Now it is possible to just boot the Digital Ocean droplet and then cut & paste the bootstrap script into the running machine, but like most cloud providers Digital Ocean supports a feature called User Data, where a script can be pasted to have it execute when the machine starts up.

19-digitalocean-create-droplet-2

AWS users can get their user data in base64 version as well by calling pupistry bootstrap with the –base64 parameter – handy if you want to copy & paste the user data into other files like CloudFormation stacks. Digital Ocean just takes it in plain text like above.

Make sure you use the right bootstrap data for the right distribution. There are variations between distributions and sometime even between versions, hence various different bootstrap scripts are provided for the major distributions. If you’re using something else/fringe, you might have to do some of your own debugging, so recommend testing with a major distribution first.

20-digitalocean-create-droplet-3

Once you create your droplet, Digital Ocean will go away for 30-60 seconds and build and launch the machine. Once you SSH into it, tail the system log to see the user data executing in the background as the system completes it’s inaugural startup. The bootstrap script echos all commands it’s running and output into syslog for debugging purposes.

21-digitalocean-connect-to-server

 

Watch the log file until you see the initial Puppet run take place. You’ll see Puppet output followed by Notice: Finished catalog run at some stage of the process. You’ll also see the Pupistry daemon has launched and is now running in the background checking for updates every minute.

21-initial-pupistry-run

If you got this far, you’ve just done a complete build and proven that Pupistry can run on your server without interruption – because of the user data feature, you can easily automate machine creation & pupistry run to complete build servers without ever needing to login – we only logged in here to see what was going on!

 

7. Using Pupistry regularly

To make rolling out Puppet changes quick and simply, Pupistry sets up a background daemon job via the puppet-pupistry companion module which installs init config for most distributions for systemd, upstart and sysvinit. You can check the daemon status and log output on systemd-era distributions with:

service pupistry status

21-pupistry-daemon-details

If you want to test changes, then you probably may want to stop the daemon whilst you do your testing. Or you can be *clever* and use branches in your control repo – Pupistry daemon defaults to the master branch.

When testing or not using the daemon, you can run Pupistry manually in the same way that you can run the Puppet agent manually:

pupistry apply

22-pupistry-manual

Play around with some of the commands you can do, for example:

Run and only show what would have been done:

pupistry apply --noop

Apply a specific branch (this will work with the sample repo):

pupistry apply --environment exampleofbranch

To learn more about what commands can be run in apply mode, run:

pupistry help apply

 

 

8. Making a change to your control repo

At this point, you have a fully working Pupistry setup that you can experiment with and try new things out. You will want to check out the repo from bitbucket with:

git clone <repo>

Screen Shot 2015-05-10 at 23.31.02

 

Your first change you might want to make is experimenting with changing some of the examples in your repository and pushing a new artifact:

23-custom-puppetcode-1

 

When Puppet runs, it reads the manifests/site.pp file first for any node configuration. We have a simple default node setup that takes some actions like using some notify resources to display messages to the user. Try changing one of these:

24-custom-puppetcode-2

Make a commit & push the change to Bitbucket, then build a new artifact:

25-custom-puppetcode-3

 

We can now see the diff command in action:

26-custom-puppetcode-4

 

If you’re happy with the changes, you can then push your new artifact to S3 and it will quickly deploy to your servers within the next minute if running the daemon.

27-custom-puppetcode-5

You can also run the Pupistry apply manually on your target server to see the new change:

28-custom-puppetcode-6

At this point you’ve been able to setup AWS, setup Git, setup Pupistry, build a server and push new Puppet manifests to it! You’re ready to begin your exciting adventure into master-less Puppet and automate all the things!

 

9. Cleanup

Hopefully you like Pupistry and are now hooked, but even if you do, you might want to cleanup everything you’ve just created as part of this walkthrough.

First you probably want to destroy your Digital Ocean Droplet so it doesn’t cost you any further money:

29-cleanup-digitialocean

If you want to keep continuing with Pupistry with your new Pupistry Bitbucket control repo and your AWS account you can, but if you want to purge them to clean up and start again:

Delete the BitBucket repo:

30-cleanup-bitbucket

Delete the AWS S3 bucket contents, then tear down the CloudFormation stack to delete the bucket and the users:

31-cleanup-aws

All done – you can re-run this tutorial from clean, or use your newfound knowledge to setup your proper production configuration.

 

Further Information

Hopefully you’ve found this walkthrough (and Pupistry) useful! Before getting started properly on your Pupistry adventure, please do read the full README.md and also my introducing Pupistry blog post.

 

Pupistry is a very new application, so if you find bugs please file an issue in the tracker, it’s also worth checking the tracker for any other known issues with Pupistry before getting started with it in production.

Pull requests for improved documentation, bug fixes or new features are always welcome.

If you are stuck and need support, please file it via an issue in the tracker. If your issue relates *directly* to a step in this tutorial, then you are welcome to post a comment below. I get too many emails, so please don’t email me asking for support on an issue as I’ll probably just discard it.

You may also find the following useful if you’re new to Puppet:

Remember that Pupistry advocates the use of masterless Puppet techniques which isn’t really properly supported by Puppetlabs, however generally Puppet modules will work just fine in master-less as well as master-full environments.

Puppet master is pretty standard, whereas Puppet masterless implementations differ all over the place since there’s no “proper” way of doing it. Pupistry hopefully fills this gap and will become a defacto standard for masterless over time.

 

 

Tagged , , , , , , , , , , | 1 Comment

Introducing Pupistry

I’ve recently been working to migrate my personal infrastructure from a very conventional and ageing 8 year old colocation server to a new cloud-based approach.

As part of this migration I’m simplifying what I have down to the fewest possible services and offloading a number of them to best-of-breed cloud SaaS providers.

Of course I’m still going to have a few servers for running various applications where it makes the most sense, but ideally it will only be a handful of small virtual machines and a bunch of development machines that I can spin up on demand using cloud providers like AWS or Digital Ocean, only paying for what I use.

 

The Puppet Master Problem

To make this manageable I needed to use a configuration management system such as Puppet to allow the whole build process of new servers to be automated (and fast!). But running Puppet goes against my plan of as-simple-as-possible as it means running another server (the Puppet master). I could have gone for something like Ansible, but I dislike the agent-less approach and prefer to have a proper agent and being able to build boxes automatically such as when using autoscaling.

So I decided to use Puppet masterless. It’s completely possible to run Puppet against local manifest files and have it apply them, but there’s the annoying issue of how to get Puppet manifests to servers in the first place…. That tends to be left as an exercise to the reader and there’s various collections of hacks floating around on the web and major organisations seem to grow their own homespun tooling to address it.

Just getting a well functioning Puppet masterless setup took far longer than desired and it seems silly given that everyone doing Puppet masterless is going to have to do the same steps over and over again.

User-data is another case of stupidity with every organisation writing their own variation of what is basically the same thing – some lines of bash to get a newly launched Linux instance from nothingness to running Puppet and applying the manifests for that organisation. There’s got to be a better way.

 

The blessing and challenges of r10k

It gets even more complex when you take the use of r10k into account. r10k is a Puppet workflow solution that makes it easy to include various upstream Puppet modules and pin them to specific versions. It supports branches, so you can do clever things like tell one server to apply a specific new branch to test a change you’ve made before rolling it out to all your servers. In short, it’s fantastic and if you’re not using it with Puppet… you should be.

However using r10k does mean you need access to all the git repositories that are being included in your Puppetfile. This is generally dealt with by having the Puppet master run r10k and download all the git repos using a deployer key that grants it access to the repositories.

But this doesn’t work so well when you have to setup deployer access keys for every machine to be able to read every one of your git repositories. And if a machine is ever compromised, it needs to be changed for every repo and every server again which is hardly ideal.

r10k’s approach of allowing you to assemble various third party Puppet modules into a (hopefully) coherent collection of manifests is very powerful – grab modules from the Puppet forge, from Github or from some other third party, r10k doesn’t care it makes it all work.

But this has the major failing of essentially limiting your security to the trustworthyness of all the third parties you select.

In some cases the author is relatively unknown and could suddenly decide to start including malicious content, or in other cases the security of the platform providing the modules is at risk (eg Puppetforge doesn’t require any two-factor auth for module authors) and a malicious attacker could attack the platform in order to compromise thousands of machines.

Some organisations fix this by still using r10k but always forking any third party modules before using them, but this has the downside of increased manual overhead to regularly check for new updates to the forked repos and pulling them down. It’s worth it for a big enterprise, but not worth the hassle for my few personal systems.

The other issue aside from security is that if any one of these third party repos ever fails to download (eg repo was deleted), your server would fail to build. Nobody wants to find that someone chose to delete the GitHub repo you rely on just minutes before your production host autoscaled and failed to startup. :-(

 

 

Pupistry – the solution?

I wanted to fix the lack of a consistent robust approach to doing masterless Puppet and provide a good way to allow r10k to be used with masterless Puppet and so in my limited spare time over the past month I’ve been working on Pupistry. (Pupistry? puppet + artistry == Pupistry! Hopefully my solution is better than my naming “genius”…)

Pupistry is a solution for implementing reliable and secure masterless puppet deployments by taking Puppet modules assembled by r10k and generating compressed and signed archives for distribution to the masterless servers.

Pupistry builds on the functionality offered by the r10k workflow but rather than requiring the implementing of site-specific custom bootstrap and custom workflow mechanisms, Pupistry executes r10k, assembles the combined modules and then generates a compressed artifact file. It then optionally signs the artifact with GPG and finally uploads it into an Amazon S3 bucket along with a manifest file.

The masterless Puppet machines then runs Pupistry which checks for a new version of the manifest file. If there is, it downloads the new artifact and does an optional GPG validation before applying it and running Puppet. Pupistry ships with a daemon which means you can get the same convenience of  a standard Puppet master & agent setup and don’t need dodgy cronjobs everywhere.

To make life even easier, Pupistry will even spit out bootstrap files for your platform which sets up each server from scratch to install, configure and run Pupistry, so you don’t need to write line after line of poorly tested bash code to get your machines online.

It’s also FAST. It can check for a new manifest in under a second, much faster than Puppet master or r10k being run directly on the masterless server.

Because Pupistry is artifact based, you can be sure your servers will always build since all the Puppetcode is packaged up which is great for autoscaling – although you still want to use a tool like Packer to create an OS image with Pupistry pre-loaded to remove dependency and risk of Rubygems or a newer version of Pupistry failing.

 

Try it!

https://github.com/jethrocarr/pupistry

If this sounds up your street, please take a look at the documentation on the Github page above and also the introduction tutorial I’ve written on this blog to see what Pupistry can do and how to get started with it.

Pupistry is naturally brand new and at MVP stage, so if you find bugs please file an issue in the tracker. It’s also worth checking the tracker for any other known issues with Pupistry before getting started with it in production (because you’re racing to put this brand new unproven app into production right?).

Pull requests for improved documentation, bug fixes or new features are always welcome, as is beer. :-)

I intend to keep developing this for myself as it solves my masterless Puppet needs really nicely, but I’d love to see it become a more popular solution that others are using instead of spinning some home grown weirdness again and again.

I’ve put some time into making it easy to use (I hope) and also written bootstrap scripts for most popular Linux distributions and FreeBSD, but I’d love feedback good & bad. If you’re using Pupistry and love it, let me know! If you tried Pupistry but it had some limitation/issue that prevented you from adopting it, let me know what it was, I might be able to help. Better yet, if you find a blocker to using it, fix it and send me a pull request. :-)

Tagged , , , , , , , | Leave a comment

Puppet 3 & 4 on FreeBSD

I’ve been playing with FreeBSD recently and wanted to run Puppet on it to provide my machines with configuration.

Unfortunately I faced the following error when I first ran Puppet with a simple Puppet manifest that tries to install NTP via usage of the puppetlabs/ntp module:

Warning: Found multiple default providers for package: pip, gem; using pip
Error: Could not set 'present' on ensure: Could not locate the pip command. at modules/ntp/manifests/install.pp

Error: /Stage[main]/Ntp::Install/Package[net/ntp]/ensure: change from absent to present failed: Could not set 'present' on ensure: Could not locate the pip command. at modules/ntp/manifests/install.pp

This collection of errors is pretty interesting – we can see that Puppet seems to understand that it’s running on FreeBSD and that the package is called “net/ntp” but it’s trying to install it with pip, which is the Python package manager :-/

Turns out that Puppet versions older than and including 4.0.0 [1] lack support for PkgNg package manager which is used in FreeBSD 10.x and since it doesn’t know how to use it, defaults to trying one of the only two providers left over – pip & gem. Not really the smartest approach…

There’s two ways to fix right now:

  1. An independently packaged provider for PkgNg was written and is available at xaque208/puppet-pkgng. This can be included into any existing Puppet 3/4 deployment and you can define it as the default for all packages globally.
  2. Pull the commit from Puppet that provides PkgNg support (thanks to author of the above project). The downside is that you then have to build your own Gem and install it.

Longer term you can either:

  1. Wait for Puppet to release a stable version that actually works. I’m guessing it will be in 4.0.0 +1 (so 4.0.1?) but I’m not familiar with their approach to releasing new feature additions. See [1] for more information.
  2. Hope that the FreeBSD developers can backport the commit that adds PkgNg support to Puppet4. I’ve actually started chatting with one of the FreeBSD developers who’s been kind enough to have a look at upgrading their package from Puppet3 to Puppet4, but it’s not as simple as it should be thanks to the terrible way Puppetlabs has packaged Puppet4 (lots of hard coded /opt paths everywhere) so it may take some time.

 

Additionally there is an issue if you choose to install Puppet 4 via RubyGems and when you attempt to run it, you will get the following error:

Error: Cannot create /opt/puppetlabs/puppet/cache; parent directory /opt/puppetlabs/puppet does not exist
Error: /File[/var/log/puppetlabs/puppet]/ensure: change from absent to directory failed: Cannot create /var/log/puppetlabs/puppet; parent directory /var/log/puppetlabs does not exist

This is because the Gem is not properly packaged and doesn’t run nicely with FreeBSD’s filesystem structure. You can hack in a “fix” with:

mkdir -p /opt/puppetlabs/puppet/cache
mkdir -p /var/log/puppetlabs/puppet

It’s far from the right way of properly fixing this, the best solution if you value OS purity right now is to stick with the FreeBSD packaged Puppet 3 version and add the xaque208/puppet-pkgng module.

 

[1] So at time of writing, *no* stable Puppet version supports PkgNg yet, but given that it’s been merged into the master git branch for Puppet, I have to *presume* that it’s going into 4.0.1… I hope :-/

Tagged , , , , | 3 Comments

FreeBSD in the cloud

This weekend I was playing around with FreeBSD in order to add support to Pupistry. Although I generally use Linux exclusively, it’s fun to play around with other platforms now and then, bit like going on vacation. Plus building support for other platforms ensures that I’m writing code that’s more portable.

FreeBSD is probably the most popular BSD in use and it’s the only one available for download from the Amazon Web Services (AWS) Marketplace and as a supported platform from Digital Ocean alongside their Linux offerings.

However as popular as FreeBSD is, it pales in comparison to Linux, which means that it doesn’t get as much love and things don’t work quite as seamlessly with these cloud providers. In my process of testing FreeBSD with both providers I ran into some interesting feature differences and annoyances.

 

FreeBSD on Digital Ocean

I started with Digital Ocean first, love them since they’re a nice simple, cheap cloud provider for personal stuff – not much need for the AWS enterprise feature set when I’m building personal machines and paying the price of a coffee for a month of compute sure is nice.

They provide a FreeBSD 10.1 image via the usual droplet creation screen, I have to give Digital Ocean credit for such a nice clean simple interface – limiting user selection does make it much more approachable for people, something Apple always understood with their products.

Screen Shot 2015-04-18 at 23.30.50

As always Digital Ocean is pretty speedy, bringing up a machine within a minute or so. Once ready, login as the freebsd user and you can just sudo to root.

Digital Ocean provides a pretty recent image with pkg already installed and ready to go, although you’ll want to run the update process to get the latest patches. You need to login initially as the freebsd user and then can sudo to acquire root powers.

Over all it’s great – so naturally there is a catch. Digital Ocean doesn’t yet support user data with their droplets. So whilst you can fill in the user data field, it won’t actually get executed.

This is pretty annoying for anyone wanting to automate large number of machines, since it now means you have to SSH to each of them to get them provisioned. I’ve raised a question on their community forum around this issue, but I wouldn’t expect a quick fix since the upstream bsd-cloudinit project they use hasn’t implemented support yet either.

It’s not going to be an end-of-the-world for most people, but it could be barrier if you’re wanting to roll out a fleet of BSD boxen.

The best feature from Digital Ocean is actually their documentation – with the launch of FreeBSD on their platform, they’ve produced some excellent tutorials and guides to their platform which can be found here and are useful to both Linux gurus and noobs alike.

Finally their native IPv6 support extends to FreeBSD, so your machines can join the internet of the 21st century from day one.

 

FreeBSD on Amazon Web Services (AWS)

Next I spun up an instance in Amazon Web Services (AWS) which is the granddaddy of cloud providers and provides an impressive array of functionality, although this comes at a cost premium over Digital Ocean’s tight pricing.

It’s actually been the first time in a long time that I’ve built a machine via the AWS web console, normally for work we just build all of our systems via Cloud Formation and it was an interesting experience to see the usability difference of AWS’s setup page vs that of Digital Ocean’s.

The fact that the launch wizard has 7 different screens says a lot and I suspect AWS is at risk of having it’s consumer user base eaten by the likes of Linode and Digital Ocean – but when a consumer user is paying $5.00 a month and an enterprise customer pays $300,000 a month, I suspect AWS isn’t going to be too worried.

Launching a FreeBSD instance is not really any different to that of a Linux one, you just need to search for “freebsd” in the AWS Market Place to find the AMI and launch as normal.

Screen Shot 2015-04-18 at 23.43.18

 

Once launched, things get more interesting. Digital Ocean’s FreeBSD instance came up in around 1 minute which is standard for their systems – but AWS took a whopping 8-10mins to launch the AMI to the level where I could login via SSH!

Digging into the startup log reveals why – it seems the AWS AMI (Amazon’s machine images/templates) for FreeBSD launches the instance, then runs a prolonged upgrade task (freebsd-update fetch install), before doing a subsequent reboot and finally starting SSH.

Whilst I appreciate the good default security posture this provides, there’s a few issues with it:

  1. It differs from most other AWS images which deal with patching by having new images built semi-frequently and leaving the patching in-between up to the admin’s choice.
  2. During the whole process, the admin can’t login which causes some confusion. I initially assumed the AMI images were broken after reviewing my security groups and seeing no reason why I shouldn’t be able to login immediately.
  3. You can’t trust the AMI images to be a solid unchanging base, which means you need to be vary wary if doing autoscaling. Not only is 10mins a bit too slow for autoscaling, having the potential risk of it not coming up due to app changes in the latest update is always something to watch out for. If doing autoscaling with these images, you’ll need to consider
  4. It caused me no end of frustration when trying to test user data since I had to wait 10mins each time to get a confirmation of my test!

The last point brings me to user data – the good news is that Amazon correctly supports user data for FreeBSD machines, so you can paste in your tcsh script (not bash remember!) and it will get invoked at launch time.

The downside is that the user data handling of FreeBSD is a lot more fragile than Linux images. Generally with Linux, the OS boots (including SSH) and then runs the user data. If the user data breaks or hangs or does anything other than expected, you can still login and debug. Whereas since FreeBSD runs the user data before starting up SSH, if something goes wrong you have no way to easily login and debug. And given the differences between tcsh and bash plus annoying commands that default to expecting user input on non-interactive ptys, changes are you’ll have more than one attempt that results in a machine getting stuck at launch.

The ultimate fix is that you’ll probably have to use Packer if using FreeBSD in any serious way on AWS to get the startup performance to an acceptable level.

Finally remember that on AWS, you need to login as the ec2-user and then su –  to become root.

 

Which one?

If you’re interested in FreeBSD and want to pick a provider to play around with, the choice seems pretty simple to me – Digital Ocean. They’re got the better pricing (~ $5/month vs $15/month) and their ridiculously simple dashboard coupled with the excellent documentation they’ve assembled makes it really attractive for anyone new to the *.nix or cloud space. Plus they’ve bothered to invest in IPv6 which I appreciate.

However if you’re doing business/enterprise systems and want user data, autoscaling or the benefit of automating entire stacks with Cloud Formation, then you will probably find AWS the more attractive offering purely due to the additional functionality offered by that platform. Just be prepared to spend a bit of time baking your own AMI to allow you to skip the overhead of having to wait for updates to apply for each instance you bring up.

Neither provider has got their FreeBSD experience to be quite as slick as that of their Linux offerings, however hopefully they improve on these deficiencies over time –  there’s not much needed to get the experience up to the same level as Linux distributions and it’s nice having a different type of unix to play with for a change.

Tagged , , , , , , | 1 Comment

Puppet facts, json and max nesting

I use Puppet for both business and pleasure and my work often involves writing custom Puppet facts to expose various bits of information.

Recently a fact I had written that worked on the development machines started throwing errors when run on our production machines:

Could not retrieve jethros_awesome_fact: nesting of 20 is too deep

 After digging around it turns out this relates to how many nested levels are inside JSON responses. By default Ruby enforces a maximum level of nesting, I guess to avoid parsing bad JSON or JSON deliberately structured to cause infinite looping.

My fact involved pulling JSON from a local application API and then providing various bits of data from the feed. In the development environments this worked without an issue, but the production systems returned a lot more information via the API feed and broke it.

The fix is pretty easy, just need to add the :max_nesting => false parameter when parsing the JSON – or set it to a different number of levels if you prefer that approach.

json         = JSON.parse(response.body, :max_nesting => false)
Tagged , , , , , , | Leave a comment

AEM/CQ 5.6.1 package file parameter missing

Having the joy and “pleasure” of patching a bunch of AEM/CQ 5.6.1 instances at work recently, I ran into a weird issue where the hotfixes would refuse to install on a particular development machine.

Any attempt to install the packages would result in the following error, regardless of whether or not the upload is done via the CRX PackMgr UI, or via Curl.

{"success":false,"msg":"package file parameter missing"}

I would have assumed that it was a bad package if it wasn’t for the fact that it worked OK on other machines, so started looking for other factors.

$ df -h /tmp
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvdb        30G   29G   88K 100% /tmp

Turns out, the /tmp volume was full on this machine thanks to a lovely collection of heap dumps being stored there. This leads to CQ being unable to write to it’s configured temporary location (which might differ on your install, check for -Djava.io.tmpdir) which is used to store the file between uploading and installing.

Clearing the tmp volume resolves the issue. You might also get this error if your tmp volume can’t be written for any other reason, such as a permissions issue or broken filesystem/mount.

Tagged , , , | 4 Comments

Ubiquiti Unifi UAP-AC in-home performance

Having completed the most important move-in task of wiring the house for ethernet and installing the Ubiquiti Unfi UAP-AC access point, I decided to run some benchmarks to see what sort of performance I could actually get.

Unifi UAP-AC just chilling out on the roof.

Unifi UAP-AC just chilling out on the roof.

Manufacturers always quote theoretical figures which can make buying hard, but even when relying on public third party tests, a particular device will perform differently in your own specific situation in a house with walls, interference, crappy consumer devices and other causes for them not to hit the theoretical maximum performance.

Ubiquiti claim that the UAP-AC can do a theoretical maximum of 1300 Mbits in 5Ghz and 450 Mbits in 2.4Ghz. Unfortunately public benchmarks for my testing laptop (Macbook Pro Retina 2013) show it capping out at 615 Mbits, so it’s not going to possible to truly test what’s possible with the UAP-AC past that speed.

Rather than a benchmark of the absolute maximum speed possible consider this benchmark a general consumer use case test in a suburban setting to see what the real world performance is going to be like with a common consumer client device.

 

Environment

The testing was done with the following equipment:

  • iperf2 as the network performance tool.
  • Ubiquity Unfi UAP-AC access point (duh)
  • Apple Macbook Pro Retina 15″ late 2013 (Running MacOS Mavericks).
  • Lenovo Thinkpad x201i laptop running Linux and connected via GigE to run the iperf server.
  • Mikrotik CRS226-24G-2S+RM switch, acting as a dumb switch to connect the AP and the Lenovo Thinkpad running iperf2.
  • House

The latest firmware was applied to the access point, all other settings like power modes are on their default values.

The Macbook Pro should be pretty typical of most high end laptops currently available and supports 802.11ac and both 2.4Ghz and 5Ghz bands, sadly I didn’t have any other 802.11ac client devices that I could run tests with – would be very keen to test with a client device that can deliver closer to the theoretical 1300 Mbits, as well as being able to test with multiple devices concurrently.

Because of where I am, there were no other WiFI access points detected (yay for elderly neighbours) so the testing was free from interference by other networks. Other sources of potential interference such as microwaves and wireless audio streaming was powered down.

Whilst they were not particular active, or being used or testing, the following other devices were also connected to the access point during the tests and could of course influence how well the AP performs.

  • Apple Iphone 5s (802.11n 2.4Ghz and 5Ghz capable)
  • Samsung Galaxy Note 2 (802.11n 2.4Ghz and 5Ghz capable)

The house itself also requires note – it’s an old single-story Wellington home with walls containing about 20mm of solid hardwood plus a layer of plasterboard. They’re thick walls and utterly punishing to 5Ghz WiFI which generally doesn’t like solid objects at the best of times.

Finally the AP is roof mounted in the rough middle of the house in the hallway – the downside is that every room gets it’s signal after passing through at least one wall, but it gives me all of house (and some garden) coverage from a single AP.

 

Test Results

I did two distinct tests – one where the client device transmits, then receives and another where the client device transmits and receives concurrently. Both tests were done using UDP.

Respectively, the following were the commands used:

  • iperf -u -c SERVERIP –tradeoff -b 1000M
  • iperf -u -c SERVERIP –dualtest -b 1000M

I performed each test a total of 3 times and the results provided are the averages result of those 3 tests. I’ve also included a column showing the general connection type that was established for most of the testing, however in some rooms it’s possible the laptop swapped between 5Ghz and 2.4Ghz throughout testing which would mess with results.

6 different locations were used:

  1. Wired GigE connection (For control purposes to verify the iperf setup and switch were not impacting performance).
  2. Hallway – This test was holding the laptop and standing directly underneath the roof mounted AP.
  3. Master Bedroom – 1 wall
  4. Office – 1 wall, but signal potentially impacted by house details.
  5. Lounge – 1-2 walls depending where the signal goes.
  6. Dining Room – 3 walls between AP and room.

 

Individual Tests Dual Tests Reported Mode
Location Transmit Receive Transmit Receive
WiredGigE 745.00 803.00 null [1] null [1]
Hallway 565.67 570.33 120.43 379.00 802.11ac 5ghz 64%
Master Bedroom 127.00 394.00 84.40 92.47 802.11ac 5ghz 12%
Office 150.00 341.33 93.37 109.13 802.11ac 5ghz 15%
Lounge 67.47 153.00 32.87 62.30 Mixed [2]
Dining Room 55.33 116.00 33.10 45.30 802.11n 2.4Ghz 47%

The locations in the results are ordered closest-furthest, so these results make sense. The fact that receive performance is often better than transmit makes sense when considering that the AP probably has far better antennas and transmission power than the laptop.

Some specific notes:

  • [1] The dual test of Wired GigE just wouldn’t work – I think this was a bug in iperf2 and it wasn’t vital to solve for the testing, hence no metrics.
  • [2] The laptop’s reported mode when in the lounge would vary. On some tests the AP would report 802.11n 2.4Ghz 50% and other times it would report 802.11ac 5Ghz 0.0% (sic), so I think the laptop was struggling to maintain 5ghz, yet could see it strongly enough to try to switch to it.

 

 

Conclusion?

So am I happy with my purchase? The answer is yes – some other access points might have scored higher performance at range but considering the punishment the walls of the house are delivering to the AP, I’m pretty happy with the speeds I’m getting in the rooms.

Considering my WAN connection is 40mbits down / 10mbits up maximum, even the worst location is still able to max the connection and if I need more capacity to push to the file server, I’ll go use one of the stronger signal rooms or just plug into one of the ethernet ports in every room of the house.

I also intentionally brought Ubiquiti’s hardware since in future it’s likely that I’ll add additional access points around the house to expand the coverage in the areas we end up spending the most time in. The software provided Ubiquiti makes configuration and management of multiple access points very easy which is a big advantage.

At just under $500 NZD, these units aren’t cheap, but compared to the time/cost of running ethernet everywhere, it wouldn’t be a bad investment to install a 3-pack spread around the house to provide maximum coverage if you really want to get as fast as possible without the pain of installing ethernet (well you still have to run it through the attic… but it’s easier than running inside the walls). You’re not going to get GigE speeds, but you should be able to get 500-600 Mbits within the same room as the AP.

Tagged , , , , , , , , , | 8 Comments