Monthly Archives: February 2013

NamedManager 1.5.1

I’ve pushed a new release of NamedManager version 1.5.1, this release is a minor bug fix release providing:

  1. Bug fix for handling of TXT records, where extra slashes would be entered into the record due to an input validator bug.
  2. The Bind configuration writer now runs the Bind-supplied validators for configuration and DNS zone files and refuses to reload Bind without them passing

The first change is naturally important if you’re using TXT records as it does fix a serious issue with the handling of TXT records (no security problems, but corrupted zonefiles would result at times).

Even if you’re not using TXT records, the second change is worth upgrading to as it makes the Bind configuration generator much more robust and prevents any potential future bugs from ever feeding Bind a bad zonefile.

Pre-1.5.1, we relied on Bind’s reload process to validate the files, however this suffers an issue where the error might not be reported back to the user and they would only discover the issue next time Bind restarts. This changes prevents a new zonefile from being loaded into place until the validator passes it, so the worst case is your DNS just refuses to accept changes, whilst logging loudly in the web interface back to you. :-)

If you upgrade, take advantage of this feature, by adding the following to /etc/namedmanager/config-bind.php or wherever you have installed your Bind component configuration file to:

$config["bind"]["verify_zone"]    = "/usr/sbin/named-checkzone";
$config["bind"]["verify_config"]  = "/usr/sbin/named-checkconf";

NamedManager 1.5.1 can be found at the project page or in my packaged repositories.

Updated Repositories

I’ve gone and updated my GNU/Linux repositories with a new home page – some of you may have been using this under my previous Amberdms branding, but it’s more appropriate that it be done under my own name these days and have it’s own special subdomain.

I want to unify the branding of a bit more of the stuff I have out there on the internet and also make sure I’m exposing it in a way that makes it easy for people to find and use, so I’m going through a process of improving site templates, linking between places and improving documentation/wording with the perspective of viewing as an outside user.

CSS3 shinyness! And it even mostly works in IE.

Been playing with new HTML5/CSS3 functionality for this site, have to say, it’s pretty awesome.

You can check out the new page at, I’ve tried to make it as easy as possible to add my repositories to your servers -I’ll be refining this a little more in coming weeks, such as adding a decent package search function to the site to make it easier to grab some of the goodies hidden away in distribution directories.

I’m currently providing packages for RHEL & clones, Debian and Ubuntu. Whilst my RHEL repos are quite sizable now, the Debian & Ubuntu repositories are much sparser, so I’m going to make an effort to bring them to a level where they at least have all my public software (see available as well tested packages for current Debian Stable and Ubuntu LTS releases.

There’s some older stuff archived on the server if you go hunting as well, such as Fedora and ancient RHEL version packages, but I’m keeping them in the background for archival purposes only.

And yes, all packages are signed with my Amberdms/Jethro Carr GPG signing key. You should never be using any repositories without GPG signed packages, since they’re ideal attack vectors to use to install malicious content with a man-in-the-middle attack otherwise.

A reminisce of Auckland

During the year (late 2011-late 2012) I spent in Auckland after moving up there to be with Lisa, I collected a number of good memories and pictures from the region that I want to share to showcase what I consider to be the best bits of Auckland in my personal experience.

Waiting at the Devonport ferry crossing

Waiting at the Devonport ferry crossing

Auckland and I certainly have a love-hate relationship, it’s easy to be negative about Auckland with it’s transportation chaos, massive sprawling size and huge (by NZ standards) population, but at the same time it faces challenges that no other NZ city faces and serves up it’s own slice of awesomeness in the face of these issues.

If you can survive the sprawling road network and the urge to murder all other drivers, Auckland isn't actually all that bad. :-)

If you can survive the sprawling road network and the urge to murder all other drivers, Auckland isn’t actually all that bad. :-)

An example of Auckland transport policy.

An example of Auckland’s transport policy.

I personally loved the Auckland region from an explorer point of view – being in a new city, especially one with lots of island and other areas I’ve never been to before was a really exciting change. Wellington has it’s collection of interesting places of course, but you always know your home city too well for it to be surprising and new after a while.

My personal highlights of my adventures in Auckland would have to be my visit to Rangitoto Island, regular Takapuna to Devonport walks and my wanders along Takapuna Beach.

At times the warmer climate of Auckland, whilst a constant source of annoyance and suffering when working from home, also served up some beautiful swimming weather during summer in which I was able to visit the beach and swim in the sea just enjoying life.

Auckland does a remarkable job of being both ugly and beautiful at the same time – sometimes you’re stuck in a bland generic corporate business park, yet an hour later you can be on the harbour bridge looking over the city whilst a ship cruises under you, up in the Sky Tower cabling servers or getting up early and exploring the near empty city as the sun rises.

Viewing Auckland CBD from up on Mt Eden.

Viewing Auckland CBD from up on Mt Eden.

Takapuna beach, a summer gem. Plus there's amazing gelato right on the beach.

Takapuna beach, a summer gem. Plus there’s amazing gelato right on the beach.

One night in Mirangi Bay

One night in Mirangi Bay

My time in Auckland was particularly people orientated. I had moved up to Auckland to be with Lisa, but at the same time I missed my Wellington friends and family terribly leading to a really weird contrast where I was happy to be with her, but sad to be away from those who have played such a big part in my life.

On the plus side, my time in Auckland strengthened some existing friendships and created some new ones, which I’m very thankful to have. I have many great memories of good times spent over bottles of wine or delicious gin, going for a swim in the beach or flooding apartments during my time there. :-)

Lisa and I outside our apartment building.

Lisa and I outside our apartment building.

My dear friend @pikelet!

Partners in crime with @pikelet

In beer we trust.

Over beer we plot how to unleash our awesomeness on the world.

I was also fortunate enough to take part in Auckland’s Thursday Night Curry (TNC), a collection of great geeks meeting at a different curry venue somewhere in Auckland city or suburbs every fortnight. TNC features a range of very smart and very awesome people and is something I really miss having left Auckland. Plus the curry was excellent. ;-)

Delicious, delicious curry!

Delicious, delicious curry!

The TNC crowd and my other friends helped me seek out some of the good food locations in Auckland – Pikelet even managed to introduce me to some decent coffee in Auckland’s CBD a task I feared impossible after Wellington’s high standard in caffeinated deliciousness.

Delicious delicious coffee with chocolatey addition.

Delicious delicious coffee with chocolatey addition.

I even discovered some great food places such as Sunflower Vegetarian Restaurant, as well as some amazing local breweries and pubs including Britomart Brewery, Galbraith’s and Brew on Quay.

Hidden vegetarian gem - Sunflower vegetarian resturant.

Hidden vegetarian gem – Sunflower vegetarian restaurant.

Whilst living in Auckland, I was also able to get out and around the city and experience different parts of it, both whilst working but also whilst exploring on a personal level.

Auckland is New Zealand’s economic power house and most large companies base their head offices here, the unfortunate side effect has been that the city keeps growing and growing as more people base themselves there for work opportunities and there’s easily a risk of the city becoming a very corporate and developing an all-business, no-play feel.

Whilst some will argue that Auckland is already a soulless corporate city, I argue that whilst it does have it’s downsides, it has more than enough great features to make up for them.

Whilst a city like Wellington is generally great all round, Auckland has a contrasting mix of horrible problems yet amazing areas to visit and great places to go.

I even found greenery inside Auckland's CBD!

I even found greenery inside Auckland’s CBD!

Marshlands around Auckland

Marshlands around Auckland

Touches of old and new throughout Auckland

Generally speaking, Auckland is a young city, but there’s still a lot of older buildings amongst the glass and steel towers.

Suburban Auckland

Suburban Auckland (view from Takapuna down Fred Thomas drive).

The waterfront has to be one of Auckland's more redeeming features.

The waterfront has to be one of Auckland’s more redeeming features.

Whilst I don’t regret leaving Auckland to spend time over in AU, and as a devout Wellingtonian, I must admit that I do have a special place for the city and I’d happily live in it again if it wasn’t for the strong pull of dear friends and family in Wellington.

I suspect Auckland is somewhere I will consider visiting semi-frequently, even if it’s just for the beach visits and warm weather during summer, the region with it’s harbour and islands is one of the best feature and a strong pull for an excuse to visit the city.

ip6tables: ipv6-icmp vs icmp

I run a fully dual stacked IPv6+IPv4 network on my servers, VPNs and home network – part of this is that I get to discover interesting new first-adopter pains with living in the future (like Networkmanager/Kernel bugs, Munin being stupid, CIFS being failtastic and providers still stuck in the IPv4 only 1980s).

My laptop was experiencing frustrating issues where it was unable to load content from some IPv6 enabled website providers. In my specific case, I was having lots of issues with page loads from WordPress and Gravatar timing out when connecting to them via IPv6, but no issues when using IPv4.

I noticed that I was still able to ping6 the domains in question and telnet to port 80 successfully, which eliminates basic connectivity issues from being the cause. Issues like this where connectivity tests succeed, but actual connections fail, can be a symptom of MTU discovery issues which are a particularly annoying networking glitch to experience.

If you’re behind a WAN link such as ADSL, you’re particularly likely to be affected since ADSL and PPP overheads drop the size of the packets which can be used – in my case, I can only send a maximum of 1460 byte packets, whereas the ethernet default that my laptop will use is 1500 bytes.

In a properly functioning network, your computer will try and send 1500 byte packets to the internet, but the router which has the 1460 byte uplink to your ISP will refuse the packet and advise your computer that this packet is too large and that it needs to break it into smaller ones and try again. This happens transparently and is a standard feature of networking.

In a fucked up improperly functioning network, your computer will try and send the 1500 byte packet to the internet, but no notification advising the correct MTU size is returned or received. In this case your computer keeps trying to re-send the packet until a timeout occurs – from your computer’s perspective, the remote host is unreachable.

This MTU notification is performed by the ICMP protocol, which is more commonly but incorrectly known as being “ping” [whilst ping is one of the functions performed by ICMP, there are many other it’s responsible for, including MTU discovery and connection refused messages].

It’s not uncommon for MTU to be broken – I’ve seen too many system and network administrators block ICMP entirely in their firewalls “for security”, not realising that there’s a lot in ICMP that’s needed for proper operation of a network. What makes the problem particularly bad, is that it’s inconsistent and won’t necessarily impact all users, which leads to those administrators disregarding it as not being an issue with their infrastructure and even blaming the user.

Sometimes the breakage might not even be in a network you or the remote endpoint control – if there’s a router somewhere between you and the website you’re trying to access which has a smaller MTU size and blocks ICMP, you may never receive an MTU notification and you lose the ability to connect to the remote site.

At other times, the issue might be more embarrassing – is your computer itself refusing the helpful MTU notifications being supplied to it by the routers/systems it’s attempting to talk with?

I’m pretty comfortable with iptables and ip6tables, Linux’s IPv4 and IPv6 firewall implementations and use them for locking down servers, laptops as well as conducting all sorts of funky hacks that would horrify even the most bitter drugged up sysadmin.

However even I still make mistakes from time to time – and in my case, I had made a big mistake with the ICMP firewalling configuration that made me the architect of my own misfortune.

On my laptop, my IPv4 firewall looks something like below:

iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A INPUT -p icmp -j ACCEPT
iptables -A INPUT -j REJECT --reject-with icmp-host-prohibited
  • We want to trust anything from ourselves (duh) with -i lo -j ACCEPT.
  • We allow any established/related packets being sent in response to whatever connections have been established by the laptop, such as returned traffic for an HTTP connection – failure to define that will lead to a very unhappy internet experience.
  • We trust all ICMP traffic – if you want to be pedantic you can block select traffic, or limit the rate you receive it to avoid flood attacks, but a flood attack on Ethernet against my laptop isn’t going to be particularly effective for anyone.
  • Finally refuse any unknown incoming traffic and send an ICMP response so the sender knows it’s being refused, rather than just dropped.

My IPv6 firewall looked very similar:

ip6tables -A INPUT -i lo -j ACCEPT
ip6tables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
ip6tables -A INPUT -p icmp -j ACCEPT
ip6tables -A INPUT -j REJECT --reject-with icmp6-adm-prohibited

It’s effectively exactly the same as the IPv4 one, with some differences to reflect various differences in nature between IPv4 and IPv6, such as ICMP reject options. But there’s one horrible, horrible error with this ruleset…

ip6tables -A INPUT -p icmp -j ACCEPT
ip6tables -A INPUT -p ipv6-icmp -j ACCEPT

Both of these are valid, accepted ip6tables commands. However only -p ipv6-icmp correctly accepts IPv6 ICMP traffic. Whilst ip6tables happily accepts -p icmp, it doesn’t effectively do anything for IPv6 traffic and is in effect a dud statement.

By having this dud statement in my firewall, from the OS perspective my firewall looked more like:

ip6tables -A INPUT -i lo -j ACCEPT
ip6tables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
ip6tables -A INPUT -j REJECT --reject-with icmp6-adm-prohibited

And all of a sudden there’s a horrible realisation that the firewall will drop ALL inbound ICMP, leaving my laptop unable to receive many important messages such as MTU and rejected connection notifications.

By correcting my ICMP rule to use -p ipv6-icmp, I instantly fixed my MTU issues since my laptop was no-longer ignoring the MTU notifications. :-)

My initial thought was that this would be horrible bug in ip6tables, surely it should raise some warning/error if an administrator tries to use icmp vs ipv6-icmp. The man page states:

 -p, --protocol [!] protocol
    The  protocol of the rule or of the packet to check.  The speci-
    fied protocol can be one of tcp, udp, ipv6-icmp|icmpv6, or  all,
    or  it  can be a numeric value, representing one of these proto-
    cols or a different one.

So why is it accepting -p icmp then? Clearly that’s a mistake, it’s not in the list of accepted protocols…. but further reading of the man page also states that:

A protocol name from /etc/protocols is also allowed.


$ cat /etc/protocols  | grep icmp
icmp       1    ICMP         # internet control message protocol
ipv6-icmp 58    IPv6-ICMP    # ICMP for IPv6

Since /etc/protocols defines both icmp and ipv6-icmp as being known protocols by the Linux OS, ip6tables accepts the protocol argument of icmp without complaint, even though the kernel effectively will never be able to do anything useful with it.

In some respects it’s still a bug, ip6tables shouldn’t be letting users select protocols that it knows are wrong, but at the same time it’s not a bug, since icmp is a valid protocol that the kernel understands, it’s just that it simply will never encounter it on IPv6.

It’s a total newbie mistake on my part, what makes it more embarrassing is that I managed to avoid making this mistake on my server firewall configurations yet ended up doing it on my own laptop. Yet it’s very easy to do, hence this blog post in the hope that someone else doesn’t get caught with this in future. day 5

Final day of – I’m about a week behind schedule in posting, but that’s about how long it takes to catch up on life following a week at LCA. ;-)

uuuurgggh need more sleep

uuuurgggh need more sleep

I like that guy's idea!

I like that guy’s idea!

Friday’s conference keynote was delivered by Tim Berners-Lee, who is widely known as “the inventor of the world wide web”, but is more accurately described as the developer of HTML, the markup language behind all websites. Certainly TBL was an influential player in the internets creation and evolution, but the networking and IP layer of the internet was already being developed by others and is arguably more important than HTML itself, calling anyone the inventor of the internet is wrong for such a collaborative effort.

His talk was enjoyable, although very much a case of preaching to the choir – there wasn’t a lot that would really surprise any attendee. What *was* more interesting than his talk content, is the aftermath….

TBL was in Australia and New Zealand for just over 1 week, where he gave several talks at different venues, including as part of the “TBL Down Under Tour“. It turns out that the 1 week tour cost the organisers/sponsors around $200,000 in charges for TBL to speak at these events, a figure I personally consider outrageous for someone to charge non-profits for a speaking event.

I can understand high demand speakers charging to ensure that they have comfortable travel arrangements and even to compensate for lost earnings, but even at an expensive consultant’s charge rate of $1,500 per day, that’s no more than $30,000 for a 1 week trip.

I could understand charging a little more if it’s an expensive commercial conference such as $2k per ticket per day corporate affairs, but I would rather have a passionate technologist who comes for the chance to impart ideas and knowledge at a geeky conference, than someone there to make a profit any day –  the $20-40k that Linux Australia contributed would have paid several airfares for some well deserving hackers to come to AU to present.

So whilst I applaud the organisers and particularly Pia Waugh for the efforts spend making this happen, I have to state that I don’t think it was worth it, and seeing the amount TBL charged for this visit to a non-profit entity actually really sours my opinion of the man.

I just hope that seeing a well known figure talking about open data and internet freedom at some of the more public events leads to more positive work in that space in NZ and AU and goes towards making up for this cost.

Outside the conference hall.

Outside the conference hall.

Friday had it’s share of interesting talks:

  • Stewart Smith spoke a bit about SQL databases with focus around MySQL & varieties being used in cloud and hosted environments. Read his latest blog post for some amusing hacks fun to execute on databases.
  • I ended up frequenting a few Linux graphical environment related talks, including David Airlie talking about improvements coming up in the server, as well as Daniel Stone explaining the Wayland project and architecture.
  • Whilst I missed Keith Packard’s talk due to a scheduling clash, he was there heckling during both of the above talks. (Top tip – when presenting at LCAs, if one of the main developers of the software being discussed is in the audience, expect LOTS of heckles). ;-)
  • Francois Marier presented on Persona (developed by Mozilla), a single sign on system for the internet, with a federated decentralised design. Whilst I do have some issues with parts of it’s design, over all it’s pretty awesome and it fixes a lot of problems that plagued other attempts like OpenID. I expect I’ll cover Persona more in a future blog post, since I want to setup a Persona server myself and test it out more, and I’ll detail more about the good and the bad of this proposed solution.

Sadly it turns out Friday is the last day of the conference, so I had to finish it up with the obligatory beer and chat with friends, before we all headed off for another year. ;-)

They're taking the hobbits to Isengard! Or maybe just back to the dorms via the stream.

They’re taking the hobbits to Isengard!

A dodgy looking charactor with a wire running into a large duffle bag.....

Hopefully not a road-side bomber.

The fuel that powers IT

The fuel that powers IT


Incoming! day 4

Another successful day of Linux geeking has passed, this week is going surprisingly quickly…

Some of the days highlights:

  • James Bottomley spoke on the current state of Linux UEFI support and demonstrated the tools and processes to install and manage keys and hashes for the installed software. Would have been interesting to have Matthew Garrett at LCA this year to present his somewhat different solution in comparison.
  • Avi Miller from Oracle did an interesting presentation on a new Linux feature called “Transcendent Memory“, which is a solution to the memory ballooning problems for virtualised environments. Essentially it works by giving the kernel the option to request more memory from another host, which could be the VM host, or even another host entirely connected via 10GigE or Infiniband, and having the kernel request and release memory when required. To make it even more exciting, memory doesn’t have to be just RAM, SSDs are also usable, meaning you could add a couple memory hosts to your Xen (and soon KVM) environments and stack them with RAM and SSD to then be provided to all your other guests as a memory ballooning space. It’s a very cool concept and one I intended to review further in future.
  • To wrap up the day, Michael Schwern presented on the 2038 bug – the problem where 32-bit computers are unable to keep time any further and reset to 1901, due to the limits of a 32-bit time buffer (see wikipedia). Time is something that always appears very simple, yet is extremely complex to do right once you consider timezones and other weirdness like leap years/seconds.
The end of time is here! Always trust announcements by a guy wearing a cardboard and robes.

The end of time is here! Always trust announcements by a guy wearing a cardboard and robes.

The conference presentations finished up with a surprise talk from Simon Hackett and Robert Llewellyn from Red Dwarf,  which was somewhat entertaining, but not highly relevant for me – personally I’d rather have heard more from Simon Hackett on the history and future expectations for the ISP industry in Australia than having them debate their electric cars.

Thursday was the evening of the Penguin Dinner, the (usually) formal dinner held at each LCA, this year rather than the usual sit down 3-course dinner, the conference decided to do a BBQ-style event up at the Observatory on Mount Stromlo.

The Penguin Dinner is always a little pricey at $80, but for a night out, good food, drinks and spending time with friends, it’s usually a fun and enjoyable event. Sadly this year had a few issues that kind of spoilt it, at least for me personally, with some major failings on the food and transport which lead to me spending only 2 hours up the mountain and feeling quite hungry.

At the same time, LCA is a volunteer organised conference and I must thank them for-making the effort, even if it was quite a failure this year – I don’t necessarily know all the behind the scenes factors, although the conflicting/poor communications really didn’t put me in the best mood that night.

Next year there is a professional events coordinator being hired to help with the event, so hopefully this adds value in their experience handling logistics and catering to avoid a repeat of the issue.

On the plus side, for the limited time I spent up the mountain, I got some neat photographs (I *really* need to borrow Lisa’s DSLR rather than using my cellphone for this stuff) and spent some good time discussing life with friends lying on the grass looking at the stars after the sun went down.

Part of the old burnt-out observatory

Part of the old burnt-out observatory

Sun setting along the ridge.

Sun setting along the ridge.

What is it with geeks and blue lights? ;-)

What is it with geeks and blue LEDs? ;-)

The other perk from the penguin dinner was the AWESOME shirts they gave everyone in the conference as a surprise. Lisa took this photo when I got back to Sydney since she loves it [1] so much.



[1] She hates it.