Tag Archives: geek

Anything IT related (which is most things I say) :-)

DHCP, I/O and other virtualisation fun with KVM

I recently shifted from having two huge server racks down to having a single speedy home server running KVM virtual machines, with the intent of packaging all my servers – experimental, development, staging, etc, into a single reliable system which will reduce power and maintenance costs.

As part of this change, I went from having dedicated DHCP & DNS servers to having everything located onto the KVM host.

The design I’ve used, has the host OS running with minimal services – the host just runs KVM, OpenVPN, DHCP and a DNS caching nameserver – all other services run as guest VMs, with a virtual network for the guests and host to communicate over.

Guests run as DHCP clients – this makes it easy to assign or adjust addressing if needed and get their information from the host OS.

However this does mean you can’t get away with hammering the host too badly – for example, running an I/O and network intensive backup can cause some interesting problems when you also need the host for services, such as DHCP.

Take a look at the following log messages from a mostly idle VM – these were taken whilst another VM on the server was running a bonnie++ process to test performance:

Mar  6 10:18:06 virtguest dhclient: 5 bad udp checksums in 5 packets
Mar  6 10:18:27 virtguest dhclient: DHCPREQUEST on eth0 to 10.8.12.1 port 67
Mar  6 10:18:45 virtguest dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67
Mar  6 10:19:00 virtguest dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67
Mar  6 10:19:07 virtguest dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67
Mar  6 10:19:15 virtguest dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67
Mar  6 10:19:15 virtguest dhclient: 5 bad udp checksums in 5 packets

That’s some messed up stuff – what you’re seeing is that the guest VM is trying to renew the DHCP address with the host server – but the host is so sluggish with having to run the I/O intensive virtual machine that is actually corrupting or dropping the UDP packets, preventing the guest VM from renewing it’s address.

This of course raises the most important question: What happens if the guest can’t renew it’s IP address?

In this case, the Linux/CentOS 5 guest VM actually completely lost it’s IP address after a long period of DHCPREQUEST attempts, fell off the network entirely and caused my phone to go nuts with Nagios alerts.

Now of course in any sane production environment, nobody would be running a bonnie++ processes on a VM on an active server – however there’s some pretty key points still made here:

The isolation is a lie: Guests are only *somewhat* isolated from one another – one guest can still mess with another and effectively denial-of-service attack the other VMs by utilising all the available resources.
Guests can be jerks: Organisations running KVM (or some other systems) with untrusted guest VMs should carefully consider how they are going to monitor and protect the service from users running crazily resource intensive processes. (after all, there will be someone who wants to bonnie++ test their new VM simply for the lols).
cgroups to the rescue? Linux cgroups does have an I/O controller (blkio-cgroup) although whilst this controls read/write flow, it won’t restrict seeks which can also badly impact spinning rust based servers.
WTF DHCP? The approach of the guests simply dropping their DHCP address after losing contact with the DHCP server is a pretty bad design limitation – if the DHCP server is unreachable, it should keep the original address (of course if the “physical” ethernet connection dropped, that would be a different situation, and it should drop it’s address to match).
Also: I wonder what OSes/distributions have the above behavior?

I’m currenting running a number of bonnie++ tests on my KVM server and will have a blog post in the near future detailing these findings in more detail, I’m also planning to look into cgroups and other resource control and limiting functions and will report back on how these fare when you have guest VMs running heavy processes.

Overall it made my weekend of geekery that bit more exciting. :-D

CentOS, RHEL and future possibilities?

10 Replies

Those who know me will know that I’m a long term CentOS user – this actually started from my love of RHEL, back in my early Linux using days when I was running Red Hat 8.0.

Whilst it made financial sense for Red Hat to switch to making their product only available in binary form for their customers, at the same time I can’t help but feel this has damaged the appeal of Red Hat for geeks like myself – I’m no longer able to setup friends, family or customers without the funds for RHEL with a quality, enterprise-grade free (as in beer + freedom) distribution.

I do wonder if this contributes to reduced market awareness in the small business space and also whether it reduces the likeliness of geeks like myself promoting the software – after all, if I can’t run RHEL myself, I’m likely to look at other distributions and options and end up promoting those.

With the lack of a free Red Hat enterprise-grade distribution, there are only a couple options for wanting a Red Hat-style experience:

Fedora – the community developed distribution that forms the future base of RHEL, a fantastic distribution in it’s own right, but with only 12 months support per release, not suitable for server deployments.
CentOS – the community free re-spin of RHEL with their trademarks removed to make it legally redistributable.

I’ve been using CentOS heavily on my servers and Fedora on my workstations, however there are a number of security delays that are concerning me about CentOS which have been recently highlighted in an LWN article.

Essentially, the core problem is that the latest version of CentOS is still only 5.5, whilst Red Hat have had 5.6 out for some time, with numerous security updates in it that have yet to be released for CentOS…..

Having systems vulnerable to known exploits with no upstream patches is always a pretty serious concern to any system administrator…. this is leading me to re-think my usage of CentOS and to consider whether I should consider other platforms.

I’ve never been a huge fan of Debian in the past, but I’m considering giving it a more detailed look and try – Debian has the advantages of a strong community (like Fedora has) but without the limitation of a short support life – although then again, Debian’s releases and support spans are a little less rigid than Red Hat, which is somewhat annoying.

There’s a few server platforms that come to mind – Ubuntu LTS, Mint Linux, Debian, Open/SuSe or of course, Fedora.

The other option is that I could spin my own distribution – based on the number of custom RPMs I already build, rebuilding Red Hat’s update packages for my own needs wouldn’t be too hard, but I really don’t want to get caught up in distribution maintenance for the next 5 years plus it’s not suitable for customer deployments – so even if I decide that a custom built system is best for me, it still doesn’t solve the “what do I install for others?” question.

Maybe I need Fedora LTS – long term support for specific versions of Fedora – 3 or 5 years would be wonderful and meet the needs of server administrators.

This was tried once before, with the Fedora Legacy project, but it didn’t last long – possibly the goal of supporting *all* the releases was too much to reasonably handle, so an approach of selection even/odd number releases only might make it more feasible – I know that I’d certain be willing to contribute.

Anyway, this is a late night concerned system administrator brain dump about the problem, interested in thoughts and comments from others here about distributions they use/would consider in the server environment.

Auckland Visits

Leave a reply

I’m heading up to Auckland on business a couple times in the next few months.

22nd & 23rd February
7th, 8th & 9th of March

I’m expecting to be too busy to do anything on the evening of the 22nd, however I’m keen to try and meet up with some Aucklanders of the evening of the 7th or 8th – most likely the 8th, somewhere in the CBD.
Planning to meet up at the Northern Steamship (Macs Pub) at 19:00, as per a suggestion by @pikelet.

I’ve created a twtvite here for those of you using twitter to RSVP to – nice to know if people actually care enough to come along ;-)

Freebies: DDR RAM

Leave a reply

More freebies! This time a pile of DDR RAM sticks I have.

You can either collect for free from me at my office (Lambton Quay) or home, or you can pay $10 for my time and postage and I’ll ship it to anywhere in NZ.

512MB DDR-400 CL 3 (Infineon)
512MB DDR-400 CL 2.5 (A-Data)
512MB DDR-333 CL 2.5 (SimpleTech)
256MB DDR-333 CL 2.5 (SimpleTech)
256MB DDR-266 CL ?? (Legend)

As far as I’m aware, all these sticks work fine as I pulled them from running systems.

PHP Podcast Interview

Leave a reply

Whilst at was at linux.conf.au I got interviewed by Steve from the Coding by Numbers podcast series, along with @LGnome about PHP, some of it’s background, future and general thoughts.

Listen to episode 23 interview regarding PHP.

Melbourne: Day 02

2 Replies

For my final day in Melbourne, @MissNickiBee had organised the greatest tour of all time – a visit to CISRAC at Melbourne Museum and then to Monash University’s computer museum.

After starting the day with coffee, we headed off the Melbourne Museum on foot through the mean streets of Melbourne suburbs.

Melbourne architects seem to love sticking turrets on their brick buildings.

Exhibition hall thingy

It took a bit of effort to find CSIRAC since Melbourne Museum had moved it out of the main area to a separate public area.

CSIRAC! :-D

Lots and lots of wring in this thing

Diagram of the components of CSIRAC in horribly bad photography by yours truly

Sadly the Cray Supercomputer and Mainframe mentioned on their website are not available for public display :-( So I spent a couple hours looking around their general exhibits at the museum, which are quite interesting.

There’s a very large geology section with just about every imaginable rock type, if you’re a geology geek you’d probably have a lot of fun.

The pyscology and Melbourne history sections are also very interesting and it would be easy to spend a lot of time there.

After the museum, we headed off to the Monash University’s Museum of Computing History, a very impressive range of machines from early prototype era through to mainframes and through to the early microprocesser generation.

I took a lot of photos, here’s a few specific ones, but there’s a lot more to the collection:

Early digital calculator

Early IBM System 360 mainframe (this is just the console, actual thing would be about a room full of refigerator sized units)

VT100 console! We still refer to terminals as being "VT100 compatible" even now in the UNIX world.

Early microprocessors - recognise any famous models? :-)

Large early generation machine - memory bank visible

Delay Line Memory (I belive these are Nickel Relays)

A VAX, one of the early machines that UNIX was written on. Much fanboy squeee ensured.

Paper tapes. Yes, this did actually exist, it's not a tale to scare young geeks.

That’s some of the pics, I’ll upload others when I have more time one day – huge thank you to Monash university for putting this display together so professionally and making it open to the public, really made my day. :-)

Kind of a shame that the Melbourne Museum’s publicly assessable “Technology Collection” only consisted of CSIRAC, when there is so much more amazing technology they have in their collection.

After Monash, I headed back into the city for coffee before heading out to the airport on the skybus for my return to Wellington NZ – had a great time in Melbourne and many thanks to @MissNickiBee for the personal tour. :-)

Hobart: Day 00.1

1 Reply

Some additional pictures from yesterday’s excursion into the wilds of Tasmania by @chrisjrn:

Photographic proof that yes indeed, I can sometimes go outside.

Hehehe, I got a dirty txt! ;-)

I'm going to twitpic these rocks!

I have conqured nature!

These bars can't restrain my awesome!

hmmm maybe they can actually.....

Come maul me babe..... ;-)

I was a little tooo naughty so had to go spend some time there.....

That’s all the dodgy pictures for now – come back soon for more blog posts as I tour AU :-)

Hobart: Day 00

3 Replies

Whilst I did technically land in Hobart yesterday, I only started looking around Hobart and Tasmania in general with the native @chrisjrn today.

We started the trip by going up Mt Wellington, but sadly were impacted by a lot of low-lying cloud making the views difficult.

Here’s a view from about half way up the mountain down onto Hobart:

With all the cloud instead of the amazing views I had been promised, I just had lots and lots of white, which whilst kind of charming in it’s own way, is clearly false advertising ;-)

After visiting Mt Wellington Fog, we headed onto the road around to Port Arthur with detours to interesting places along the way.

Always an excuse to stop for random self-camwhoring.

Chris took me to some really interesting natural rock formations at a cove, the geological activity forces the rocks to break in straight lines, they look like man cut blocks!

Check out those rocks!

@chrisjrn checking out the rocks

Following mystical rock cove (maybe not not real name) we decided to go checkout a blowhole.

Sadly it turned out to be a natural geological formation rather than some dodgy deviant makeout facility.

Natural tunnel formation - watch the water coming in before it splashes up

Splash! It's pretty hard to capture on camera really :-(

Of course the best part about the blowhole action was the salty salty deliciousness:

Fresh from the deep fryer, all chrispy and nommmmm

After refuelling at the blowhole, we headed to Port Arthur, where Tasmania had it’s convict colony to look at all the historical buildings:

Main prison block in the foreground

I can never say no to exploring a dungeon....

There’s some more dodgy pictures involving bars and chains that I’ll have to wait for Chris to upload at a later stage.

Meanwhile, here’s Chris looking dodgy and being reminded why long hair may be cool to look at whilst being very annoying to actually have:

Also, we went on a boat:

Boat View!

Captain Jethro!

After the boat trip, we headed back home – although did get interrupted by an (amazing for me) sight of a bridge being swung to allow a sailing boat to pass through the bridge.

Note the mast!! O_o

Also, today’s WTF moments:

Um, historical laptop anyone? To be fair, that model is probably about 10 years old...

Oh what the fuck, is that comic sans ms?!!?

And on that note, I’m off to bed for more adventures tomorrow. :-)

LCA2011: Day 07

Leave a reply

OK, technically there is no LCA day 07, but seeing as it was the last day in Brisbane I figured I could get away with it, without needing to create a separate heading. ;-)

Firstly, I found an awesome pic of me by Andrew McMillian aka Karora on Flickr which clearly demonstrates my need for a haircut:

Me at the Professional Delegates Networking Session (pic by karora)

I spent the morning catching up on sleep and then after packing, I headed into Brisbane Times Square to attend a Fedora meeting at the library.

After that, I headed back to Urbanest by walking from the library, across a bridge and then along southbank back to the accommodation.

Hai Gais! (pic by @chrisjrn on a very awesome high quality lens)

Photoshop the bottle & caption if you dare ;-) (pic via @chrisjrn)

I then caught the AirTrain with @chrisjrn and another guy to the Brisbane Airport – I have to say, I love the AirTrain – it’s fast, easy, comfortable and cheap to get to/from the airport or any station along the gold coast.

I'm on a train!!! (pic by @chrisjrn)

Fast train is fast!

Travelling in style!

Brisbane Airport is pretty decent, didn’t take long to get through security, although I got explosive tested *again* which I find somewhat amusing, since they test for explosives far more often than drugs and I’d bet good money as to which one 20-something European males typically carry….

Flight was delayed a little, but made it to Melbourne with 30mins to spare to connect to the Melbourne-Hobart flight.

One oddity from arriving in Hobart was the way they advertise their seal touristy things: