Monthly Archives: April 2013

Holy Relic of the Server Farm

At work we’ve been using New Relic, a popular software-as-a-service monitoring platform to monitor a number of our servers and applications.

Whilst I’m always hesitant of relying on external providers and prefer an open source solution where possible, the advantages provided by New Relic have been hard to ignore, good enough to drag me away from the old trusty realm of Munin.

Like many conventional monitoring tools (eg Munin), New Relic provides good coverage and monitoring of servers, including useful reports on I/O, networking and processes.

Bro, I'm relaxed as bro.

Bro, I’m relaxed as bro. (server monitoring with New Relic)

However where New Relic really provides value is with it’s monitoring of applications, thanks to a number of agents for various platforms including PHP, Ruby, Python, Java and .NET.

These agents hook into your applications and profile their performance in detail, showing details such as breakdown of latency by layer (DB, language, external, etc), slow DB queries and other detailed traces.

For example, I found that my blog was taking around 1,000ms of processing time in PHP when serving up page content. The VM itself had little load, but WordPress is just not a particularly well oiled application.

Before and after installing W3 Total Cache on my blog.

Before and after installing W3 Total Cache on my blog. Next up is to add Varnish and drop server times even further.

What's my DB up to?

Toss out your crufty DBA, we have a new best friend! (just kidding DBAs, I still love ya)

New Relic will even slip an addition into the client-side content which measures the browser-side performance and experience for users visiting your website or application, allowing you to determine cause of slow page loads.

Generally my issue is too much large content + slow links

Generally my issue is too much large content + slow links

There’s plenty more offered, I haven’t even looked at all the options and features yet myself – best approach is to sign up for a free account and trial it for a while to see if it suits.

New Relic recently added a Mobile Application agent for iOS and Android developers, so it’s also attractive if you’re writing mobile applications and want to check how they’re performing on real user devices in the wild.

 

Installation of the server agent is simply a case of dropping a daemon onto the host (with numerous distribution packages available). The application agents vary depending on language, but are either a case of loading the agent with the application, or bundling a module into your application.

It scales well performance wise, we’ve installed the agent on some of AU’s largest websites with very little performance impact in most cases and the New Relic interface remains fast and responsive.

Only warning I’d make is that the agent uses HTTP by default, rather than HTTPS – whilst the security impact is somewhat limited as the data sent isn’t too confidential, I would really prefer the application use HTTPS-only. (There does appear to be an “enterprise security” mode which forces HTTPS agents only and adds other security options, so do some research if it’s a concern).

 

Pricing is expensive, particularly for the professional account package with the most profiling. Having said that, for a web company where performance is vital, New Relic can quickly pay for itself with reduced developer time spend on issues and fast alerting to performance related issues. Both operations and developers have found it valuable at work, and I’ve personally found this a much more useful tool than our Splunk account.

If you’re only interested in server monitoring you will probably find better value in a traditional Munin setup, unless you value the increased simplicity of configuration and maintenance.

 

Note that New Relic is also not a replacement for alert-monitoring such as Nagios – whilst New Relic can generate alerts for performance issues and other thresholds, my advice is to rely on Nagios for service and resource overload/failure and rely on New Relic monitoring for alerting to abnormal or negative performance trends.

I also found that I still find Awstats very useful – whilst New Relic has some nice browser stats and geography stats, Awstats is more useful for the “how much traffic and data has my website/application done this month” type questions.

It’s not for everyone’s requirements and budget, but I do highly recommend having an initial trial of it, whether you’re running a couple of servers or a massive enterprise.

KVM instances dying at boot

I recently encountered a crashing KVM instance, where my VM would die at boot once the bootloader tried to unpack initrd.

A check of the log in /var/log/libvirt/qemu/vmname.log showed the following unhelpful line:

Guest moved used index from 6 to 229382013-04-21 \
06:10:36.029+0000: shutting down

The actual cause of this weird error and crash occurs when the host OS lacks disk space on the host server’s filesystems. In my particular case, my filesystem was at 96% full, so whilst the root user could write to disk, the non-root processes including Libvirt/KVM were refused writes.

I’m not totally sure why the error happens, all my VM disks are based on LVM volumes rather than the host root filesystem, I suspect the host OS disk is being used for a temporary file such as unpacking initrd and this need for a small amount of disk leads to this failure.

If you’re having this problem, check your disk space and add some Nagios alerting to avoid a repeat issue!

Age of Empires 2 HD on VirtualBox

It’s been a few years since I’d last played it, but Age of Empires is still one of my all time favourite games. I started playing it back as a young noobling on Windows 98 with a Celeron 433mhz machine and loved the perfect balance the game achieved between simplicity and flexibility.

AOE 2 offered a large number of different civilisations with different research options, yet once you learnt the basics, it was quick and easy to pick up the rest and run with it – generally it made for a very fun game, both against the AI but also against friends multiplayer over a 10mbit LAN. ;-)

Whilst AOE 2 still ran on modern Windows, the game was showing it’s age with issues like hard coded resolutions (800×600 or 1024×768 anyone?), assumption of an optical media drive for the music to be loaded from and dated multiplayer functions.

Recently Microsoft re-released AOE 2 as “Age of Empires 2 HD” on steam, taking the opportunity to fix up the above issues and port the game to newer DirectX versions and adding in Steam support providing better multiplayer support.

AOE 2 HD on Steam

AOE 2 HD on Steam

Now as a Linux geek I used to have a dedicated Windows PC hanging around for gaming – however with my move to AU, I now only have my Linux laptop and I wasn’t very keen to go back to a dual-booting world, having last done dual boot over 5 years ago.

Instead I have a Windows 7 VM inside VirtualBox which is a very good virtualisation product for desktop users offering easy management of VMs, good guest OS integration (eg desktop resizing) and also basic 3D acceleration.

I had tested the old AOE 2 original game in my VM which ran OK, but after AOE 2 HD was installed, I found the newer game would refuse to start with:

Error on start, subcode=1

The forums discuss a range of issues for this, but the general consensus is that with the move to a newer DirectX, the game would fail if some of the newer features were absent (eg shaders). I was worried about my Intel GPU being too poor as some posts suggested, but thankfully even the Intel GPUs from a few years ago are enough to play this game.

The issue was that the level of 3D acceleration being provided by VirtualBox was too low – this was easily verified by running “dxdiag” utility.

No 3D

By default, Virtualbox delivers DirectDraw Acceleration but not Direct3D.

By default if you enable 3D acceleration and install the Guest Additions, the acceleration is limited to just the stable basic features. For more advance 3D features, a module is available, but needs to be specifically selected during the installation as it’s not “stable” yet.

The stability of this feature is also rapidly changing – after turning everything on and running the game, I was causing the VM to crash entirely on the Linux OS-side.

ShCrOpenGL[23043]: segfault at 7fd40d7c3f90 ip 00007fd3f47c2f9d sp
00007fd3f4a8ba90 error 4 in VBoxSharedCrOpenGL.so[7fd3f4732000+cb000]

Generally I’m not a fan of Segfaults on my system. In order to avoid any nasty surprises, you must upgrade to the absolutely latest upstream version (at time of writing, this was 4.2.12-84980). Make sure you grab the latest version from Oracle/VirtualBox themselves, as if you’re using a distribution version their latest version in the distribution repositories may not be as up-to-date.

After installing/upgrading VirtualBox, adjust your virtual machine to be using 3D Acceleration and give it the full 256MB video memory.

You kind of want to enable 3D acceleration for 3D to work ;-)

You kind of want to enable 3D acceleration for 3D to work ;-)

Once booted, install the latest version of Guest Additions (“Devices->Install Guest Additions” from VirtualBox VM menu). Older VirtualBox drivers still work for most tasks, but you must do this installation/upgrade in order to get the absolute latest 3D fixes for the game to run.

When installing, make sure you select “Direct3D Support (Experimental)” from the component selector.

3D support is an option, not a standard feature.

3D support is an option, not a standard feature.

Note that VirtualBox will try and talk you into installing the stable basic 3D drivers. Make sure you select No and get the fully featured (but less stable) drivers.

No is Yes in our case.

No is Yes in our case.

Once installed, you’ll need to reboot and your Windows system should now feature the much better 3D support.

To verify, run “dxdiag” again to check the report:

Ready to party!

Ready to party!

Once done, you should be good to play! :-D

Recommend going to full screen in VirtualBox, before launching the game, the directional edge-of-screen scrolling works a bit weirdly otherwise. There’s also a quirk with the launch videos, where they play but display a blank screen. Just click to skip through them and to get to the game launch screen.

And we're live!

And we’re live! Back to the mines peons!

Performance and stability seems to be OK – I haven’t been able to take advantage of the higher resolution support much thanks to my laptop having a wonderfully crap display of 1280×768, so I can’t be sure whether it works well on higher resolutions or if the 3D acceleration can’t handle it.

My test was done on a x86_64 Debian Wheezy system running kernel 3.6.7 using the Oracle provided VirtualBox package (4.2.12-84980~Debian~wheezy) and a fully patched 64-bit Windows 7 Home Premium guest with latest Guest Addition drivers.

I have not tested on MacOS, but I would assume the process and support to be the same, since VirtualBox uses OpenGL on the HostOS side for the 3D Acceleration Pass-through.

Marriage Equality

New Zealand has passed legalisation for allowing same-sex marriage this evening! I’m so happy for my many LGBT friends and love the fact that no matter whom any of us love, we can get the same recognition and rights in not just law, but also have society recognise these choices.

I love the fact that my beautiful lady friend can settle down with another beautiful lady friend should she chose to. I love the fact my awesome male friend can find another awesome male to spend life with should he chose. And I love the fact that their relationship does not impact on the meaning of my own conventional relationship with Lisa in any way.

Unleash the Gayroller 2000!

Run conservatives! (credit to theoatmeal.com)

NZ has had civil unions for some time giving same-sex couples similar legal rights, but this removes the last barriers to equality and gives hetrosexual and homosexual relationships the same footing in society.

This may sound like a small thing, but it has huge ramifications – it removes the distinction of same-sex relationships as being different in some negative sense. In the eyes of the law, there is no difference in any marriage no matter what genders the participants are and this will reflect over time in society as well

It’s also solved the same-sex adoption issue, with the legalisation opening the door for the first gay adoptions to take place.

Generally New Zealand is very liberal – our government is almost entirely secular, generally mentioning and promoting religious views in government is the realm of fringe 1 or 2 MP parties, which has really helped with this legalisation as it hasn’t gotten so tangled up with religious arguments.

Whilst there were still a few conservative fucktards disturbed individuals who voted against the bill, the support has been strong across the entire house and across parties.

I’ve personally noted that the younger generations are the most open and comfortable around same-sex relationships, whereas the older generations tend to support their rights in general, but still raise their eyebrows when they see a same-sex couple, a hangover from decades of stigma where they still feel the need to point it out as something unusual.

There’s still some stigma and fights remaining – the law still won’t recognise polygamous marriage and government departments still tend to vary a lot on the gender question (although the new marriage laws allow you to be Partner/Partner rather than Husband/Husband, Wife/Wife or Husband/Wife; and NZ passports now have a X/Other gender option, so it’s getting better.).

There’s also some more understanding needed in society around the fluid nature of sexuality – if I sleep with guys and then settle down with a lady (or vice versa), it’s not a case of going through a phase before setting on straight or gay – there’s a whole range of gray areas in between which are very hard to define, yet I feel that society in general is very quick to label and categorise people into specific categories when the reality is that we’re just undefinable.

But despite a lot of remaining work, I’m extremely happy with this legalisation and happy that I voted for a party that supported it 100% as an undeniable human rights issue. Maybe even had a momentary twinge of patriotism for NZ. :-)

Launch Paywall!

So far my time working at Fairfax Media AU has been pretty much non-stop from day one – whilst the media companies are often thought of as dull and slow moving, the reality is that companies like Fairfax are huge and include a massive range of digital properties and are not afraid to invest in new technologies where there are clear business advantages.

It’s great stuff for me, since I’m working with a really skilled group of people and being given some awesome and challenging projects to keep me occupied.

Since January I’ve been working with the development team to build up the infrastructure needed to run the servers for our new paywall initiative. It’s been a really fun experience working with a group of skilled and forwards thinking group of developers and being able to build infrastructure that is visible by millions of people and at such a key stage in the media business is a very rare opportunity.

Our new paywall has just been launched a few days ago on Sydney Morning Hearld and The Age websites for select international countries (including the US and UK)  before we roll it out world wide later in the year.

Paywalls on SMH and The Age, as seen for select international countries.

Paywalls on SMH and The Age, as seen for select international countries.

Mobile hasn't been neglected, a well polished interface has been provided there too.

Mobile hasn’t been neglected, a well polished interface has been provided there too.

Obviously paywalls are a pretty controversial topic, there’s already heaps of debate online  ranging from acceptance to outright rage for the idea of having to pay for daily news content and plenty of reflection over long term sustainability of Fairfax itself.

I won’t go into too much detail, I have somewhat mixed personal views on the idea, but generally I think Fairfax has created a pretty good trade off between making content casually available and sharable. Rather than a complete block, the paywall is porus and allows a select number of article reads a month, along with liberal social media sharing and reading of links shared by others.

Changing a business model is always hard and there will be those whom are both happy and unhappy with the change, time will tell how well it works. Thankfully my job is about designing and managing the best infrastructure to run the vision that comes down from management rather than trying to see into the future of the consumption and payment of media content, which has got to the be the hardest job around right now….

The future is digital!

Whatever your format of choice, we have your fix!

It’s been an interesting project working with a mix of technologies including both conventional and cloud based virtualisation methods and using tools such as Puppet for fast deployment and expansions of the environment, along with documenting and ensuring it’s an easy environment to support for the future engineers who operate it.

I’ve only been here 6 months and have already picked up a huge range of new skills, I’m sure that the next 6 months will be even more exciting. :-)