I’ve been looking forwards to Munin 2 for a while – whilst Munin has historically been a great monitoring resource, it’s always been a little bit too fragile for my liking and the 2.x series sounds like it will correct a number of limitations.
Munin 2.0.6 packages recently became available in the EPEL repository, making it easy to add Munin to your RHEL/CentOS/OracleEL 5/6 servers.
Unfortunately the upgrade managed to break value collection for all my hosts, thanks to the fact that I run a dual-stack IPv4/IPv6 network. :-(
Essentially there were two problems encountered:
- Firstly, the Munin 2.x master attempts to talk to the nodes via IPv6 by default, as it typical of applications when running in a dual stack environment. However when it isn’t able to establish an IPv6 connection, instead of falling back to IPv4, Munin just fails to connect.
- Secondly, the Munin nodes weren’t listing on IPv6 as they should have been – which is the cause of the first problem.
The first problem is an application bug, or possibly a bug in one of the underlying libraries that Munin-node is using. I haven’t gone to the effort of tracing and debugging it at this stage, but if I get some time it would be good to fix properly.
The second is a packaging issue – there are two dependency issues on EL 5 & 6 that need to be resolved before munin-node will support IPv6 properly.
- perl-IO-Socket-INET6 must be installed – whilst it may not be a package dependency (at time of writing anyway) it is a functional dependency for IPv6 to work.
- perl-Net-Server as provided by EPEL is too old to support listening on IPv6 and needs to be upgraded to version 2.x.
Once the above two issues are corrected, make sure that the munin configuration is correctly configured:
host * allow ^127\.0\.0\.1$ allow ^192\.168\.1$ allow ^fdd5:\S*$
I configure my Munin nodes to listen to all interfaces (host *) and to allow access from localhost, my IPv4 LAN and my IPv6 LAN. Note that the allow lines are just regex rather than CIDR notation.
If you prefer to allow all connections and control access by some other means (such as ip6tables firewall rules), you can use just the following as your only allow line:
allow ^\S*$
Once done, you can verify that munin-node is listening on an IPv6 interface. :-)
ipv4host$ netstat -na | grep 4949 tcp 0 0 0.0.0.0:4949 0.0.0.0:* LISTEN ipv6host$ netstat -na | grep 4949 tcp 0 0 :::4949 :::* LISTEN
I’ve created packages that solve these issues for EL 5 & EL 6 which are now available in my repos – essentially an upgraded perl-Net-Server package and an adjusted EPEL Munin package that includes the perl-IO-Socket-Net package as a dependency.
Hmm, i’m still using munin 1.x.
Don’t forget to do a blog post when you get your netflow thingy working ;)
(Actually, are you githubbing it?)
I will do :-) Almost done, I’ve finished the code for the 1.0.0 initial beta and have been using it for a couple of weeks internally.
Just working on getting the documentation sorted now, really need to get the docs done so that it’s easy to pickup and start playing with the application as well as writing some basic developer docs to assist with adding different structures of netflow database to the application.
It’ll be up on http://projects.jethrocarr.com once done, it’s where I put all my open source work rather than using providers like github.