Puppet exported resources is a pretty awesome feature – essentially it allows information from one node to be used on another to affect the resulting configuration. We use this for clever things like having nodes tell an Icinga/Nagios server what monitoring configuration should be added for them.

Of course like everything in the Puppet universe, it’s not without some catch – the biggest issue I’ve run into is that if you have a mistake and generate bad exported resources it can be extremely hard to find which node is responsible and take action.

For example, recently my Puppet runs started failing on the monitoring server with the following error:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: A duplicate resource was found while collecting exported resources, with the type and title Icinga2::Object::Service[Durp Service Health Check] on node failpet1.nagios.example.com

The error is my fault, I forgot that exported resources must have globally unique names across the entire fleet, so I ended up with 2x “Durp Service Health Check” resources.

The problem is that it’s a big fleet and I’m not sure which of the many durp hosts is responsible. To make it more difficult, I suspect they’ve been deleted which is why the duplication clash isn’t clearing by itself after I fixed it.

Thankfully we can use the Puppet DB command line tools on the Puppet master to search the DB for the specific resource and find which hosts it is:

# puppet query nodes \
--puppetdb_host puppetdb.infrastructure.example.com \
"(@@Icinga2::Object::Service['Durp Service Health Check'])"

durphost1312.example.com
durphost3436.example.com

I can then purge all their data with:

# puppet node deactivate durphost1312.example.com
Submitted 'deactivate node' for durphost1312.example.com with UUID xxx-xxx-xxx-xx

In theory deleted hosts shouldn’t have old data in PuppetDB, but hey, sometimes our decommissioning tool has bugs… :-/