JSON alerts for Munin

Munin is a great and easy way to monitor your servers. The documentation for Munin is kind of all over the place, and sometimes not all that clear.

  * There is the recent ["Munin Guide"](https://munin.readthedocs.io/en/latest/) which is quite informative and well laid out.

• Sometimes additional info and background can be found on the old munin wiki.

Getting the standard mail alerts working on munin is simplest, but as soon as you want custom layouts or send alerts as a push message to a phone, it can take quite some work. Here's what I figured out to get the Munin alert available to you as JSON inside your own Python script so you can do anything you want with it.

On the munin master, create the file /etc/munin/munin-conf.d/alerts.conf with the following contents:

[sourcecode gutter="false" language="text"] contact.python.command /etc/munin/scripts/alert.py contact.python.text { "group":"${var:group}", "host":"${var:host}",
"graph_category":"${var:graph_category}",
"graph_title":"${var:graph_title}",
"warning":[
${loop<,>:wfields {"label":"${var:label}",
"value":"${var:value}",
"w":"${var:wrange}",
"c":"${var:crange}",
"extra":"${var:extinfo}"} }
],
"critical":[
${loop<,>:cfields {"label":"${var:label}",
"value":"${var:value}",
"w":"${var:wrange}",
"c":"${var:crange}",
"extra":"${var:extinfo}"} }
],
"unknown":[
${loop<,>:ufields {"label":"${var:label}",
"value":"${var:value}",
"w":"${var:wrange}",
"c":"${var:crange}",
"extra":"${var:extinfo}"} }
]
} [/sourcecode]

This will make Munin output JSON to the "python" user, you can chose another name if needed. After this, you can create a Python script looking somewhat like this:

[sourcecode gutter="false" language="python"] #!/usr/bin/python

Save as /etc/munin/scripts/alert.py

chown munin:munin /etc/munin/scripts/alert.py

chmod +x /etc/munin/scripts/alert.py

import sys import os import json

def humanreadable( jsonalert ): hascritical = (len(jsonalert['critical']) > 0) haswarning = (len(jsonalert['warning']) > 0) hasunknown = (len(jsonalert['unknown']) > 0)

report = ""

if hascritical: for i in jsonalert['critical']: report += "CRITICAL: {0} {1}: {2}={3}".format(jsonalert['host'], jsonalert['graph_title'], i['label'],i['value'])

if haswarning: for i in jsonalert['warning']: report += "WARNING: {0} {1}: {2}={3}".format(jsonalert['host'], jsonalert['graph_title'], i['label'],i['value'])

if hasunknown: for i in jsonalert['unknown']: report += "UNKNOWN: {0} {1}: {2}={3}".format(jsonalert['host'], jsonalert['graph_title'], i['label'],i['value'])

if hascritical + haswarning + hasunknown == 0: report += "OK: {0} {1}".format(jsonalert['host'],jsonalert['graph_title'])

return report

Get the output from Munin stdout and print it.

for line in sys.stdin: j = json.loads(line) readable = humanreadable(j) print readable [/sourcecode]

There is no debugging without testruns. To see if this works, you can run this command:

[sourcecode gutter="false" language="bash"] sudo -u munin /usr/share/munin/munin-limits --force [/sourcecode]

This should give you output which looks somewhat like this:

[sourcecode gutter="false" language="text"] WARNING: monitor.local Logged in users: pts=4.00 OK: monitor.local Inode usage in percent OK: monitor.local eth0 errors OK: monitor.local File table usage OK: monitor.local Disk usage in percent OK: monitor.local routerboard CPU usage OK: monitor.local Disk latency per device :: Average latency for /dev/mmcblk0 [/sourcecode]

Bingo! Munin sends JSON to your Python script! Now you can use your awesome Pyhton skills to change this into whatever you want or send. To get you inspired, here's a piece of code I use to send alerts to a Mattermost server:

[sourcecode gutter="false" language="python"] import requests

mattermost_url = "https:///hooks/"

def sendmattermost( alerts ): payload = '{"username": "Monitor", "text": "'

if "WARNING:" in alerts or "CRITICAL:" in alerts: payload += ':rotating_light:\n' else: payload += ":pager:\n"

payload += alerts payload += '"}'

print "Sending payload: '" + payload +"'." r = requests.post(mattermost_url, data=payload) if r.status_code != 200: print "PANIC " + r.reason +" : " + r.text [/sourcecode]

You can find this trick and other munin plugin stuff in my munin-plugins repository.

Happy alerting!