When munin df does not list all disks

May 23, 2021 · 2 min read ·

Share on:

I am a great fan of using munin for monitoring my servers and I have quite an extensive setup that even sends me Telegram messages when something is wrong.

This weekend I ran into an interesting problem with a backupserver where the ‘df’ plugin on the node did not want to list the backup disk, and I’ve spent quite some time trying to figure out what the problem was.

Here’s the fstab setup for my machine:

1proc                  /proc            proc    defaults          0       0
2PARTUUID=698b7a16-01  /boot            vfat    defaults          0       2
3PARTUUID=698b7a16-02  /                ext4    defaults,noatime  0       1
4UUID=6ef2cb75         /home/pi/usbdisk ext4    defaults          0       2

When I run ‘df’ on my machine it returns the expected results:

 1# df
 2Filesystem      1K-blocks       Used  Available Use% Mounted on
 3/dev/root        29522500    1377884   26912408   5% /
 4devtmpfs           465280          0     465280   0% /dev
 5tmpfs              498560          0     498560   0% /dev/shm
 6tmpfs              498560      19128     479432   4% /run
 7tmpfs                5120          4       5116   1% /run/lock
 8tmpfs              498560          0     498560   0% /sys/fs/cgroup
 9/dev/mmcblk0p1     258095      49767     208329  20% /boot
10/dev/sda1      3845568008 1698470112 1951731164  47% /home/pi/usbdisk
11tmpfs               99712          0      99712   0% /run/user/1000

When I use munin-run to check the output of the ‘df’ plugin this also works:

1# munin-run df fetch
2_dev_root.value 4.87051883381055
3_dev_shm.value 0
4_run.value 3.83664955070603
5_run_lock.value 0.078125
6_sys_fs_cgroup.value 0
7_dev_mmcblk0p1.value 19.2823600520736
8_dev_sda1.value 46.5308618230838

But for some mysterious reason, /dev/sda1 did not show up when connecting through the network:

 1# nc localhost 4949
 2# munin node at backuppi
 3fetch df
 4_dev_root.value 4.87051883381055
 5_dev_shm.value 0
 6_sys_fs_cgroup.value 0
 7_run.value 3.83664955070603
 8_run_lock.value 0.078125
 9_dev_mmcblk0p1.value 19.2823600520736
10.

After hours of debugging the perl script (I can not recommend that to anyone) and trying restarting, rebooting and other voodoo, I found a non-related redit post about systemd and protecting home directories. The keen-eyed reader has spotted that my disk is mounted on /home/pi/usbdisk (let’s say that’s for artistic reasons).

A quick peek in /lib/systemd/system/munin-node.service shows that this is property indeed set to ProtectHome=true (which makes sense). Of course I have the option to mount the disk somewhere else, but since my raspberry is behind 2 (!) firewalls and only operated by myself I solved the problem as follows:

Edit /lib/systemd/system/munin-node.service and set ProtectHome=false.

1# systemctl daemon-reload
2# systemctl restart munin-node

A quick check shows that the disk is now indeed listed:

 1# nc localhost 4949
 2# munin node at backuppi
 3fetch df
 4_dev_root.value 4.87051883381055
 5_dev_shm.value 0
 6_sys_fs_cgroup.value 0
 7_run.value 3.83664955070603
 8_run_lock.value 0.078125
 9_dev_mmcblk0p1.value 19.2823600520736
10_dev_sda1.value 46.5308618230838
11.

Success!