Monitoring

A short tutorial about the setup process of an open source monitoring system.
June 27, 2022 Translations: 

A working monitoring setup is a must-have for a proper home lab. In this blog article as well as the video, I explain the basics regarding a monitoring system based on Grafana, Prometheus and MQTT.

(German YouTube video)

This installation is just an example and is not production ready!

Install Grafana

I recommend referring to the official Documentation. The basic steps are shown below.

Install dependencies

sudo apt install -y apt-transport-https
sudo apt install -y software-properties-common wget

Add GPG key

wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -

Add apt source

echo "deb https://packages.grafana.com/enterprise/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

Install Grafana

sudo apt update
sudo apt install grafana-enterprise

Enable the service

sudo systemctl enable --now grafana-server

Build Grafana MQTT datasource from source

This step isn’t necessary any more as the MQTT datasource is now officially available"

The original build process is outlined in a GitHub Issue.

Install Go

wget -O -https://raw.githubusercontent.com/mauleyzaola/scripts/master/go/go.install.sh | sh
source ~/.bashrc

Install Mage

sudo apt install git
git clone https://github.com/magefile/mage
cd mage
go run bootstrap.go

Install Node.js

curl -fsSL https://deb.nodesource.com/setup_16.x | sudo -E bash -
sudo apt install -y nodejs

Enable Yarn

sudo corepack enable

Modify the Grafana plugin configuration.

sudo nano /etc/grafana/grafana.ini

Modify these variables:

plugins = /var/lib/grafana/plugins
allow_loading_unsigned_plugins = grafana-mqtt-datasource

Create plugin directory

mkdir /var/lib/grafana/plugins

Build MQTT datasource

cd /var/lib/grafana/plugins
git clone https://github.com/grafana/mqtt-datasource.git
cd mqtt-datasource
yarn add @grafana/toolkit
yarn build
yarn install

Restart Grafana

sudo systemctl restart grafana-server

Install and configure Mosquitto

Install Mosquitto

sudo apt install mosquitto

Configure Mosquitto

Create the configuration file

sudo nano /etc/mosquitto/conf.d/mosquitto.conf

Insert the configuration:

listener 1883 0.0.0.0
log_dest syslog
log_dest stdout
log_dest topic
log_type error
log_type warning
log_type notice
log_type information
connection_messages true
log_timestamp true
allow_anonymous false
password_file /etc/mosquitto/pwfile

Create the password file

sudo nano /etc/mosquitto/pwfile

Add users to this file in the <username>:<password> scheme.

Hash the password file

sudo mosquitto_passwd -U /etc/mosquitto/pwfile

Start the Mosquitto service

sudo systemctl enable --now mosquitto

Install and configure Prometheus

Create a service account

I recommend using an unprivileged service account for Prometheus.

sudo adduser prometheus
sudo su - prometheus

Download and extract Prometheus

wget https://github.com/prometheus/prometheus/releases/download/v2.36.0/prometheus-2.36.0.linux-amd64.tar.gz
tar -xzvf prometheus-2.36.0.linux-amd64.tar.gz
mv prometheus-2.36.0.linux-amd64/ prometheus/

Create the Prometheus service

The following commands have to be executed as an admin.

sudo nano /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus Server
After=network-online.target

[Service]
User=prometheus
Restart=on-failure
ExecStart=/home/prometheus/prometheus/prometheus \
--config.file=/home/prometheus/prometheus/prometheus.yml \
--storage.tsdb.path=/home/prometheus/prometheus/data

[Install]
WantedBy=multi-user.target

Start the Prometheus service

sudo systemctl enable --now prometheus

Install the Node Exporter

I recommend using a service account here, too. The process of creating one is the same as for Prometheus.

Download and extract the Node Exporter

wget https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz
tar -xzvf node_exporter-1.3.1.linux-amd64.tar.gz
mv node_exporter-1.3.1.linux-amd64/ node_exporter/

Create the Node Exporter service

sudo nano /etc/systemd/system/node-exporter.service
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
ExecStart=/home/prometheus/node_exporter/node_exporter

[Install]
WantedBy=default.target

Start the Node Exporter Service

sudo systemctl enable --now node node-exporter

Add the Node Exporter to Prometheus

The config file is located in the prometheus folder in the home directory of the Prometheus service account.

nano prometheus.yaml

This configuration has to be added to the job configuration section of the configuration:

- job_name: 'node_exporter'
  static_configs:
   - targets: ['localhost:9100']

Restart the Prometheus service

sudo systemctl restart prometheus

Done!