Monitoring of a system is key to its smooth functioning. Going to the battlefield (production) without having proper monitoring setup done is like making your platform vulnerable, hence to obtain full control it becomes a must; as the popular say goes “Failing to plan, is planning to fail”. In this article I’m going to show how you can monitor a system using Prometheus, node_exporter and the Grafana UI.
Difference between Pull and Push based monitoring architecture
Simply put, in push based architecture each target node periodically sends metrics to a central collector. Examples of push architectures include: sFlow, Ganglia, Graphite, collectd and StatsD. Whereas, in pull based architecture the central collector periodically requests each of the target node to send metrics to it. Examples of pull architectures include: SNMP, JMX, WMI, libvirt, prometheus,etc.
Prometheus is primarily a pull-based system, however it can act as a push based system by using pushgateway
Installation and setup
We will install
prometheus, which will pull metrics from the target server,
node_exporter which will make the target system’s metrics available at an HTTP port for
prometheus to pull; and
Grafana is the UI for an amazing visualisation.
Installation procedure is pretty simple, I’m going to show how to install on two platforms, Arch linux and Ubuntu. For others you can definitely follow the official docs
Arch has package for prometheus, which is great because then you don’t need to explicitly write unit file for the service.
sudo pacman -S prometheus
sudo systemctl enable prometheus(to create systemlink to the unit file in the systemd directory to that the systemd can always start it at boot)
sudo systemctl start prometheusto start the service right away.
Now if you wish to check whether it is running properly or not you can run
systemctl status prometheus.
tar -xzf prometheus-2.8.0.linux-amd64.tar.gz
- Move the
prometheusbinary executable to PATH
mv prometheus-2.8.0.linux-amd64/prometheus /usr/local/bin/
Configure the prometheus config file (
prometheus-2.8.0.linux-amd64/prometheus.yml) with simple configurations:-
global: scrape_interval: 15s # By default, scrape targets every 15 seconds. # Attach these labels to any time series or alerts when communicating with # external systems (federation, remote storage, Alertmanager). external_labels: monitor: 'codelab-monitor' # A scrape configuration containing exactly one endpoint to scrape: # Here it's Prometheus itself. scrape_configs: # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config. - job_name: 'prometheus' # Override the global default and scrape targets from this job every 5 seconds. scrape_interval: 5s static_configs: - targets: ['localhost:9090']
Now write a unit file for the prometheus service. For more info on systemd and unit files see here.
sudo nano /etc/systemd/system/prometheus.service
Add the following contents :-
[Unit] Description=Prometheus Wants=network-online.target After=network-online.target [Service] Type=simple ExecStart=/usr/local/bin/prometheus --config.file /home/ubuntu/prometheus-2.8.0.linux-amd64/prometheus.yml [Install] WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl enable prometheus
- Now if you wish to check whether it is not or not execute
systemctl status prometheus
Alternatively, you can use apt to install prometheus as well.
Now you can visit
<ip-address>:9090on browser to see promethues running. Kudos you’ve successfully set up prometheus on your monitoring server. (make sure port 9090 is open for http)
The Prometheus Node Exporter exposes a wide variety of hardware- and kernel-related metrics, which prometheus can scrape metrics from. Typically
node_exporter is installed on the target machine which you want to monitor and
prometheus is installed on a vertically scaled (it demands heavy resource, generally) server which is primarily used as the master to monitor the target servers.
Here I will show you how you can install
node_exporter on Debian and Windows server. Prometheus will keep pulling metrics from them and hence monitor.
sudo apt-get install prometheus-node-exporter
2) This by default enables and starts the node exporter service but you can cross check by
systemctl status node_exporter.service
node_exporter is not well-supported on windows and hence we will use an alternative
2) I would recommend to download and run the
.msi as can setup most of the things for you.
By default the service will start running on port 9182 so make sure to open that port to prometheus server.
Now, make this newly created node_exporter a target for prometheus to pull.
Add the following lines to promethues configuration file i.e
For the above mentioned
debian target server:-
- job_name: 'node_exporter' static_configs: - targets: ['<target-server-ip>:9100']
Similar for the above mentioned
windows target server but change the port address to
These ports are the default values and can be changed according to need by making necessary changes to configuration file of the exporters.
For the new target to get ready to be configured we need to restart the
prometheus service so that it read the updated configuration i.e
sudo systemctl restart prometheus.service.
The downside to this
restart is that this will cause a downtime during the restart process. To get around this downtime follow this neat trick.
The idea is to send a hang-up single to the
prometheus service which will make it reload the configuration.
For that first we need to know the PID (process ID) of the process.
ps aux | grep prome*
[souvik@quiche ~]$ ps aux | grep prome* prometh+ 29115 0.1 1.5 1142212 126020 ? Ssl Mar26 1:32 /usr/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus/data souvik 33410 0.0 0.4 891840 39956 pts/1 S+ 03:12 0:00 journalctl -u prometheus -f souvik 33445 0.0 0.0 6268 2316 pts/2 S+ 03:18 0:00 grep prome* [souvik@quiche ~]$
So now we know that the PID of the prometheus service is
Now we need the hang up signal to it.
sudo kill -s HUP 29115
This will make prometheus reload the configuration as evident in the below logs:-
Mar 27 03:20:36 quiche prometheus: level=info ts=2019-03-27T10:20:36.618684382Z caller=main.go:724 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml Mar 27 03:20:36 quiche prometheus: level=info ts=2019-03-27T10:20:36.619961054Z caller=main.go:751 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml
Now visit the
Status --> Targets on the prometheus’s address in the browser and your target server will appear there.
Now you can query for basic metrics and see it’s corresponding graph on the dashboard. But Grafana makes it cooler. Let’s grab it now.
Link to docs
sudo dpkg -i grafana_6.0.2_amd64.deb
sudo systemctl start grafana-server.service
sudo systemctl enable grafana-server.service
sudo pacman -S grafana
sudo systemctl enable grafana
sudo systemctl start grafana
Now open port 3000 because by default Grafana listens on port 3000.
Congratulations, your pretty dashboard is now setup correctly!
Now we need to configure Grafana to set
prometheus as a data-source.
2) Click on
Add data source
4) Let the defaults be. Check if the address the alright.
5) Import a pre-built dashboard but clicking on
405 as Dashboard ID.
Voila you have the pretty dashboard ready!
So that’s a brief walk-through of the setup of monitoring system for a linux and windows server. Now you can customize further and setup alerts for different scenarios using
Alert Manager. You can go through this cool video tutorial if you are more of a video person.
Thanks and I hope I was able to at least get you started with the popular and amazing monitoring tool Prometheus.