Setting up Prometheus and Grafana Monitoring

In this post I want to talk about how to get started with some very nice monitoring tools, namely

• Prometheus
• Grafana

At the job I was at previously we used these tools a lot. They served as the backbone to our monitoring for our system overall, and we were chucking a lot of metrics into these tools, and without these tools I think its fair to say we would not have had such a good insight to where our software had performance issues/bottlenecks. These 2 tools really did help us a lot.

But what exactly are these tools?

Prometheus

Prometheus is the metrics capture engine, that come with an inbuilt query language known as PromQL. Prometheus is setup to scrape metrics from your own apps. It does this by way of a config file that you set up to scrape your chosen applications. You can read the getting started guide here : https://prometheus.io/docs/prometheus/latest/getting_started/ and read more about the queries here : https://prometheus.io/docs/prometheus/latest/querying/basics/

As I just said you configure Prometheus to scrape your apps using a config file. This file is called prometheus.yml where a small example is shown below. This example is set to scrape Prometheus own metrics and also another app which is running on port 9000. I will show you that app later on.

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090','localhost:9000']

So lets assume you have something like this in place, you should then be able to launch Prometheus using a command like this prometheus.exe –config.file=prometheus.yml, and then navigate to the following url to test out the Prometheus setup http://localhost:9090/

This should show you something like this

From here you can try and also go to this url which will show you some of the inbuilt Prometheus metrics : http://localhost:9090/metrics, so taking one of these say go_memstats_alloc_bytes, we could go back to http://localhost:9090/ and build a simple graph

So once you have verified this bit looks ok. We can then turn our attention to getting Grafana to work.

Grafana

As we just saw Prometheus comes with fairly crude graphing. However Grafana offers a richer way to setup graphs, and it also comes with inbuilt support for using Prometheus as a data source. To get started you can download from here  : https://grafana.com/docs/grafana/latest/guides/getting_started/. And to start Grafana you just need to launch the bin\grafana-server.exe, but make sure you also have Prometheus running as shown in the previous step. Once you have both Prometheus and Grafana running, we can launch the Grafana UI from http://localhost:3000/

Then what we can do is add Prometheus as a data source into Grafana, which can be done as follows:

So once you have done that we can turn our attention into adding a new Graph, this would be done using the “Add Query” button below.

If we stick with the example inbuilt metrics that come with Prometheus we can use the go_memstats_alloc_bytes one and add a new Panel and use the “Add Query” button above, where we can enter the following metric go_memstats_alloc_bytes{instance=”localhost:9090″,job=”prometheus”}, once configured we should see a nice little graph like this

I won’t go into every option of how to create graphs in Grafana, but one thing I can tell you that is very very useful is the ability to grab one graphs JSON representation and use it ton create other graphs. or you can also duplicate this is also very useful, both of these can be done by using the dropdown at the top of the graph in question

This will save you a lot of time when you are trying to create your own panels

Labels

I also just wanted to spend a bit of time talking about labels in Grafana and Prometheus. Labels allow you to to group items to together on one chart but distinguish them in some way based on the label value. For example suppose we wanted to monitor the number of GET vs PUT through some API, we could have a single metric for this but could apply some label value to it somehow. We will see how we can do this using our own .N ET code in the next section, but for now this is the sort of thing you can get with labels

This one was setup like this in Grafana

We will see how we did this is the next section.

Custom Metrics Inside Your Own .NET App

To use Prometheus in your own .NET code is actually fairly simple thanks to Prometheus.NET which is a nice Nuget package, which you can read more about at the link just provided. Once you have the Nuget installed, its simply a matter of setting up the metrics you want and adding your app to the list of apps that are scraped by Prometheus, which we saw above in the Yaml file at the start of this post. Essentially Prometheus.NET will expose a webserver at the port you chose, which you can then configure to be scraped.

Let’s see an example of configuring a metric. We will use a Gauge type, which is a metric that can go up and down in value. We will also make use of labels, which is what is driving the chart shown above.

using Prometheus;
using System;
using System.Threading;

namespace PrometheusDemo
{
    class Program
    {

        private static readonly Gauge _gauge =
            Metrics.CreateGauge("sampleapp_ticks_total", 
                "Just keeps on ticking",new string[1] { "operation" });
        private static double _gaugeValue = 0;
        private static Random _rand = new Random();

        static void Main(string[] args)
        {
            var server = new MetricServer(hostname: "localhost", port: 9000);
            server.Start();
            while (true)
            {
                if(_gaugeValue > 100)
                {
                    _gaugeValue = 0;
                    _gauge.Set(0);
                }
                _gaugeValue = _gaugeValue + 1;
                if(_rand.NextDouble() > 0.5)
                {
                    _gauge.WithLabels("PUT").Set(_gaugeValue);
                }
                else
                {
                    _gauge.WithLabels("GET").Set(_gaugeValue);
                }

                Console.WriteLine($"Setting gauge valiue to {_gaugeValue}");
                Thread.Sleep(TimeSpan.FromSeconds(1));
            }

            Console.ReadLine();
        }
    }
}

That is the entire listing, and that is enough to expose the single metric sampleapp_ticks_total with 2 labels

• POST
• GET

Which are then used to display 2 individual line graphs for the same metric inside of Prometheus

Conclusion

And that is all there is to it.Obviously for more complex queries, you will need to dig into the Prometheus PromQL query syntax. But this gets you started. The other thing I have not shown here is that Grafana is also capable of creating other types of displays such as these