Observability is a very important topic to consider in any application just like Tests. It helps understand a lot about how our system works, and measure a lot of aspects to make data-driven decisions.

It can mostly be a go-to guide to what is happenning in your system, and how to improve it.

Parts of Observability

I’d like to categorize Observability into 3 parts:

  • Logging: Logging is the logs we generate from our application usually in levels like info, debug, error, warn, etc. These logs are custom messages that we can use to debug our application or monitor each event that happens in our application.
  • Metrics: Metrics or data points are the data we collect from our application to monitor the performance of our application. These metrics can be anything like the number of requests, response time, etc.
  • Tracing: Tracing is the process of monitoring the flow of a request through our application. It helps us understand how a request is processed in our application.

Setting up Observability

For logging

Loki and Promtail are very popular tools for setting up logging in your application. They are open-source tools that can be used to collect and store logs from your application.

Loki is a log aggregation system that can be used to collect logs from different sources and store them in a time-series database.

Promtail is a log collector that will collect the logs from your application and send them to Loki.

Docker compose example

services:

  grafana:
    image: grafana/grafana
    ports:
      - 3000:3000
  
  loki:
    image: grafana/loki
    ports:
      - 3100:3100
    volumes:
      - ./loki-config.yml:/etc/loki/local-config.yaml
  
  promtail:
    image: grafana/promtail
    volumes:
      - ./app.log:/var/log/app.log

We can explore logs from loki via Grafana.

log files can be explored via query: {filename="/var/log/app.log"}

For Metrics

Grafana and Prometheus are the most popular tools for setting up Observability in your application. They are open-source tools that can be used to monitor your application.

Grafana is a visualization tool that can be used to visualize the data collected by different monitoring tools.

Prometheus is a monitoring tool that will collect the data from your application and store it in a time-series database.

Docker compose example

Suppose we have an application running in port 8080 and we expose metrics in /metrics endpoint. We can use the following docker-compose file to set up Prometheus and Grafana.

services:
  
  prometheus:
    image: prom/prometheus
    ports:
      - 9090:9090
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
  
  grafana:
    image: grafana/grafana
    ports:
      - 3000:3000

Prometheus Configuration example

Prometheus works in a pull-based model. It will pull the metrics from the application. We need to configure Prometheus to scrape the metrics from our application.

global:
  scrape_interval: 15s

scrape_configs:
    - job_name: 'prometheus'
        static_configs:
        - targets: ['localhost:9090']
    - job_name: 'myapp'
        static_configs:
        - targets: ['localhost:8080']

PushGateway example

A service named pushgateway can be used to push metrics from an application which can then be scraped by Prometheus.

docker-compose example:

services:
  
  prometheus:
    image: prom/prometheus
    ports:
      - 9090:9090
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
  
  pushgateway:
    image: prom/pushgateway
    ports:
      - 9091:9091

prometheus.yml:

global:
  scrape_interval: 15s

scrape_configs:
    - job_name: 'prometheus'
        static_configs:
        - targets: ['localhost:9090']
    - job_name: 'pushgateway'
        static_configs:
        - targets: ['localhost:9091']

We can send request to pushgateway to push metrics:

curl -X POST -d 'metric_name{label="value"} 100' http://localhost:9091/metrics
  • POST request can be used to add / append the metrics.
  • PUT request can be used to replace the metrics.
  • DELETE request can be used to delete the metrics.

Using such tools, we can set up Observability in our application and monitor the performance of our application. We can also set up alerts to notify us when something goes wrong in our application.

This is very important and useful for any application.