Selected work

One place for logs, metrics and traces

Logs, metrics and traces in one place, so on-call engineers can follow an incident without SSH access.

Client
US clients, via a consultancy and directly
Role
Site reliability engineer and team lead, then consultant
Years
2023–present

Situation

Several US clients ran on Kubernetes. The goal was the same each time: when something breaks, the on-call engineer can follow the incident through logs, metrics and traces in one place, without SSH access to the machines.

What I did

  • Deployed and operated the Grafana stack inside Kubernetes: Prometheus for metrics, Loki for logs, Tempo for traces and Mimir for long-term metric storage.
  • Wired Prometheus service discovery, so new services are scraped when they are deployed.
  • Set retention for each signal, so the stack keeps what investigations need without growing without limit.
  • Wrote the alert rules and built the dashboards, so they ship with the service.
Observability stack inside KubernetesServices on Kubernetes produce three signals. Prometheus collects metrics, using service discovery, and stores them long term in Mimir. Loki collects logs. Tempo collects traces. Grafana reads Mimir, Loki and Tempo and holds the dashboards and alert rules for all three signals in one place. Alerts go to the on-call engineer, who follows an incident in Grafana without SSH access.
Each signal has its own store; Grafana brings them together for dashboards, alerts and incident work.

Result

On-call engineers correlate logs, metrics and traces from one Grafana instead of logging in to machines.

Tools

  • Prometheus
  • Grafana
  • Loki
  • Tempo
  • Mimir
  • Kubernetes

All selected work

Have a similar platform problem? Tell me about it.

Book a 30-minute call