One place for logs, metrics and traces
Logs, metrics and traces in one place, so on-call engineers can follow an incident without SSH access.
- Client
- US clients, via a consultancy and directly
- Role
- Site reliability engineer and team lead, then consultant
- Years
- 2023–present
Situation
Several US clients ran on Kubernetes. The goal was the same each time: when something breaks, the on-call engineer can follow the incident through logs, metrics and traces in one place, without SSH access to the machines.
What I did
- Deployed and operated the Grafana stack inside Kubernetes: Prometheus for metrics, Loki for logs, Tempo for traces and Mimir for long-term metric storage.
- Wired Prometheus service discovery, so new services are scraped when they are deployed.
- Set retention for each signal, so the stack keeps what investigations need without growing without limit.
- Wrote the alert rules and built the dashboards, so they ship with the service.
Result
On-call engineers correlate logs, metrics and traces from one Grafana instead of logging in to machines.
Tools
- Prometheus
- Grafana
- Loki
- Tempo
- Mimir
- Kubernetes