How to Design an Observability Stack
Observability rests on three pillars: metrics (what's wrong), logs (why), and traces (where). A good stack combines them so you can go from 'something's slow' to the exact cause. Here's how to design one.
The three pillars
| Pillar | Answers | Example tool |
|---|---|---|
| Metrics | What is wrong + when - rates, errors, latency over time | Prometheus / Grafana |
| Logs | Why - the detailed events around a specific request/error | Loki / ELK |
| Traces | Where - which service/span in a request is slow | Tempo / Jaeger / OpenTelemetry |
Each answers a different question. The mistake is picking one and hoping it covers all three.
How they work together on a real incident
"A perf regression hit production two days ago." To actually debug it you need:
- Metrics to see it and when it started - a latency graph showing the step change two days ago.
- Logs (or traces) to find why - the slow requests' details, the error, the change.
A metrics-only stack tells you something's slow but not why. A logs-only stack drowns you without a way to spot the trend. Metrics + logs (and ideally traces) is what lets you go from symptom to cause.
Designing the stack
- Start with metrics + logs - the two that cover the most incidents for the least setup. Add distributed tracing when you have multiple services and "which hop is slow?" becomes the question.
- Correlate them - put a request/trace id in logs and metrics labels so you can jump from a spiking metric to the exact logs/trace. Correlation is what makes the stack greater than its parts.
- Instrument once with OpenTelemetry - it emits metrics, logs, and traces in a vendor- neutral format, so you can swap backends without re-instrumenting.
- Alert on symptoms, not causes - alert on user-facing SLOs (latency, error rate), then use logs/traces to find the cause; don't page on every internal metric.
The takeaway
Observability isn't one tool - it's metrics to detect, logs to explain, and traces to localize, correlated by shared ids. Choose the combination that lets you answer the questions your incidents actually pose.
Want to try it hands-on? HeyDevJob gives you this exact setup in a live cloud workspace in your browser - edit it, run it, and see it work. Free, nothing to install.
Try it in a workspace →What you'll practice
- Mapping incidents to the pillar that answers them
- Correlating metrics/logs/traces with a shared request id
- Choosing the minimal stack that can actually debug your scenarios
FAQ
What are the three pillars of observability?
Metrics (aggregate rates/errors/latency over time - what's wrong and when), logs (detailed event records - why), and traces (the path of a request across services - where the time goes). A good stack combines them.
What's the difference between metrics and logs?
Metrics are numeric aggregates over time, great for spotting trends and alerting (e.g. latency rose two days ago). Logs are detailed per-event records that explain a specific request or error. You need both: metrics to detect, logs to diagnose.
Do I need distributed tracing?
Once you have multiple services and the question becomes 'which hop is slow?', yes - traces show where in a request the time goes. For a single service, metrics + logs usually suffice.
What are the three pillars of observability?
Metrics (numeric time series for trends and alerts), logs (discrete event records for detail), and traces (the path of a request across services). Together they let you detect, investigate, and pinpoint a production incident.
Keep learning
Learn it by doing. Open this in a live cloud workspace, make the change yourself, and keep a record of the work you can share.
Open the workspace →