Prometheus was built at SoundCloud starting in 2012 and joined the Cloud Native Computing Foundation in 2016 as its second project after Kubernetes. It has since become the default way to collect metrics in Kubernetes and cloud-native systems, and its exposition format is supported by a large ecosystem of software.
How Prometheus works
- Pull model. The Prometheus server scrapes a
/metricsendpoint on each target at a set interval, often every 15 to 60 seconds. Applications expose metrics with a client library. - Exporters. Software that can't expose metrics itself gets an exporter that translates them: Node Exporter for Linux hosts, exporters for MySQL, PostgreSQL, Redis and many more, and the Blackbox Exporter for probing endpoints.
- Service discovery. Targets are found automatically from Kubernetes, cloud APIs, Consul or files, so new pods are scraped without config changes.
- Labels. Each time series is a metric name plus key-value labels, for example
http_requests_total{method="GET", status="500"}. - PromQL. The query language slices and aggregates series.
rate(http_requests_total{status=~"5.."}[5m])gives the per-second rate of 5xx responses over five minutes. - Alerting. Prometheus evaluates alert rules, and Alertmanager deduplicates, groups and routes the alerts to email, Slack, PagerDuty and other receivers.
- Dashboards are usually built in Grafana.
Metric types
- Counter: only goes up, such as total requests. You usually query its rate.
- Gauge: goes up and down, such as memory in use or queue length.
- Histogram: counts observations in buckets, such as request durations, so you can calculate percentiles.
- Summary: calculates quantiles in the client.
More detail: Prometheus gauges and counters.
Limits to plan for
- Single-node storage. A Prometheus server stores data on its local disk. Long retention, high availability and a global view across clusters need extra systems such as Thanos, Cortex or Grafana Mimir, or a remote-write backend. See Prometheus architecture and scalability.
- Cardinality. Labels with many values, such as user IDs or request IDs, multiply the number of series and memory use.
- Metrics only. Logs and traces need separate tools.
- You run it. Upgrades, scaling, storage and the alerting pipeline are your team's job.
Prometheus monitoring with Atatus
The Atatus Infrastructure Agent can scrape any Prometheus-format endpoint, with include and exclude filters and a namespace prefix, and stores the results as custom metrics you can chart and alert on next to APM, log and infrastructure data. Existing exporters keep working as they are. To compare options, see Prometheus alternatives.