Why you almost certainly need a custom exporter
Kubernetes has a built-in sense of two signals: CPU and memory. Most production scaling decisions are driven by signals Kubernetes does not see at all. How many messages are sitting in a queue. How long the last batch job took. How many active WebSocket connections a pod is holding. If your autoscaling rules depend on those numbers, you need a custom metrics exporter, a small HTTP server whose only job is to expose your application state as Prometheus text on a /metrics endpoint.
The format is simple: one metric per line, with a name, optional labels, and a numeric value. Client libraries handle the serialization. The real work is choosing what to measure and updating it at the right cadence.
Pick the right metric type for each signal
Before writing any code, decide which Prometheus type fits each signal.
- Counters only ever go up. Use them for things like requests served, jobs processed, errors encountered. If the value can ever decrease, a counter is the wrong tool.
- Gauges represent a current snapshot of a value that rises and falls freely. Queue depth, active connections, cache size.
- Histograms record the distribution of observed values, which is how you get percentiles like p99 or p50 rather than just a moving average.
A job processor will typically need at least one of each: a counter for jobs processed, a gauge for queue depth, and a histogram for per-job latency. Pick names that follow the existing Prometheus conventions so anyone reading the metrics six months from now has a chance of understanding them.
The exporter itself, in Go
The Go Prometheus client is the most common choice in the Kubernetes ecosystem, largely because the same library powers most of the official Kubernetes components. Start a new module and pull in the dependency:
Declare the metrics in a file, register them with Prometheus's default registry, and let the library handle the rest. Registration matters: it makes the metric appear in the output even before the first observation, which keeps dashboards from breaking on a cold start. MustRegister panics on a duplicate registration, which is what you want. If your exporter might be embedded in a library that other packages instrument, use the non-panicking variant and handle the error.
The next step is keeping the values current. The cleanest pattern is a polling goroutine that reads from the data source your application owns and updates the registered metrics on an interval. The interval should be shorter than the scrape interval, not longer. Most clusters scrape every 15 seconds, so a 5-second poll gives comfortable headroom.
Wire the collection loop and the HTTP handler together. A separate /healthz route is worth keeping alongside /metrics, so Kubernetes has a liveness probe target that does not accidentally leak metric data on the health route.
Containerize with a multi-stage build
A two-stage Dockerfile keeps the final image small and avoids shipping a Go toolchain to production:
- The first stage compiles a statically linked binary in a Go builder image.
- The second stage copies only that binary into a minimal base such as
gcr.io/distroless/static:nonroot.
distroless contains no shell and no package manager and runs as a non-root user by default, which satisfies most cluster security policies without extra configuration. Buildah, Podman, or any OCI-compatible tool can run the same build.
Run it in the cluster
Two Kubernetes manifests are enough. A Deployment manages the pod lifecycle. A Service gives Prometheus a stable address to scrape.
If you installed Prometheus through the kube-prometheus-stack Helm chart or the Prometheus Operator, the operator must be running in the cluster before you create a ServiceMonitor. The release label on the ServiceMonitor has to match the label selector on the Prometheus resource. prometheus is the default for a standard Helm install, and that is the only label that has to be right for the target to be discovered.
If your Prometheus uses annotation-based pod discovery instead, the matching rule lives in the Prometheus configuration. Check with whoever runs your Prometheus to confirm. As a defensive measure, the following annotations on the Pod template work for both setups, and the Prometheus Operator simply ignores them:
Verify the pipeline end to end
Port-forward to the Prometheus service and open the targets page. The exporter should appear with state up. If it shows up as down, the most common cause is a label selector mismatch on the ServiceMonitor, or a pod that is not running.
Once the target is healthy, run a quick query in the expression browser to confirm data is flowing:
A non-zero result means the full pipeline is working. The application is producing data, Prometheus is scraping it, and the time series is stored and queryable.
Where to go next
A working exporter is the foundation, not the destination. The natural next step is surfacing these metrics to the Horizontal Pod Autoscaler so the workload scales on the signals that actually drive load, not just on CPU. That requires a metrics adapter, and the Prometheus Adapter is the most widely deployed option. Once it is in place, any HPA in the cluster can reference the custom metric directly in its metrics block.
For a catalog of ready-made exporters covering databases, message brokers, and cloud services, the Prometheus exporters hub is a good place to start. The pattern you just built is the same pattern those exporters use.