Operating AI/ML Workloads on Kubernetes: A Headlamp Plugin for Kubeflow

Kubeflow exposes ML workloads as Kubernetes custom resources. A new Headlamp plugin brings those resources into a general-purpose UI, helping operators troubleshoot without switching tools.

axonn bots
axonn bots
·4 min read
The Headlamp Kubeflow plugin brings Kubeflow's custom resources into a general-purpose Kubernetes UI, helping cluster operators troubleshoot ML workloads without switching between tools. It reads directly from the Kubernetes API server and supports Notebooks, Katib, Pipelines, and other Kubeflow components.

Kubernetes has quietly become the default platform for AI and machine learning. Notebook servers, distributed training jobs, hyperparameter tuning, and multi-step ML pipelines all land on Kubernetes clusters. Kubeflow is one of the most popular ways to assemble that stack, and it does so the Kubernetes-native way: every capability is exposed as a Custom Resource Definition (CRD)[reference:24].

That design is a gift to cluster operators, because it means ML workloads can be observed and managed with the same primitives as everything else in the cluster. But in practice, the specialized ML dashboards that ship with these platforms hide the Kubernetes layer underneath. When a notebook is stuck or a training run fails, the operator is often left dropping back to kubectl to find out what actually happened at the Pod level[reference:25].

A plugin that closes the gap

The Headlamp Kubeflow plugin bridges this gap by displaying Kubeflow's custom resources directly inside a general-purpose Kubernetes UI[reference:26]. Headlamp itself is an extensible Kubernetes web UI maintained under Kubernetes SIG UI, and its plugin system lets anyone add first-class views for custom resources[reference:27].

The plugin reads directly from the Kubernetes API server. It shows Pod conditions, Kubernetes failure reasons, and resources across namespaces without requiring an intermediary ML service or database[reference:28]. This matters because purpose-built ML dashboards help data scientists submit experiments and pipelines, but cluster operators and SREs ask different questions: Why is this notebook pending? What Pod failure reason is blocking this training run? Which resources are consuming all the GPU memory in this namespace? The Headlamp Kubeflow plugin helps answer these questions directly.

What the plugin shows

Kubeflow is modular, and teams often install only the components they need. The plugin discovers the Kubeflow API groups on a cluster and displays only the corresponding sections.

The Notebook detail view shows Pod conditions and their status and reason fields. It also shows CPU, memory, and GPU requests and limits; volume mounts and their backing types (PersistentVolumeClaim, ConfigMap, Secret, or emptyDir); environment variables that reference Secret or ConfigMap objects; sidecar containers; and node tolerations. This view consolidates information that would otherwise require several kubectl commands.

The Katib views show the tuning algorithm, search space, every Trial with its live status, and the current best Trial with its metric values and parameter assignments. They also show the early-stopping configuration and the number of Trial resources that stopped early, so you can follow the search without leaving the cluster UI.

The Pipelines views read Kubernetes API resources directly and do not query the Kubeflow Pipelines API service or backend database. You can inspect stored pipeline state even when that service is unavailable. The Pipeline detail view compares the latest and previous PipelineVersion specifications in a side-by-side YAML diff. Run views show state and duration, RecurringRun views show human-readable schedules, and the artifacts view aggregates values from recent Run resources.

The plugin also registers a graph view that renders Notebook, Profile, PodDefault, Experiment, Pipeline, SparkApplication, and TrainJob resources as graph nodes. It draws edges between supported resources based on owner references. Headlamp also shows inline summaries for these resource types when you hover over them.

A pattern for any CRD-heavy platform

Kubeflow illustrates a broader pattern. Platforms often model domain-specific workflows with custom resources. Their dashboards focus on those workflows, while Kubernetes operators also need the state of the underlying API resources and Pods. A CRD-driven plugin in a general Kubernetes UI can expose that state without making operators switch between unrelated tools[reference:29].

The Headlamp Kubeflow plugin uses the Apache 2.0 license and is developed under Kubernetes SIG UI. The documentation explains installation and local-cluster setup, including a lightweight CRD-only path for evaluation. Because the plugin discovers installed API groups, you can use it with an existing modular Kubeflow installation or create an evaluation cluster with only the CRDs and sample resources.