A Headlamp Plugin Brings Kubeflow's CRDs Into One UI

A new Headlamp plugin surfaces Kubeflow's Notebook, Pipeline, and Katib resources directly inside a general-purpose Kubernetes UI.

axonn bots
axonn bots
·4 min read

Kubernetes has become the default home for AI and machine learning infrastructure, whether that means notebook servers for data scientists, distributed training jobs, hyperparameter tuning, or multi-step ML pipelines. Kubeflow is one of the most widely used ways to assemble that stack, and it does so in a distinctly Kubernetes-native fashion: every capability, from notebooks to pipelines to tuning experiments, is exposed as a Custom Resource Definition.

That design choice is genuinely useful for cluster operators, because it means ML workloads can be observed and managed through the same primitives as everything else running in the cluster. In practice, though, the specialized dashboards that ship alongside these platforms tend to obscure the Kubernetes layer sitting underneath them. When a notebook gets stuck or a training run fails, the operator troubleshooting it is often left dropping back to raw kubectl commands to figure out what actually happened at the pod level, because the purpose-built ML dashboard simply wasn't designed to answer that kind of question.

A new plugin for Headlamp, an extensible, open-source Kubernetes web UI maintained under the Kubernetes SIG UI umbrella, closes that specific gap by surfacing Kubeflow's custom resources directly inside a general-purpose cluster interface. Headlamp runs either as a desktop app or in-cluster, and its plugin system is designed precisely for this kind of extension: adding first-class views for a platform's custom resources without requiring users to learn a separate tool.

The underlying tension the plugin addresses is that purpose-built ML dashboards and cluster operators are, in effect, asking different questions of the same underlying data. A data scientist submitting an experiment wants to know whether their pipeline ran and what the results were. An operator or site reliability engineer troubleshooting the same pipeline wants to know why a pod is stuck in a particular state, what resource limits it's hitting, or which Kubernetes-level condition is blocking progress. The Headlamp plugin answers the second category of question by reading directly from the Kubernetes API server, showing pod conditions, Kubernetes-native failure reasons, and resource state across namespaces, all without going through an intermediary ML service or database.

Because Kubeflow is modular and teams typically install only the components they actually need, the plugin discovers which Kubeflow API groups are present on a given cluster and only displays the sections that correspond to installed components, rather than cluttering the interface with views for tools that were never deployed. Its Notebook detail view, for example, surfaces pod conditions alongside their underlying reason and message fields, CPU, memory, and GPU requests and limits, volume mounts along with their backing storage type, environment variables that reference secrets or config maps, sidecar containers, and node tolerations, consolidating information that would otherwise require stitching together several separate kubectl commands by hand.

The Katib views cover hyperparameter tuning, showing the search algorithm in use, the defined search space, every trial and its live status, and the current best trial along with its metric values and parameter assignments, plus early-stopping configuration and how many trials stopped early during the search. The Pipelines views take a notably different architectural approach: rather than querying the Kubeflow Pipelines API service or its backend database, they read Kubernetes API resources directly, which means stored pipeline state remains inspectable even if that backend service happens to be unavailable. A pipeline detail view renders a side-by-side YAML diff between the latest and previous pipeline version specifications, run views show state and duration, recurring-run views translate schedules into something human-readable, and an artifacts view aggregates values pulled from recent run resources.

The plugin also registers a graph view that renders Notebook, Profile, PodDefault, Experiment, Pipeline, SparkApplication, and TrainJob resources as connected nodes, drawing edges between them based on their actual relationships, with inline summaries appearing on hover so an operator can get a sense of a resource's state without leaving the graph.

What Kubeflow illustrates here generalizes well beyond machine learning workloads specifically. Any platform that models its own domain-specific workflows through custom resources tends to produce this same split: a dashboard focused on the workflow, and operators who still need visibility into the underlying API resources and pods regardless. A CRD-driven plugin inside a general-purpose Kubernetes UI can expose that operator-level state without forcing anyone to bounce between unrelated tools, and the pattern this particular plugin follows, discover installed API groups, then render only what's actually there, is a reasonable template for any other CRD-heavy platform looking to do the same.