1. Docs
  2. Changes and releases
  3. Observability

Observability

  • Observability grafana-app v2.39.0

    Changed

    • Fix ABS config to not override AppVersion in Chart.yaml
    • Upgrade grafana chart: 12.3.0 => 12.10.3
    • Upgrade grafana (appVersion): 13.0.1 => 13.1.2
    • Upgrade postgresql-cnpg: 18.3 => 18.4 — the rolling DB upgrade can cause ~5min of Grafana downtime.

    Removed

    • The upstream chart no longer creates a Role/RoleBinding when grafana.rbac.namespaced is false (our default). Those objects were empty, sidecar permissions come from the ClusterRole.
  • Observability observability-bundle v3.3.1

    Changed

    • Values: Generate schema for Alloy PodLogs CRDs.
    • Values: Add Cilium as dependency for Alloy apps & Kube Prometheus Stack.
    • Values: Update Alloy apps to v0.21.2.

    Removed

    • Values: Remove unused catalog.
  • Observability dashboards v4.29.1

    Added

    • Add a nav bar between the four Envoy Gateway dashboards, like the Cilium ones have.

    Changed

    • Give Envoy Gateway | Clusters a readable uid. Links to its old random uid stop resolving.
    • Rename the cluster variable on Cilium Agent logs to cluster_id, matching the other Cilium dashboards so the selection carries across links.
    • Rename workload_cluster to cluster_id on the Envoy Gateway dashboards, so the selection carries when moving between them. On Clusters its label is K8s Cluster, to tell it apart from the Envoy cluster selector.
    • Hide the datasource selector on the Cilium and Envoy Gateway dashboards and point it at the default datasource, instead of the exported uids that only resolved in the Grafana a board came from.

    Removed

    • Drop the Cluster selector from Envoy Gateway | Clusters. Every panel also matched on the namespace, resource and rule selectors, which compose the same Envoy cluster name.

    Fixed

    • Give Cilium | Agent its own uid instead of reusing the retired Cilium metrics dashboard’s, which kept it from being created.
    • Give Envoy Gateway | Control Plane its own uid instead of reusing the retired Global dashboard’s, which kept it from being created.
  • Observability alloy-app v0.21.2

    Changed

    • Chart: Install Vertical Pod Autoscaler independent of API availability.
  • Observability alloy-app v0.21.1

    Fixed

    • Setting alloy.enabled: false now disables the whole chart on its own and renders nothing. Previously it only dropped the upstream Alloy chart, and the Giant Swarm templates (NetworkPolicy, Kyverno PolicyException, VerticalPodAutoscaler, PodLogs, extra env Secret) failed to render with no template "alloy.chart" associated with template "gotpl" unless networkPolicy.flavor and kyvernoPolicyExceptions.enabled were also overridden.

    Added

    • Add a make test-chart target and a test-chart workflow that render the chart with every ci/*-values.yaml file and assert it renders nothing when disabled and all of its objects when enabled. Unlike ct lint, these use a current Helm, which is required to catch template rendering errors.
  • Observability dashboards v4.29.0

    Added

    • Add the Envoy Gateway | Overview dashboard: a lean single-pane health view (traffic, latency, error rate, saturation) that avoids duplicating the detail boards.
    • Add a downstream request-latency heatmap to Envoy Gateway | Clusters.
    • Add an Envoy Gateway controller logs panel to the Control Plane dashboard.
    • Publish the Flux Logs dashboard to Shared Org / GitOps. It covers the Flux controllers on the management cluster; workload-cluster Flux logs are not ingested yet, because the flux-system namespace there has no tenant assignment and Alloy drops untenanted pod logs.
    • Add scripts/update-flux-dashboards.sh (make update-flux-dashboards), which syncs the Flux dashboards from fluxcd/flux2-monitoring-example and re-applies our patches. It runs in the monthly dashboard-update workflow and fails loudly if a patch stops applying.
    • Add Organization and Cluster selectors to both Flux dashboards. They are deployed to the management cluster’s Grafana, whose Mimir holds metrics for every workload cluster, so until now every panel aggregated across a customer’s whole fleet.
    • Replace the Cilium dashboards with a set of four: Cilium | Overview for triage, Cilium | Agent, Cilium | Datapath & Policy and Cilium | Operator. Rows load collapsed and the per-agent panels use topk, so the entry point is no longer a 355 kB dashboard that fires 75 queries at once.
    • Add Cilium | Operator. The cilium-operator PodMonitor has always been enabled, but none of its metrics were graphed: identity garbage collection, workqueues, unmanaged pods and LB-IPAM.

    Changed

    • Consolidate the Envoy / Gateway API dashboards down to four: Overview, Clusters, Access Logs and Control Plane. The Control Plane board (formerly Global) absorbs the config push status panels, and Logs is renamed to Access Logs.
    • Synced Flux dashboards from fluxcd/flux2-monitoring-example@7ab65dc.
    • Anchor the etcd-health dashboard cluster selector to etcd_server_id instead of up, so clusters with a managed control plane (aks, eks) no longer appear as empty options.

    Removed

    • Remove the redundant Envoy Gateway | Proxy Global and Envoy Gateway | Configuration dashboards, folded into the boards above.

    Fixed

    • Fix the Flux Cluster Stats dashboard, which showed no data at all: every panel queried gotk_reconcile_condition, a metric the Flux controllers no longer expose. It now uses gotk_resource_info, the same metric our Flux alerting rules use, and gains upstream’s Suspended Objects panel.
    • De-duplicate gotk_resource_info on the Flux Cluster Stats dashboard. On management clusters both flux-ksm and kube-state-metrics export identical series, so the resource counts read exactly double.
    • Scope the Flux Control Plane dashboard to the Flux namespace. Its controller_runtime_* panels matched any operator exposing the same controller label values (for example controller="helmrelease" is also emitted by dex-operator and team-stamper).
    • Widen the Flux Control Plane range windows from [1m] to [$__rate_interval]. With our 60s scrape interval a one-minute window cannot hold two samples, so the reconciliation and API-request panels were empty. The ops/min panels use rate(...) * 60 so they stay per-minute rather than reporting the count over the whole (range-dependent) window.
    • Point the Flux Cluster Stats targets at the $datasource variable. Seven of them were pinned to a prometheus datasource uid that does not exist here, and the target-level datasource overrides the panel.
    • Restrict the Flux namespace selector on Flux Control Plane to the Flux controller pods. It was derived from a generic controller-runtime metric, so on a management cluster it also offered external-secrets, giantswarm and kube-system, and the dashboard could open on another operator’s data.
    • Fix the worker count expression on the Nodes Overview dashboard.
    • Fix the monthly dashboard-update workflow, which called a make update-mixin target that no longer exists (it was renamed to update-all-mixin), so the automation failed instead of opening a PR.

    Removed

    • Remove the Cilium Metrics and Cilium performance dashboards, replaced by the set above. 16 of their panels queried metrics Cilium no longer exposes (the kvstore and BPF syscall sections, the L7 proxy counters and the policy trigger panels), and Cilium performance duplicated the Overview.
  • Observability mimir-app v0.30.0

    Changed

    • Upgrade mimir-distributed chart to version 6.1.0.

    Fixed

    • Remove the unused enterprise/legacyLabels branches from the mimir.labels, mimir.podLabels and mimir.selectorLabels helpers, aligning them with upstream. The mimir-distributed 6.1.0 chart dropped the default enterprise block from its values, which made these helpers fail with a nil-pointer error during templating.

    Removed

    • Remove the prometheus.io/service-monitor: "false" label from the gateway service, reverting the 0.29.1 bugfix. It is no longer needed as the mimir-distributed 6.1.0 chart removes the gateway serviceMonitor upstream.
  • Observability tempo-app v0.18.0

    Changed

    • Fix ABS config to not override AppVersion in Chart.yaml
    • Upgrade Tempo Vulture chart from 0.12.10 to 0.13.0
      • Bumps the tempo-vulture image from 2.10.5 to 3.0.2, which is the only change in the chart
      • Pins -tempo-metrics-backoff-duration=0s: the TraceQL metrics check went from disabled to enabled (every 10s) upstream, and it cannot pass here because we do not enable the metrics-generator local-blocks processor
      • Drops support for the -rhythm-rf1-after flag (unused here) and starts honouring -tempo-push-tls, which was previously ignored (we leave it disabled)
    • Upgrade Tempo chart from 2.23.0 to 2.26.2
      • Upgrades Tempo from 2.10.5 to 2.10.7
      • Bumps the memcached image from 1.6.42-alpine to 1.6.43-alpine
      • Bumps the rollout-operator sub-dependency from 0.49.0 to 0.50.0 (not enabled here)
      • The metrics-generator now gets a livenessProbe (/ready, 60s initial delay), which it did not have when running as a StatefulSet (chart 2.25.4)
      • Ingester and metrics-generator moved onto the shared pod template; container ports are reordered and the ingester grpc port is now declared via ingester.extraPorts. Rendered output is otherwise unchanged, so this is a rolling restart only
      • Adds (disabled by default) ingest/Kafka write path with the experimental block-builder and live-store components, plus ingester.enabled to turn the ingester off (chart 2.24.0)
      • Adds a dnsConfig value on every component with defaults/tempo fallback, deprecating dnsConfigOverides (chart 2.26.1)
      • Adds distributor.shutdownDelay for graceful load-balancer draining, disabled by default (chart 2.26.0)
  • Observability mimir-app v0.29.1

    Fixed

    • disabled gateway metrics servicemonitor. Had been added in 0.29.0 but not working.
  • Observability mimir-app v0.29.0

    Changed

    • Fix ABS config to not override AppVersion in Chart.yaml
    • Upgrade chart to version 6.0.6
    • Upgrade mimir to 3.1.4