1. Docs
  2. Changes and releases
  3. Observability

Observability

  • Observability observability-bundle v3.6.0

    Changed

    • !Breaking change: Turn apps.<appName>.dependsOn from a list into a map from dependency name to true/false, so that a provider can turn off a single dependency (for example kyverno-crds on clusters without the security bundle) instead of overriding the whole list.

    Added

    • KSM custom resource state: Add status_parent_accepted and status_parent_resolved_refs gauges for Gateway API HTTPRoute, one series per parent with the condition reason as label.
    • KSM custom resource state: Add target_info and status_ancestor_accepted metrics for Envoy Gateway SecurityPolicy, BackendTrafficPolicy, ClientTrafficPolicy and EnvoyExtensionPolicy.
  • Observability observability-platform-api v0.6.0

    Changed

    • Change the missing X-Scope-OrgID response body from Bad Request to Missing X-Scope-OrgID header in the Loki, Mimir, Tempo and Basic Auth headers-check filters. The status stays 401.
    • Fail the render when loki.enabled, mimir.enabled or tempo.enabled is true and auth.jwt.providers is empty.

    Removed

    • Remove the Loki OTLP gRPC route and loki.write.grpc. Loki serves OTLP over HTTP only.

    Fixed

    • The helm.sh/chart label is valid for long chart versions: the 63-character cut trims the whole trailing run of -, . and _.
  • Observability dashboards v4.32.0

    Added

    • Add AWS NLB Metrics dashboard to the Envoy Gateway folder.
    • Add Log export dashboard to the Observability folder.
  • Observability grafana-app v2.41.0

    Added

    • Added the Grafana image renderer configuration, disabled by default.
    • Added a CiliumNetworkPolicy for the image renderer.
    • Documented image rendering in the chart README.

    Changed

    • Updated values.schema.json for the imageRenderer values.
    • Upgrade grafana chart: 12.11.2 => 13.2.7
    • Upgrade postgresql-cnpg: 18.4 => 18.6

    Fixed

    • The helm.sh/chart label is valid for long chart versions: the 63-character cut trims the whole trailing run of -, . and _.
    • Trim trailing . from the helm.sh/chart label, to fix build.
  • Observability alloy-app v0.24.0

    Added

    • Add crossplane.aws.iam.enabled, rendering a Crossplane IAM Role for IRSA even while alloy.enabled is false. Off by default.

    Changed

    • Upgrade Alloy upstream chart from 1.12.1 to 1.13.0 (CHANGELOG)
      • This bumps the version of Alloy from 1.19.2 to 1.20.0 (CHANGELOG).
      • Alloy Breaking changes
        • [otelcol.processor.k8sattributes] Default label and annotation keys use the singular form (k8s.pod.label.<key>). container.image.tags replaces container.image.tag. deployment_name_from_replicaset is removed.
        • [otelcol.processor.transform] Base64Decode is removed. set(target, nil) sets target to nil.
        • See Alloy v1.20.0 release notes
      • Update prometheus-config-reloader from v0.91.0 to v0.94.0.
      • Add controller.dnsConfig and service.trafficDistribution values.

    Fixed

    • The helm.sh/chart label is valid for long chart versions: the 63-character cut trims the whole trailing run of -, . and _.
  • Observability dashboards v4.31.0

    Removed

    • Remove the agent_platform sub-chart and its LLM usage dashboard.
  • Observability mimir-app v0.30.1

    Added

    • Allow InPlaceOrRecreate and InPlace as verticalAutoscaling.updateMode values in values.schema.json.

    Changed

    • Set the ruler’s VPA updateMode to Initial (was Recreate).
  • Observability observability-bundle v3.5.0

    Changed

    • Update alloy apps to 0.23.1 (Alloy v1.19.2).
    • Update prometheus-operator-crd to 24.0.0 (Prometheus Operator CRDs v0.94.0).
    • Update kube-prometheus-stack to 24.0.0 (chart 91.2.3, Prometheus Operator v0.94.0).

    Added

    • Point kube-prometheus-stack’s control-plane ServiceMonitors at the alloy-metrics token Secret.

    Fixed

    • KSM custom resource state: Set the Gateway API TCPRoute and UDPRoute collectors to v1, which is the version the API server serves.
  • Observability kube-prometheus-stack-app v24.0.0

    Changed

    • Upgraded chart dependency to kube-prometheus-stack-91.2.3 (from 88.6.3, via 90.0.0)
      • prometheus-operator (and prometheus-config-reloader) from v0.93.1 to v0.94.0
      • grafana subchart from 12.11.2 to 13.2.4, which switches the Grafana image from 13.2.0 to 13.2.1-distroless, enables readOnlyRootFilesystem on the Grafana container (with a new /tmp emptyDir) and sets plugins.preinstall_auto_update = false
      • k8s-sidecar (Grafana dashboard/datasource sidecar) from 2.10.1 to 2.11.2
      • kube-state-metrics subchart from 8.4.1 to 8.4.2 (KSM image unchanged at v2.20.0)
      • prometheus-node-exporter subchart from 4.56.3 to 4.57.0 (we keep nodeExporter.enabled: false, no impact)
      • Alertmanager, Prometheus, Thanos and kube-webhook-certgen images are unchanged.
    • Breaking (chart 90.0.0): the control-plane ServiceMonitors (kubelet, kube-apiserver, kube-controller-manager, kube-scheduler, kube-etcd, kube-proxy, coredns) no longer use bearerTokenFile/tlsConfig.caFile. They now authenticate through a kubernetes.io/service-account-token Secret (<release>-prometheus-token, newly created by the chart) and read the CA from the kube-root-ca.crt ConfigMap. This makes the ServiceMonitors work with arbitraryFSAccessThroughSMs.deny and with Grafana Alloy’s prometheus.operator.servicemonitors component (>= v1.19.0), which silently dropped every control-plane target before.
    • Breaking (chart 91.0.0 / operator v0.94.0): the Prometheus Operator ClusterRole no longer grants wildcard (*) verbs. Verbs are now explicit per resource group, and the operator only gets get/list/watch on the monitoring.coreos.com CRs plus writes on their /status and /finalizers subresources.
    • prometheusOperator.admissionWebhooks.matchConditions changed type from a map ({}) to a list ([]). We do not set it, so no impact.
    • Fixed the chart appVersion, which was still v0.92.0, to match the bundled Prometheus Operator (v0.94.0).
    • Added global.controlPlaneScrapeAuth to configure the Secret the control-plane ServiceMonitors authenticate with.
    • Dropped the scrape credential for coreDns, kubeEtcd and kubeProxy.
    • Removed insecureSkipVerify from the kubeControllerManager and kubeScheduler ServiceMonitors.

    Note: release the matching prometheus-operator-crd app (CRDs chart 32.0.0, Prometheus Operator v0.94.0) before this one. The CRDs shipped by this chart live in charts/crds/crds/ and are therefore only applied by Helm on install, never on upgrade.

  • Observability prometheus-operator-app v24.0.0

    Changed

    • Upgraded chart dependency to kube-prometheus-stack-91.2.3 (from 88.6.3, via 90.0.0)
      • prometheus-operator (and prometheus-config-reloader) from v0.93.1 to v0.94.0
      • grafana subchart from 12.11.2 to 13.2.4, which switches the Grafana image from 13.2.0 to 13.2.1-distroless, enables readOnlyRootFilesystem on the Grafana container (with a new /tmp emptyDir) and sets plugins.preinstall_auto_update = false
      • k8s-sidecar (Grafana dashboard/datasource sidecar) from 2.10.1 to 2.11.2
      • kube-state-metrics subchart from 8.4.1 to 8.4.2 (KSM image unchanged at v2.20.0)
      • prometheus-node-exporter subchart from 4.56.3 to 4.57.0 (we keep nodeExporter.enabled: false, no impact)
      • Alertmanager, Prometheus, Thanos and kube-webhook-certgen images are unchanged.
    • Breaking (chart 90.0.0): the control-plane ServiceMonitors (kubelet, kube-apiserver, kube-controller-manager, kube-scheduler, kube-etcd, kube-proxy, coredns) no longer use bearerTokenFile/tlsConfig.caFile. They now authenticate through a kubernetes.io/service-account-token Secret (<release>-prometheus-token, newly created by the chart) and read the CA from the kube-root-ca.crt ConfigMap. This makes the ServiceMonitors work with arbitraryFSAccessThroughSMs.deny and with Grafana Alloy’s prometheus.operator.servicemonitors component (>= v1.19.0), which silently dropped every control-plane target before.
    • Breaking (chart 91.0.0 / operator v0.94.0): the Prometheus Operator ClusterRole no longer grants wildcard (*) verbs. Verbs are now explicit per resource group, and the operator only gets get/list/watch on the monitoring.coreos.com CRs plus writes on their /status and /finalizers subresources.
    • prometheusOperator.admissionWebhooks.matchConditions changed type from a map ({}) to a list ([]). We do not set it, so no impact.
    • Fixed the chart appVersion, which was still v0.92.0, to match the bundled Prometheus Operator (v0.94.0).
    • Added global.controlPlaneScrapeAuth to configure the Secret the control-plane ServiceMonitors authenticate with.
    • Dropped the scrape credential for coreDns, kubeEtcd and kubeProxy.
    • Removed insecureSkipVerify from the kubeControllerManager and kubeScheduler ServiceMonitors.

    Note: release the matching prometheus-operator-crd app (CRDs chart 32.0.0, Prometheus Operator v0.94.0) before this one. The CRDs shipped by this chart live in charts/crds/crds/ and are therefore only applied by Helm on install, never on upgrade.