Fixed
- (deps) Update module github.com/valkey-io/valkey-go to v1.0.78 in #1230 by @renovate[bot]
Full Changelog: https://github.com/giantswarm/muster/compare/v5.19.15...v5.19.16
Full Changelog: https://github.com/giantswarm/muster/compare/v5.19.15...v5.19.16
Full Changelog: https://github.com/giantswarm/klaus/compare/v0.1.16...v0.1.17
Full Changelog: https://github.com/giantswarm/agent-platform/compare/v4.19.0...v4.19.1
klausGateway.observability.otlpEndpoint was empty on every installation, so the gateway’s tracer provider was a no-op, the A2A call carried no trace context and the kagent controller’s SendStreamingMessage trace in Tempo had no parent — a Slack round trip could only be reconstructed from five logs. klaus-gateway 1.10.0 opens one span per turn (slack.turn, the parent of the controller’s trace and of the actor’s spans under it), takes a URL endpoint and an otlpHeaders knob; the meta chart now sets both to the target kagent’s exporters use — http://otlp-gateway.kube-system.svc:4317 with X-Scope-OrgID: giantswarm — under a new klausGateway.observability.enabled: auto that follows the resolved global.observability.metrics.serviceMonitor.enabled the way kagent.otel.*.enabled does (an explicit true/false wins; off, the release gets an empty endpoint and no headers; the knob itself never reaches the klaus-gateway chart, whose closed observability block takes the endpoint and headers only — components.klaus-gateway.omitKeys). The connectivity chart renders -klausgateway-otlp-egress exactly while the endpoint is set — DNS plus the endpoint’s namespace on its port (the cluster entity for a host that is not an in-cluster Service), in both policy flavours — through the destination parsing and the rule the kagent controller’s and the actors’ OTLP egress already used, now shared as agent-platform.otlpTarget / agent-platform.otlpEgressRule. components.klaus-gateway.versionRange moves to >=1.10.0 <2.0.0 (the floor: an older chart’s schema refuses the headers knob) and the example BOM pins 1.10.0. make verify-klausgateway-otlp asserts the forwarding, the auto resolution and the overrides, both flavours of the policy, the guards, and that the kagent controller’s rule is unchanged. Every installation with the klaus-gateway component on rolls it once (the new chart and the two new arguments) and starts exporting one trace per Slack turn to its Tempo tenant.initialize and the agent’s first tools/list answered only after the last server had connected, 2.7-3.2 s on an installation with 81 such servers, although tools/list returns the meta-tools and needs none of the connections. The fan-out now starts on the first request and runs in the background; initialize and tools/list answer at once. A call_tool (or resource read, or prompt) for a server still connecting waits for that server’s connect alone, resolved from the tool name’s prefix or its family, not for the whole fan-out; list_tools, filter_tools and describe_tool wait for the fan-out so a session’s first listing is complete. The person’s subject-scoped grants (GitHub-style connectors) are adopted in the same fan-out instead of sequentially on the first call that misses them. Requests arriving while the fan-out runs share it; the sign-in flow still connects before it issues the access token. The fan-out logs its total duration, outcome counts and slowest server (SSO: fan-out finished … duration_s=… slowestServer=…), each connect logs its duration, and a held request logs how long it waited (SSO: tool call waited for its server's connect). muster test: a step’s max_duration bounds one invocation and fails the step when it ran longer; an OAuth-protected mock takes connect_delay to stand in for a backend that is slow to connect. Scenario: oauth-sso-session-start-answers-during-fan-out.karpenter.k8s.aws/instance-generation next to the vendor in kagent.substrateWorkerPool.template.nodeSelector — is documented, render-guarded and verified (giantswarm/agent-platform#457, the caveat #430 recorded). One pool runs one CPU feature set: an actor’s golden snapshot is a gVisor checkpoint that restores only on a host whose CPU offers every feature it recorded, and #429’s vendor pin left the generation open — on gazelle (2026-09-14) Karpenter put two workers on an m7a.2xlarge (generation 7, AVX-512) and two on c5ad.xlarge (generation 5), the goldens were taken on the m7a workers, and every cold turn placed on a c5ad worker failed incompatible FeatureSet: missing features: … avx512f … twice (30 s each), then SendStreamingMessage Internal, “the turn failed” in Slack; the next morning’s pool (c6a, c5a ×2, m6a) restored everywhere only because the goldens had been re-taken on a gen-5 worker. The pin is a second label in the same map, set per installation (the fleet template renders it from the installation’s values the way it renders the vendor): pick a generation that is one CPU model across its families — for AMD on AWS 6 (c6a/m6a/r6a, EPYC Milan) or 7 (c7a/m7a/r7a, Genoa); 5 mixes Naples (m5a/r5a) with Rome (c5a/c5ad) and is no feature-set pin — and one the goldens restore on (an older generation’s golden restores on a newer one, never the reverse), and check the spot pool it leaves (the generation’s families × the NodePool’s sizes; three families is the floor). The meta chart now refuses a nodeSelector value that is not a string naming the key (agent-platform.validateWorkerPool): the label is "6", and a bare 6 rendered, passed the open kagent schema and would have failed only when helm-controller applied the kagent release on every installation carrying it. make verify-workerpool (in CI) asserts the default forwards the architecture alone, an installation’s three-label pin reaches the kagent release verbatim with every value a string, the unquoted generation fails the render, and the kagent chart the range resolves to renders the pin unchanged into the one WorkerPool’s spec.template.nodeSelector — the lab has no Karpenter labels, so this render is where the pin is proven. Docs: values.yaml, the chart README, the root README’s Substrate section and UPGRADE.md name the generation next to the vendor. A restore that fails with incompatible FeatureSet is retried on the same worker; placing it on another worker is Substrate’s (giantswarm/giantswarm#37742 row 12), not this chart’s. No rendered object changes for an installation that sets no generation.core_*, workflow_<name>) are kept for five minutes as the backstop for a definition change no invalidation saw; the meta-tool call that found them older paid the rebuild – one list of every Workflow resource from the API server plus their conversion, 1.9-2.05 s with 282 workflows – while a warm session had been answering the same calls in 7-20 ms. Now that call is served the catalogue it has and starts one rebuild in its own goroutine; concurrent calls share it, and the rebuilt catalogue is swapped in when it lands. A call without a catalogue – the first of the process, or after a workflow was created, updated or deleted – still waits for the rebuild, so a definition change stays visible on the very next call; a change that arrives while a rebuild runs makes it build again before anything is served. A change in the server registry – a backend server registered, lost, or re-registered by the reconciler’s status sync – no longer invalidates the core catalogue either: it depends on the tool providers alone, not on which servers are registered, yet every registry change ran the capability refresh that threw it away, so on an installation with 87 servers the meta-tool calls after a muster restart paid the rebuild again and again (1.2-3.3 s calls minutes apart, none of them at an age boundary). Every rebuild is logged at debug (Core catalogue rebuilt, with its trigger and duration_s). MUSTER_CORE_CATALOGUE_MAX_AGE overrides the age (the integration test harness runs it at 3 s).oauth.server.storage did not answer at that moment – muster and Valkey starting together, a Valkey OOM or a node roll while muster restarts – muster logged Failed to create Valkey client for session stores, falling back to in-memory and kept in-memory stores: muster:cap:* and muster:auth:* never appeared in Valkey while sessions worked, the next restart lost every session’s view and SSO state, and the OAuth token store, which retries its own connection, ended up in Valkey regardless. Now muster dials the configured Valkey again with backoff (1 s doubling to 8 s) for up to 20 s (under the chart’s 30 s liveness window), logging Valkey for the session stores did not answer, retrying in between and serving nothing until it answers; when it stays away the start fails with Configured Valkey did not answer; refusing to serve sessions on in-memory stores and muster serve exits non-zero, so the kubelet restarts the pod. In-memory stores remain the behaviour of a deployment that configured no Valkey. The backend in use is published as the gauge muster_session_store_backend{backend="valkey|memory"} (1 for the backend in use, 0 for the other). The Valkey client’s client-side cache, which no store reads through, is off (CLIENT TRACKING is no longer requested per connection).agent-platform-connectivity follows the meta chart’s own version; a pin on it is refused (giantswarm/agent-platform#450, the root cause behind #431, #441 and #339). The wiring chart is published off the same git tag as this chart — two charts, one commit, one version — yet its versionRange was a range of its own (>=4.0.0 <5.0.0) that a BOM pinned separately, so every BOM’s connectivity lagged the moment the meta chart moved: three releases in a row forwarded a key (gateway.parameters.podAnnotations, vm-manager / vmManager, postgres.affinity / imagePullSecrets) that the pinned connectivity’s closed schema refused, and the release failed on every BOM-pinned installation. The example BOM’s own pin was fourteen releases behind. The connectivity entry is now releasedWithChart: true with an empty versionRange: the meta chart renders the OCIRepository at its own exact version (agent-platform.chartVersion — build metadata dropped, a pre-release kept, the same derivation the self range floors on), so Flux rolls the connectivity release exactly when the meta chart rolls and never ahead or behind. A versionRange on such an entry is refused when it would pull from the repository this chart’s releases come from (gitops.self.repository) with no semverFilter — that is the lagging pin; the two development shapes keep their knobs (a semverFilter for a dev channel, another repository for a chart pushed by hand — the ATS’s in-cluster registry, a lab registry). examples/customer-bom.yaml no longer pins connectivity and tests/verify-components.py refuses one; the ATS smoke and own-flux scenarios point the component at the in-cluster registry and let the version derive, proving it against a real source-controller with a pre-release version. Every installation sees its connectivity OCIRepository move from the range to the exact version of the meta chart it runs — the same chart the range resolved to for an installation on the current release, an upgrade for one whose connectivity lagged (UPGRADE.md).make verify-components-charts renders every component of the meta chart’s roster, and the BOM has to pin all of it (giantswarm/agent-platform#278 follow-up, #450). The check rendered thirteen components named in the script — the seven extras, the two managers, the kagent and Substrate lines — so agent-platform-connectivity, klaus-gateway, muster, agentgateway, valkey, agent-platform-mcps, agent-sandbox, dicebear and vm-manager were never rendered against the values the meta chart forwards them; that list is exactly what let #278’s three breaks reach main. The set is now the meta chart’s own roster (every components.* entry of values.yaml with a chart), and examples/customer-bom.yaml must match it both ways: a roster entry the BOM does not pin, a pin that names no component, a pin the render carries differently than the BOM’s line, a pin no registry publishes, and a pin on a chart released with the meta chart each FAIL. Every component is rendered at the range’s resolution with the defaults’ forwarded values and at the BOM pin with the BOM render’s forwarded values (one render when both agree); agent-platform-connectivity, released with the chart, is rendered from the working tree — the pair an installation gets is always the pair in the commit, and a key added to both charts at once exists nowhere else until the tag is cut (#339). The BOM and the roster are read as YAML (PyYAML is in the CI image), every Helm call is bounded, and the four UNRELEASED entries whose releases exist are gone: a range that admits nothing published is still rendered against the newest chart the line has while UNRELEASED names the release it waits for, a BOM pin never is. The inputs an installation supplies and the meta chart leaves empty (the resource servers’ OAuth clients, a public Gateway for the routes, valkey’s ACL secret) are ALL_ON_INPUTS, deliberately apart from QUICKSTART, which tests/verify-kagent-tools-namespace.py imports for renders with a much smaller component set. 22 components, 43 renders.tests/verify-bom-charts.py and its make verify-meta step (giantswarm/agent-platform#450; added by #278, carried #339’s working-tree rule for the connectivity chart). make verify-components-charts does strictly more: it renders the charts through where that one only proved the schema accepted the values (reporting the rest as needs-input), it covers the range’s resolution as well as the pin, and it renders the pin with the BOM render’s forwarded values. The BOM-completeness guard, the pin-reaches-the-render assertion and the working-tree rule moved with it.Full Changelog: https://github.com/giantswarm/muster/compare/v5.19.11...v5.19.12