AI Agents
Fixed
make verify-gpu-pool passes on a GOLDEN_REF that carries modelServing.gpuPool (#482). The untainted case asserts the serving render is byte-identical to GOLDEN_REF (origin/main by default) but for the discovery block — with the golden rendered at its own defaults. From the commit that merged #482 onto main that golden is tainted, so an untainted head could never equal it and test-ingress-modes failed on every branch cut from main, main included. The golden now takes the same modelServing.gpuPool.taint.key= override when its values.yaml declares the key, and the discovery block is cut from both sides: the assertion is that equal inputs render alike, which is what it was always meant to be. A golden from before #482 still renders at its defaults, unchanged.
Added
modelServing.gpuPool: the GPU node pool’s taint as a toleration and its label as a node selector on everything the serving layer schedules onto the pool, published for model-manager; the model-manager line moves to >=0.23.0 (giantswarm/agent-platform#315; bumblebee-plans#46, the plan’s D3/D5; epic giantswarm/giantswarm#37639). A GPU node pool created through the platform (the gpu-node-pool chart) arrives tainted nvidia.com/gpu NoSchedule and labelled giantswarm.io/machine-pool=<cluster>-<pool>; the serving layer tolerated no taint, so the vLLM runtime and every preset’s predictor would land nowhere on such a pool. New: modelServing.gpuPool.taint (key, value, effect; the default is the pool chart’s taint — nvidia.com/gpu NoSchedule with no value, tolerated with operator: Exists, so a pool whose taint carries a value is covered too; a value narrows it to Equal; an empty key is an untainted pool and renders nothing) and modelServing.gpuPool.nodeSelector (the pool’s label, per pool and empty by default — the serving slice’s values set it). The connectivity chart applies both to the ClusterServingRuntime (KServe copies them onto every predictor pod of the runtime; runtime.tolerations come after, runtime.nodeSelector keys win) and to every published preset’s scheduling block (the pool’s toleration first, the preset’s own after it, an equal entry once; the pool’s selector under the preset’s own keys) through one helper (agent-platform.modelServing.poolScheduling), and publishes them in the discovery ConfigMap as spec.gpuPool.taint.{key,value,effect} and spec.gpuPool.nodeSelector — the keys model-manager 0.23.0 reads (giantswarm/model-manager#86) to schedule the LLMInferenceServices it composes, its download Jobs and its inventory scan pods by the same input (a registered backend document may override it). Platform components never carry it. Three render guards: the effect is one of the three taint effects, the key a qualified name, every selector value a string. components.model-manager.versionRange moves to >=0.23.0 <1.0.0 (the floor: the release that reads the keys) and the example BOM pins 0.23.0. make verify-gpu-pool (tests/verify-gpu-pool.py, fixture ci/test-model-serving-gpu-pool-values.yaml) asserts the three sites in the default, pool-selected, valued and untainted shapes, the guards, the meta chart’s forwarding, and that an empty taint key leaves the serving render byte-identical to origin/main but for the discovery block. The default render is unchanged (model serving is off by default). Every installation with model serving on rolls its predictors once (the runtime gains the toleration and KServe reconciles every InferenceService of it; the models reload from the cache) and model-manager once (0.23.0). README of the connectivity chart, “The GPU node pool”.
Added
components.cluster-manager: the Agent Platform’s cluster write surface as a chart component, off by default (giantswarm/agent-platform#316; giantswarm/giantswarm#37637; bumblebee-plans#42 and #46; epic giantswarm/giantswarm#37639). One OCIRepository + HelmRelease of the catalog’s cluster-manager chart (>=0.4.2 <1.0.0, dependsOn: [muster] — the floor is the first release that tolerates a cluster without the Cluster API group: list_clusters answers an empty list with a note, a dry run on a cluster that does not exist is refused naming it, get_info reports clusterApi and the real version) in model-manager’s mold: the cluster-manager: block forwarded verbatim — the pinned Service name, the OAuth 2.1 resource server acting as the caller (oauth.downstream), the chart’s own MCPServer CR with forwardToken and the dex-k8s-authenticator audience the kube-apiserver trusts, so muster registers x_cluster-manager_* and the portal’s Clusters pages call the tools through muster as the signed-in user (MCP is its only surface: no route, no JWT policy, no proxy entry) — with global.* injected and modelManager.namespace derived from the platform’s own namespace (an own value must agree). The connectivity chart renders the wiring from the clusterManager: block: network policies in both flavors (ingress from muster and the kubelet’s probes; egress to DNS, this installation’s Kubernetes API, the identity provider, the workload clusters’ API servers — clusterManager.networkPolicy.workloadClusters: names, addresses, ports: [443, 6443] — and clusterManager.networkPolicy.egress on 443) and the render-time guards (muster off with the MCPServer on, a missing identity input, a bad CIDR, an empty port list, clusterManager.flux.requireApi). The two blocks travel to the connectivity release only while the component is on (gatedValues, vm-manager’s shape); the default render is unchanged but for the roster entry. examples/customer-bom.yaml pins 0.4.2; make verify-cluster-manager (tests/verify-cluster-manager.py) covers all of it; README “Cluster manager”. The flip to default-on is a decision for after the first lab proof.
Added
- The serving slice:
components.kserve-runtime-configs, the models Gateway with one Dex JWT policy, examples/serving-slice.yaml, two 24 GB presets (giantswarm/agent-platform#326; bumblebee-plans#46, the plan’s D4/D6/D8/D9; epic giantswarm/giantswarm#37639). components.kserve-runtime-configs (giantswarm/kserve, 0.2.x, dependsOn: [kserve-llmisvc-crd], into the release namespace — where the llm-d controller resolves well-known configs from, besides the LLMInferenceService’s own) installs the well-known LLMInferenceServiceConfigs with kserve.llmisvcConfigs.enabled: true, kserve.servingruntime.enabled: false and no registry value (the chart’s gsoci default, giantswarm/kserve#78); its block is held back from connectivity. modelServing.modelsGateway (off by default — the Gateway is the slice’s) renders one agentgateway Gateway models on models.<global.domain> with the platform’s wildcard (or a Certificate from tls.issuerRef), the external-dns hostname, and one AgentgatewayPolicy (Strict, inheritance: Override, audience dex-k8s-authenticator, JWKS from the issuer’s host on 443); KServe’s kserve.controller.gateway.ingressGateway.kserveGateway is derived onto the kserve-resources release so every LLMInferenceService route attaches to it, and the discovery ConfigMap names the endpoint and the /<namespace>/<model>/v1 convention. examples/serving-slice.yaml is the one-release-per-target-cluster profile: the five kserve components and the switch, runtimeClassName: nvidia, agentgateway off beside the platform’s release and on for a workload cluster (nothing is detected). Presets qwen3-4b-instruct and qwen3-8b-fp8 (one L4-class GPU, tools on, no image; the schema gains spec.template). The default render is byte-identical but for the roster entry. make verify-serving-slice; README “The serving slice and the models Gateway”; docs/authentication.md section 6.
Changed
- model-manager ships with every installation by default, with no backend; the model-manager line moves to
>=0.22.0 (giantswarm/agent-platform#329, bumblebee-plans#46 D3/D4/D5/D7, epic giantswarm/giantswarm#37639): components.model-manager.enabled defaults to true and the forwarded block names no backend — neither model-manager.backend nor model-manager.backends — so the release starts with zero backends (model-manager 0.22.0’s zero-backend start-up) and backends are registered at runtime as labelled ConfigMaps (the add_backend tool, the portal’s Serving page, cluster-manager for a GPU node pool; model-manager’s docs/backends.md). The connectivity guards demand an endpoint only for a backend that is listed statically, so backend: ollama / backends: [ollama] without ollama.endpoint fail as before; the static forms render as before (the golden holds them equal). Model-manager’s OAuth guard follows the muster guard’s convention exactly: a model-manager.oauth.dex.* / existingSecret value that disagrees with global.identity fails the render (one login provider), checked only where both sides are set; a missing input is reported by the child release as for muster, so a render that sets no global.identity (or global.domain / global.identity.issuerUrl alone) is untouched. With components.kagent off the meta chart derives kagent.disableWiring: true for the model-manager release (it must not wire ModelConfigs into a kagent the installation does not run), with components.muster off muster.mcpServer.enabled: false (the MCPServer CRD ships with muster, so the MCP surface follows it; the REST API stays), and with muster’s OAuth server off (muster.muster.oauth.server.enabled, the platform’s one login — the lab shape of examples/kind-lab-dex.yaml) oauth.enabled: false (no issuer to trust, the way the muster discovery label derives from the same toggle); tests/verify-components.py asserts all three, both ways, and the two connectivity guards that failed the render for those cases are gone. Opt out with components.model-manager.enabled: false. The network policy for a backend registered at runtime is giantswarm/agent-platform#478.
Added
components.gpu-operator: NVIDIA’s GPU operator as a chart component, off by default and independent of model serving (giantswarm/agent-platform#327; bumblebee-plans#46, the plan’s D3; epic giantswarm/giantswarm#37639). One OCIRepository + HelmRelease of Giant Swarm’s gpu-operator-app — the catalog’s gpu-operator wrapper chart on 1.x — into kube-system (the release history there too, so the GPU guide’s hand-installed release upgrades in place) with crds: CreateReplace, no dependsOn, gitops.target.kubeConfig.secretRef honoured. The operator’s configuration is the new gpu-operator: block, forwarded under the wrapper’s subchart key — a table of two rows and no third: Flatcar, driver and toolkit off (the default); nodes with a pre-installed driver, gpu-operator.toolkit.enabled: true. In every configuration the operator creates the nvidia RuntimeClass GPU pods use (modelServing.serving.runtimeClassName: nvidia where the platform’s own release serves), so the chart adds none. One owner per cluster: with the component on, a ClusterPolicy, HelmRelease or App of the operator that is not this release’s fails the render naming what was seen and the handover — delete cluster-manager’s <cluster>-gpu-operator (its detection then sees the component and never re-creates it) or the hand-installed release, then switch the toggle on, or leave it off; the ClusterPolicy CRD is under ownedCrds as the backstop. The default render is unchanged but for the roster entry forwarded to connectivity (components.gpu-operator.enabled: false), which make verify-target’s golden now drops as the one difference a new component is allowed. make verify-gpu-operator asserts the offline half; tests/fixtures/gpu-operator-foreign-owner.yaml shows the refusal on a cluster; examples/customer-bom.yaml pins 1.3.0. README “The GPU operator”.
Added
- One release of this chart per target cluster: the target kubeconfig knob,
enabled for muster and dicebear, one owner per cluster-scoped component, the slice fixtures (giantswarm/agent-platform#328; bumblebee-plans#46, epic giantswarm/giantswarm#37639). The serving slice (#326) and the runtime slice (#317) are values profiles cluster-manager installs as one <cluster>-agent-platform release per target cluster, the slices as toggles in its values — beside the platform’s own release on the installation’s cluster, or onto a workload cluster that runs no Flux. Three things in the chart stood in the way. New: gitops.target.kubeConfig.secretRef (name, key) — when set, every component HelmRelease the loop renders carries spec.kubeConfig.secretRef verbatim (the connectivity release included), the field shape the fleet installs every workload-cluster app with, so the installation’s helm-controller installs each component into the target from the OCIRepositories in the release namespace, dependsOn intact, the connectivity chart’s lookup guards evaluated against the target; unset, the render is byte-identical to before; refused with the bundled engine on (agent-platform.validateTarget — no Flux is installed into a workload cluster or into a cluster that runs one), and no Helm hook renders with it (the kagent storage-version pair would run against the installation’s CRDs). components.muster.enabled and components.dicebear.enabled (default true; the two were force-on) — off renders no release of theirs and the roster tells the connectivity chart, which now drops every reference: muster’s /mcp HTTPRoute, the -mcp-egress policy, the agentgateway data-plane and controller rules, the kagent controller and Substrate rules that select muster’s pods (the managers’ already read the roster); the avatars host in the portal’s CSP. components.<name>.ownedCrds — a CRD the component installs that already exists and whose Flux labels (helm.toolkit.fluxcd.io/name, /namespace, which helm-controller stamps on every object of a release, crds/ included) name another HelmRelease fails the render naming the release to remove first (agent-platform.validateCrdOwners; agentgateway, kserve-crd, kserve-llmisvc-crd, kagent-crds and muster carry one representative CRD each; skipped with the target knob, where the lookups see the installation; a CRD without the labels is left to Helm as before). make verify-target (tests/verify-target.py) asserts the offline half: the stamp and the byte-identity of five renders against origin/main, the toggles on both charts, the engine guard, the hooks, and the serving- and runtime-shaped toggle sets (ci/test-slice-serving-values.yaml, ci/test-slice-runtime-values.yaml, ci/test-target-values.yaml — the profiles themselves are #326’s and #317’s) alone, combined (the union; the first slice’s documents unchanged when the second is switched on — an in-place upgrade) and targeted, agentgateway off beside the platform’s release and on for a workload cluster; the two lookup guards are asserted on a cluster (README, “One release per target cluster”). Default render unchanged.
Added
muster test: an installation-shaped scale fixture with budgets enforced in CI. internal/testing/fixtures/scale is a committed, generated fixture – 87 MCPServers (84 session-authenticated, in five families with an instanceArg on 28 installations, three in-house), 18 distinct capability documents shared by the family members, 282 workflows over their tools, 450 session records; every name invented, nothing from an installation – rendered by go generate from a seeded generator, with a test that fails when the committed YAML drifts. A scenario boots from it with pre_configuration.fixture: scale and adds or overlays on top. Budgets are hard limits with headroom over the value measured on CI’s medium executor, never “faster than last run”, and a failing budget names the metric, the measured value and the limit. In Go (internal/aggregator, run by make test): a warm list_tools, filter_tools, describe_tool or call_tool over the fixture costs at most 100 ms (measured 1–2 ms), 2 store commands (HGETALL, HKEYS) and 0 definition reads (API-server requests) – the catalogue is built once; list_tools’ default page stays under 40 KB (measured 14 KB); the capability store holds at most 16 KB per session over 450 sessions (measured 7.2 KB: 84 references and a share of 18 shared documents). On the wire (scenario scale-fixture-budgets, on a Valkey stand-in): the default page under 40 KB, under 32 KB of capability store per session (measured 9.9 KB over 8 sessions), a warm meta-tool request costing the store a constant 16 commands at most (measured 11: the two of the catalogue plus the OAuth middleware’s per-request session bookkeeping, nothing per server), and a new session’s first request answered within 2 s while its fan-out connects 84 servers two of which take four seconds. New test tools test_measure_meta_tool (duration, response bytes, store commands by name per call) and test_valkey_footprint (the store’s content by prefix, capability bytes per session); a new expectation json_path_max bounds numeric response fields, and a step whose expectations are not met now reports which one and how (json_path_max "valkey_commands": measured 177, budget 16) instead of “expectations not met”. The scenario fails on the release that had each bug, at the budget’s step: list_tools unbounded on v5.15.6 (response_bytes 61,651 against 40,960, #1193); every session storing every server’s document inline on v5.19.4 (capability_bytes_per_session 105,332 against 32,768, 672 inline entries and no shared document, #1217); a listing reading every server’s entry and document on its own on v5.19.10 (valkey_commands 177 against 16: 84 HGET and 87 GET per call, #1225); a new session’s first request held by the fan-out on v5.19.14 (4.02 s against a 2 s max_duration, #1226). Documented in “Installation scale: the fixture and the budgets” of docs/contributing/testing/scenarios.md. (#1240)
Fixed
- A
forEach can iterate a step result’s field. items: "{{ .results.<id>.<field> }}" failed every workflow with items expression … resolved to string, expected a list: a reference deeper than one key below .input, .results or .vars was rendered as text, so only a list passed in as a workflow argument could be iterated. Template references that are a pure path now keep their Go type at any depth – the navigator the spec.output template already uses – so a forEach over the pods a previous step listed works, {{ .vars.pod.name }} inside the loop body is the string it names, and a numeric field referenced as a tool argument arrives as a number. Anything more than a pure path still renders to text. The returned document also showed the loop’s last result on every iteration’s record (the records share the body step’s ID); each record now carries its own iteration’s result and an iteration index. Scenario workflow-foreach-step-result-items.
Added
muster self-update --check reports the running and the latest release without installing
anything and exits with status 125 when a newer one exists (devctl’s convention for
version check), for scripts. Every command a person runs starts with a one-line hint on stderr
while a newer release is out – a hint, never a gate: the GitHub round trip is capped at two
seconds, its answer is cached for an hour under the user cache directory
(~/.cache/muster/latest-release.json on Linux, ~/Library/Caches/muster/ on macOS) and a failed
attempt is remembered for ten minutes, so a machine without internet is not held up.
MUSTER_NO_UPDATE_CHECK=1 silences the hint; serve, standalone, agent, test, version,
self-update, help and completion never print it, and dev builds never check. The same
internal/update package as agentlab’s, so the two CLIs behave alike.
Changed
muster version and muster --version print one line, muster version v5.23.5 (commit 361cdef, built 2026-09-15T19:38:51Z), with the details the binary knows; the commit: and built: lines
and their unknown placeholder are gone, and muster version still adds the aggregator’s version
when one is running. A go build from a checkout reports Go’s own stamp – the tag at a tag,
+dirty over local edits, a pseudo-version between tags (v5.23.6-0.20260915195350-977012d023ba:
after v5.23.5, before v5.23.6) – which self-update and the hint compare as what it is; a binary
without any version says dev, and self-update refuses it with a pointer to
go install github.com/giantswarm/muster/v5@latest. The self-update messages follow agentlab’s
(“Nothing newer than v5.23.5 on GitHub (latest release v5.23.5).”; a newer release is announced
with its URL instead of its full release notes), and its logic moved from cmd to
internal/update.
