AI Agents

  • Fixed

    • postgres.imagePullSecrets and postgres.affinity, declared by both charts and rendered as Cluster.spec.imagePullSecrets and Cluster.spec.affinity on the platform Postgres Cluster (#311, gap 4). affinity is CNPG’s AffinityConfiguration, not a core Kubernetes Affinity: any other key fails the render, a core podAffinity or podAntiAffinity with a message of its own. Both render no field while unset.
    • A network policy for the Backstage app pods, in both flavours, with components.backstage.enabled and networkPolicy.enabled (#311, gap 1). Ingress: the Gateway that serves the portal’s route — the proxy pods of the namespaces the route names as its parents (backstage.parentRefs, else global.gatewayApi.parentRefs), or the agentgateway data plane while the route attaches to it (backstage.parentRefs wins over gatewayApi.gateway.create for the policy as it does for the route) — and the kubelet’s probes, on backstage.port. Egress: DNS, the kube-apiserver, the identity provider, the edge, muster, the portal’s own CNPG cluster on backstage.database.engine: postgresql, and the scaffolder catalog location on backstage.catalogs.version. The egress leg to the edge follows the Gateways the routes the app-config calls attach to (ingress.parentRefs, kagent.controllerRoute.parentRef, modelManager.route.parentRef), not the portal’s own route, and is opened on 443 and 10443: an Envoy Gateway proxy binds the listener’s port plus 10000, so the pod behind a 443 listener listens on 10443.
  • Fixed

    Full Changelog: https://github.com/giantswarm/muster/compare/v5.19.9...v5.19.10

  • Fixed

    • The boot pass no longer reports an MCPServer the reconciler registered a moment earlier as a failure (#1222). The orchestrator’s boot pass and the MCPServer reconciler’s first pass run concurrently: for a definition the boot loop had not reached yet, the reconciler found no service and StartService registered it lazily, and when the loop then got there the refused duplicate registration was logged as ERROR Failed to create MCPServer service: <name> ... service <name> already registered – one misleading error per affected server per boot (two of 87 definitions on one installation) for a service that existed and was being started by the caller that registered it. The registry’s duplicate error now wraps services.ErrServiceAlreadyRegistered and the boot pass treats it as a debug-level no-op (MCPServer <name> was registered through StartService while the boot pass ran; that caller starts it); every other error is reported as before and the suspended-server skip is unchanged.
  • Fixed

    Full Changelog: https://github.com/giantswarm/klaus/compare/v0.1.14...v0.1.15

  • Fixed

    • mcp-oauth’s metrics reach muster’s metrics endpoint, the token store’s included. The OAuth server’s instrumentation now records through muster’s own OpenTelemetry meter provider (mcp-oauth 1.4.0, instrumentation.Config.MeterProvider), and the Valkey and in-memory token stores share it. Until now mcp-oauth built a Prometheus collector of its own on the default registerer, which the :9464 endpoint – served from a registry of its own – never gathered: not one oauth_* series was ever exported, and the token store was not instrumented at all. Now oauth_http_requests_total, oauth_token_endpoint_failures_total{grant_type,error_code} and the other oauth_* series, storage_operation_total{operation,result} (result is success, error or timeout), the storage_operation_duration_milliseconds histogram and the storage_*_count size gauges are served next to the muster_* series. A Valkey outage shows as storage_operation_total{result="timeout"} and oauth_token_endpoint_failures_total{error_code="temporarily_unavailable"}.
  • Fixed

    • A suspended MCPServer is not started when muster boots (#1216). The orchestrator’s boot pass created and started a service for every definition with autoStart: true without looking at spec.suspended (what core_service_stop and a portal’s Deactivate write), and the reconciler’s first pass then stopped it again: per restart and per suspended server one Creating MCPServer service / Starting MCP server service / Suspending MCPServer service <name> (spec.suspended=true) sequence in the log, one connection attempt against a backend the operator had switched off, a MCPServerStarting and a MCPServerStopped event, status.lastAttempt moved to the boot time, and – when the connect won the race against the stop – a window in which the deactivated server’s tools were registered in the aggregator and callable. The boot pass now skips a suspended definition the way it skips autoStart: false (Skipping MCPServer <name>: Suspended=true at debug level); the server keeps reading Disconnected (stdio: Stopped) through the reconciler’s no-service branch, its status keeps its pre-restart lastAttempt, and spec.suspended: false still starts it. muster test: a pre-configured mock server accepts suspended: true, which also takes it out of the readiness gate. Scenario: mcpserver-suspended-not-started-at-boot.
  • Fixed

    • klaus-gateway’s egress reaches its stores: the platform’s Valkey and the Kubernetes API (giantswarm/agent-platform#443; a Cilium installation, 2026-09-14). The connectivity chart’s two policies selecting the gateway pod (-klausgateway-a2a-egress: DNS and the agentgateway data plane on 8080; -klausgateway-obo-egress: world / cluster on 443 and 10443) put it in Cilium default-deny egress and neither named the Valkey pods or the kube-apiserver entity — the API’s Service address is resolved by Cilium’s socket-LB before the policy applies, so the cluster port rule never matched it. Every installation ran the shapes that never leave the pod (memory routing, the bolt link store, no controller), so nothing showed it until klaus-gateway 1.3.0’s Secret link store (klausGateway.obo.store: secret, giantswarm/klaus-gateway#245) was switched on: the gateway crash-looped at start on read secret …/klaus-gateway-obo-links: … context deadline exceeded (not forbidden: the Role was there, the route was not) and was rolled back by hand after about 95 s of downtime; the Valkey routing store this release adds (klausGateway.routing.store: valkey, klaus-gateway 1.6.0, giantswarm/klaus-gateway#252) would have hung every binding write and the readiness probe the same way. A third policy, -klausgateway-store-egress, now renders exactly while a store or the controller reaches beyond the pod, one rule per store: the platform’s Valkey pods (agent-platform.valkey.podSelector, the release namespace) on valkey.valkey.service.port with routing.store: valkey and the valkey component on — an out-of-band Valkey (the component off) gets no rule, its egress is the installation’s; the kube-apiserver entity (cilium) / networkPolicy.kubernetes.apiServerCIDR (kubernetes) with obo.store: secret and OBO on, routing.store: configmap or crd, or controller.enabled; DNS as in the siblings (agent-platform.klausGateway.valkeyEgress / apiServerEgress; the keys are the klaus-gateway chart’s, forwarded by the meta chart and read with that chart’s defaults when unset). The default shape renders as before, byte for byte; the five policies selecting or targeting the pod now share one agent-platform.klausGateway.fullname helper. make verify-klausgateway-netpol asserts every gate (each store alone, both together, the port following the Service, the out-of-band Valkey, the Secret store with OBO off, the bolt routing store), both flavours, the selector following klausGateway.fullnameOverride and the off switches; ci/test-klausgateway-store-values.yaml lints the cilium render with both stores, ci/test-netpol-kubernetes-values.yaml the kubernetes one. Not reproducible in agentlab (networkPolicy.enabled: false, kind runs no Cilium): verified by render and the chart’s CI; the live proof is a store switch on a Cilium installation (hubble observe --verdict DROPPED on the gateway pod stays empty). No installation acts: every one runs the default shape; a switch to obo.store: secret or routing.store: valkey needs this release first — UPGRADE.md.
  • Added

    • The gateway’s routing store can be the platform’s Valkey with one line (giantswarm/klaus-gateway#252, klaus-gateway 1.6.0’s routing.store: valkey). With klausGateway.routing.store: valkey the klaus-gateway release’s routing.valkey.url, existingSecret and passwordKey are filled from the valkey release — its Service (<valkey.valkey.fullnameOverride>:6379, the bare name muster’s own storage.valkey.url uses), the Secret its default user authenticates from (valkey.valkey.auth.usersExistingSecret, else the one muster reads, else global.identity.existingSecret) and that user’s passwordKey — so every thread’s binding to its agent instance and the record of a turn in flight survive a gateway restart without a volume or API-server access. An operator’s own url, Secret or key wins (an out-of-band Valkey); with any other store nothing about Valkey is forwarded, so an installation whose klaus-gateway release still resolves 1.5.x sees no new key. make verify-klausgateway-valkey asserts the three cases. On a Cilium installation the switch also needs the gateway pod’s egress to the Valkey pods on 6379 (#443, the Valkey half) — until that rule ships, routing.store: valkey is for installations without the Cilium policies only.
  • Fixed

    • muster-valkey gets a memory bound of its own (giantswarm/agent-platform#446; gazelle 2026-09-14). The valkey.valkey block set no maxmemory, so Valkey ran with maxmemory 0 and the container limit was its only bound: once muster’s capability cache had grown the dataset to ~246 MB (giantswarm/muster#1217) the kernel killed the process, the pod restarted, loaded the RDB back to the same size and the next background save — a fork, copy-on-write on top of the dataset — tipped it over again: 19 OOM kills in 90 minutes, nine orphaned temp-*.rdb files on the PVC, every muster token refresh blocked while the store was down, the muster connector unusable in the Dev Portal and Claude, until the limit was raised by hand (the entry above makes that limit the default). Now valkey.valkey.valkeyConfig — the upstream subchart’s raw valkey.conf fragment, which the chart had left empty — carries maxmemory 640mb and maxmemory-policy volatile-lru (evict the least recently used key that carries a TTL; every muster record does, and the store is a cache: an evicted token is one re-sign-in, an evicted capability entry one re-list on the next connect): maxmemory at ~60 % of the 1Gi limit leaves the fork’s copy-on-write pages, the client buffers and the allocator their headroom under the cgroup, and a store that grows past it evicts instead of dying. An installation’s own valkeyConfig replaces the fragment whole. make verify-valkey (tests/verify-valkey.py) holds the fragment, maxmemory at or under two thirds of the limit and above the request, the policy, no AOF, and the override’s passage (helm/agent-platform/ci/test-valkey-override-values.yaml). The config checksum rolls the muster-valkey pod once more (Recreate, a few seconds without the store; the RDB on the PVC carries the data across) — UPGRADE.md. Lab proof (agentlab, platform.chartPath from this branch): the live muster-valkey answers CONFIG GET maxmemory = 671088640 and maxmemory-policy = volatile-lru, agentlab platform-test passes.
  • Added

    • tests/verify-bom-charts.py, wired into make verify-meta: every chart the BOM pins is rendered against the values the meta chart forwards to it (giantswarm/agent-platform#278). verify-meta rendered the meta chart alone — it proved the HelmReleases and their values were shaped as intended, never that the chart on the other end accepts them, so a key forwarded unconditionally against a pin whose schema forbids it was invisible until an installation failed to reconcile. The check pulls each pinned chart at the version and from the registry its rendered OCIRepository names (so the ghcr.io components — kagent, Substrate — and cloudnative-pg are covered, not just the gsoci ones) and renders it with exactly the values its release carries, failing only on a schema rejection: a later template error means the values were accepted and the chart wanted an input the BOM alone does not carry, which is reported apart and counted apart. Every BOM-pinned component is enabled for the render, so a pin cannot drop out of the check because the golden render leaves that component off, and a pin with no OCIRepository or one resolved to another version fails rather than skipping silently — 21 of 21 pins covered. A self-test proves against the installed Helm that a schema rejection and a template failure are still told apart, so a reworded Helm message fails the check loudly instead of downgrading the break it exists for into a passing line. Deliberately stdlib-only, like tests/verify-toolset-presets.py: the CI image has no PyYAML. It found the two pins above on unmodified main on its first run.

    Fixed

    • The actors’ egress opens the host model servers model-manager fronts (giantswarm/agent-platform#278). model-manager with a host backend (ollama, lemonade, lmstudio) wires every pulled model into kagent as a ModelConfig pointing at that server — but only model-manager’s own policy opened it, and an actor’s inference request leaves through Agent Substrate’s egress gateway (substrate-atenet-egress), whose allow-list carries muster, the kagent controller, the tool server and world:443. A host model server is on neither 443 nor HTTPS, so every such request was dropped while model-manager reported the model as wired and healthy. The egress gateway now opens each host backend at the address the ModelConfigs actually carry — model-manager.<backend>.agentHost, falling back to .endpoint — as a toCIDR for an IP and a toFQDNs plus cluster arm for a hostname, and renders nothing while model-manager is off, for a kserve-only backend list or in the kubernetes flavour. A hostname arm also switches that policy’s DNS rule to agent-platform.dnsEgressWithProxy: Cilium fills its FQDN cache only from answers its own proxy sees, so without it the toFQDNs selector never matches and the name stays blocked; an IP-only backend list keeps the plain rule, so no installation gains an L7 DNS proxy on every actor lookup it did not need. model-manager.<backend>.agentHost is now guarded like .endpoint — an http(s) URL with a host — because the policies parse it with urlParse, which answered an empty matchName with port 80 for a scheme-less value and died on a raw parse error for a host:port one. make verify-kagent-netpol covers all of it.
    • The customer BOM pins two charts that reject the values the meta chart forwards to them (giantswarm/agent-platform#278, found by the check below). gateway.parameters.podAnnotations and klausGateway.podDisruptionBudget / .podAnnotations travel unconditionally since #431, and vm-manager / vmManager since #441, while the BOM pinned agent-platform-connectivity 4.8.0 and klaus-gateway 1.0.2 — both additionalProperties: false, so a BOM-pinned installation got two releases that could not install or upgrade whatever the operator set (additional properties 'podAnnotations' not allowed; additional properties 'vmManager', 'vm-manager' not allowed). Checked release by release: connectivity is rejected through 4.10.0 on the #431 keys and through 4.10.6 on the #441 ones, and accepted from 4.11.0; klaus-gateway is rejected at 1.0.3 and accepted from 1.1.0 — the release the #431 entry already names for its budget knob. A BOM-pinned installation should take these pins.