AI Agents
Fixed
- A deactivated MCPServer refuses sign-ins and reports itself down (#1211). An OAuth MCPServer with
spec.suspended: true (what core_service_stop and the portal’s Deactivate write) keeps its pending-auth registry entry so that sign-ins work again once it is activated – but that entry also let core_auth_login hand out a sign-in link while the server was suspended. The person completed the flow, the session connected, the event handler failed a global registration (Failed to register MCP server … no MCP client available (service state inconsistent)), the reconciler stopped the service again within milliseconds, and auth://status then said connected with a tools count while the session’s tool list had nothing for the server; a session that had signed in before the deactivation saw the same contradiction. Now core_auth_login refuses a suspended server with Server '<name>' is deactivated (spec.suspended=true); activate it with core_service_start before signing in. and creates no challenge; the OAuth callback for a server deactivated between challenge and callback stores the token and establishes no connection (the first core_auth_login after core_service_start connects with it, no browser needed); auth://status reports a server whose service is down as disconnected whatever the session’s auth mark says, with a new "suspended": true flag naming the reason (muster auth status prints Deactivated); and list_tools no longer names a down server under servers_requiring_auth. The spec is read through the MCPServer manager – the source the reconciler acts on – so the aggregator keeps no copy of it. The service state inconsistent ERROR turned out not to be the suspension’s doing: the event handler tried a global registration on every per-session OAuth sign-in (the skip covered SSO servers only), failed because such a service holds no client, and emitted a MCPServerToolsUnavailable event the moment a person connected; it now skips a server that authenticates per session the way it skips SSO servers. Scenario: mcpserver-suspended-refuses-auth-login; test_simulate_oauth_callback gains an auth_url argument to complete a challenge an earlier step obtained.
Fixed
- Duration histograms use bucket boundaries for seconds (#995).
muster_tool_call_duration_seconds, muster_downstream_tool_call_duration_seconds and muster_workflow_execution_duration_seconds are recorded in seconds but inherited the OTel SDK’s default boundaries (0 … 10000), which are spaced for milliseconds: every observation landed in the first bucket and the other 14 series stayed empty, so histogram_quantile over them could resolve nothing finer than “under 5 seconds”. A sdkmetric.View matched on unit s now applies 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120, 300 to every seconds-unit histogram, current and future. The boundaries are documented in docs/explanation/observability.md next to the p95 example query.
Changed
describe_tool states that the tool it describes is only reachable through call_tool. Its response gained an invocation line — Call it through the call_tool meta-tool: call_tool with {"name": "<tool>", "arguments": {...}}, arguments per inputSchema. Tools inside muster are not callable by name directly — only the meta-tools are. — which the meta-tool’s own description names, next to what it returns, and call_tool’s own description now states the rule itself (“Aggregated tools are callable only this way, never by their own name”), so a model reads it when it decides how to execute rather than only after a lookup. An MCP client sees only the meta-tools, so a model that took a name out of describe_tool and issued it as a tool call of its own got an error with nothing in the detail it had just read to recover from. The field is additive: description, inputSchema, server, kind and annotations are unchanged, and the consumers that parse the response (the muster CLI, the REPL describe) ignore what they do not know. Scenario: list-tools-paged.
Fixed
- The s3proxy façade authenticates to Azure Blob with Workload Identity (giantswarm/agent-platform#436; found on glean’s cut-over, 2026-09-13). With
kagent.harness.snapshotStore.crossplane.provider: capz the connectivity chart left JCLOUDS_CREDENTIAL unset so that s3proxy would defer to DefaultAzureCredential — but the image (gaul/s3proxy 4.1.1) carries JCLOUDS_CREDENTIAL="remote-credential" as a Dockerfile default, and s3proxy takes the DefaultAzureCredential path only while the credential is empty: the façade signed every request as the account with that literal as the shared key and Azure’s SDK refused it on the base64 decode ('base64Key' was not a valid Base64 scheme … Illegal base64 character 2d on every PUT …/snapshots/…), so every golden boot crashed before its snapshot was taken. The capz branch now sets JCLOUDS_CREDENTIAL: "" explicitly next to AZURE_CLIENT_ID / AZURE_TENANT_ID; the account-key branch is unchanged. make verify-substrate-store asserts both shapes. Live proof on glean: with the variable empty the golden snapshot of sre-agent landed in giantswarm-glean-substrate through the façade on the next boot (Workload Identity end to end, the half agentlab cannot exercise — its façade proof runs the account-key branch against Azurite).
Changed
- The Substrate line re-pins to
v0.0.27-gs.9, the kagent line to v0.11.0-gs.12 (giantswarm/giantswarm#37773): components.substrate / components.substrate-crds move to >=0.0.27-gs.9 <0.0.28-0, components.kagent / components.kagent-crds to >=0.11.0-gs.12 <0.11.1-0 — the kagent release whose chart stamps the 0.0.27-gs.9 worker image, where the gVisor worker’s terminate collects a workload whose runsc containers are already gone (giantswarm/substrate#30). Before it, an AgentTemplate changed while its previous golden boot was crash-looping left the superseded revision uncollectable — failed to collect runtime revision … runsc state: exit status 128 every minute per template — and its golden actor pinned one worker of the pool for good (gazelle 2026-09-13: two of four). Proven in agentlab on the fork’s dev build: on 0.0.27-gs.7 the superseded revision was never collected in 56 minutes; on the patch the sweep after the template change collected it and freed the worker. The WorkerPool rolls once with the kagent release, which by itself frees the workers an installation on 4.10.x has pinned. UPGRADE.md carries the recognition and the by-hand recipe (delete the hosting worker pod) for a Substrate release before gs.9. BOM, tests/verify-components.py and tests/verify-kagent-crds.py follow the pins.
Fixed
- The cut-over Job’s egress admits the registry’s blob-storage front (giantswarm/agent-platform#433): the connectivity chart’s
CiliumNetworkPolicy for the agent-manager migrate Job named api.github.com and the agent chart’s registry only; Azure Container Registry answers a chart blob GET with a redirect to *.blob.core.windows.net, which the policy dropped, so on gazelle’s cut-over (giantswarm/giantswarm#37746, connectivity 4.10.1) the Job logged agent chart registry unavailable … blob.core.windows.net … context deadline exceeded every 30 s and completed its expand phase with 0 release(s) rewritten … 8 pending — the portal-created agents stayed on chart 0.x until a hand-applied bridge policy and a clone of the Job (expand done (10 release(s) on 1.x values) within 30 s). The agent chart’s sources — the registry host, the storage front and GitHub (agentManager.networkPolicy.egress.fqdns, .cidrs) — now come from one helper for every pod that reads the chart, agent-platform.agentManager.chartSourcesEgress.<flavor>: agent-manager’s own egress and the Job’s policy render the same destinations in both flavours (the kubernetes flavour’s Job policy now also honours agentManager.networkPolicy.egress.cidrs the way agent-manager’s does, instead of opening every public destination regardless; agent-manager’s kubernetes egress is unchanged). make verify-migration asserts the Job’s policy admits the blob front and, knob by knob, exactly the FQDN and CIDR set agent-manager’s egress does in both flavours; UPGRADE.md names the redirect host next to the Job’s prerequisites. Render-verified only: agentlab has no Cilium (its kubernetes flavour already admitted the redirect), so the Cilium proof is gazelle’s bridge policy. Installations act: an installation whose Job already ran under 4.10.1 or earlier re-runs it per UPGRADE.md once this release is live — the pod template is unchanged, so the completed Job does not run again by itself; gazelle’s bridge policy agent-manager-migrate-acr-blob-cutover-bridge is removed once the release is live there.
Added
- The platform’s single-replica pods refuse voluntary disruption (giantswarm/agent-platform#431; on graveler a Karpenter consolidation evicted the agentgateway data plane mid-turn).
karpenter.sh/do-not-disrupt: "true" on the pods that carry live streams — the agentgateway data plane (gateway.parameters.podAnnotations, merged onto the pod template through AgentgatewayParameters by the connectivity chart), muster (muster.podAnnotations), the kagent controller (kagent.controller.podAnnotations) and klaus-gateway (klausGateway.podAnnotations) — and a PodDisruptionBudget minAvailable: 1 on muster (muster.podDisruptionBudget, the muster chart’s knob), the kagent controller (kagent.controller.pdb with maxUnavailable: "" and unhealthyPodEvictionPolicy: AlwaysAllow, the kagent chart’s knob), klaus-gateway (klausGateway.podDisruptionBudget, the klaus-gateway chart’s knob, 1.1.0+) and agent-manager (agentManager.podDisruptionBudget, rendered by the connectivity chart as PodDisruptionBudget agent-manager in the release namespace; exactly one of minAvailable / maxUnavailable, the render refuses both, neither and a policy outside the API’s enum). Every knob is on by default and enabled: false / a "false" annotation opts out. Every installation rolls the four annotated pods once; the budgets make Karpenter report DisruptionBlocked and a node drain wait for its drain timeout on these pods — UPGRADE.md. make verify-disruption asserts the render, the knobs off and the guards.
Added
- The chart provisions Agent Substrate’s snapshot store on CAPZ (giantswarm/agent-platform#417, giantswarm/giantswarm#37756):
kagent.harness.snapshotStore.crossplane.provider: capz renders, through the connectivity chart, the Azure storage Account (TLS-only, no public blob access, seven-day soft delete; never deleted by Crossplane, kept by Helm), the blob Container with a lifecycle ManagementPolicy (capz.lifecycleDays, 30 by default), a UserAssignedIdentity with a FederatedIdentityCredential against the cluster’s OIDC issuer for the s3proxy ServiceAccount and a RoleAssignment (Storage Blob Data Contributor, scoped to the container; provider-kubernetes Objects bridge the identity’s generated principalId, clientId and tenantId the way the Loki chart does), and the s3proxy façade (kagent.harness.snapshotStore.s3proxy): Substrate speaks S3 only, so a stateless two-replica substrate-s3proxy Deployment in the release namespace (the gsoci mirror of gaul/s3proxy, non-root on 8080 behind a Service on 80, a PodDisruptionBudget, network policies in both flavours admitting only atelet and ate-api-server, Substrate’s egress to it next to the bundled store’s) translates aws-v4 — a per-installation key pair generated once and kept, the Secret substrate-s3proxy in the release namespace and in ate-system — to azureblob as the identity (DefaultAzureCredential, azure.workload.identity/use). The meta chart derives kagent.harness.snapshotLocation = s3://<capz.containerName>/<prefix> and hands the substrate release AWS_ENDPOINT_URL, AWS_S3_USE_PATH_STYLE, AWS_REGION and the key pair on atelet.extraEnv and ateApiServer.extraEnv next to an installation’s own entries. The façade also runs alone (s3proxy.enabled: true, the Crossplane block off) in front of an account provisioned by hand or a lab’s Azurite (s3proxy.azure.*, an account key). Guards: the provider, each provider’s inputs, the account name’s shape, the endpoint’s scheme, the bundled store on, an own S3 variable, a disagreeing snapshotLocation or s3proxy.azure.*. Off and aws render byte-identically to 4.9.x; make verify-substrate-store extended, ci/test-substrate-store-capz-values.yaml, tests/verify-components.py pins the façade’s image to gsoci and the BOM; hack/agentlab/ carries the lab’s Azurite fixture and overlay. Measured in agentlab against Azurite: one 2 GiB object, in-cluster aws-cli: PUT 9.8 s (209 MB/s), GET 7.4 s (276 MB/s) through the façade into Azurite, against PUT 5.8 s (356 MB/s), GET 3.0 s (684 MB/s) on the bundled RustFS — the façade’s hop costs about 1.7× on writes and 2.5× on reads in the lab; an actor’s golden snapshot restores from the store in ~0.24 s (manifest 11 ms, download 37 ms, restore 190 ms).
Changed
- The WorkerPool is one CPU feature set, not one architecture (giantswarm/agent-platform#429):
kagent.substrateWorkerPool.template.nodeSelector pinned the pool to kubernetes.io/arch: amd64 and the values comment, the READMEs and UPGRADE.md said an actor’s golden snapshot is architecture-specific. The constraint is narrower: the snapshot is a gVisor checkpoint, and gVisor restores it only on a host whose CPU offers every feature the checkpoint recorded — two amd64 nodes of different vendors are not compatible. On the first 4.x installation (2026-09-13) Karpenter had mixed AMD c5a and Intel r6i nodes under the pool; the sre-agent golden snapshot was taken on an AMD worker, every restore on the Intel worker failed incompatible FeatureSet: missing features: cmp_legacy, cr8_legacy, sse4a, misalignsse, topoext, mmxext, fxsr_opt (all AMD-only CPUID flags) every 30 s, and the portal turn surfaced as actor "ai-…" request timed out — the actor never recovers because it stays placed on that worker. The chart’s default stays kubernetes.io/arch: amd64 (the coarse case, and the only label every provider carries); the values comment, the chart README, the README’s “Scheduling” paragraph and UPGRADE.md now name the feature-set constraint and the pin per provider: on CAPA the vendor karpenter.k8s.aws/instance-cpu-manufacturer: amd|intel (Karpenter consolidation may replace a node with another vendor’s at any time) and, where a generation change must be ruled out too, the family karpenter.k8s.aws/instance-family; on CAPZ and the other providers the node pool (giantswarm.io/machine-pool) or node.kubernetes.io/instance-type; and the recognition recipe (the worker log line, kubectl get nodes -L <the label> showing two values under the pool). No render change. Installations act: a CAPA installation whose workers Karpenter provisions adds the vendor its golden snapshots were taken on to kagent.substrateWorkerPool.template.nodeSelector — the fleet template carries karpenter.k8s.aws/instance-cpu-manufacturer: amd for every CAPA installation on the 4.x line (shared-configs, agentPlatform.workerPoolCpuManufacturer), so a Giant Swarm installation acts only to tighten or to opt out. The durable fix is Substrate’s — a restore placed only on a worker whose CPU covers the snapshot’s feature set (giantswarm/giantswarm#37742 row 12).
Fixed
- The storage-version backup hook stops the 0.x kagent release from re-applying the CRDs it deletes (giantswarm/agent-platform#416): on graveler’s cut-over (giantswarm/giantswarm#37745, meta 3.23.1 → 4.9.1 through the cluster’s Flux) the still-installed kagent 0.3.3 wrapper
HelmRelease — crds: CreateReplace, in an upgrade-retry loop because the fleet’s flag PR (agentPlatform.kagentApiV2: true) had landed before the bound lift, its render refused on every retry (additional properties 'harness' not allowed) — re-applied modelconfigs, remotemcpservers and modelproviderconfigs.kagent.dev one second after <release>-kagent-storage-version-backup had deleted them, and kagent-crds failed six times with status.storedVersions[0]: Invalid value: "v1alpha2" and stalled; recovery was by hand. The backup hook now patches spec.install.crds and spec.upgrade.crds to Skip on every HelmRelease a deleted CRD’s helm.toolkit.fluxcd.io/name + namespace labels name before it deletes (the 4.x manifest carries no crds policy for the kagent release: helm-controller’s client-side three-way merge drops the patch on apply, a server-side apply leaves it inert on a release whose chart ships no crds/; a gone owner is tolerated), then watches the CRDs stay absent for 60 s — one that comes back is deleted again and the hook fails naming the owner, so Flux’s retried upgrade finds them gone. Rehearsed on a 3.23.1 agentlab upgraded in place to this chart with the 0.x release held in the loop and reconciled every 5 s: the CRDs stayed absent, kagent-crds installed on its first attempt. make verify-kagent-storage-version asserts the patch, its order and the watch; UPGRADE.md (“3.x → 4.0”, both sections) names the race and the order of the two fleet PRs. Installations act: none — the hook runs on the cut-over upgrade; an installation that lands the flag and the bound lift within one Flux interval never enters the loop at all.
Changed
- The kagent line re-pins to
v0.11.0-gs.9 (components.kagent and components.kagent-crds floor, examples/customer-bom.yaml; giantswarm/giantswarm#37742 row 41): the line’s ninth release carries the Harness template that drops an empty-valued selector label (above); the pin and the other carried patches are those of gs.8.
Fixed
- The platform Harness admits templates by the platform label alone on an installation whose kagent HelmRelease pre-existed the cut-over (giantswarm/agent-platform#418): the meta chart removed the kagent chart’s own default selector key
kagent.dev/harness by forwarding a null (components.kagent.nullKeys), which deletes the key on coalesce — but a kagent HelmRelease that existed on 3.x is patched by the upgrade, and a JSON merge patch removes a null key instead of storing it, so the chart default came back and the live Harness required both agent-platform.giantswarm.io/harness and kagent.dev/harness while the Generic agent chart 1.x stamps only the first: no AgentTemplate was admitted (graveler, 2026-09-13; a freshly created release — spidertron — kept the null). The meta chart now forwards kagent.dev/harness: "" in kagent.harness.allowedAgentTemplates.selector.matchLabels, a value create and patch store alike, and the kagent line’s Harness template drops every selector label whose value is empty (v0.11.0-gs.9, giantswarm/kagent-upstream#26; a selector left with nothing fails the render there). components.kagent.nullKeys and the agent-platform.setNull helper are removed — nothing else used them. verify-kagent-harness now also replays the forwarded values as a merge patch onto a pre-existing release, coalesces the chart’s default over them and asserts the effective selector is the platform label alone, and refuses any null in the kagent release’s values; verify-components-charts asserts the Harness the resolved kagent chart renders from the forwarded values selects by agent-platform.giantswarm.io/harness only.
