AI Agents

  • Added

    • muster test: faults are named steps and time is a clock the scenario moves. test_redeploy_mock_server replaces a mock backend’s process behind its port – every MCP session forgotten, tools kept, no refused connection in between, the way a rolled pod takes over behind its Service; test_set_mock_server_auth: {required: true|false} flips a running mock between answering anonymously and 401-with-metadata, the rollover of a backend to an OAuth resource server (a mock with a token validator and oauth.required: false starts anonymous). test_advance_clock: {duration} moves muster serve’s clock and every mock authorization server’s clock forward together: the reconnect backoff of a remote MCPServer, the orchestrator’s retry and health ticks and the age of the core catalogue now read a process clock (internal/clock) that the harness advances through a control socket MUSTER_TEST_CLOCK selects – a tick that has become due fires at once, nothing waits, and production binaries keep the system time. pre_configuration.intervals: production leaves those timers on their production defaults (30 s initial backoff, 2 min cap, 30 s ticks, 5 min catalogue age) for scenarios that assert the production schedule; the default short keeps the seconds-long environment knobs. The fault steps and the clock are documented together in “Faults and time” of docs/contributing/testing/scenarios.md. Four scenarios: a backend redeployed behind the same port has its session recovered on the next call, no health probe in between (fails on v5.15.0, #999); on the production schedule a remote server whose gateway answers 504 four times waits 30 s, 1 min, 2 min and again 2 min – never 4 – and is connected within the cap once the gateway serves (fails on v5.9.10 with the clock, #1163); the core catalogue found older than its five-minute age is served as it is and rebuilt once in the background (fails on v5.19.13 with the clock, #1231); a forwardToken backend rolled over from an anonymous pod to a 401 pod and back is never registered globally and keeps serving the signed-in session (fails on v5.7.17, #1135). (#1238)
  • Fixed

    Full Changelog: https://github.com/giantswarm/muster/compare/v5.21.0...v5.21.1

  • Fixed

    Full Changelog: https://github.com/giantswarm/klausctl/compare/v0.4.6...v0.4.7

  • Added

    • The Substrate worker pool gets a PodDisruptionBudget; the pool template’s Karpenter knobs are documented and verified; a spread value is refused until the Substrate line carries it (giantswarm/agent-platform#472). The platform runs every agent on the one WorkerPool (kagent.substrateWorkerPool, four gVisor workers, one actor each), and nothing guarded its workers against voluntary disruption: no budget selected them (the kagent namespace carried budgets for the controller and the database only), no annotation kept consolidation off their node — while on gazelle (2026-09-15) Karpenter had bin-packed all four onto one spot m6a.xlarge, so one consolidation or reclaim took the whole pool down: every turn in flight failed, every session waiting for a person was lost (its pause checkpoint is node-local, giantswarm/giantswarm#37795), new turns failed no free workers available. Decided 2026-09-15: the pool stays on spot, the workers get a budget only. New: kagent.substrateWorkerPool.podDisruptionBudget (enabled: true, maxUnavailable: 1, unhealthyPodEvictionPolicy: AlwaysAllow) — a PodDisruptionBudget named after the pool in the kagent namespace, rendered by the connectivity chart the way muster-valkey’s is (#439; the kagent chart has no budget template), selecting the worker pods by Substrate’s ate.dev/worker-pool: <pool> label, so a voluntary drain moves one worker at a time (ALLOWED DISRUPTIONS 1 with four Ready workers); the key never reaches the kagent release (components.kagent.omitKeys); the four budgets of the connectivity chart now share one spec helper and its guards (agent-platform.podDisruptionBudget.spec, the render unchanged). Documented in values.yaml, both READMEs and UPGRADE.md, unset by default: template.annotations karpenter.sh/do-not-disrupt: "true" (consolidation, drift and expiry never drain a worker node; a drift roll then waits on the NodePool’s terminationGracePeriod) and template.nodeSelector karpenter.sh/capacity-type: on-demand (the only knob that covers a spot interruption; Karpenter-only — never on CAPZ, on-prem or a cluster-aws node pool), and the roll every spec.template change causes (the Deployment once, one worker at a time under the budget, turns in flight on a replaced worker lost, goldens untouched). The render now refuses what WorkerPool.spec.template would prune silently (agent-platform.validateWorkerPool): topologySpreadConstraints and podAntiAffinity — carried by no published Substrate release yet (the carried patch giantswarm/giantswarm#37797) and pruned by the CRD’s structural schema, so a spread value looked applied and did nothing — are refused naming the key and the range until components.substrate.versionRange’s floor is the release that carries them (agent-platform.substrate.workerPoolSpreadFloor, empty until then) and forwarded verbatim from it on; any other key the template does not have is refused the same way (a typo was pruned in silence); the floor is read by the new agent-platform.semverRangeFloor. kagent.controller.substrate.defaultWorkerPool is marked inert (the kagent chart renders SUBSTRATE_DEFAULT_WORKERPOOL_*, nothing in the line reads them). make verify-workerpool asserts all of it: the budget from the connectivity chart of the working tree with the pool’s selector and maxUnavailable: 1, gone with enabled: false, never on the kagent release; the annotation and the capacity-type selector reaching the kagent release and the resolved kagent chart’s WorkerPool verbatim; the two spread keys and an unknown key refused; still exactly one WorkerPool. make verify-disruption counts the worker budget among the connectivity chart’s; the golden render leaves it off on both sides. Every installation with kagent on gets one new object (the budget); no pod rolls.
  • Added

    • muster test: scenarios can run their instance in Kubernetes mode. pre_configuration.mode: kubernetes starts one envtest control plane per run from the binaries KUBEBUILDER_ASSETS points at, installs the CRDs from helm/muster-crds, applies the scenario’s mcp_servers and workflows as CRs into a namespace of its own and runs muster serve with kubernetes: true against it – informers, reconciler, boot pass and CR status the way an installation has them. A TCP proxy the harness owns sits between muster and the API server: apiserver.reachable_after keeps it closed for that long after the process started (the API server late at start) and test_set_apiserver_reachable: false|true cuts and restores it mid-run. test_patch_cr applies a merge patch to a CR (spec.suspended, labels, spec.auth.authorizationServer) so the reconciler is driven by real CR updates; test_get_cr reads a CR with the status muster wrote; test_set_mcpserver_labels and test_pin_mcpserver_authorization_server update the CR in this mode. muster test --mode filesystem|kubernetes runs one definition source; without KUBEBUILDER_ASSETS the Kubernetes-mode scenarios are reported as skipped with the reason, never as passed, and make test-envtest (the test-envtest CI job) runs them against envtest. Four scenarios reproduce bugs the filesystem-mode suite could not express: a suspended server is not started by the boot pass (fails on v5.19.6, #1216), a suspended remote server is stopped once and not on every resync tick (fails on v5.19.0, #1212), the boot pass and the reconciler’s first pass register a dozen CRs without an ERROR (fails on v5.19.8, #1222), and an API server unreachable at start is waited for instead of swapped for the filesystem client (fails on v5.7.18, #1143); two more cover a boot from CRs and an API server gone mid-run. (#1237)

    Fixed

    • muster test: the process test_restart_instance starts no longer dies when the restart step returns. It was bound to the step’s context, so a restart step with a timeout killed the new muster serve right after it had passed readiness and the next step found the connection reset.
  • Fixed

    Full Changelog: https://github.com/giantswarm/muster/compare/v5.20.1...v5.20.2

  • Changed

    • The muster and muster-crds chart READMEs no longer render a version badge (chart.badgesSection removed from both README.md.gotmpl files): a release PR bumping Chart.yaml’s version or appVersion no longer changes the checked-in README.md, so the helm-docs pre-commit hook no longer fails on it. (giantswarm/devctl#2180)
  • Added

    • make verify-all: every verify-* target of Makefile.custom.mk in one target, which is what CI runs. The list is read out of that file, so a new verify-* target reaches CI by existing and nothing names the set twice; make help lists what each one asserts. The test-ingress-modes job named all 37 of them on one line, which drifted from the Makefile on every addition.
    • make e2e KUBECONFIG=… [SCENARIO=…] [CLUSTER_TYPE=…] [VALUES=…]: runs one ATS scenario against any cluster. It packages the chart and hands the suite its inputs. SCENARIO is the pytest marker (smoke or functional), CLUSTER_TYPE picks the scenario defaults (kind or eks), and VALUES replaces the scenario’s values-file list. E2E_DOMAIN, E2E_ISSUER_URL, E2E_CLIENT_ID and E2E_IDP_SECRET_NAME (the name of a Secret already on the cluster, not a credential) become a temporary values overlay layered last and removed on exit; tests/e2e_overlay.py renders it, and the target prints its line count, never a value. Each of those names one fact the chart and the tests both need, so the target derives the suite’s ATS_ISSUER_URL, ATS_CLIENT_ID, ATS_IDP_CA_SECRET and — on a cluster type other than kind — ATS_MUSTER_BASE_URL from them, and an ATS_ variable already in the environment wins. A KUBECONFIG with a leading ~ is expanded, which zsh does not do for an argument of that form. Documented in tests/ats/README.md.
    • helm/agent-platform/examples/kind-lab-dex.yaml: the kind lab shape as a shipped example, moved from tests/test-values.yaml with its values unchanged (its header comment carries the install recipe and the no-secret statement). It is the file the ATS smoke installs, and test_the_scenario_installs_the_example_file asserts the scenario names an example file the repository ships. .ats/main.yaml and make verify-self’s lab-shape render follow the new path.

    Changed

    • The ATS suite takes its cluster-specific inputs from a Scenario (tests/ats/scenarios.py, one per cluster shape) instead of kind literals, so the same assertions run on any cluster. Five groups become fields, each with its own environment variable: the values files (ATS_VALUES, plus ATS_OVERLAY_VALUES layered last), the identity provider (ATS_ISSUER_URL, ATS_CLIENT_ID, ATS_CLIENT_SECRET, ATS_REGISTRATION_TOKEN, ATS_IDP_USER, ATS_IDP_PASSWORD, ATS_IDP_CA_SECRET), the OAuth redirect (ATS_OAUTH_CALLBACK), muster’s base URL and how it is reached (ATS_MUSTER_BASE_URL, ATS_MUSTER_REACH) and the budgets (ATS_INSTALL_TIMEOUT, ATS_UNINSTALL_TIMEOUT, ATS_UNINSTALL_BUDGET_S, ATS_READY_TIMEOUT_S). A port is read out of the URL that carries it, so the two can never disagree. ATS_CLUSTER_TYPE gains eks, whose scenario installs no lab Dex, port-forwards nothing and carries no default issuer, client, base URL or values file. An unknown cluster type, an unknown reach, a budget that is not an integer, a missing issuer, client or base URL, a values file that does not exist, and a non-kind scenario without ATS_VALUES, each fail the run with a message naming the variable. make verify-scenarios asserts the loader and the make e2e overlay offline. Every field defaults to the value it had, so ATS_CLUSTER_TYPE=kind with nothing else set installs the same three values files with the same ports, budgets and credentials.
  • Changed

    • The kagent line moves to v0.11.0-gs.16: context compaction in the Go ADK runtime and spec.context.compaction on the AgentTemplate (giantswarm/giantswarm#37792). After prompt caching (4.16.0) the remaining cost of a long agent turn was the model’s first pass over new tokens: every tool result stayed in the history verbatim and every later model call of the turn re-read it — on gazelle the status question’s ten calls grew from 6 k to 62 k input tokens, waited 9.2 s for the first token after 16 k new tool-result tokens and 18.8 s after 21.9 k, model time 59 s of the turn’s 68 s. kagent 0.11.0-gs.16 carries kagent-dev/kagent#2790 (giantswarm/kagent-upstream#41): google.golang.org/adk/v2 v2.3.0 with its native session/compaction, spec.context.compaction on the v1alpha3 AgentTemplate CRD (tokenThreshold / eventRetentionSize for tail retention inside a turn, compactionInterval / overlapSize for a sliding window over turns, an optional summarizer.modelConfig + promptTemplate, CEL for the half configurations), the kagent compiler writing it into config.json and the Go runtime handing it to the ADK runner (context compaction enabled … in the actor’s startup log). components.kagent / components.kagent-crds move to >=0.11.0-gs.16 <0.11.1-0: the floor, because the platform’s agent chart (giantswarm/agent 1.3.0) renders a compaction default on every platform agent — tail retention at 24 000 prompt tokens keeping the last four events, the summaries on the agent’s own model — and an older CRD would prune the field at admission, leaving every long turn re-reading its whole history in silence. The meta chart renders no AgentTemplate itself; the default and its knobs (context.compaction.*, enabled: false to opt out) live in the agent chart, which every agent HelmRelease follows on range 1.x. tests/verify-kagent-crds.py validates the connectivity chart’s kagent objects against the gs.16 CRDs; the BOM example and tests/verify-components.py follow the pin. Every installation rolls kagent once (the controller, then the platform Harness recompiles every admitted AgentTemplate; about 20 s per template) and, once the agent chart release is admitted, every agent recompiles once more with its spec.context — UPGRADE.md.
  • Added

    • muster test: scenarios can run their instance on Valkey storage and restart it. pre_configuration.storage: {type: valkey} starts an in-process Valkey stand-in (miniredis) per instance and points every backed store – session auth, capabilities, OAuth tokens, state, client credentials, the OAuth server’s own store – at it, the way an installation’s stores outlive a pod; start_delay makes it answer only after muster serve started, for “Valkey is late” scenarios. test_restart_instance stops and starts muster serve on the same configuration while the store and the mock servers keep running and reconnects every user client with the bearer it held, so the steps after it act as the sessions that lived through a rollout; test_stop_valkey / test_start_valkey take the store away and bring it back with its data. Three scenarios reproduce bugs that were first seen on an installation because the suite could not express them: a family tool call after a restart without a listing (fails on v5.18.2, #1204), a Valkey that answers late at start being waited for rather than replaced by memory stores (fails on v5.19.12, #1229), and a pinned bare-401 server staying connectable and its grant revocable after a restart with no login in the new process (fails on v5.8.3, #1150, #1154). (#1236)

    Fixed

    • The DPoP replay cache’s Valkey client no longer enables valkey-go’s client-side cache: nothing reads through it, and its CLIENT TRACKING handshake failed against a server without the feature, which left muster’s OAuth server in degraded mode (service_unavailable on every request). The session stores and the mcp-oauth store already ran without it since v5.19.13.