AI Agents

  • Fixed

    • Image verification renders the egress Kyverno’s admission controller needs, so a model pod is admitted on an installation whose Kyverno may reach the API server only (giantswarm/agent-platform#599). A verifyImages rule is evaluated by Kyverno’s admission controller itself: it fetches the image’s Sigstore bundle from the registry and verifies it against the Sigstore trust root. The fleet’s Kyverno chart gives the admission controller a CiliumNetworkPolicy allowing egress to the API server only, and Cilium’s default-deny takes the rest — DNS included —, so every fetch failed before it started (failed to fetch bundles: … dial tcp: lookup gsoci.azurecr.io: i/o timeout), and since verifyImages runs in the mutating webhook with failurePolicy: Fail, the error denied the pod whatever failureAction said (Audit too): from the switch’s default-on (4.43.0) every model pod on such an installation was refused, its ReplicaSet retried with backoff, the LLMInferenceService stayed MinimumReplicasUnavailable and no GPU node was launched — seen on an installation minutes after the release rolled, invisible to the lab’s proof, whose Kyverno runs without that policy. The connectivity chart now renders, where the image-verification ClusterPolicy renders and the network-policy flavor is cilium, one CiliumNetworkPolicy <release>-model-serving-image-verification-egress in Kyverno’s namespace selecting the admission controller pods, with DNS through Cilium’s DNS proxy and TCP 443 to the registry hosts (gsoci.azurecr.io, the Azure Container Registry data endpoints *.*.data.azurecr.io a blob read is redirected to) and the Sigstore hosts (tuf-repo-cdn.sigstore.dev, the trust root; rekor.sigstore.dev, the transparency log the keyless attestor names) — additive to whatever Kyverno’s own policies allow. The block is modelServing.imageVerification.kyvernoEgress (enabled, namespace, podSelector, hosts as toFQDNs entries), mirrored in both charts; enabled: false renders none for a Kyverno that reaches the registry already; an empty podSelector (the default) selects the upstream chart’s admission-controller labels and a set one renders alone; the guards refuse an empty namespace or host list, a selector that is no mapping, a host that is no toFQDNs entry, a nulled block and an unknown key, naming the key. make verify-model-serving-policies asserts the default render, the switches, the guards and the meta chart’s forwarded values. UPGRADE.md, “image verification renders the egress Kyverno’s admission controller needs”.
  • Fixed

    • The hook Jobs get memory headroom, and the storage-version backup no longer parses this chart’s own release (giantswarm/agent-platform#593; a lab, 2026-09-19). The pre-upgrade backup hook pod ran with a 128Mi limit and was OOM-killed three seconds into its first kubectl calls — the kernel’s report shows the pod’s cgroup at 128Mi with only sh and one kubectl (17 MiB) alive, the rest held by nothing that was still running — and the Job’s retry pod finished nine seconds later: a headroom problem, not a leak. Measured in the hook image against a large installation (1069 release Secrets, 715 CRDs): one kubectl run peaks near 50 MiB with a cold discovery cache, the decode of the largest release Secret (4.3 MB of JSON) near 35 MiB, helm get values on this chart’s release near 80 MiB — nothing a hook does scales with the cluster, so every hook Job (agent-platform.hooks.job and the self-management hooks) now runs with limits.memory: 512Mi, the requests unchanged at 10m / 32Mi. The one allocation that did grow with the payload is gone: the backup’s release scan pre-filtered with a plain grep for kagent.dev/v1alpha2, which this chart’s own release matches on every installation — the hook script is stored under .hooks[] — so jq parsed every kept revision of the meta release on every upgrade; the pre-filter is now the JSON-encoded marker apiVersion: kagent.dev/v1alpha2\n (the version followed by an encoded newline, what only a manifest document carries; script text encodes the same newline as a doubled backslash), and a 4.x installation’s upgrade runs no jq at all. make verify-kagent-storage-version asserts the marker, refuses the plain grep, runs the marker against two fixtures (this chart’s rendered backup Job stored as a hook next to a manifest without kagent objects; a manifest naming a v1alpha2 object) and pins both hook Jobs’ resources.
  • Changed

    • The substrate chart’s third-party image defaults are pinned to their gsoci copies in the forwarded substrate.images block, and make verify-images sees a bare Docker Hub name (giantswarm/agent-platform#575, #580). The substrate release ran postgres:18-alpine@…, rustfs/rustfs:… and amazon/aws-cli:… from Docker Hub — short names whose registry is the node runtime’s call — and the atenet router’s and egress’s agentgateway from the line’s ghcr build: the chart’s own images: defaults, forwarded verbatim by a block that set none of them, so the check that holds every default to gsoci never saw them. substrate.images.{postgres,rustfs,agentgateway} now name the gsoci.azurecr.io/giantswarm/… copies at the same digest (postgres:18.4-alpine@… — the mirror carries Docker Hub’s floating 18-alpine digest under its own tag — rustfs:1.0.0-beta.3@…, agentgateway:v1.5.1-gs.4); images.awsCli is forwarded at the chart’s own short name on purpose — the rustfs-bucket-init Job is a release resource whose pod template is immutable, so a changed image fails the upgrade of every installation running the bundled store — and moves with the Substrate release whose Job can be recreated. tests/verify-images.py fails a string under a key named image, or an entry of a map or list named images, that is shaped like an image reference off gsoci — a bare name included — and tolerates the bucket-init image by name until then; the new make verify-substrate-images pulls every Substrate release the range admits and holds each pin to the release default’s digest or tag under gsoci, renders the release with the forwarded block and the bundled store on, and refuses a third-party image left off gsoci, so a Substrate re-pin that moves a default (another agentgateway build, a newer postgres) cannot leave the meta chart forwarding the old one. README “Agent Substrate”; UPGRADE.md.
  • Fixed

    Full Changelog: https://github.com/giantswarm/muster/compare/v5.27.7...v5.27.8

  • Changed

    • (deps) Update gsoci.azurecr.io/giantswarm/alpine docker tag to v3.24.2 in #215 by @renovate[bot]

    Full Changelog: https://github.com/giantswarm/klaus-operator/compare/v0.0.163...v0.0.164

  • Fixed

    Full Changelog: https://github.com/giantswarm/klaus/compare/v0.1.22...v0.1.23

  • Removed

    • The classic KServe InferenceService serving path — llm-d is the one serving path. Breaking (giantswarm/agent-platform#574; the platform’s decision that no model is served on the classic vLLM path any more, no fallback, no deprecation window). The connectivity chart rendered two paths for a served model: the classic InferenceService on the chart’s kserve-vllm ClusterServingRuntime (modelServing.runtime, modelServing.additionalRuntimes, the classic predictor pod shape in the Kyverno policies, the network policies and the PolicyException, the serving.kserve.io/v1beta1 check behind kserve.requireApi, the runtime / runtimes keys of the discovery ConfigMap) and the llm-d path (LLMInferenceService on the well-known LLMInferenceServiceConfigs, the models Gateway), while every shipped preset was composed onto the llm-d template by model-manager. Now the chart renders no serving.kserve.io object at all: model-manager composes a published preset into an LLMInferenceService, and the serving namespace’s policies, network policies and exception select that workload pod alone (kserve.io/component=workload, app.kubernetes.io/part-of=llminferenceservice, container main, port 8000). Removed, and refused by the render naming the key: templates/model-serving/clusterservingruntime.yaml; the values modelServing.runtime, modelServing.additionalRuntimes (4.36.0, #550), modelServing.networkPolicy.predictor (its additionalIngressNamespaces is modelServing.networkPolicy.additionalIngressNamespaces), modelServing.serving.deploymentStrategyType and .timeoutSeconds; the preset fields spec.runtime and spec.predictor (the shipped qwen3-8-27b, qwen3-8-27b-l40s and qwen3-8-flash-next-nvfp4 lose their predictor.timeout, the Flash-Next preset its runtime: kserve-vllm-b12x — its B12X serving image is the well-known config’s to name); the discovery ConfigMap’s spec.runtime and spec.runtimes. modelServing.kserve.requireApi and modelManager.kserve.requireApi check serving.kserve.io/v1alpha2 (LLMInferenceService) and are satisfied by the kserve-llmisvc-crd + kserve-llmisvc-resources components. The meta chart drops the classic controller’s components kserve-crd and kserve-resources and their values blocks (refused too): kserve-llmisvc-crd ships every CRD the llm-d path needs (llminferenceservices, llminferenceserviceconfigs, clusterstoragecontainers), and the kserve-llmisvc-resources release renders the control plane’s shared objects itself (kserve.createSharedResources: true, the chart’s default; a false is refused) with controller.deploymentMode: Standard and gateway.disableIngressCreation: true, receives the derived models-Gateway reference (controller.gateway.ingressGateway.kserveGateway) and depends on kserve-llmisvc-crd alone; the connectivity release depends on no kserve component (it renders no serving.kserve.io CR), model-manager depends on kserve-llmisvc-resources and its range is >=0.23.0 <2.0.0 (model-manager 1.0.0 composes LLMInferenceServices only; nothing the chart forwards names the values it drops). examples/serving-slice.yaml, ci/test-slice-serving-values.yaml, the CI fixtures and the BOM switch the llm-d components on alone; tests/verify-runtimes.py, its fixture and the classic predictor pod fixture are gone, and verify-wiring asserts that the classic keys and components fail the render naming them and that the serving render carries no serving.kserve.io object. UPGRADE.md, “the classic KServe InferenceService path is removed”. Closes giantswarm/agent-platform#561 (removed with the feature).
  • Added

    • modelServing.imageVerification is on by default: a model pod’s images are verified at admission against the Giant Swarm signing identity (giantswarm/agent-platform#575; epic giantswarm/giantswarm#37853). 4.41.0 gave the block its defaults — images: [gsoci.azurecr.io/giantswarm/*], one keyless attestor for the identity the architect orb’s cosign keyless signing gives every image a Giant Swarm CircleCI project builds or re-signs (issuer https://oidc.circleci.com, subject the pipeline definition that ran), type: SigstoreBundle — and left the switch off until every image a model pod runs was signed. Every one is now, verified with cosign verify against that identity: the curated model image the shipped oci:// preset names, gsoci.azurecr.io/giantswarm/models/qwen3-8-flash-next-nvfp4:7c4f1bc1a2d6, which giantswarm/models builds, signs and publishes (giantswarm/agent-platform#554); the llm-d runtime and sidecars of the well-known LLMInferenceServiceConfigs at the llm-d-fast/ prefix (llm-d-cuda:v0.8.0, llm-d-router-disagg-sidecar:v0.9.0, the endpoint picker, tokenizer and latency-predictor images; giantswarm/llm-d signs every mirror and repack); the KServe storage-initializer:v0.20.0 and agent:v0.20.0 retagger copies (giantswarm/retagger#1230); the pre-pull DaemonSet’s pause:3.10.1; and the vllm-b12x:20260913 mirror an installation serves the flash-next preset with. Both charts’ defaults carry enabled: true, so a serving shape under Kyverno renders <release>-model-serving-image-verification without being asked: one verifyImages rule per pod shape over the model pods of the serving namespace at CREATE and UPDATE, every verified image pinned to its digest, a signature required, Enforce. A container image matching no pattern is left alone, so an installation serving from a registry of its own is untouched until it names its pattern and its signer; an installation whose model pods run an image under the platform’s namespace that none of these signed adds its signer or turns the switch off before it upgrades (UPGRADE.md). make verify-model-serving-policies asserts the default render carries the policy with its defaults and that the switch off or no Kyverno renders none; make verify-target holds the switch on both sides of the golden comparison until origin/main carries it. Proof: kyverno apply of the rendered default policy over a pod of the llm-d workload shape naming every image above admits it with every image rewritten to its digest, and denies a pod naming an unsigned image under the pattern with sigstore bundle verification failed; the same two cases on a lab’s Kyverno.
  • Fixed

    • The LLM listener attributes a kagent API v2 agent’s model calls to the agent, not to the Substrate egress (giantswarm/agent-platform#586). gateway.metricLabels’ agent and agent_namespace were the source workload resolved by IP (source.unverifiedWorkload.serviceAccount / .namespace), the shape of kagent 0.10 — one Deployment and one ServiceAccount per agent. Under kagent API v2 an agent is an AgentTemplate executed as a Substrate actor in the shared WorkerPool, and every actor connection leaves the worker as a mTLS CONNECT through the Substrate egress (atenet-egress), which authenticates the actor at CONNECT time and tunnels the bytes opaquely: by source IP every agent’s model call was the egress pod, every agent’s spend one row named after a proxy, and user unknown (actor traffic carries no bearer). The egress cannot add a header. The kagent runtime sends the identity on every model call as request headers — x-kagent-agent and x-kagent-agent-namespace (the template’s name and namespace) and x-kagent-user (the person of the turn) — and the three default expressions read them when, and only when, the source workload is the egress (agent-platform.substrate.egressCall: the namespace and ServiceAccount from the chart’s Substrate names, agent-platform.substrate.egressServiceAccount); every other caller keeps its own ServiceAccount, so no pod relabels itself by sending the header, and a call from the egress without the header reads unknown — the egress ServiceAccount itself appears in no new sample. The identity stays an accounting identity, never an authorization input. Mirrored in the meta chart. make verify-metric-labels asserts the three expressions with the predicate and the header names (parsed, since toYaml folds a long expression across lines), the person label’s hold and claim knob behind the predicate, an installation’s own expression replacing the default, and the meta chart forwarding the expressions unrendered; verify-target holds the old expressions equal on both sides of its golden comparison until origin/main carries this. README “Metric labels”. Installations act: none — the -metrics policy is re-rendered on the upgrade; the runtime half is the kagent line’s carried patch, and until an installation runs it the egress row reads unknown instead of the egress ServiceAccount.
  • Changed

    • The kagent line’s images, the Substrate line’s images and the bundled Flux engine’s controllers come from their gsoci copies (giantswarm/agent-platform#580; the copies giantswarm/retagger#1229; epic giantswarm/giantswarm#37853). retagger copies every release of the two lines and every Flux controller of the 2.9+ line to gsoci.azurecr.io under the upstream path — giantswarm/kagent/{controller,ui,golang-adk,claude-harness}, giantswarm/substrate/{ateapi,atecontroller,atelet,atenet,podcertcontroller,ateom-gvisor}, giantswarm/fluxcd/{source-controller,helm-controller} — with the index digests preserved, so one registry value moves per line and everything the charts stamp stands: kagent.registry is gsoci.azurecr.io (the controller and UI at the chart’s tag, the Go ADK and Claude harness images at the digests the chart carries, the kagent-images ConfigMap Substrate’s image cache reads), substrate.image.registry is gsoci.azurecr.io/giantswarm/substrate (the control plane at the chart’s tag; the worker image the meta chart derives from the Substrate floor follows it, as does the helper’s default) and flux-engine.instance.distribution.registry is gsoci.azurecr.io/giantswarm/fluxcd (the operator composes <registry>/<controller>:<tag>; Flux 2.9.5’s source-controller:v1.9.5 and helm-controller:v1.6.4 under the pinned operator). The two lines’ chart sources stay on ghcr.io (components.kagent*.repository, components.substrate*.repository): retagger copies images, not charts, and the lines publish their release charts to gsoci through their own tag workflows once that lands — the three chart references (and the upstream CloudNativePG chart, behind a Giant Swarm wrapper release on the same operator line) are what make verify-images still tolerates by name. A dev build’s images are not copied: the dev channel sets kagent.registry: ghcr.io and substrate.image.registry: ghcr.io/giantswarm/substrate next to its filter (README “Dev channel”). verify-components, verify-worker-image, verify-kyverno and verify-engine assert the new defaults; verify-target holds them on both sides of the golden comparison until origin/main carries them. UPGRADE.md.