AI Agents

  • Changed

    • (golden) Drop the #575 holds — origin/main carries the gsoci image defaults, the kserve 0.4.x ranges and the imageVerification defaults, so holding them equal on both sides only narrowed the comparison in #584 by @teemow

    Full Changelog: https://github.com/giantswarm/agent-platform/compare/v4.41.0...v4.41.1

  • Changed

    • Every image default of both charts is a gsoci.azurecr.io reference, and modelServing.imageVerification defaults to the Giant Swarm signing identity (giantswarm/agent-platform#575; epic giantswarm/giantswarm#37853). The meta chart’s kubectl hook images (gitops.hooks.image, gitops.hooks.helmImage), the bundled Flux engine’s operator image (flux-engine.operator.image) and the connectivity chart’s hook images (hooks.kubectlImage, hooks.opensslImage) move to their gsoci copies — giantswarm/kubectl, giantswarm/alpine-k8s, giantswarm/alpine-openssl (mirrored by giantswarm/retagger#1228, the same digests as upstream) and giantswarm/flux-operator (the Renovate manager that moves the subchart’s appVersion with the operator tag names the gsoci image). modelServing.imageVerification gains defaults: images: [gsoci.azurecr.io/giantswarm/*] and one keyless attestor — issuer https://oidc.circleci.com, subject ^https://circleci\.com/api/v2/projects/[a-f0-9-]+/pipeline-definitions/[a-f0-9-]+$ — the identity the architect orb’s cosign keyless signing gives every image a Giant Swarm CircleCI project builds or re-signs (the certificate’s subject is the pipeline definition that ran; the orb verifies with the same pair), and a new type (SigstoreBundle, the default | Cosign): cosign 3 attaches the signature to the image as a Sigstore bundle in an OCI referrer and Kyverno reads one format per rule, so an orb-signed image verified with Kyverno’s default Cosign type fails with no signatures found (kyverno apply against a signed gsoci image: Cosign fails, SigstoreBundle passes with the attestor). The switch stays off in the defaults until the curated model images are published and signed (giantswarm/agent-platform#554); the runtime and the llm-d sidecars (the llm-d line signs every mirror and repack since v0.5.0) and every image retagger copies (giantswarm/retagger#1230) already carry the identity. New make verify-images (in CI through verify-all): every container image of every pod template, every OCIRepository url and every registry and reference the meta chart forwards in the rendered defaults of both charts — the engine and every component on — is on gsoci; the references whose copies are still being published are tolerated by name with the issue that lands each — the kagent and Substrate lines’ charts and images, the CloudNativePG chart and the Flux controllers (giantswarm/agent-platform#580, giantswarm/retagger#1229), the classic runtime’s vLLM image (giantswarm/agent-platform#574) — and a stale entry fails the check. The five KServe components (kserve-crd, kserve-resources, kserve-llmisvc-crd, kserve-llmisvc-resources, kserve-runtime-configs) move to 0.4.x, the giantswarm/kserve line whose every image default is a gsoci reference (giantswarm/kserve#81: the controllers, agent, router, storage-initializer, the local-model images, kube-rbac-proxy; the classic runtimes ship no image any more, which this chart’s servingruntime.enabled: false never rendered); 0.4.1 is the version the storage-initializer’s resources follow kserve.storage.resources from (giantswarm/kserve#86) — the BOM example pins it. The connectivity CI fixtures name the gsoci pgvector mirror. README: “Verifying the model images”. Lab proof (a fresh agentlab on the branch’s dev build, Kyverno 1.17.2): platform-test all PASS; every hook Job ran from gsoci.azurecr.io/giantswarm/alpine-k8s / alpine-openssl and the Flux operator from gsoci, the only images off gsoci the pending set; the rendered verifyImages policy with the defaults admitted the orb-signed kserve-controller:0.2.4 pinned to its digest (kyverno.io/verify-images … pass) and the retagger-signed kubectl:v1.37.0 likewise, left a docker.io image alone (no pattern matches), denied the unsigned alpine-k8s:1.37.0 and — with the subject pattern swapped for a foreign identity — denied the signed image (sigstore bundle verification failed: no matching signatures found). Installations act: none for the defaults — a hook Job is created per event, nothing rolls; an installation with modelServing.imageVerification.enabled: true and a signer of its own that writes cosign 2’s .sig tags sets type: Cosign (UPGRADE.md).
  • Changed

    Full Changelog: https://github.com/giantswarm/klausctl/compare/v0.4.8...v0.4.9

  • Fixed

    • A session connecting to an MCPServer no longer reconciles it. With the reconciler’s duplicate cadence gone (#1285), every server was still reconciled – and its CR status written – once per session that connected to it: the aggregator reports each successful session connection to api.UpdateMCPServerState, which queued a reconcile unconditionally, so every new session’s SSO fan-out cost one pass per server it reached, several times per ten minutes on a busy installation, while nothing changed. The handler now only hands the state to the service. A state that actually changes (Auth Required to Connected on the first session, back on the last session’s loss) reaches the CRD status through the service’s state-change event and the StateChangeBridge, as every other state change does; an unchanged state is logged at debug and costs nothing. TriggerReconcile is gone from the reconcile manager’s API; it had no other caller. (#1290)
  • Fixed

    • Filesystem mode: a status sync no longer rewrites the definition file. The status the reconciler records for an MCPServer or Workflow (state, last error, last connection) was written into the definition’s own YAML file: the sync read the file, applied the status and renamed its copy over the original. A spec written by another process between that read and that rename – an operator’s editor, a GitOps sync – was overwritten with the stale spec, and the reconciler then saw the old definition: the change never happened. In CI this was the flaky oauth-auth-config-change-resets-live-sessions (the pin of an authorization server lost to a status write of the connect it followed, so the session stayed Connected and the reset never came) and oauth-pinned-authorization-server-change-takes-effect (the same loss, surfacing as the sign-in rate limit). Status now lives under status/ at the definition’s relative path (status/mcpservers/<name>.yaml), the definition files are only ever written by core_mcpserver_create/core_mcpserver_update and their workflow counterparts, a status: block left in a definition file by an earlier muster is ignored, and a status write no longer fires the change detector that watches the definition directories. (#1288)
  • Fixed

    • An OAuth server’s spec.timeout reaches the client a tool call builds on a pool miss. #1284 gave the login-time client of a server with auth.type: oauth the server’s timeout; the client the tool-call path builds when a session is already authenticated but holds no pooled connection – after a restart of the aggregator, after an eviction – was still built without it (and without spec.meta), so the first call after such a restart was cut at the default 30 s although the server declared more. Both are handed to that client too. (#1283)
  • Changed

    • components.kserve-runtime-configs follows the 0.3.x line (giantswarm/kserve#82). kserve-runtime-configs 0.3.0 adds kserve.llmisvcConfigs.images.<preset>.<container>, a per-config image override beside imageRegistry; its default render is byte-identical to 0.2.4’s, and the meta chart’s kserve-runtime-configs: block passes nothing new. The BOM example pins 0.3.0.

    Fixed

    • modelServing.prepull.nodeSelector set by an installation renders alone (giantswarm/agent-platform#562). Karpenter’s karpenter.k8s.aws/instance-gpu-manufacturer: nvidia was the values default in both charts, and Helm merges maps: a selector an installation set rendered next to the default key — a DaemonSet no node matched — and a null for the key was coalesced away when the meta chart forwarded its values, so the README’s “or another selector here” was not reachable, and an installation whose GPU nodes are not launched by Karpenter stripped the key with a post-renderer. Both values files now carry nodeSelector: {} and the template holds the default (agent-platform.modelServing.prepull.nodeSelector): empty renders Karpenter’s label, a set map renders alone, the pool’s label (gpuPool.nodeSelector) is merged under either; a non-string label value fails the render naming the key. tests/verify-mirrored-values.py counts an empty mapping as a leaf, so a copy that kept the map is drift; make verify-model-serving-policies asserts the selector cases, the guard, and a selector set on the meta chart reaching the DaemonSet alone. README “Pre-pulling the runtime image”; UPGRADE.md.
    • The pre-pull DaemonSet is a hook object: an image that cannot be pulled yet no longer fails the connectivity upgrade (giantswarm/agent-platform#563). As a release resource the DaemonSet was in the release’s wait, and a release’s wait counts a DaemonSet ready by its pods: helm-controller from 1.6 waits with the kstatus poller, which needs every pod of a DaemonSet Ready (Helm 3’s legacy waiter, still on helm-controller 1.4, takes maxUnavailable: 100% as ready once the pods are scheduled — the fleet runs both). A pod whose image cannot be pulled is never Ready, so one listed image not yet pullable — a runtime image or a prepull.modelPresets image still being built, a registry outage — held the pods in ImagePullBackOff, timed the wait out and marked the whole connectivity release failed and retried, every other object applied. The DaemonSet now carries helm.sh/hook: post-install,post-upgrade,post-rollback (weight 0, before-hook-creation): created after the release’s objects are applied and waited for, waited for by nobody — a hook’s WatchUntilReady watches Jobs and Pods only, in Helm 3, Helm 4 and helm-controller alike — so a missing image leaves the pods retrying on the kubelet’s backoff and the release Ready; the next event replaces the DaemonSet and the pods come back on images already present. A hook object is not Helm’s to delete on uninstall, so a pre-delete hook Job (<release>-model-serving-prepull-cleanup, the hook include agent-platform.hooks.job; the identity <release>-hooks is created for that event too, with delete on exactly that DaemonSet) removes it. The deny-all policy stays a release resource. make verify-model-serving-policies asserts the hook shape, the cleanup Job, the identity’s rule and that prepull.enabled: false renders neither; verify-serving-slice holds the identity to the pre-delete event alone with the cache off; verify-gpu-pool leaves the DaemonSet out of the golden comparison while origin/main renders it as a release resource. README “Pre-pulling the runtime image”; UPGRADE.md.
  • Changed

    • The connectivity chart README names the GPU node’s locked-memory requirement (giantswarm/agent-platform#564): a container inherits containerd’s RLIMIT_MEMLOCK and has no CAP_IPC_LOCK, so a serving runtime that mlock()s its weights dies at engine start under the unit’s 8 MB default; a pool node created through the platform runs containerd with LimitMEMLOCK=infinity (gpu-node-pool’s memlock.conf drop-in), and a GPU node not created through the platform needs the same drop-in. No rendered object changes.

    Fixed

    • The storage-initializer’s memory limit is 8Gi, headroom for a download that overlaps the runtime image’s unpack (giantswarm/agent-platform#576). modelServing.policies.storageInitializerMemoryLimit defaulted to 4Gi, sized after KServe’s 1Gi default was OOM-killed by an 8 GB download; the limit also holds the page-cache pages of the files the initializer writes, which the kernel charges to the container’s cgroup until write-back has drained them to the disk. On a 4 vCPU / 16 GiB L4 node the GPU became allocatable about 77 s before the runtime pre-pull had finished gunzipping the 8.8 GB image, so a 9.4 GB download into the emptyDir on the node’s local disk ran inside a CPU-bound unpack (load average 30, write-back saturated), its dirty pages stayed charged to the initializer and the init container was OOM-killed (exit 137) a minute in; the same download had passed whenever it started after the pre-pull, and a 16 vCPU node with the same overlap was fine. The zstd re-layered runtime image (giantswarm/agent-platform#568, 4.40.0) unpacks in about 30 s and ends the pre-pull before the GPU appears, which removes the collision on the L4 size; 8Gi is the headroom for a download that still overlaps an unpack (a bigger model, a slower disk). It fits the smallest curated L4 size: init and main containers do not run at once, so the pod’s effective request stays the predictor’s own 10Gi, within the ~12 GiB a 16 GiB node leaves to pods after the kubelet’s reservations. The meta chart mirrors the default; the README says what the limit must hold.
  • Fixed

    • An MCPServer in sync is left alone. In Kubernetes mode the reconciler logged Reconciling MCPServer: <name> for every server about every 14 s – some 250 passes per server per hour, each a definition read and a status write against the API server, while nothing changed. Two periodic sources of the same length ran out of phase: every successful pass requeued itself after 30 s (the status sync from before the periodic resync existed), and the Manager’s resync enqueued every server every 30 s on top. A pass that finds the server in sync now asks for nothing further – the next one comes from a definition change, a runtime state change (StateChangeBridge) or the resync – and the resync is a safety net for a lost event rather than a heartbeat: its default is ten minutes instead of 30 s. MUSTER_RECONCILER_RESYNC_INTERVAL still overrides it and muster test still runs instances at 2 s. DefaultStatusSyncInterval is gone. (#1285)
  • Fixed

    • An OAuth server’s spec.timeout reaches the client a person’s grant connects. #1282 wired the budget into the token-exchange and token-forwarding clients only; the client establishConnection builds for a server with auth.type: oauth – the core_auth_login callback and the grant reuse on first use – kept the zero timeout, so a tool call on such a server was cut after the default 30 s (no answer within the server's timeout of 30s) although its definition declared more. The entry’s spec.timeout is read with its spec.meta and handed to the DynamicAuthClient (new WithTimeout) and to the static-bearer client alike. (#1283)