agent-platform release v4.28.16

Fixed

• The model pods download their weights from Hugging Face without the Xet path (giantswarm/agent-platform#520; proof 1 of giantswarm/giantswarm#37639). On a Giant Swarm cluster a served model’s weights never arrived: KServe’s storage-initializer fetched the six small repository files over HTTPS from huggingface.co and then hung forever — no error, no restart, 0 CPU, eight sleeping threads (gazelle 2026-09-17 02:14Z and 02:30Z, chart 4.28.15, huggingface_hub 0.32.4 + hf_xet 1.5.1 + hf_transfer 0.1.9 in the initializer image, HF_HUB_ENABLE_HF_TRANSFER=1, HF_XET_HIGH_PERFORMANCE=1, HF_XET_NUM_CONCURRENT_RANGE_GETS=8). hubble observe --verdict DROPPED on the pod showed every SYN to the Xet CDN (15.236.85.177:443, world) policy-denied, and the endpoint’s cilium fqdn cache held addresses for cas-server.xethub.hf.co, us.aws.cdn.hf.co and huggingface.co — not that one. The connectivity chart’s serving-namespace CiliumNetworkPolicies are a toFQDNs allow-list (huggingface.co, *.huggingface.co, *.hf.co, *.*.hf.co on 443): Cilium admits exactly the addresses its DNS proxy saw resolved for an allowed name, for the answer’s TTL; the Xet CDN (CloudFront) answers with ~30 s TTLs and rotating address sets, and hf_xet — a Rust client with its own resolver cache and one connection per shard range-GET — keeps connecting to addresses the proxy has forgotten or never saw, so the shards’ SYNs are dropped and the client waits on its connect timeouts without reporting. The allow-list stays as it is — a model download is no reason to open world on 443 to the serving pods, and a CIDR block for a CDN is a list that rots — and the client comes off the Xet path: the pods ClusterPolicy gains one strategic-merge rule per pod shape, model-pod-env-<shape>, that merges modelServing.policies.env (default HF_HUB_DISABLE_XET=1) by name onto the storage-initializer’s and the shape’s runtime container’s env (kserve-container / main; every model pod with a storage-initializer, named model or not; the containers’ own entries stay, so HF_HUB_ENABLE_HF_TRANSFER=1 is kept). With Xet off, huggingface_hub downloads the shards with hf_transfer from the LFS fronts (cdn-lfs*.hf.co, cdn-lfs.huggingface.co — within the patterns) through the system resolver: every connection is a DNS query the proxy sees, and its address is admitted. The values document the mechanism, that widening the egress is not the fix, and the cluster-level alternative the chart does not own — a tofqdns-min-ttl on the Cilium agent long enough for the client’s cache, after which an installation may set env: [] to have Xet back; a mirror’s HF_ENDPOINT belongs in the same list. make verify-model-serving-policies (kyverno apply of the rendered policy over a fixture pod of each shape, the initializer now carrying KServe’s three HF variables and the classic runtime the ClusterServingRuntime’s two) holds the entry next to the containers’ own env on both containers of both shapes, added where a container has none, on a pod without a model name too, a no-op over the policy’s own output (webhook reinvocation), the download Job’s pod and a pod without an initializer untouched, no env rule with the list empty, and the mutated pod still passing the fleet’s restricted PSS with the chart’s exception. The live half — the 8 GB of qwen3-4b-instruct on the claim, the predictor Ready and /v1/chat/completions answering, a second load downloading nothing — is the proof’s. model-manager’s own download Job takes the same path on its side.