agent-platform release v4.83.0
Changed
gemma-4-31bserves 32k on one L40S:--kv-cache-dtype=fp8with--max-model-len=32768(was 8k) (giantswarm/agent-platform#611). With vLLM’s bf16 KV cache the 48 GB card held 8640 tokens beside the 31 GiB of weights. The fp8 cache halves the need: on one L40S (g6e.2xlarge) with vLLM 0.23.0 the preset reached Ready with 49,885 tokens of KV cache (1.5 full-length requests at a time) and served tool calls through the gemma4 parser.