TL;DR
- Long-context MoE pressure rises: Tencent Hy3 and Poolside Laguna XS pushed sparse open models toward cheaper long-context serving, while vLLM and SGLang hardened cache, sparse attention, and MoE paths.1
- Blackwell and AMD paths got real work: TensorRT-LLM, FlashInfer, ROCm, and Triton moved low-bit kernels, NVFP4, FP8, and backend fixes for new GPU targets.2
- Local stacks chased cloud parity: llama.cpp, Ollama, Apple MLX, oMLX, and Osaurus improved MLX, GGUF, speculative decoding, model import, and multi-backend stability.3
- Edge inference became a packaging fight: LiteRT, ExecuTorch, OpenVINO, Qualcomm GenieX, sherpa-onnx, and ArmNN advanced NPU, QNN, WebGPU, Core ML, and mobile-serving paths.4
- Inference infra money stayed hot: SambaNova and Together AI funding showed that open-model serving clouds and custom inference hardware remain capital magnets.5
This Week in Inference
Tencent released Hy3, a Hunyuan-family sparse MoE model with about 295B total parameters, about 21B active parameters, and a 256K context window, aimed at coding, agents, workflow planning, and long-form tasks (announcement).6 Poolside released Laguna XS, a 33B total, roughly 3B active MoE coding model with a context window near 262K tokens, aimed at private developer infrastructure and long-horizon codebase work (coverage).7 Anthropic’s Claude Sonnet raised proprietary cloud expectations with 1M-token context, 128K output, and stronger tool and terminal workflows, which puts more pressure on open serving stacks to make long-context memory use predictable.8
The systems trend was clear across the repos: speculative decoding, KV-cache control, and low-bit kernels now define the cost curve. vLLM added heterogeneous draft and target vocabulary support for speculative decoding, SGLang hardened DeepSeek and GLM sparse-attention paths, and LightLLM added Qwen MTP speculative decoding for dense and MoE models (vLLM).1 Long-context work also showed up in llama.cpp server and prompt-cache fixes, OpenVINO prefix-cache and block KV-cache work, TensorRT-LLM disaggregated serving, and ROCm ATOM cache-aware routing experiments (OpenVINO).9
Hardware news stayed focused on deployment rather than new chips. NVIDIA pushed Blackwell, Jetson Thor, CUDA toolchain, CuTe DSL, and NVFP4 MoE paths across TensorRT-LLM, CUTLASS, and TensorRT Edge-LLM (TensorRT Edge-LLM).10 AMD’s ROCm stack moved AITER and ATOM in lockstep for Instinct serving, while Qualcomm, Google, OpenVINO, Arm, and sherpa-onnx made QNN, NPU, XNNPACK, and mobile CPU heuristics more usable for edge deployment (ROCm AITER).11 The money followed the same story: SambaNova raised $1B at an $11B valuation, and Together AI had closed a large round days earlier, both betting that inference infrastructure will absorb the next wave of open models (SambaNova).5
Top Stories
vLLM and SGLang work through the MoE serving mess vLLM added Token-Level Intersection for speculative decoding with different draft and target vocabularies, fixed Qwen reasoning and tool-call behavior, and advanced DFlash, FlashInfer, MXFP4, DeepGEMM, and XQA attention paths (vLLM).1 SGLang hardened DeepSeek and GLM sparse attention, fused top-k, HiCache, CUDA graph, and disaggregated decode paths while refactoring resolved server configuration (SGLang).12 These changes matter because Hy3, Laguna XS, DeepSeek, Kimi, and GLM-class workloads stress cache layout, expert routing, and speculative verification before they stress raw FLOPs (SGLang planning).13
NVIDIA turns low-bit inference into a full-stack release train NVIDIA pushed TensorRT-LLM across sampling, dynamic speculation, guided decoding, FlashInfer-backed GDN MTP dispatch, DSV4 RoPE fusion, RopeQuant fusion, AutoDeploy, and disaggregated serving work (TensorRT-LLM).2 TensorRT Edge-LLM added DGX Spark, DFlash speculative decoding, Qwen Omni NVFP4, Gemma text inference, C++ audio preprocessing, Jetson Thor support, and wider NVFP4 MoE coverage (TensorRT Edge-LLM).10 CUTLASS and Edge-LLM also aligned around CuTe DSL artifacts and Blackwell targets, which makes NVIDIA’s low-bit path feel less like a kernel drop and more like a deployable stack (CUTLASS).14
Local inference stacks absorb model churn faster than before
llama.cpp merged a wide backend sweep across Vulkan, OpenCL, CUDA, Metal, SYCL, HIP, Hexagon, CPU, and quantization paths, while also moving llama-cli toward an HTTP-based implementation and improving server streaming progress (llama.cpp).3
Ollama rewrote its MLX create path, fixed Qwen architecture detection, improved thinking-parser behavior, and added an agent foundation with tool plumbing (Ollama).15
Apple MLX, oMLX, Osaurus, and Blaizzy’s MLX projects moved the same direction: more native kernels, more speculative decoding, better memory handling, and fewer fragile fallback paths (MLX).16
Edge inference gets serious about NPUs and packaging Google’s AI Edge stack moved low-bit XNNPACK kernels, LiteRT delegate support, LiteRT-LM APIs, NPU export, and Android Gallery polish in parallel (LiteRT).17 Qualcomm pushed GenieX and AI Hub Models around QAIRT, Qwen support, scorecards, device metadata, and app release operations (GenieX).18 sherpa-onnx added QNN export and runtime paths for Parakeet and Nemotron ASR, which shows how speech inference is joining the same NPU packaging track as LLMs and VLMs (sherpa-onnx).19
The serving control plane is becoming part of inference performance
LiteLLM added MCP OAuth, token exchange, semantic tool filtering, Model Armor scans, provider bridge fixes, pricing controls, rate limits, and observability work in one high-volume week (LiteLLM).20
Ray Serve added direct ingress streaming paths, HAProxy metrics, gRPC hardening, request cancellation accounting, and Data/LLM timeout controls (Ray).21
ai-dynamo advanced KV-aware routing, cache_salt isolation, vLLM-Omni multimodal paths, RDMA tensor transfer, and Kubernetes operator work, which places routing policy closer to the model runtime (ai-dynamo).22
Deeper Dive
Everything below is for readers who want the full picture. Feel free to scroll.
Code Changes by Category
Cloud & Datacenter Serving
vLLM had one of the busiest weeks in the ecosystem, with core work on MRV layout cleanup, async scheduling, CPU offload cache policy, memory profiling reuse, heterogeneous speculative decoding, FlashInfer, DFlash, MXFP4, DeepGEMM, XQA decode, ROCm, XPU, TPU, and Gaudi plugin fixes (vLLM, vllm-gaudi).1 SGLang matched that intensity with DeepSeek and GLM sparse-attention fixes, DSA fused top-k work, HiCache reliability, diffusion serving, Intel XPU speculative decoding, and a resolved-configuration refactor (SGLang).12
TensorRT-LLM pushed serving internals across sampling, dynamic speculation, guided decoding, AutoDeploy, KV/disaggregated serving, FlashInfer MLA, DSV4 RoPE fusion, and MTP paths (TensorRT-LLM).2 Ray focused on Serve direct ingress, streaming gRPC, HAProxy metrics, /classify and /pooling endpoints, FastAPI compatibility, queued-request cancellation, and Data/LLM timeout control (Ray).21 ai-dynamo moved KV-aware routing, selected-worker reporting, cache_salt tenant isolation, vLLM-Omni multimodal serving, RDMA tensor transfer through NIXL, and Kubernetes operator support (ai-dynamo).22
LiteLLM remained the busiest gateway layer. The project added MCP OAuth and token exchange work, Entra OBO support, semantic tool filtering, Model Armor scans, Claude and Codex compatibility fixes, provider bridges, pricing controls, Prometheus and OTEL improvements, and admin UI hardening (LiteLLM).20 LMDeploy tightened guided decoding under speculative decoding, fixed Qwen MTP distributed warmup, reduced multimodal memory retention, and improved streaming tool-call parsing (LMDeploy).24
Local LLM Runtimes
llama.cpp merged a wide backend sweep covering Vulkan, OpenCL, CUDA, Metal, SYCL, HIP, Hexagon, CPU, ARM NVFP4, and official Q2_0 quantization, with server and CLI fixes around HTTP mode, /responses, prompt cache, speculative loading, and DeepSeek KV behavior (llama.cpp).3 Ollama added an agent foundation, rewrote MLX model creation into staged import phases, fixed Qwen architecture detection, tightened thinking parser behavior, added JetPack CUDA fallback, and bumped both llama.cpp and MLX (Ollama).15
LocalAI shipped a large reliability pass across reasoning and tool-call compatibility, ROCm/HIP backend assets, vLLM and llama.cpp integration, SSRF protection, Prometheus metrics, OIDC behavior, image generation, model capability metadata, and backend logs (LocalAI).25 oMLX added Lightning MTP speculative decoding, Qwen prefill kernels, DeepSeek and GLM custom kernels, Hy3 support, oQe imatrix quantization, memory lifecycle fixes, and OpenAI-compatible serving improvements (oMLX).26 Osaurus turned native search into a first-class surface, hardened Hugging Face downloads, added benchmarking, improved warm-up KV caching, and expanded MCP, Agent Channels, memory pressure, and vMLX reliability paths (Osaurus).27
Apple Silicon & MLX Ecosystem
Apple MLX shipped generated qmm implementations, CUDA CI swap enablement, qmm K-tail fixes, and a set of correctness patches for NAX split-K GEMM, fp-quantized matvec, grouped conv gradients, CPU gather transposes, CUDA RMSNorm, and negative-strided compiled kernels (MLX).16 mlx-swift fixed iOS build behavior and added FP8 conversion APIs for weight quantization and dequantization (mlx-swift).28 mlx-lm fixed a Transformers tokenizer registration crash, while mlx-swift-lm had open work on Qwen, Gemma, safetensors, and CI reproducibility (mlx-lm).29
Blaizzy’s MLX stack expanded MOSS audio support, MOSS Swift parity, Qwen ASR batching, Nemotron chunking, and native VLM text backends for Llama, Qwen, Mistral, Gemma, Nemotron, DeepSeek, GLM MoE, and Mamba-style paths (mlx-audio, mlx-vlm).31 Mozilla’s llamafile restored HTTPS downloads and TLS serving through cpp-httplib plus Mbed TLS, added Metal transcription acceleration for transcribefile, and delegated CLI help to llama.cpp’s option parser (llamafile).32 WallyAI added an Apple MLX backend across C++ catalog routing, Swift runtime, lifecycle, LLM, VLM, embeddings, STT, and TTS paths (WallyAI).33
Mobile & Edge Frameworks
Google’s AI Edge portfolio moved as one stack: XNNPACK added low-bit kernels, LiteRT picked up YNNPACK two-bit and four-bit enablement plus delegate fixes, LiteRT-LM added thinking controls, Kotlin APIs, NPU sliding-window support, prefix caching, web-demo upload, and JS cloning, and litert-torch advanced ASR export plus NPU compiler and calibration flows (LiteRT, LiteRT-LM).17 Meta ExecuTorch added portable runtime and EXIR coverage, Arm TOSA and VGF work, Qualcomm QNN improvements, NXP Neutron, WebGPU backend work, Gemma quantization, BF16 DFlash GGUF loading, and multi-batch LLM quantization and evaluation (ExecuTorch).35
Qualcomm pushed GenieX and AI Hub Models around QAIRT, Qwen assets, GenieX scorecards, device metadata, Windows validation, and Trusted Publishing (GenieX).18 k2-fsa’s sherpa-onnx added QNN SDK setup, Parakeet TDT QNN export and runtime, Nemotron streaming QNN export and runtime, JavaScript per-stream hotwords, Qwen ASR homophone post-processing, and a large ONNX Runtime update (sherpa-onnx).19 ArmNN added a graph-level CpuAcc SME/SVE/NEON shape policy to avoid SME2 regressions on awkward quantized workloads while exposing faster kernels where useful (ArmNN).36
Alibaba MNN added C4 export, packed RoPE optimization, CPU KV-cache fixes for Eagle speculative decoding, Qualcomm QNN Resize/Interp support, and RISC-V build fixes (MNN).37 Tencent ncnn expanded reshape semantics for batch movement, fixed pnnx compatibility after a yanked export path, improved pooling and broadcast conversion, moved CI to newer Torch expectations, and fixed a MultiHeadAttention out-of-bounds read (ncnn).38
Compilers, Runtimes & Graph Engines
PyTorch released a major framework update while core work moved cuBLASLt grouped GEMM, CUDA package trimming, Inductor fixes, AOTInductor user-stream support, ROCm import reliability, MPS correctness, XPU support, Dynamo fixes, and release engineering (PyTorch).39 TensorFlow and OpenXLA shared a huge compiler week around ROCm, oneAPI, XLA GPU reshape validation, collectives, dynamic-slice runtime fixes, autotuning, profiling, and TensorFlow Lite delegate hardening (TensorFlow, OpenXLA).41
ONNX and ONNX Runtime hardened pytest coverage, shape inference, fuzzing, CUDA plugin EP defaults, CudaQuantizer, MatMulNBits, QMoE graph capture, GQA correctness, WebGPU GEMM, ARM64 two-bit MLAS kernels, QNN Gather, Core ML minimal builds, and packaging security fixes (ONNX Runtime, ONNX).42 TVM and TVM-FFI stabilized Optional and Variant ABI layouts, moved TVM to the new FFI requirement, removed SizeVar, advanced TIRx and Relax cleanup, and improved runtime thread safety (TVM, TVM-FFI).44
Triton updated its LLVM pin, improved split and join layout handling, added partition-aware allocator placement for MXFP8 by MXFP4 workloads, added Blackwell row-reduction fusion, improved AMD gfx11, gfx12, and gfx1250 paths, and fixed sanitizer and top-k behavior (Triton).46 TileLang shipped LLVM backend support, backend registry work, scheduler APIs, pass visualization, CUDA intrinsics, FP8 casts, TMA, GMMA, UMMA lowering, cache fixes, and DeepSeek TopK boundary handling (TileLang).47
Models, Quantization & Optimization
FlashInfer focused on Blackwell and SM100 kernels, FP4 and NVFP4 support, CuTeDSL TGV BF16 GEMM, MXFP8 split-K GEMM, NVFP4 paged KV append, TRT-LLM Gen decode scale overrides, VSA block-sparse integration, MoE EP packaging, and GLM and DeepSeek routing NaN fixes (FlashInfer).48 Dao-AILab FlashAttention added FA3 package install support, ROCm CK backward support for RDNA, and SM120 FA4 CuTe frontend and Pack-GQA work (FlashAttention).49
ROCm AITER and ATOM coordinated around fused all-reduce, RMSNorm, quantization, GLM, Qwen, DeepSeek, MiniMax, Kimi, FP8, MXFP, OPUS, FlyDSL, Triton conv2d, attention, MLA, RoPE, and serving process hardening (AITER, ATOM).11 Intel Neural Compressor expanded JAX and Keras quantization, including INT8 ViT, BF16-model quantization, Conv2D quantization, dot-product attention quantization, and Gemma cached-attention alignment (Neural Compressor).51 OpenVINO NNCF added INT2 and INT3 symmetric OpenVINO weight-compression types with backend and test coverage (NNCF).52
Other Notable Changes
OpenVINO core advanced NPUW Qwen VLM, block KV-cache, linear-attention hybrid model building, flash-attention mask skipping, GQA, long-RoPE prefix caching, GPU MoE weight offload, grouped matmul, cache-blob bounds checks, ONNX and PyTorch frontend KV formats, and GenAI Gemma, Qwen, Eagle, structured-output, and TTS work (OpenVINO, OpenVINO GenAI).9 CTranslate2 shipped a security-focused release with safer legacy checkpoint loading, a model-load heap overflow fix, a Whisper alignment divide-by-zero fix, and Windows CI restoration (CTranslate2).54
Cactus added Apple Metal support across build, engine, model, completion, embedding, and transcription paths, then added benchmark infrastructure for desktop and mobile runners (cactus).55 FluidAudio rebuilt downloads around ModelHub, resumable range downloads, byte-level progress, Hugging Face pagination, ASR and TTS fixes, and text-processing-rs NeMo parity work (FluidAudio).56 Zetic shipped its public iOS package for Melange and added cross-platform demos for pose motion, image-to-3D, and medical segmentation (ZeticMLangeiOS).57
Community Pulse
Issue-heavy projects told the week’s story. Open WebUI had no merges, yet users and maintainers triaged Safari blank-message regressions, OAuth session expiry, workspace model ACL bypasses, STT duplication, SearXNG behavior, Gemini endpoint compatibility, and knowledge/file-context semantics (Open WebUI).58 exo had no landed code, but users and contributors paired detailed reports with proposed fixes for Claude Code compatibility, MLX/JACCL long-context hangs, stale node identities, event-log races, elections, and orphan runner cleanup (exo).59
Candle stayed unmerged but active around f16 and bf16 accumulation correctness, Metal layout bugs, quantized serving, SDPA contiguity, FlashInfer decode attention, and paged KV seams (Candle).60 WebLLM had service-worker reliability reports around Workbox caching, model initialization callbacks, and cached inference hangs (WebLLM).61 DeepSeek’s repo had no code activity, but users reported repeated English-to-Chinese language routing failures in web chat, which points to product-layer inference controls as much as model behavior (DeepSeek-V3).62
Community Debates
MLX maintainers kept a firm backend roadmap. They closed AVX, CPU parallelism, fp64 SIMD, AddMM fusion, and zero-VJP proposals while pointing contributors toward the planned Highway and thread-pool migration or cleaner autodiff semantics (MLX).63 Triton maintainers took the same stance on scope, closing Apple Metal backend and old AMD architecture proposals while also pruning deprecated block-pointer paths (Triton).64
OpenVINO reviewers rejected workaround-style fixes where the tradeoff could harm long-prompt memory or mask real NPU accuracy problems, including GPU PA micro-SDPA padding and an INT4 NPUW workaround (OpenVINO).65 TensorFlow closed a TFLite buffer-offset overflow fix after UBSan concerns and moved the work to subtraction-based checks, which shows the bar for security hardening remains high even for urgent fixes (TensorFlow).66
vLLM and SGLang both had many superseded PRs as maintainers split large changes into smaller paths. vLLM redirected FlashMLA sparse DCP, NCCL symmetric-memory collectives, MIG UUID handling, and KV offload synchronization into replacement PRs (vLLM).67 SGLang closed a Unix-domain-socket server flag after conflicts with multi-tokenizer mode and Prometheus metrics, while broader MiniMax and AMD DeepSeek bring-up work moved into smaller integration efforts (SGLang).68
Worth Watching
Hy3 support will become a cross-stack test. oMLX already landed Hy3 support, llama.cpp has community interest, and ds4 has open work around GLM and sparse SSD-streaming patterns that overlap with Hy3-style deployment needs (oMLX, ds4).69
KV-cache compression is moving from research headline to product requirement. OpenVINO, vLLM, SGLang, TensorRT-LLM, ai-dynamo, ROCm ATOM, and LiteRT-LM all touched cache layout, prefix cache, offload, or routing this week (OpenVINO, vLLM).9 GGUF also keeps spreading beyond llama.cpp-native local use, with ExecuTorch and CTranslate2 release work showing the format and its model families moving into more deployment targets (ExecuTorch, CTranslate2).54
Watch the Blackwell, AMD, and edge NPU triangle. FlashInfer, TensorRT-LLM, CUTLASS, Triton, ROCm AITER, ATOM, Qualcomm GenieX, sherpa-onnx, and LiteRT all spent this week on vendor-specific paths, but the APIs above them still converge on OpenAI-compatible serving, model catalogs, and portable quantization metadata (FlashInfer, Qualcomm GenieX).74
Major Releases
Hugging Face shipped transformers v5.13.0 and diffusers v0.39.0, with the main theme split between new model coverage and attention/export hardening. Transformers highlighted Kimi architecture support and multimodal agent models, while Diffusers added new image and video pipelines plus Qwen-style and Cosmos-related components. Transformers release notes.75
Apple shipped MLX v0.32.0 and mlx-swift 0.31.6, centered on generated qmm implementations, CUDA CI and quantized qmm fixes, and iOS build repair. The core release matters most because it improves low-level matrix and quantized execution paths that downstream MLX LLM projects now depend on. MLX release notes.16
BerriAI shipped eight LiteLLM releases from v1.89.5 through v1.93.0-dev.1, with a fast cadence around MCP OAuth, provider compatibility, enterprise controls, signed Docker image verification, and admin reliability. The most important theme was turning LiteLLM into a safer policy and tool gateway for agent traffic. Latest release.76
ggml shipped 52 llama.cpp build releases from b9860 through b9929, with the dominant theme being backend and server hardening rather than a single feature launch. The latest builds include Vulkan, Hexagon, HTTP CLI, fused-op, recurrent batching, Metal, OpenCL, CUDA, SYCL, HIP, prompt-cache, and quantization fixes. Latest release.77
Google shipped LiteRT v2.1.6, focused on header-only C++ API refactoring, C++ usage without Abseil linkage, ARMv7 prebuilts, ATS single-op coverage, and built-in kernel improvements for higher-dimensional tensors. This release anchors the broader AI Edge work across XNNPACK, LiteRT-LM, litert-torch, Gallery, and MediaPipe..17
Meta shipped pytorch v2.13.0, with release notes covering a large framework update and a visible highlight around FlexAttention on Apple Silicon and MPS. The surrounding week also advanced Inductor, AOTInductor, ROCm, MPS, XPU, ExecuTorch, and on-device GenAI paths..39
NVIDIA shipped CUTLASS v4.5.3 and TensorRT Edge-LLM v0.9.0. CUTLASS focused on CuTe DSL compile-time regression fixes, while TensorRT Edge-LLM delivered the more visible inference update with DGX Spark, DFlash, Qwen3-Omni NVFP4, Gemma 4, C++ audio preprocessing, Jetson Thor, and expanded NVFP4 MoE support. TensorRT Edge-LLM release notes.10
Dao-AILab shipped fa4-v4.0.0.beta21, a prerelease bundling FA3 package install support, ROCm RDNA backward support, SM120 compile-time argument fixes, and SM120 Pack-GQA forward work. The release shows FlashAttention broadening across packaging, AMD, and Blackwell-era CUDA paths at once..49
FlashInfer shipped v0.6.14 plus nightly builds, with the main release changing install paths after PyPI storage limits forced flashinfer-cubin and JIT cache assets onto alternate wheel indexes. The technical theme was Blackwell, FP4, NVFP4, MoE EP packaging, paged KV, TRT-LLM Gen decode, and attention correctness..48
FluidInference shipped FluidAudio v0.15.5, text-processing-rs v0.2.3, and text-processing-rs v0.2.4. FluidAudio rebuilt downloads around ModelHub, resumable HTTP Range support, byte-level progress, validation, and retry policy, while text-processing-rs pushed English normalization toward NeMo parity. FluidAudio release notes.56
Arm shipped armnn v26.07, focused on CpuAcc SME, SVE, and NEON shape policy, Compute Library updates, and Android binary page-size fixes. The release aims to expose faster Arm Compute Library kernels while avoiding SME2 regressions on quantized or awkward GEMM-like shapes..36
Intel shipped neural-compressor v3.9, focused on JAX and Keras quantization coverage. The release adds INT8 ViT quantization in JAX plus Keras and JAX support for out-of-place quantization, BF16-model quantization, Conv2D quantization, and dot-product attention quantization..51
k2-fsa shipped sherpa-onnx v1.13.4, a large release with 289 assets. The release highlights Dart iOS package fixes, Parakeet CTC QNN export and runtime, split-package sdist metadata, JavaScript per-stream hotwords, and platform build updates..19
LocalAI shipped v4.6.0, v4.6.1, and v4.6.2, all focused on production local serving. The week’s release train covered AMD ROCm speed, distributed load fixes, realtime warm-up, model capability metadata, OIDC logging, CI sharding, and backend reliability. Latest release.78
Osaurus shipped eight releases from 0.21.4 through 0.21.11, with native search, Agent Channels, warm-up, RAM ceilings, chat UX, Hugging Face download improvements, benchmarking, localization, and crash fixes moving in rapid sequence. The latest release consolidated timestamps, onboarding curation, faster downloads, connectivity triage, osaurus bench, and Agent Channels readiness. Latest release.79
OpenNMT shipped CTranslate2 v4.8.1, focused on security and runtime hardening. The release adds Gemma4 12B dense model support, safer legacy converter checkpoint loading, a model-load heap overflow fix, and a Whisper alignment divide-by-zero fix..54
Qualcomm shipped ai-hub-apps v0.32.2, ai-hub-models v0.57.1, and GenieX v0.3.14. The dominant theme was GenieX and QAIRT enablement across apps, model metadata, performance data, signed Hexagon HTP catalogs, Qwen assets, and developer-preview packaging. GenieX release notes.18
ROCm shipped AITER v0.1.17-rc0 and ATOM v0.1.6-rc0, paired release candidates for AMD Instinct serving. The releases focus on ROCm wheel builds, OPUS and clang fixes, fused all-reduce plus RMSNorm plus quantization, MiniMax fixes, and ROCm serving compatibility. AITER release notes.11
Ollama shipped v0.31.2 as a prerelease, focused on CI parallelism, CUDA toolkit lookup fixes, cloud retirement docs, JetPack CUDA fallback, and the new agent harness core. The surrounding commits also rewrote MLX model creation and fixed Qwen and thinking-parser behavior..80
jundot shipped oMLX v0.5.0.dev1, a development release centered on Lightning MTP speculative decoding, custom-kernel acceleration, oQe imatrix quantization, model support, admin UX, and memory and build hardening. The release is one of the clearest signs that Apple Silicon local serving is moving toward speculative, MoE-aware production behavior..81
TileLang shipped v0.1.12, with 91 commits since the prior release. The release adds LLVM backend support, scheduler and backend registry work, pass visualization, CUDA intrinsics, FP8 cast optimization, cache and build fixes, and compiler correctness fixes..47
Blaizzy shipped mlx-vlm v0.6.4, adding server TTS and STT endpoints, diffusion prefill-policy support, mxfp4 embedding dequantization fixes, and a continuous-batching cache guard. The release landed amid a much larger MLX audio and VLM model-support push..31
Zetic shipped ZeticMLangeiOS 1.9.0, updating the public Swift Package to a new XCFramework asset. The release adds LFM-VL vision-language support, function calling, RAG, and runtime reliability improvements for on-device AI apps..57