The latest AI news, in focus.

Migrating vLLM to the LibTorch Stable ABI

How vLLM migrated its CUDA build and compiled dependencies to the LibTorch Stable ABI, enabling compatible PyTorch upgrades without rebuilding vLLM.

How NVIDIA Uses AI Agents to Streamline AI Factory Operations

As GPU clusters scale, operations span compute, network, hardware, and scheduling. Site reliability engineers (SREs) often have to examine multiple dashboards...

NVIDIA CUDA 13.5 Brings NVIDIA Vera Rubin GA Support

As artificial intelligence and accelerated computing continue to push the boundaries of what's possible, the latest NVIDIA CUDA release arrives to unlock...

Configure, Certify, and Operate Kubernetes GPU Clusters with NVIDIA DSX OS

One underperforming GPU can slow an entire distributed training job. The node may pass basic health checks and show normal utilization, leaving operators to...

vLLM Delivers Day-0 Support for Kimi K3.1 Pro

vLLM supports Kimi K3.1 Pro on day 0: sparse attention on a KDA hybrid, a packed KV cache with FP8 or NVFP4 latents and MXFP4 index keys, token-sharded prefill, decode context parallelism, and fused low-latency decode.

Build Specialized Agents on a Stack You Control with NVIDIA NemoClaw

Building an autonomous agent requires a model, a harness, and domain-specific tools. To deploy an agent more securely, developers need a runtime that enforces...

vLLM Semantic Router Releases d3-ultra: A 27B Open Multimodal Decision Model

vLLM Semantic Router released d3-ultra, a 27B open-weight multimodal decision model. It reads text, images and video once, then returns probabilities for every answer, scoring 87.2 on the team’s SR-Text test. The post vLLM Semantic Router Releases d3-ultra: A 27B Open Multimodal Decision Model appeared first on MarkTechPost.

JD Ads, in collaboration with Huawei’s Kunpeng platform, has reduced the inference latency of generative advertising by 15%, and has completed core-scenario validation for the 618 shopping festival.

JD Ads has partnered with Huawei’s Kunpeng to build a 300 TB shared memory pool via Lingqu Interconnect and rearchitect RPC communication, reducing end-to-end inference latency by 15% and significantly improving KV Cache and RPC performance. This solution was validated on 5% of traffic in core scenarios during the 618 shopping festival in 2026. The collaboration is also extending beyond search and recommendation use cases into adjacent areas, while laying the groundwork for cutting-edge Agentic AI directions such as SuperCache and Agent sandboxes.

Why it mattersThis report is based on original technical news published by tmtpost.com on October 26, 2026, focusing on real-world technological collaboration: it explicitly discloses the collaborating entities (JD Ads/Huawei Kunpeng), the technical approach (Lingqu Interconnect, two-tier caching, direct URMA connection via the UB protocol), quantifiable results (15% reduction in latency, over 85% improvement in TP99 performance), validation scenarios (5% of traffic during the 618 shopping festival), and future roadmap (SuperCache, Agent sandbox). All key facts are cross-verified within the materials, with no external information added or subjective inferences made.

Tsinghua University and The Chinese University of Hong Kong jointly open-source AKV: an engineering service system for KV caching tailored to LLM agents.

On October 26, 2026, researchers from Tsinghua University and the Chinese University of Hong Kong open-sourced AKV—a KV cache engineering server built on SGLang and specifically designed for large language model (LLM) agents. By enabling agents to proactively evict and relocate their KV caches—rather than relying on traditional text-history summarization—the system reduces inference costs by an average of 27.7% across four agent benchmarks and improves task scores by 1.7 points on average. The code is licensed under the Apache-2.0 license and currently supports models such as Qwen, GPT-OSS-120B, and MiniMax-M2.7.

Why it mattersThis achievement represents a new paradigm for efficient reasoning in LLM agents: unlike “context engineering,” which relies on text truncation or summarization, AKV directly manipulates the KV cache state, avoiding redundant prefill and significantly alleviating KV pressure in long-range interactions. Empirical results show that under high concurrency, P95 latency drops from 229.9 seconds to 30.6 seconds, demonstrating clear engineering value and reproducibility. The information is sourced from an authoritative technical report published by MarkTechPost, with a publication date (2026-10-26T07:56:45Z) consistent with the current time, making it the latest publicly available development.

Underdog Saluki 27B: Technical Analysis of the 2-bit Qwen3.8-27B Tool-Calling Model Based on GSQ-RCO

This article introduces Saluki 27B, a model released by Underdog—a 7.9 GB, 2-bit quantized variant of Qwen3.8-27B—built using the GSQ-RCO method from ISTA-DASLab. It claims to outperform the original 54 GB full-precision model on the BFCL v4 benchmark across 120 tasks; however, all performance conclusions are based solely on the developers’ assertions and have not been independently verified.

AnyLanguageModel 1.0 released: a unified Swift interface supporting both local and hosted models, compatible with the entire Apple platform family of operating systems 27

Hugging Face has officially released AnyLanguageModel 1.0, an open-source Swift framework fully compatible with the Apple Foundation Models API. It supports seamless switching between local models (MLX, llama.cpp, Core ML) and hosted models (OpenAI, Anthropic, Gemini, Hugging Face Inference Providers, etc.), and introduces new features such as image input, tool invocation, guided generation, and automatic adaptation of inference chunks.

Google DeepMind partners with the Singaporean government to develop a lightweight edge AI model for real-time interception of private chat scams.

Google DeepMind and GovTech Singapore have jointly launched a lightweight edge AI model based on the Gemma series, optimized for local execution on users’ devices. It can detect and issue real-time alerts for conversational scams without uploading any chat data to the cloud.

Run a vLLM Server on HF Jobs in One Command

Run a vLLM Server on HF Jobs in One Command

IBM Granite 3.0 models

Ollama partners with IBM to bring Granite 3.0 models to Ollama.

Llama 3.2 goes small and multimodal

Ollama partners with Meta to bring Llama 3.2 to Ollama.

Reduce hallucinations with Bespoke-Minicheck

Bespoke-Minicheck is a new grounded factuality checking model developed by Bespoke Labs that is now available in Ollama. It can fact-check responses generated by other models to detect and reduce hallucinations.

Tool support

Ollama now supports tool calling with popular models such as Llama 3.1. This enables a model to answer a given prompt using tool(s) it knows about, making it possible for models to perform more complex tasks or interact with the outside world.

An entirely open-source AI code assistant inside your editor

Continue enables you to easily create your own coding assistant directly inside Visual Studio Code and JetBrains with open-source LLMs.

Llama 3 is not very censored

Compared to Llama 2, Llama 3 feels much less censored. Meta has substantially lowered false refusal rates. Llama 3 will refuse less than 1/3 of the prompts previously refused by Llama 2.