Blogs

29

2026

2025

2024

2023

2022

Talks

9

From Memory Budget to Cost Per Token: Serving Agentic Traffic with Large Open Weight Models

Yuchen Fama (Red Hat), Ashish Kamra (Red Hat)

Open Source in Finance Forum New York 2026 November 4, 2026

GuideLLM v0.7.0+: Benchmarking Reasoning, Tool Calling, and Agentic Traces

Samuel Monson (Red Hat), Jared O'Connell (Red Hat)

vLLM Office Hours #56 September 20, 2026

KV Cache-Aware Routing and P/D Disaggregation on Kubernetes

Yuchen Fama & Ashish Kamra (Red Hat)

AI Engineer World's Fair 2026 July 2, 2026

A Cross-Industry Benchmarking Tutorial for Distributed LLM Inference on Kubernetes

Samuel Monson (Red Hat), Ganesh Kudleppanavar (NVIDIA), Jason Kramberger (Google), Jing Chen (IBM Research)

KubeCon + CloudNativeCon Europe 2026 March 24, 2026

Routing Stateful AI Workloads in Kubernetes

Maroon Ayoub (IBM Research), Michey Mehta (Red Hat)

KubeCon + CloudNativeCon North America 2025 November 11, 2025

Learn How to Run an LLM Inference Performance Benchmark on NVIDIA GPUs

Samuel Monson (Red Hat), Ashish Kamra (Red Hat)

DevConf.US 2025 September 20, 2025

Multi-Node Finetuning LLMs on Kubernetes: A Practitioner's Guide

Ashish Kamra (Red Hat), Boaz Ben Shabat (Red Hat)

KubeCon + CloudNativeCon India 2024 December 11, 2024

Optimizing Load Balancing and Autoscaling for Large Language Model (LLM) Inference on Kubernetes

David Gray (Red Hat)

KubeCon + CloudNativeCon North America 2024 November 13, 2024

Efficiently Deploying and Benchmarking LLMs in Kubernetes

Nikhil Palaskar (Red Hat)

DevConf.US 2024 August 14, 2024

Publications

1

llm-tuna: Hyperparameter Optimization for LLM Inference

An open-source framework that automates vLLM inference hyperparameter optimization using Bayesian search via Optuna, achieving up to 32.9% throughput improvement on mixture-of-experts models.

ACM Web Conference 2026 (WWW '26)
Thameem Abbas Ibrahim Bathusha, Aanya Sharma, Andy Huynh, R.C. Samaratunga, Ashish Kamra

Featured Projects

8
vllm-project

GuideLLM

SLO-aware benchmarking and evaluation platform for LLM deployments that simulates production workloads against OpenAI-compatible and vLLM-native servers.

Maintainer
openshift-psap

Performance-Analysis-Agent

AI-powered analysis of vLLM benchmark data across models, versions, accelerators, and workloads.

Creator
openshift-psap

vllm-profiler

Kubernetes mutating admission webhook that transparently injects PyTorch profiler instrumentation into vLLM serving pods, capturing CPU and CUDA traces with no code changes or rebuilds.

Creator
openshift-psap

fournos

Kubernetes operator that schedules benchmark jobs via Kueue and executes them as Tekton PipelineRuns on remote clusters through the FORGE framework.

Creator
openshift-psap

forge

Framework for Orchestrating Runtime Gen-AI Experiments — a CI-first harness for reproducible, observable performance and scale testing of AI/ML workloads on OpenShift.

Creator
openshift-psap

inference-playbooks

Curated, reproducible deployment recipes that connect a model, serving stack, hardware profile, workload, and auditable benchmark evidence for running LLMs on OpenShift/Kubernetes.

Creator
openshift-psap

serveit-studio

Automated LLM inference optimization for Kubernetes — deploys real vLLM instances, sweeps configurations across aggregated, disaggregated, and expert-parallel architectures, and ranks the optimal setup by TTFT and throughput.

Creator
openshift-psap

self-hosted-llm-cost-calculator

Interactive cost calculator comparing self-hosted open-weight LLM inference against frontier API providers.

Creator