How to Run gpt-oss-120b on a Single GPU VPS (96 GB VRAM)

gpt-oss-120b is OpenAI’s open-weight reasoning model, and it runs on a single GPU VPS with 96 GB of VRAM. Its mixture-of-experts weights ship natively in 4-bit MXFP4, so OpenAI says the model runs within 80 GB of memory, which leaves room for the KV cache. This guide covers the memory budget, Ollama and vLLM setup, and realistic expectations.

What Is gpt-oss? Models, Licence and Architecture

gpt-oss is a pair of open-weight mixture-of-experts reasoning models from OpenAI, released on 5 August 2025 as gpt-oss-120b and gpt-oss-20b. “Open-weight” means you can download the trained weights and run them yourself. It is not the same as open source in the full sense: the weights are published, but that says nothing about training data or training code.

Both models are released under the Apache 2.0 licence. Hugging Face’s launch post adds that it comes with a small complementary use policy, so read both texts before shipping a commercial product. Internally, gpt-oss-120b has 36 layers and 128 experts per layer, with 4 active per token (gpt-oss-20b: 24 layers, 32 experts, 4 active). Attention alternates between full-context layers and layers with a 128-token sliding window, and the models support up to 128K tokens of context.

gpt-oss VRAM Requirements: Does 120b Fit in 96 GB?

Yes. The Ollama library lists the gpt-oss:120b weights at 65GB, so a 96 GB card has about 31 GB left for the KV cache, activations and the CUDA context. Weights are a fixed cost; the KV cache grows with every token of context and every concurrent request.

The KV cache formula is: bytes per token = 2 (keys and values) × layers × KV heads × head dimension × bytes per element. For gpt-oss-120b that is 2 × 36 layers × 8 KV heads × 64 head dimension × 2 bytes (BF16) = 73,728 bytes, about 72 KiB per token. Head and dimension values come from NVIDIA’s gpt-oss model documentation. This is the worst case, with every layer caching the full context. Half the layers use a 128-token sliding window, so an engine that exploits that needs less, depending on engine and version.

Read it as a ceiling: the 26 GB pool holds roughly two full 128K sequences, about ten at 32K, or about forty at 8K. To adapt it to another card, swap the VRAM and weight rows and rerun the arithmetic. For the general method, see our VRAM sizing guide for LLMs.

Prepare the GPU VPS: Drivers and Docker

Get the host ready first. A short checklist for a fresh GPU VPS:

• Confirm the driver sees the card. nvidia-smi should list the GPU and roughly 96 GB of memory.
• Install Docker, following our guide to install Docker on Ubuntu, if you plan to use vLLM or Open WebUI.
• Add GPU access to containers with the NVIDIA Container Toolkit guide rather than repeating driver steps here.
• Do not expose an inference port to the internet. Bind it to localhost or restrict it, and learn how to configure UFW before opening anything.

nvidia-smi
docker run –rm –gpus all ubuntu nvidia-smi

Pick an Inference Engine: Ollama, vLLM or llama.cpp

Hugging Face’s launch post lists transformers, vLLM, llama.cpp and Ollama among the engines that run gpt-oss. The choice follows your workload rather than the model.

Start with Ollama for a working endpoint in minutes; move to vLLM when concurrent requests matter more than setup time.

Run gpt-oss-120b with Ollama on a GPU VPS

Install Ollama, then pull gpt-oss:120b. Avoid the gpt-oss:120b-cloud tag: it runs on Ollama’s service, not on your server.

curl -fsSL https://ollama.com/install.sh | sh
ollama pull gpt-oss:120b
ollama run gpt-oss:120b

Confirm the model is on the GPU rather than split with the CPU: ollama ps shows the loaded model and its processor split, and nvidia-smi should show roughly 65 GB or more of VRAM in use.

ollama ps
nvidia-smi

The API listens on port 11434, with OpenAI-compatible endpoints under /v1. Ollama applies its own default context length, which can be far below the model’s 128K, so check it and raise it deliberately using the table above to budget the KV cache.

curl http://localhost:11434/v1/chat/completions \
  -H “Content-Type: application/json” \
  -d ‘{“model”:”gpt-oss:120b”,”reasoning_effort”:”medium”,
       “messages”:[{“role”:”user”,”content”:”Explain MXFP4 in two sentences.”}]}’

Serve gpt-oss-120b with vLLM and an OpenAI-Compatible API

This section covers only what is specific to gpt-oss; for the general container setup, use our vLLM Docker guide. vLLM’s recipe asks for vLLM 0.10.1 or newer and uses the vllm/vllm-openai image.

docker run –gpus all –ipc=host -p 127.0.0.1:8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  –model openai/gpt-oss-120b \
  –max-model-len 32768 \
  –gpu-memory-utilization 0.90 \
  –tool-call-parser openai \
  –enable-auto-tool-choice

• –max-model-len defaults to the model’s maximum; setting 32,768 caps the KV cache per sequence. Pick it from the budget table.
• –gpu-memory-utilization sets the fraction of VRAM vLLM may claim; the recipe’s examples use 0.85 to 0.95.
• –tool-call-parser openai with –enable-auto-tool-choice turns on function calling; vLLM supports only tool_choice set to auto.
• The 0.90 and 32,768 values are starting points from the budget above, not vendor recommendations. Pin a tested image tag once it works.

vLLM’s recipe pages name H100, H200 and B200 among supported NVIDIA GPUs and do not name the RTX PRO 6000, so read the release notes for the tag you pull. Once it starts, the endpoint answers at http://localhost:8000/v1.

gpt-oss Reasoning Levels, Tool Use and Context Length

• Reasoning effort: low, medium and high trade latency against answer quality, and medium is the default in Ollama. The model card says the level can be set in the system prompt, for example “Reasoning: high”. Ollama also accepts reasoning_effort on /v1/chat/completions.
• Reasoning output: the model produces a chain of thought before the answer. Ollama returns it in a separate message.thinking field; vLLM can skip it with include_reasoning set to false. Keep it out of end-user views by default.
• Function calling and structured outputs: both are native capabilities. Ollama accepts tools on /v1/chat/completions but not tool_choice, and offers JSON mode; vLLM needs the two flags above.
• Context length: 128K tokens natively, which costs about 9.7 GB of KV cache per full-length sequence in the worst case above.
• Prompt format: the model only works correctly with the harmony format, which engines apply through chat templates, so use the chat endpoints.

What Performance Should You Expect from gpt-oss-120b on One GPU?

We are not quoting tokens per second: the published figures we found do not name this GPU, and a number from another card would mislead you. These factors drive the result.

• Active parameters: only 5.1B of 117B parameters run per token, which helps decoding on one card.
• Batch size: vLLM batches concurrent requests, which raises total throughput while each request slows down.
• Context length: long prompts cost prefill time, and a longer cache means more memory read per generated token.
• Reasoning effort: high produces more reasoning tokens before the answer, so latency rises at the same token speed.
• Engine and version: kernels differ between Ollama, vLLM and llama.cpp, and gpt-oss support keeps moving.

Load-test with your real prompts and watch time to first token, per-request speed and total throughput together. A GPU server for AI work that feels fast in one chat can look very different under ten.

Add a Chat Interface for gpt-oss with Open WebUI

Open WebUI talks to Ollama or any OpenAI-compatible endpoint, so it works with either setup above. Our Open WebUI how-to covers the container; these are the gpt-oss specifics.

• For Ollama, use the base URL on port 11434. For vLLM, add an OpenAI-compatible connection with a base URL ending in /v1 and a placeholder key.
• From inside a container, localhost is the container itself, so use the host’s address.
• Reasoning arrives separately from the answer. For a fixed effort level, put “Reasoning: high” in the system prompt of a model preset.

Can You Run gpt-oss-20b on a Smaller Server?

Yes. gpt-oss-20b has 21B total and 3.6B active parameters, OpenAI says it runs within 16 GB of memory, and the Ollama download is 14GB. A GPU with 16 GB of VRAM or more is the comfortable route.

Without a GPU it is possible but slower. Ollama says the 20b model runs on systems with as little as 16GB of memory, but the weights alone take 14GB, so a CPU-only VPS needs well more for the operating system and context. Speed depends on memory bandwidth and core count; we quote no numbers. A CPU server with plenty of RAM fits one user, a prototype or a low-traffic agent. Move to a GPU for concurrent users, long contexts or gpt-oss-120b, whose 65GB weights are impractical on a CPU VPS.

Why Run gpt-oss on a Contabo GPU VPS

The memory budget above needs one 96 GB card, and that is what Contabo’s GPU VPS provides: a dedicated NVIDIA RTX PRO 6000 Blackwell Server Edition with 96 GB of GDDR7 VRAM. Per Contabo’s launch post, the plan also includes 18 vCPU, 96 GB of RAM and 900 GB of NVMe storage, at €999 per month excluding VAT.

An inference endpoint or agent that runs around the clock suits a fixed monthly price better than hourly GPU billing, because the bill does not move with usage. To weigh the options, compare GPU VPS pricing across providers. For gpt-oss-20b or a small experiment, a Cloud VPS with plenty of RAM is the cost-effective start, which is where the RAM-per-Euro advantage of the Core line pays off. For background, see our guide to LLM inference on a GPU VPS.

FAQ: Running gpt-oss on a GPU VPS

Disclaimer: Product specifications, features, and prices mentioned in this article are subject to change and may vary by region, billing term, and active promotions. Please check each provider’s or brand’s official website for current figures, pricing, and local currency rates.