How to Run LLM Inference on a GPU VPS (2026 Guide)

In short. Running LLM inference on a GPU VPS gives you a private model endpoint, full data sovereignty, and flat predictable cost instead of per-token billing. You need a GPU VPS with enough VRAM for your model, Ubuntu 22.04 or later, CUDA, and either Ollama or vLLM as the runtime. From order to first inference call takes about 15 to 30 minutes.

Running your own inference gives you control that managed APIs don’t: the model runs on your hardware, and prompts and responses never leave your server. Cost works differently too. You pay a flat monthly rate whether you process a thousand tokens or a billion, with no per-token bill that climbs as usage grows. A managed API is fine for low-traffic prototyping, but it adds latency and ties you to a vendor’s uptime and pricing decisions. This guide deploys a working LLM endpoint on a GPU VPS from scratch.

What You Need to Run LLM Inference

Before provisioning the server, confirm you have the following:

1. A GPU VPS with sufficient VRAM for your target model. Look for a plan with at least 48 GB VRAM to run 70B quantized models on a single card; for FP8 or FP16 70B inference, 80 GB or more is the practical requirement. The VRAM sizing table in the next section maps each configuration.
2. Ubuntu 22.04 LTS or 24.04 LTS. All commands in this guide target Ubuntu. NVIDIA drivers and CUDA are typically pre-installed on Contabo server images.
3. Root SSH access to the server.
4. An LLM inference runtime: Ollama for fast single-user setup, or vLLM for production APIs with high concurrency. The comparison table later in this guide walks through the decision.
5. Storage for model weights. A 7B model in FP16 takes about 14 GB on disk; a 70B model at Q4 quantization runs to around 40 GB. Contabo server plans include at least 1 TB of storage, which covers most model configurations.

For a self-hosted AI inference server designed to run an LLM locally without external dependencies, all five requirements need to be confirmed before you begin.

Sizing VRAM for Your Model

VRAM is the binding constraint. If the model does not fit in GPU memory, the runtime falls back to system RAM and inference slows substantially. Use this table to choose the right GPU for LLM inference before ordering a plan:

Q4 quantization cuts VRAM by roughly 75% compared to FP16, with a modest accuracy trade-off that works for most LLM hosting and inference tasks. For FP8 or FP16 70B inference, look for a GPU with at least 80 GB VRAM on a single card.

For most self host LLM deployments, a 48 GB GPU is the practical starting point. It handles any 70B quantized model and leaves room for multiple concurrent sessions. Step up to a larger card only if your workload requires unquantized accuracy or pushes context window limits.

Step 1: Order a GPU Server

1. Provision a GPU server with enough VRAM for your target model. Use the sizing table in the previous section to confirm your requirement before ordering.
2. Select your preferred operating system. All commands in this guide are tested on the latest Ubuntu distribution. Pick the region closest to your users and complete the order.
3. SSH in as root:
   ssh root@YOUR_SERVER_IP
4. Confirm the GPU is present and the NVIDIA driver is loaded:
   nvidia-smi

A healthy output shows the installed GPU model name in the title row. The header line reports the driver and CUDA versions: for pre-Blackwell GPUs, expect driver 535.x or higher; for Blackwell and newer, 570.x or higher. CUDA 12.x or later is expected on both paths. If nvidia-smi returns “command not found”, the driver is not loaded.

Step 2: Install CUDA and an Inference Runtime

The correct driver path depends on your GPU architecture. NVIDIA splits Linux GPU support into two tracks from Blackwell onwards:

• Blackwell and newer: require the open GPU kernel module driver, available from version 570. An older proprietary driver will not work on Blackwell hardware.
• Pre-Blackwell (Hopper, Ada Lovelace, Ampere, and earlier): use the traditional proprietary kernel module driver, available from version 535 onwards.

Confirm the driver and CUDA are both available:

nvidia-smi
nvcc --version

For pre-Blackwell GPUs, a healthy nvidia-smi reports a driver of 535.x or higher. For Blackwell and newer, expect 570.x or higher. If the wrong driver is installed for your architecture, or if nvcc returns “command not found”, install the correct version:

# Blackwell and newer: open GPU kernel module path (driver 570+)
sudo apt update && sudo apt install -y nvidia-open

# Pre-Blackwell: traditional proprietary path (if reinstallation is needed)
sudo apt update && sudo apt install -y nvidia-driver-535

# CUDA toolkit (required by both paths if not already present)
sudo apt update && sudo apt install -y cuda-toolkit-12-4

With CUDA confirmed, choose your runtime.

Ollama

Ollama installs in one command and registers as a system service immediately:

curl -fsSL https://ollama.com/install.sh | sh
ollama --version

For the automation angle, including wiring Ollama into n8n workflow tools, see What Is Ollama and How to Use It with n8n.

vLLM

vLLM requires Python 3.9 or later and CUDA 11.8 or later. Install via pip:

pip install vllm
python -c "import vllm; print(vllm.__version__)"

vLLM also requires at least 8 GB of system RAM in addition to GPU VRAM. If you are working in a shared Python environment, set up a virtual environment first:

python -m venv .venv && source .venv/bin/activate
pip install vllm

Step 3: Serve the Model and Test It

Ollama

Pull a model by name and test the local endpoint:

# Download the model weights
ollama pull llama3.2:8b

# Ollama starts as a service on port 11434 automatically
# Send a synchronous test request
curl http://localhost:11434/api/generate \
-d '{"model":"llama3.2:8b","prompt":"What is 2+2?","stream":false}'

A successful response is a JSON object with a response field containing the model’s answer. That is your LLM inference endpoint running locally. To expose the server to other machines on your network, set OLLAMA_HOST=0.0.0.0 before starting the service.

vLLM

Start the OpenAI-compatible API server and send a test completion:

# Start the server
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-3.2-8B-Instruct \
--host 0.0.0.0 --port 8000

# Test
curl http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{"model":"meta-llama/Llama-3.2-8B-Instruct","prompt":"What is 2+2?","max_tokens":50}'

Llama 3 models on Hugging Face require accepting Meta’s license agreement before the download proceeds. Run huggingface-cli login and approve the terms at huggingface.co/meta-llama first. Any OpenAI-compatible client can then point to http://YOUR_SERVER_IP:8000/v1 with no other code changes.

Ollama vs vLLM: Which Runtime?

The decision comes down to one variable: how many concurrent users will be hitting the endpoint?

Pick Ollama if you are building an internal tool or iterating solo, or any time you want an agent talking to a local model with minimal setup. You go from install to first inference call in about two minutes, with no configuration to manage. The OpenAI-compatible API also means your client code does not need to know the difference.

Pick vLLM when you are running a shared API endpoint across a team or serving a production application with concurrent traffic. Its PagedAttention algorithm handles the memory management that makes high-concurrency inference possible, at the cost of a more involved initial setup. Since both runtimes expose the same OpenAI-compatible API surface, migrating from Ollama to vLLM later is a single base-URL change in your client configuration.

FAQ: LLM Inference on a GPU VPS