In short. A GPU VPS is the standard compute unit for running AI workloads in 2026, whether that means serving an LLM to users or fine-tuning one on your own data. Inference is a burst workload sized by model parameters. Training holds the GPU at full utilization for hours or days and needs several times more VRAM. The right GPU VPS size depends on which of these two tasks you’re running, and at what model scale.
The best GPU for AI and the right GPU VPS size are really the same question asked two ways, and the answer changes completely depending on whether you’re running inference or training. This guide covers both paths: sizing VRAM correctly for each, running LLM inference with Ollama, vLLM, and llama.cpp, fine-tuning a model with LoRA and QLoRA, and why GPUs dominate this workload category over CPUs in the first place.
AI Training vs Inference: Different GPU Requirements
Training and inference stress a GPU in almost opposite ways, and conflating the two is the most common sizing mistake.
| Dimension | Training | Inference |
|---|---|---|
| GPU utilization | Sustained 90-100% for the full run | Bursty, often idle between requests |
| VRAM use | Weights, gradients, and optimizer state together | Weights only, plus a small KV cache |
| Duration | Hours to weeks per run | Milliseconds to seconds per request |
| Precision | FP16 or BF16 mixed precision | INT8 or INT4 quantized, often |
| Cost model | Time-boxed run, provision and release | Always-on, sized for sustained serving |
A GPU for AI training needs headroom for gradients and optimizer state on top of the model weights themselves, which is why a 7B training run needs several times the VRAM of a 7B inference deployment. Inference, by contrast, is dominated by weights and a comparatively small per-request cache, which is why quantization has such an outsized effect on inference VRAM specifically.
How Much VRAM Do You Actually Need?
VRAM is the binding constraint for both training and inference, and the math differs sharply between them. Sizing GPU VRAM correctly up front avoids the single most common mistake in this space: provisioning a card that runs out of memory mid-job. For inference at FP16 precision, the rule of thumb is straightforward: parameter count times 2 bytes gives you the minimum VRAM. A 7B model needs roughly 14 GB just to load the weights. Training multiplies that by roughly 4 to 6 times, since the GPU now has to hold gradients, optimizer state (Adam-style optimizers keep two additional copies per parameter), and activation memory alongside the weights themselves.
Quantization changes the inference side of this equation dramatically. Dropping to INT4 cuts the weight memory by roughly 4x compared to FP16, which is exactly why quantized models unlock much larger parameter counts on the same card.
VRAM for AI workloads scales predictably with model size:
| Model size | Inference VRAM (FP16) | Inference VRAM (INT4) | Fine-tuning VRAM | Full training VRAM |
|---|---|---|---|---|
| 7B | 14 GB | 5 GB | 16 GB | 56 GB |
| 13B | 26 GB | 9 GB | 28 GB | 104 GB |
| 34B | 68 GB | 20 GB | Multi-GPU | Multi-GPU |
| 70B | 140 GB | 40 GB | Multi-GPU | Multi-GPU |
The jump from 13B to 34B is where single-GPU VPS deployments generally stop working for full training, though INT4 inference and LoRA-style fine-tuning still fit on a single card well past that point.
Running LLM Inference on a GPU VPS
Three tools cover most GPU inference deployments, each trading off simplicity against control.
Ollama is the easiest path to local LLM inference. Ollama GPU mode runs GGUF-format models with a single command, falls back to CPU when no GPU is available, and handles quantization automatically. It’s the right starting point for anyone who wants a model running in minutes rather than an hour of configuration.
vLLM is built for production inference serving rather than quick local use. Its PagedAttention memory management technique lets it batch many concurrent requests efficiently, and it exposes an OpenAI-compatible API, so existing client code often works against it with only a base URL change.
llama.cpp offers the most flexible quantization options of the three, running everything from full-precision weights down to aggressive sub-4-bit quantization, and it underlies Ollama’s own inference engine.
| Tool | Model format | Quantization support | GPU utilization | Best for |
|---|---|---|---|---|
| Ollama | GGUF | Automatic, several presets | Full or partial GPU offload | Quick local LLM inference |
| vLLM | HuggingFace / safetensors | FP16, INT8, AWQ, GPTQ | Full GPU, optimized batching | Production API serving |
| llama.cpp | GGUF | Extensive, down to 2-bit | Full, partial, or CPU-only | Maximum quantization flexibility |
| TGI | HuggingFace / safetensors | FP16, INT8, bitsandbytes | Full GPU, continuous batching | HuggingFace-native deployments |
The VRAM tier you provision determines which model sizes are actually usable: 16 GB comfortably runs a quantized 13B model through Ollama or llama.cpp, while a 24 GB card opens up full-precision 13B inference or quantized 34B models without headroom problems.
GPU Training: Fine-Tuning an LLM on a GPU VPS with LoRA and QLoRA
Full fine-tuning of a large model needs multi-GPU infrastructure, but LoRA and QLoRA make GPU deep learning practical on a single GPU VPS for 7B to 13B base models. Choosing the best GPU for AI training at this scale usually comes down to VRAM rather than raw compute, since LoRA and QLoRA are deliberately designed to work within a single card’s memory ceiling. LoRA fine-tunes a small set of additional weights rather than the full model, and QLoRA adds 4-bit quantization on top of that, which is what makes fine-tuning a 7B model fit inside 16 GB of VRAM.
The standard tooling for this kind of machine learning GPU work has converged around a few tools: HuggingFace PEFT paired with bitsandbytes handles the quantization and adapter mechanics directly, Axolotl wraps that into a configuration-driven workflow, and Unsloth optimizes the same process for faster training on a single card. All three build on PyTorch, which remains the standard deep learning framework for this work, and finding the best GPU for deep learning generally means matching VRAM to whichever of these tools and model sizes you plan to run.
A practical fine-tuning checklist before starting a run:
- Dataset size. More than 1,000 examples is a reasonable floor for a LoRA fine-tune to show a real effect.
- Base model choice. Mistral, Llama 3, and Qwen are common current starting points with strong community tooling support.
- QLoRA configuration.
r=16, alpha=32is a reasonable default starting point for the adapter rank and scaling. - Batch size. Start at 4 and increase until you hit an out-of-memory error, then step back down.
- Gradient checkpointing. Trades training speed for VRAM headroom, and is usually worth enabling on a single-GPU setup.
GPU vs CPU for AI: Why GPUs Dominate
CPUs have high single-core speed but only 8 to 128 cores even in server-class hardware. Transformer models run on millions of parallel matrix multiplications per forward pass, which is exactly the workload GPU architecture was built for: thousands of smaller cores doing the same operation simultaneously rather than one core doing it fast. This is why GPU for AI has become the default rather than CPU for anything beyond the smallest models, though the actual speedup varies far more than most guides admit: for a 7B model, some benchmarks show a modern CPU with an optimized inference engine coming within a factor of 1.3 to 2x of GPU throughput, while others show a 10 to 30 times gap, depending heavily on the specific CPU, GPU, quantization level, and inference engine used. The gap widens sharply for larger models, where a CPU’s limited memory bandwidth becomes the binding constraint rather than core count.
An NVIDIA GPU for AI is effectively the default choice today given how much of the ecosystem, from PyTorch to vLLM to Ollama’s GPU backend, targets CUDA first. TPUs and NPUs exist and offer real advantages inside the hyperscalers that built them, but they remain niche outside that context. Nearly all self-hosted AI infrastructure runs on GPUs.
Choosing a GPU VPS for AI: Key Specs to Evaluate
A handful of specs decide whether a VPS with GPU for AI work actually fits the job, and the same checklist is how you find the best GPU for AI at any budget:
- VRAM. 24 GB is a practical sweet spot: enough for 13B inference at FP16 and a 7B fine-tune on the same card.
- GPU generation. Ampere-generation cards or newer are needed for BF16 support, which most current training and inference code assumes by default.
- NVMe storage. Model weights range from a few GB to well over 100 GB; NVMe keeps model loading and checkpoint writes from becoming the bottleneck.
- CPU RAM. Should be at least equal to VRAM, so data preprocessing and model loading don’t bottleneck the GPU.
- Network egress. Pulling models from HuggingFace repeatedly adds up; check egress terms if you’re rotating through many models.
Sizing a GPU server for AI work follows the same shape at every tier: match VRAM to the largest model and task combination you plan to run, then confirm CPU RAM and storage aren’t the actual bottleneck once the GPU is sized correctly.
Object Storage for AI Datasets and Model Checkpoints
Training datasets and model checkpoints add up fast, often into the hundreds of gigabytes across a single fine-tuning project. Staging that data through S3-compatible Object Storage with rclone or s5cmd keeps it off the GPU VPS disk between runs: copy the dataset in at job start, push checkpoints out after each epoch, and the VPS disk never needs to hold more than the current run’s working set.
Streaming the dataset instead of copying it in is also an option for training, but only when it’s done right: shard the data into tar files and read them with a loader built for it, such as WebDataset or MosaicML Streaming, which pull data sequentially in large chunks. What does not work is pointing a training loop at a FUSE mount and doing per-sample random reads of many small files, which is slow at best and unreliable at worst. Pick sequential streaming with a real streaming loader, or stage the data to local disk first. Do not treat a FUSE mount as a local filesystem for random access.
This decouples compute from storage. A GPU VPS can spin down completely between training runs without losing any checkpoints or datasets, since everything durable lives in object storage rather than on the server itself. This is one of the practical advantages of GPU server hosting over a fixed local workstation: storage and compute scale independently.
Why Run AI Workloads on Contabo
Three factors matter most when choosing a GPU VPS provider for sustained AI training and inference workloads, and this is where the approach at Contabo differs most from a typical hyperscaler, starting with the billing model.
Flat-rate billing removes the surprise-invoice risk that comes with per-second hyperscaler billing. A long training run or a continuously served inference endpoint costs the same predictable amount on Contabo regardless of exactly how many hours it runs, which matters far more for AI workloads than for typical web hosting, since a stuck training job on per-second billing can be an expensive surprise.
RAM matters as much as VRAM for AI workloads specifically, since data preprocessing, tokenization, and dataset loading all happen in system memory before anything reaches the GPU. GPU server plans at Contabo pair generous CPU RAM alongside the GPU, avoiding a second, easy-to-miss bottleneck.
EU data center locations matter for GDPR-compliant AI inference, particularly for any deployment processing user-submitted text or documents where data residency is a genuine compliance requirement rather than a nice-to-have. The EU-based infrastructure at Contabo covers this by default without extra configuration.
FAQ: GPU VPS for AI Training and Inference
For Ollama GPU inference, 16 GB of VRAM comfortably runs quantized 13B models, and 24 GB adds headroom for full-precision 13B or quantized 34B models. A GPU VPS sized for your target model, with CPU RAM at least matching VRAM, covers the vast majority of local LLM use cases without needing multi-GPU infrastructure.
Full training needs roughly 4 to 6 times the VRAM of inference at the same model size, since gradients, optimizer state, and activations all fit alongside the weights. A 7B model needs about 56 GB at the low end of that range, which pushes past most single-GPU VPS tiers. LoRA and QLoRA cut that dramatically, fitting a 7B fine-tune into as little as 16 GB.
Yes, for 7B to 13B models using LoRA or QLoRA rather than full fine-tuning. QLoRA’s 4-bit quantization is specifically what makes this practical on a single card: a 7B fine-tune fits inside 16 GB of VRAM, and a 13B fine-tune fits inside roughly 28 GB. Full training of the same models needs multi-GPU infrastructure well beyond what a single GPU VPS provides.
GPU training holds the model, its gradients, and optimizer state in VRAM simultaneously while running at sustained high utilization for hours or days. GPU inference only needs the model weights plus a small per-request cache, running in short bursts rather than continuously. This is why a model that needs 56 GB to train can run inference comfortably on a 14 GB card.
Renting a GPU server on flat billing is typically cheaper than AWS or other hyperscaler GPU instances billed by the second, for workloads that run most of the day. For genuinely intermittent workloads that sit idle most of the time, hyperscaler billing can work out cheaper, since you pay nothing when the GPU is idle. Rent GPU server capacity once daily utilization crosses roughly 12 to 16 hours.
Transformer models run as millions of parallel matrix multiplications per forward pass, and a GPU’s thousands of smaller cores handle that kind of parallel math far faster than a CPU’s much smaller core count. This is the core reason why GPU for AI is the default: the workload shape matches GPU architecture almost exactly, in a way it does not match a CPU’s design for fast sequential execution.
