{"id":31985,"date":"2026-06-11T12:47:58","date_gmt":"2026-06-11T10:47:58","guid":{"rendered":"https:\/\/contabo.com\/blog\/?p=31985"},"modified":"2026-07-07T13:29:08","modified_gmt":"2026-07-07T11:29:08","slug":"how-to-run-llm-inference-on-a-gpu-vps-2026-guide","status":"publish","type":"post","link":"https:\/\/contabo.com\/blog\/how-to-run-llm-inference-on-a-gpu-vps-2026-guide\/","title":{"rendered":"How to Run LLM Inference on a GPU VPS (2026 Guide)"},"content":{"rendered":"\n
\"How<\/figure>\n\n\n\n

In short.<\/strong> Running LLM inference on a GPU VPS gives you a private model endpoint, full data sovereignty, and flat predictable cost instead of per-token billing. You need a GPU VPS with enough VRAM for your model, Ubuntu 22.04 or later, CUDA, and either Ollama or vLLM as the runtime. From order to first inference call takes about 15 to 30 minutes.<\/p>\n\n\n\n

\n\n\n\n

Running your own inference gives you control that managed APIs don’t: the model runs on your hardware, and prompts and responses never leave your server. Cost works differently too. You pay a flat monthly rate whether you process a thousand tokens or a billion, with no per-token bill that climbs as usage grows. A managed API is fine for low-traffic prototyping, but it adds latency and ties you to a vendor’s uptime and pricing decisions. This guide deploys a working LLM endpoint on a GPU VPS from scratch.<\/p>\n\n\n\n

What You Need to Run LLM Inference<\/h2>\n\n\n\n

Before provisioning the server, confirm you have the following:<\/p>\n\n\n\n

    \n
  1. A GPU VPS with sufficient VRAM for your target model. Look for a plan with at least 48 GB VRAM to run 70B quantized models on a single card; for FP8 or FP16 70B inference, 80 GB or more is the practical requirement. The VRAM sizing table in the next section maps each configuration.<\/li>\n\n\n\n
  2. Ubuntu 22.04 LTS or 24.04 LTS. All commands in this guide target Ubuntu. NVIDIA drivers and CUDA are typically pre-installed on Contabo server images.<\/li>\n\n\n\n
  3. Root SSH access to the server.<\/li>\n\n\n\n
  4. An LLM inference runtime: Ollama for fast single-user setup, or vLLM for production APIs with high concurrency. The comparison table later in this guide walks through the decision.<\/li>\n\n\n\n
  5. Storage for model weights. A 7B model in FP16 takes about 14 GB on disk; a 70B model at Q4 quantization runs to around 40 GB. Contabo server plans include at least 1 TB of storage, which covers most model configurations.<\/li>\n<\/ol>\n\n\n\n

    For a self-hosted AI inference server designed to run an LLM locally without external dependencies, all five requirements need to be confirmed before you begin.<\/p>\n\n\n\n

    Sizing VRAM for Your Model<\/h2>\n\n\n\n

    VRAM is the binding constraint. If the model does not fit in GPU memory, the runtime falls back to system RAM and inference slows substantially. Use this table to choose the right GPU for LLM inference before ordering a plan:<\/p>\n\n\n\n

    Model<\/th>Precision<\/th>Approx. VRAM<\/th>Min. GPU VRAM<\/th><\/tr><\/thead>
    7B<\/td>FP16<\/td>~14 GB<\/td>24 GB<\/td><\/tr>
    7B<\/td>Q4<\/td>~4 GB<\/td>8 GB<\/td><\/tr>
    13B<\/td>FP16<\/td>~26 GB<\/td>32 GB<\/td><\/tr>
    13B<\/td>Q4<\/td>~8 GB<\/td>16 GB<\/td><\/tr>
    70B<\/td>Q4<\/td>~38 GB<\/td>48 GB<\/td><\/tr>
    70B<\/td>FP8<\/td>~70 GB<\/td>80 GB<\/td><\/tr>
    70B<\/td>FP16<\/td>~140 GB<\/td>141 GB<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

    Q4 quantization cuts VRAM by roughly 75% compared to FP16, with a modest accuracy trade-off that works for most LLM hosting and inference tasks. For FP8 or FP16 70B inference, look for a GPU with at least 80 GB VRAM on a single card.<\/p>\n\n\n\n

    For most self host LLM deployments, a 48 GB GPU is the practical starting point. It handles any 70B quantized model and leaves room for multiple concurrent sessions. Step up to a larger card only if your workload requires unquantized accuracy or pushes context window limits.<\/p>\n\n\n\n

    Step 1: Order a GPU Server<\/h2>\n\n\n\n
      \n
    1. Provision a GPU server with enough VRAM for your target model. Use the sizing table in the previous section to confirm your requirement before ordering.<\/li>\n\n\n\n
    2. Select your preferred operating system. All commands in this guide are tested on the latest Ubuntu distribution. Pick the region closest to your users and complete the order.<\/li>\n\n\n\n
    3. SSH in as root:
      ssh root@YOUR_SERVER_IP<\/code><\/li>\n\n\n\n
    4. Confirm the GPU is present and the NVIDIA driver is loaded:
      nvidia-smi<\/code><\/li>\n<\/ol>\n\n\n\n

      A healthy output shows the installed GPU model name in the title row. The header line reports the driver and CUDA versions: for pre-Blackwell GPUs, expect driver 535.x or higher; for Blackwell and newer, 570.x or higher. CUDA 12.x or later is expected on both paths. If nvidia-smi returns “command not found”, the driver is not loaded.<\/p>\n\n\n\n

      Step 2: Install CUDA and an Inference Runtime<\/h2>\n\n\n\n

      The correct driver path depends on your GPU architecture. NVIDIA splits Linux GPU support into two tracks from Blackwell onwards:<\/p>\n\n\n\n