How to Self-Host an LLM Gateway for AI Agents on a VPS

In short. A self-hosted LLM gateway puts one authenticated endpoint in front of your MCP tool servers and your LLM providers, so API keys never touch agent code and all traffic stays on your VPS. This guide sets one up in two parts: an MCP gateway self-hosted with docker-mcp-gateway for tool access, and LiteLLM as the LLM router with automatic fallback. Both run as lightweight Docker containers on one Contabo Cloud VPS.

What Is a Self-Hosted AI Agent Gateway?

A self-hosted AI agent gateway is a service you run yourself that sits between your AI agents and everything they need to call: MCP tool servers and LLM providers. Instead of each agent holding its own API keys and server addresses, every agent talks to one endpoint, and the gateway handles authentication, routing, and logging.

In practice, this pairs two pieces: an MCP gateway self-hosted for tool access, and an LLM gateway for model routing. Two problems explain why that pairing helps. Hardcoded API keys scattered across agent configs are hard to rotate and easy to leak, and every agent that talks directly to a provider or an MCP server needs its own connection logic, so behavior drifts between agents as your setup grows. Combining both gateways fixes this: keys live in one place, and every agent gets the same connection, retry, and fallback behavior, regardless of which model or tool it ends up calling.

The two tools in this guide handle one job each. docker-mcp-gateway authenticates and routes MCP tool calls. LiteLLM authenticates and routes LLM calls, with automatic fallback if a provider errors or hits a rate limit. Running both on the same VPS gives an agent a single pair of endpoints instead of a growing list of direct provider and tool connections.

Prerequisites

Setting up a self-hosted LLM gateway needs the following, all working before you start:

• A Contabo Cloud VPS with Docker installed. Cloud VPS 4 covers the gateway containers alone. See the VPS recommendation further down if you plan to add local inference with Ollama.
• Docker and Docker Compose installed on the VPS.
• At least one LLM provider API key (OpenAI, Anthropic, or Google), or a local Ollama endpoint if you are routing to a self-hosted model instead of an external provider.
• SSH access to the VPS, ideally with a non-root user in the docker group.

Part 1: Set Up a Unified MCP Tool Gateway with docker-mcp-gateway

docker-mcp-gateway is a Docker image that gives your agents authenticated access to multiple MCP tool servers from a single endpoint, an MCP gateway self-hosted entirely on your VPS. It bundles MCPHub with a Caddy reverse proxy that enforces Bearer token authentication on every request except its health check, and it ships with eight MCP servers built in: filesystem, fetch, GitHub, Brave Search, Git, PostgreSQL, memory, and sequential-thinking. Pair it with an LLM gateway (Part 2) and agents get one authenticated door to both tools and models.

Start the container:

docker run \
  --name mcp \
  --restart=always \
  -v mcp-data:/var/lib/mcp \
  -p 3000:3000/tcp \
  -d hwdsl2/mcp-gateway

On first start, the gateway generates a Bearer API key and writes it to the container logs. Retrieve it with:

docker logs mcp
# or, for scripting:
MCP_KEY=$(docker exec mcp mcp_manage --getkey)

Test the endpoint. The fetch server is enabled by default, so it responds without extra configuration:

curl http://localhost:3000/mcp \
  -H "Authorization: Bearer $MCP_KEY"

A successful response confirms the MCP gateway self-hosted setup is live and authenticating requests correctly, before you connect any agent to it.

Enable Additional MCP Servers

GitHub, Brave Search, and PostgreSQL need their own API keys or connection strings, set as environment variables in the container’s env file before those servers activate. Add the relevant keys, restart the container, and the newly configured servers appear alongside the defaults at the same /mcp endpoint. No separate port, container, or gateway is needed per tool; every enabled server shares the one authenticated endpoint.

Secure the Gateway for Internet-Facing Deployments

For a VPS reachable from the public internet, bind the container to localhost instead of exposing the port directly, and put a reverse proxy in front of it for HTTPS:

docker run \
  --name mcp \
  --restart=always \
  -v mcp-data:/var/lib/mcp \
  -p 127.0.0.1:3000:3000/tcp \
  -d hwdsl2/mcp-gateway

The Bearer token still authenticates every request, but this flag change stops the raw port from being reachable directly from the internet. The reverse proxy in front of it handles the HTTPS termination instead.

Part 2: Set Up an LLM Model Router with LiteLLM

LiteLLM is a self-hosted LLM gateway that exposes one OpenAI-compatible endpoint and routes requests to more than 100 providers, including OpenAI, Anthropic, Gemini, Groq, and Ollama. Point your existing OpenAI SDK at LiteLLM’s base URL and nothing else in your agent code changes; LiteLLM handles the provider-specific translation behind the scenes.

Start the container:

docker run \
  --name litellm \
  --restart=always \
  -v litellm-data:/etc/litellm \
  -p 4000:4000/tcp \
  -d hwdsl2/litellm-server

On first start, the proxy generates a master API key and writes it to the container logs:

docker logs litellm
# to display it again later:
docker exec litellm litellm_manage --showkey

Add a model with a provider API key using the built-in helper script. The proxy applies the change and restarts automatically:

docker exec litellm litellm_manage --addmodel anthropic/claude-3-6-sonnet-latest --key sk-ant-...
docker exec litellm litellm_manage --addmodel openai/gpt-4o --key sk-...

As an LLM router, LiteLLM’s real value shows up in its fallback routing: define a primary model and a fallback in config.yaml, and LiteLLM switches automatically on a provider error or rate limit, without the agent noticing.

model_list:
  - model_name: primary
    litellm_params:
      model: anthropic/claude-5-sonnet-latest
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: fallback
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

router_settings:
  fallbacks:
    - primary:
        - fallback

This is the core of what makes LiteLLM function as a self-hosted LLM gateway rather than a single-provider proxy: one config file, several providers, automatic failover between them.

Connect the Two: MCP Tools and LLM Routing on One VPS

docker-mcp-gateway and LiteLLM are built to work together, and hwdsl2’s LiteLLM image wires the connection through two environment variables rather than a manual config edit. Set them in litellm.env:

LITELLM_MCP_URL=http://mcp:3000/mcp
LITELLM_MCP_API_KEY=mcp-xxxx...

Get the MCP gateway’s key with docker exec mcp mcp_manage --showkey, then restart LiteLLM:

docker restart litellm

On restart, LiteLLM injects an mcp_servers block into its own config.yaml automatically, so no manual YAML editing is needed. Run both containers on the same Docker network so they can reach each other by container name instead of a public IP:

docker network create ai-gateway

Add --network ai-gateway to both docker run commands above, then restart each container. Agents send model calls to LiteLLM on port 4000, and LiteLLM forwards tool calls to the MCP gateway self-hosted on port 3000, using the LITELLM_MCP_URL wiring above. Both containers use restart=always, so a VPS reboot brings the whole self-hosted LLM gateway back up on its own, with no manual recovery step.

Why Contabo for a Self-Hosted AI Gateway?

Both containers are lightweight on their own. docker-mcp-gateway and LiteLLM together use a small fraction of the RAM on a Cloud VPS 4 (4 vCPU cores, 8 GB RAM, 75 GB NVMe), which covers a gateway-only setup comfortably.

Running local inference on the same box changes the math. Add Ollama for local models and RAM becomes the limiting factor, not CPU. A Cloud VPS 8 (8 vCPU cores, 24 GB RAM, 200 GB NVMe) gives enough headroom for a mid-size local model alongside both gateway containers, without the model and the gateways competing for memory. The existing Ollama setup guide on the Contabo blog recommends this same pair of plans for the same reason.

Either way, this is a self-hosted LLM gateway, and an AI gateway self-hosted the same way covers the MCP tools too: no per-request markup from a third-party SaaS layer, and no data leaving the VPS unless a request is routed to an external provider you configured yourself.

FAQ: Self-Hosted AI Agent Gateway