voxa.host rents AMD EPYC 9004 bare metal for CPU LLM inference, from the Single EPYC 9254 (24 cores, 128 GB DDR5 ECC, €449/mo) to the Dual EPYC 9754 (256 cores, 1 TB DDR5 ECC, €1,519/mo). Every build includes two 1 TB NVMe drives and full root, billed monthly with no setup fee. Ollama, llama.cpp and vLLM run on the CPU alone, with no GPU and no hypervisor in the way. By the usual rule of thumb a 70B model quantised to 4 bits needs roughly 35–42 GB of RAM, so even the entry server holds one with room to spare.
Not every AI workload needs a GPU. CPU inference for small and mid-sized language models, embedding generation, batch classification and NLP preprocessing runs efficiently on high-core, high-memory EPYC servers, often more cost-effectively than renting GPU cloud instances. The trade-off is generation speed on large models: a CPU server holds a 70B model easily but produces text from it more slowly than a GPU does, which suits batch jobs, retrieval pipelines and internal tools better than a busy public chat service.
Memory bandwidth is the main limit on CPU token generation, because producing each token means reading the model's weights from RAM. AMD EPYC 9004 is a 12-channel DDR5 platform, against DDR4 on our Xeon builds, and its Zen 4 cores support AVX-512, which llama.cpp, Ollama and vLLM's CPU backend all use for their matrix maths.
You get full root on bare metal with the operating system of your choice at checkout. EPYC builds are made to order, so plan the delivery estimate shown at checkout into your schedule. No shared tenancy, no noisy neighbours — every FLOP on the system belongs to your workload.
Running Ollama on an AMD EPYC server
Ollama is built on llama.cpp and runs models on the CPU when it finds no GPU, so on EPYC bare metal the setup is the same as on a laptop: install it, pull a quantised model and point your application at its HTTP API on port 11434. Ollama also answers OpenAI-style requests under /v1, so most client libraries only need a new base URL. If you would rather run llama.cpp directly, its llama-server binary offers the same kind of endpoint.
A few settings matter more than the choice of server. Set the thread count to the number of physical cores, not SMT threads (num_thread in Ollama, -t in llama.cpp), since the extra threads rarely speed up inference. Keep the model loaded between requests with OLLAMA_KEEP_ALIVE, because reading weights from disk on every call dominates response time. On the dual-socket builds, bind each instance to one socket with numactl so its memory reads stay local, or run one instance per socket behind a load balancer.
For several users at once, OLLAMA_NUM_PARALLEL sets how many requests one loaded model serves together, and OLLAMA_MAX_LOADED_MODELS how many models stay in memory at the same time. Each parallel request reserves its own context memory, and that is where RAM beyond the model file goes: a 70B model at 4 bits leaves most of a 128 GB server free for long contexts, and the 512 GB and 1 TB builds can keep several large models resident for a routing setup.
Ollama's API has no authentication of its own and listens on localhost by default. Keep it there, and put a reverse proxy with TLS and authentication in front before anything outside the server can reach it. We do not publish tokens-per-second figures: speed on a CPU depends on the model, the quantisation, the context length and the number of parallel requests, so measure your own prompts before you size a fleet. llama.cpp ships llama-bench for exactly that.
How much RAM a language model needs on CPU
On a CPU server the whole model sits in system RAM, so memory size decides what fits. The rule of thumb: a model quantised to 4 bits needs about 0.5–0.6 GB per billion parameters, 8 bits about 1.0–1.1 GB, and 16 bits about 2 GB, plus 1–2 GB for the system and more for long contexts and parallel requests. The table applies that rule to common model sizes. Treat the figures as estimates and check the file size of the exact model you plan to run.
| Model · quantisation | Approx. RAM (estimate) | Smallest EPYC that fits |
|---|---|---|
| 7–8B · 4-bit | ≈ 4–5 GB | Single EPYC 9254 (128 GB) |
| 7–8B · 8-bit | ≈ 7–9 GB | Single EPYC 9254 (128 GB) |
| 13–14B · 4-bit | ≈ 7–8 GB | Single EPYC 9254 (128 GB) |
| 13–14B · 8-bit | ≈ 13–15 GB | Single EPYC 9254 (128 GB) |
| 32–34B · 4-bit | ≈ 16–20 GB | Single EPYC 9254 (128 GB) |
| 32–34B · 8-bit | ≈ 32–37 GB | Single EPYC 9254 (128 GB) |
| 70B · 4-bit | ≈ 35–42 GB | Single EPYC 9254 (128 GB) |
| 70B · 8-bit | ≈ 70–77 GB | Single EPYC 9254 (128 GB) |
| 70B · 16-bit | ≈ 140 GB | Dual EPYC 9254 (256 GB) |
| 400B-class · 4-bit | ≈ 200–240 GB | Dual EPYC 9554 (512 GB) |
| 400B-class · 8-bit | ≈ 400–440 GB | Dual EPYC 9754 (1024 GB) |
Estimates from the per-parameter rule above, not measurements; actual file sizes vary by model and quantisation format. "Fits" means the estimate leaves at least a quarter of the server's RAM free for context, the operating system and parallel requests. The Dual Xeon builds (128–256 GB DDR4, delivered in under an hour) hold the same small and mid-sized models, without AVX-512.
Why Voxa for AI & Machine Learning Inference
Up to 1 TB DDR5 ECC
Model weights stay resident in RAM, and the 512 GB and 1 TB builds keep several large models loaded at once. All of it is ECC memory.
256 cores on EPYC 9754
Parallelise inference, preprocessing, and serving across hundreds of threads.
NVMe scratch storage
Two 1 TB NVMe drives included, up to 24 at checkout, for fast model loading, dataset staging, and checkpoint storage.
Bare metal, full root
No hypervisor and no other tenants. Install conda, Docker, llama.cpp or anything else your stack needs.