Running a Language Model on CPU: What Fits in 32 GB

You do not need a graphics card to run a language model. You need memory, memory bandwidth and patience, and the three of those decide everything about whether a given model is pleasant or useless on a given machine. This post gives the arithmetic, maps it onto real plans, and is straightforward about where a virtual machine stops being the right tool.

We do not sell GPUs. Everything below is processor inference, which is genuinely good for embeddings, classification, extraction and background batch work, and genuinely slow for interactive chat with a large model.

The only formula you need

A model quantised to roughly four bits needs about half a gigabyte of RAM per billion parameters, plus room for the context window and the operating system. So:

• A 3 billion parameter model is about 2 GB.
• A 7 or 8 billion parameter model is about 4 to 5 GB.
• A 13 billion parameter model is about 8 GB.
• A 32 billion parameter model is about 20 GB.
• A 70 billion parameter model is about 40 GB.

Add one to two gigabytes for the system and the context. Eight bit quantisation roughly doubles those figures, and running at full precision roughly quadruples them.

What fits on which plan

Plan	RAM	vCPU	Comfortably runs
Pulse	4 GB	2	Embedding models, a small classifier, a 1 to 3 billion parameter model
Core	8 GB	2	A 7 or 8 billion parameter model at four bits, with a modest context
Forge	16 GB	4	A 13 billion parameter model, or a 7 billion one with a long context
Apex	24 GB	6	A 32 billion parameter model at four bits, slowly
Titan	32 GB	8	The largest we offer as a virtual machine, a 32 billion model with headroom

Fitting and being fast are different questions. Processor inference is limited by memory bandwidth far more than by core count, and a virtual machine shares the host's bandwidth with its neighbours. Treat a VPS as the right home for embeddings, retrieval, summarising a queue of documents overnight and anything where a request can take a few seconds without a human waiting. Do not expect a large model on a virtual machine to feel like a hosted chat service.

Benchmark it for pennies before you commit

This is the one workload where the specification sheet tells you least, so measure it. Every plan is available by the hour at the monthly price divided by 720 plus the location fee divided by 720. A Titan is €0.1167 an hour in Frankfurt or Vienna, so loading your actual model and running your actual prompts for an hour costs twelve cents.

1. Order the largest plan you are considering, by the hour.
2. Load the exact model file and quantisation you intend to use.
3. Measure tokens per second on your real prompts, not a benchmark suite, and watch memory while a long context is in play.
4. Keep it or delete it. Deleting is what stops the charges, since a powered off machine still holds its disk and its address.

Anything you keep running permanently belongs on a monthly, quarterly or annual cycle, because 720 hourly units cost exactly one month. The per city rates for every plan are in our price by location breakdown.

When the answer is bare metal

Three signals, and any one of them means a virtual machine is the wrong shape.

• The model does not fit in 32 GB. A 70 billion parameter model at four bits needs around 40 GB before context, and our largest virtual machine is a 32 GB Titan.
• Throughput matters. A dedicated machine gives you every memory channel on the platform rather than a share of them, which is the single biggest lever in processor inference.
• You are serving other people. Concurrency multiplies both memory and bandwidth demand, and a noisy neighbour you cannot see is a poor foundation for a product.

Our EPYC range is built on DDR5 and is the sensible target for serious processor inference.

Server	Cores and threads	RAM	Per month
Single EPYC 9254	24 core / 48 threads	128 GB DDR5	€405
Dual EPYC 9254	48 core / 96 threads	256 GB DDR5	€659
Dual EPYC 9554	128 core / 256 threads	512 GB DDR5	€969
Dual EPYC 9754	256 core / 512 threads	1 TB DDR5	€1579

A 1 TB machine holds a 70 billion parameter model at full precision, or several large models resident at once for a routing setup. Details and the Xeon options are on the dedicated servers page, and the deeper comparison is in VPS against dedicated servers.

Practical notes that save a day

• Disk is not the bottleneck, but it fills. Model files are gigabytes each, and a plan disk runs from 20 GB to 120 GB. Extra disk is available in 50 GB units at €4 per unit per month, up to three units.
• Pull models over unmetered traffic. Downloading a few tens of gigabytes of weights is not a billing event here, and it is not free everywhere.
• Pin thread counts. Letting an inference runtime spawn threads for every visible processor usually makes it slower, not faster. Match threads to the vCPU count on the plan.
• Keep the model resident. Loading weights per request dominates the time. Run a long lived service, not a script per call.
• Choose the city for your data, not for the model. Inference is not latency sensitive in the network sense. If the documents live in a database in Warsaw, run the model in Warsaw.
• Watch what you send. A model on your own machine keeps your prompts on your own machine, which is the main reason people do this rather than call an interface. If that is why you are here, see our note on EU data residency.

Where to start

For retrieval, embeddings and background extraction, a Core or a Forge in the city where your data already lives, on a monthly cycle. For interactive use of anything above 13 billion parameters, rent a Titan by the hour, measure it honestly, and if the numbers disappoint, the next step is EPYC rather than a bigger virtual machine. Plans are on the VPS page and the workload summary is on the AI and machine learning page.