Email Hosting Explained: Self-Hosted, Hosted or Hybrid

Professional email on your own domain can be run three ways: a hosted mailbox provider, a self-managed mail server, or a hybrid where you send transactional mail through a relay and keep mailboxes elsewhere. Self-hosting gives you complete control and costs you deliverability work forever. This guide covers what each route actually involves so you choose deliberately rather than by default.

The honest recommendation first

For most businesses, do not self-host mailboxes. Deliverability is the hard part of email, and it is a reputation problem rather than a technical one: a new IP address has no sending history, the large providers treat it with suspicion, and building trust takes weeks of consistent, low-volume, well-authenticated sending. A hosted mailbox provider inherits reputation you did not have to build.

Self-hosting is the right answer when you have a specific reason: data residency requirements that a provider cannot meet, volumes where per-mailbox pricing becomes significant, or a genuine need to control the entire path. Those are real reasons. "It is cheaper" usually is not, once the maintenance is counted.

The three routes compared

	Hosted mailboxes	Self-hosted mail server	Hybrid (relay for sending)
Setup effort	DNS records, minutes	Days, plus ongoing	Hours
Deliverability	Inherited	Yours to build and defend	Inherited for outbound
Ongoing maintenance	None	Patching, spam tuning, blocklist monitoring	Minimal
Data control	Provider's terms	Complete	Split
Cost shape	Per mailbox per month	One VPS plus your time	Per message sent
Best for	Almost everyone	Residency or control requirements	Applications sending transactional mail

The DNS records that decide whether mail arrives

These matter in every scenario, including hosted. Getting them wrong is the most common cause of mail landing in spam.

; where mail for the domain goes
example.com.        MX  10 mail.example.com.

; who is allowed to send as this domain
example.com.        TXT "v=spf1 mx a:mail.example.com -all"

; the public half of your signing key
sel1._domainkey.example.com. TXT "v=DKIM1; k=rsa; p=MIIBIjANBg..."

; what receivers should do when SPF or DKIM fail, and where to report
_dmarc.example.com. TXT "v=DMARC1; p=quarantine; rua=mailto:[email protected]; pct=100"

• SPF lists the servers permitted to send for your domain. End it with -all (hard fail), not ~all, once you are confident the list is complete. SPF has a ten-DNS-lookup limit — exceed it and the record silently fails.
• DKIM signs each message so receivers can verify it was not altered. Use a 2048-bit key.
• DMARC ties the two together and tells receivers what to do on failure. Start at p=none with reporting, read the reports for a fortnight, then move to quarantine and eventually reject. Publishing p=reject on day one will block your own legitimate mail from systems you forgot about.
• Reverse DNS (PTR) on the sending IP must resolve to the hostname the server announces in HELO, and that hostname must resolve back to the IP. Mismatched rDNS is an instant spam signal.

If you do self-host

A modern setup is Postfix for SMTP, Dovecot for IMAP, Rspamd for filtering, and TLS certificates from Let's Encrypt. Rather than assembling those by hand, the maintained all-in-one distributions — Mailcow, Mail-in-a-Box, iRedMail — package them with sane defaults and are a far better starting point than a hand-rolled configuration you will have to remember in two years.

Practical requirements:

• A clean IP with no blocklist history. Check it against the major lists before you commit to it.
• Working reverse DNS. Non-negotiable, and it must be set by the provider.
• Outbound port 25. Many networks block it by default. Confirm before you build.
• 4 GB RAM minimum. Rspamd and ClamAV are memory-hungry; our Pulse plan at €6/month is the realistic floor and the €10 Core is more comfortable.
• Backups that include the mail store. Mail is the data people miss most — see automating backups.

Then warm the IP: start at a low daily volume and increase gradually over two to four weeks. Sending ten thousand messages on day one from a new address is the fastest route to a blocklist.

The hybrid pattern, which is what most applications should do

Keep mailboxes with a hosted provider, and send application mail — password resets, receipts, notifications — through an authenticated SMTP relay. Your VPS then never needs port 25 open outbound, never builds sending reputation, and never appears on a blocklist because of a compromised contact form.

This is the correct default for anything running on a web server. It also keeps a compromise contained: an attacker who gets into the application cannot use your mail server to send spam, because there is not one.

Security essentials either way

• Never run an open relay. Require authentication for submission on port 587, and test it from outside.
• TLS everywhere — STARTTLS on 587, implicit TLS on 465 and 993.
• Rate-limit outbound. A compromised account should hit a ceiling long before it damages your reputation.
• Monitor blocklists and read your DMARC reports. Both tell you about problems before your customers do.

General hardening applies here as everywhere: see server security best practices and the firewall guide.

Frequently asked questions

Can I run a mail server on a small VPS?

Technically yes from 2 GB, comfortably from 4 GB once spam filtering and antivirus are running. The constraint is rarely CPU.

Why does my self-hosted mail go to spam?

Almost always one of four things: missing or incorrect SPF, DKIM or DMARC; reverse DNS that does not match; an IP with prior history; or sending volume that ramped too quickly. Check them in that order.

Do I need a dedicated IP for email?

Yes if you self-host. Reputation attaches to the IP, and sharing one with unknown senders means inheriting their behaviour.

What if port 25 is blocked?

Then you cannot deliver directly and must relay through an authenticated provider. Confirm outbound 25 is available before designing around it.

Choosing

If you need mailboxes, use a hosted provider and get the DNS records right. If you need application mail, use a relay. If you have a residency or control requirement that neither satisfies, self-host on a KVM VPS with full root access in the jurisdiction you need — and budget the deliverability work honestly, because it is the part that never finishes.