Self-Hosted Mail Server on a VPS: Postfix + Dovecot Guide

Running your own mail server sounds like a solved problem until you actually try it - one wrong DNS record and your mail lands in spam, or worse, bounces outright. This guide walks through a working Postfix + Dovecot setup on a plain Ubuntu VPS, plus the exact DNS records you need so mail actually gets delivered instead of quietly disappearing.

Why run your own mail server at all

Most people are better off with Google Workspace or cPanel's built-in Exim setup - it's less to maintain and someone else handles reputation and spam filtering. Self-hosting makes sense when you want full control over a domain's mail flow, you're running a lot of mailboxes and the per-seat cost adds up, or you're building infrastructure where email is just one more service on a VPS you already manage. Go in knowing this: deliverability is a full-time concern, not a one-time setup step.

Before you start

Check these three things first - skipping them is the single biggest reason self-hosted mail servers end up in everyone's spam folder:

• Port 25 is open. Many VPS providers block outbound port 25 by default to fight spam. Confirm with your provider or test with telnet smtp.gmail.com 25 from the VPS. If it hangs, you'll need to request it unblocked.
• You control DNS for the domain. You'll be adding MX, SPF, DKIM, DMARC, and PTR records.
• The IP has no bad history. Check it against Spamhaus and MXToolbox blacklists before you commit to it as your mail IP. A recycled IP with a spam history will fight you from day one.

Step 1: Set the hostname and install the packages

Your server's hostname should match the reverse DNS (PTR) record you'll set up later - something like mail.yourdomain.com, not the bare domain.

sudo hostnamectl set-hostname mail.yourdomain.com
sudo apt update
sudo apt install postfix dovecot-imapd dovecot-lmtpd opendkim opendkim-tools certbot

During the Postfix install prompt, choose Internet Site and set the system mail name to your domain (not the mail. subdomain).

Step 2: Configure Postfix for the domain

Edit /etc/postfix/main.cf and set the essentials:

myhostname = mail.yourdomain.com
mydomain = yourdomain.com
myorigin = $mydomain
inet_interfaces = all
mydestination = $myhostname, localhost.$mydomain, localhost, $mydomain
home_mailbox = Maildir/
smtpd_tls_cert_file = /etc/letsencrypt/live/mail.yourdomain.com/fullchain.pem
smtpd_tls_key_file = /etc/letsencrypt/live/mail.yourdomain.com/privkey.pem
smtpd_use_tls = yes
smtpd_tls_security_level = may

Restart Postfix after any change: sudo systemctl restart postfix.

Step 3: Get a real TLS certificate

Point mail.yourdomain.com at the VPS IP first, then issue the certificate:

sudo certbot certonly --standalone -d mail.yourdomain.com

Certbot handles renewal via a systemd timer, but confirm it's active with systemctl list-timers | grep certbot - a lapsed certificate is a common reason mail clients start throwing TLS warnings a few months in.

Step 4: Configure Dovecot for IMAP

In /etc/dovecot/conf.d/10-mail.conf, set the mailbox location to match Postfix:

mail_location = maildir:~/Maildir

In 10-ssl.conf, point Dovecot at the same certificate pair as Postfix, and set ssl = required so nothing authenticates in plaintext. Restart with sudo systemctl restart dovecot and test locally with openssl s_client -connect localhost:993 - you should see the certificate chain come back clean.

Step 5: Sign outgoing mail with DKIM

Generate a key pair with OpenDKIM:

sudo opendkim-genkey -b 2048 -d yourdomain.com -s mail -D /etc/opendkim/keys/yourdomain.com
sudo cat /etc/opendkim/keys/yourdomain.com/mail.txt

That last command prints the exact TXT record to publish in DNS. Wire OpenDKIM into Postfix by adding it as a milter in main.cf (smtpd_milters and non_smtpd_milters pointing at the OpenDKIM socket), then restart both services.

Step 6: The DNS records that make it actually work

This is where most self-hosted setups quietly fail - the server works, but mail never arrives because DNS wasn't finished.

The PTR record is the one people forget because it isn't set in your domain's DNS zone at all - it's configured on the VPS provider's side, tied to the IP address. Most major mail providers reject or spam-flag anything without a matching PTR.

Step 7: Test before you trust it

Send a test message to mail-tester.com and check the score - anything below 8/10 usually points to a missing or misconfigured SPF, DKIM, or PTR record. Also send a real message to a Gmail and an Outlook address and check the raw headers for spf=pass, dkim=pass, and dmarc=pass.

Common problems after setup

• Mail sends but never arrives: almost always a missing or wrong PTR record, or an IP that's already blacklisted.
• Everything lands in spam: check your DMARC alignment - SPF and DKIM both need to match the domain in the From: header, not just pass on their own.
• Connection refused on port 25 from outside: your provider is blocking outbound SMTP; you'll need to request an unblock, which usually involves a short verification process.
• Dovecot won't start after a reboot: check journalctl -u dovecot - a stale PID file or a certificate path that changed after renewal are the usual culprits.

Prevention: keeping deliverability healthy long-term

Monitor your sending IP's reputation monthly, not just at setup. Keep DMARC reports flowing to an address you actually read - they'll tell you within days if something starts failing alignment. And don't skip fail2ban on port 25 and the Dovecot ports; an open mail server that gets compromised into a spam relay will burn your IP's reputation faster than any misconfiguration.