Configuring a self-hosted SMTP server (Linux and Windows)

If your application sends password resets, order confirmations, OTPs, or notification emails, those messages need a reliable way to reach your user’s inboxes. Simple Mail Transfer Protocol (SMTP) is the standard protocol that makes it possible. Without a proper SMTP configuration, important emails may fail to send, get blocked by spam filters, or never reach the recipient.

If you’re setting up a Linux or Windows server, proper SMTP configuration helps your application send emails securely and consistently. A correct setup also makes it easier to troubleshoot delivery issues and monitor email activity.

In this guide, you’ll learn why and how to configure an SMTP server on both Linux and Windows and understand the common mistakes to avoid.

Why configure SMTP on Linux or Windows?

Many applications run on Linux or Windows servers. By configuring SMTP, you can route their outgoing emails through your email service instead of the server's default mail system.

SMTP configuration is commonly used for:

• Sending alerts from Windows scheduled tasks, Linux cron jobs, or monitoring tools
• Delivering password resets, verification emails, and other user notifications
• Sending invoices, receipts, and order confirmations from business applications
• Enabling CRM and ERP systems to send automated emails
• Emailing reports from scripts and scheduled jobs

Configuring SMTP also allows your applications to send authenticated emails when your domain is properly configured with SPF, DKIM, and DMARC. These email authentication standards help improve email security, protect your domain from spoofing, and increase email deliverability.

Configure SMTP on Linux: Step by step

Most Linux mail servers use software such as Postfix, Exim, or Sendmail to handle email delivery. Among these, Postfix is the most widely used mail transfer agent (MTA) on Linux. It's relatively easy to configure, well-documented, and trusted across multiple production environments.

1. Install Postfix on your system

Start by installing Postfix. Open a terminal and run the appropriate command for your Linux distribution:

• Ubuntu/Debian:

   sudo apt install postfix

• RHEL/CentOS:

   sudo dnf install postfix

During the installation prompt, select Internet Site as the configuration type.

2. Open the main configuration file

After Postfix is installed, run the command to access the main configuration file.

sudo nano/etc/postfix/main.cf

3. Set your server's hostname

Locate the myhostname parameter and set it to your server’s fully qualified domain name (FQDN), e.g., mail.yourdomain.com. This tells Postfix what to identify itself as when communicating with other mail servers.

4. Define which domains your server will handle

Set mydestination to specify which domains this server will accept local mail for. If you’re managing mail for multiple domains, you can list them all here, separated by commas.

5. Lock down who can send mail through your server

Configure your mynetworks settings to include only trusted hosts or internal networks. This prevents unauthorized systems from using your server as an open relay.

Note: Because this configuration relies strictly on IP restrictions (mynetworks) rather than user authentication, never expose this server directly to the public internet without configuring SMTP authentication and enabling TLS encryption.

6. Restart Postfix to apply your changes

With all your changes saved, it’s time to restart the Postfix service so your new configuration takes effect. Run:

 sudo systemctl restart postfix

You can then verify that the service is running correctly with:

sudo systemctl status postfix 

Configure SMTP on Windows: Step by step

To deploy a secure SMTP service on Windows, organizations often rely on trusted, open-source options such as hMailServer. It's a widely used self-hosted SMTP server for Windows, although it's no longer actively maintained. It supports SMTP, IMAP, and POP3 and allows reliable and secure email delivery.

Here are the step-by-step instructions:

1. Download and install hMailServer

1. Download the latest installer from the official hMailServer website and run it.
2. Choose Full installation. Use the built-in database (SQL Compact) for testing only. For production, connect an external database (MySQL, PostgreSQL, or MS SQL) to bypass SQL Compact's 4GB limit and prevent crashes.
3. Set a strong main administration password (this is different from your Windows password)

After installing hMailServer, configure your DNS settings to improve email delivery. While MX records route incoming mail, ensure your outbound email sending is backed by valid A, PTR, SPF, and DKIM records to prevent your emails from being flagged as spam.

Next, allow the required SMTP ports through Windows Advanced Firewall. Instead of allowing the hMailServer executable, create a port rule to allow traffic on the standard mail transfer ports.

2. Add your domain in hMailServer

1. Go to Domains > Add domain and enter your domain name (e.g.,yourdomain.com).
2. Click Save.

3. Set up an email account

1. Go to Accounts, and add a new account there.
2. Enter a username (the part before the @), set a strong password, set a size limit if needed, and save. Repeat for each mailbox you need.

Your applications or services will use this account to authenticate and send email via the SMTP server.

4. Add your public host name

1. Go to Settings > Protocols > SMTP and, under the Settings tab, select the Delivery of e-mail tab.
2. In the local host name field, enter your fully qualified domain name (for example, mail.yourdomain.com).This tells the server how to identify itself when communicating with other mail servers.

To prevent passwords and email content from being transmitted in clear text, ensure you bind an SSL/TLS certificate to your ports by going to Settings > Advanced > SSL Certificates to enforce secure connections.

5. Specify SMTP relay

Some internet service providers block outgoing traffic on port 25. If that happens, configure hMailServer to route outgoing emails through your ISP's or a third-party's SMTP server. When you configure an external SMTP server as a relay, hMailServer sends all outgoing emails to that server, which then delivers them to the intended recipients.

1. Under the SMTP Setting, go to the Delivery of e-mail tab.
2. Choose SMTP Relayer field and enter the host name of your ISP's email server, along with your credentials, if authentication is needed. Save your changes.

6. Check IP ranges

IP ranges in hMailServer are used to define which IP addresses are permitted to send email through your server. For example, you can set up IP ranges so that only devices within your local network are allowed to use the server for sending emails.

7. Restart the hMailServer service

After making all these changes, a restart ensures every setting, port binding, and security rule is properly configured.

Troubleshooting and mistakes to avoid

FAQs

1. How can I test SMTP authentication?

You can test SMTP authentication using your email client, command-line tools like Telnet or OpenSSL, or an SMTP testing tool. If the server accepts your credentials and successfully sends a test email, your SMTP authentication is working correctly.

2. How do I know if my SMTP server is working?

After completing the configuration, send a test email. If the message is accepted by the SMTP server and successfully delivered to the recipient without authentication or connection errors, your SMTP server is configured correctly.

3. Which SMTP port should I use?

You should use port 587 with STARTTLS for authenticated email submission because it's the preferred choice for most applications. Port 465 is used for SMTP over implicit SSL/TLS, and port 25 is mainly intended for server-to-server email transfers and is frequently blocked by internet service providers for outbound mail.

4. Should I configure my own SMTP server or use an SMTP relay?

Running your own SMTP server gives you full control but requires ongoing maintenance, security, and reputation management. For most businesses, using an SMTP relay such as ZeptoMail is easier, more reliable, and helps improve email deliverability.

5. How do I configure SMTP for transactional emails?

Create SMTP credentials with your email provider, then configure your application with the SMTP host, port, username, password, and encryption settings. Use a dedicated transactional email service such as ZeptoMail to ensure fast delivery, authentication, and better inbox placement.