Setting up high-volume outgoing mail system with Linux
------------------------------------------------------

The following are my experiences setting up a system to send out emails
as fast as possible. Normally, one considers this spam. However, in my
case it was simply to send emails out to all of our own users. Usually,
these mail 'campaigns' sent mail to about 1.2 million addresses.

Note that this solution will make administrators of the domain you are
sending mail to very unhappy. Unless they are using server farms, and can
handle a very high load of incoming mail, you will get complaints. In our
case, we owned the receiving servers, so it was not a big deal.

I considered different open source mail software, but decided on Postfix
for its ease of setup and efficient design. You should be familiar with
Postfix and all of options below before using them.

Our system is composed of 3 servers, which are all dual P3-650mhz with
512M RAM and 2 9gig SCSI drives. A collection of Perl scripts were written
such that one machine would feed messages via SMTP to the other 2 boxes
concurrently. These can be written by anyone who knows Perl, so I will
not bother posting them. One machine should really be dedicated for feeding,
and it should not feed back to a Postfix running locally. Doing both
feeding and mailing on the same machine will be very slow.

Each machine was installed with Redhat Linux 6.2 according to my setup
instructions (see my other docs on this). It's important to apply at
least the kernel tuning parameters.

Postfix Configuration
---------------------

Install a stable Postfix release on the machines. Do not start the service
until applying everything below. I added the following to /etc/postfix/main.cf:

default_process_limit = 1000
smtp_destination_concurrency_limit = 0
trigger_timeout = 1
qmgr_message_active_limit = 10000
qmgr_message_recipient_limit = 10000
virtual_maps = hash:/etc/postfix/virtual
hash_queue_names = defer,deferred,incoming,active
smtp_connect_timeout = 30
qmgr_site_hog_factor = 100

When sending 1.2 million messages, quite a few bounce back with possible mail
errors (quota full, user nonexistant, etc). In our case, I didn't care
about such bounces, so in /etc/postfix/virtual I had:

from_address@from_domain	some_user

and in /etc/aliases:

some_user:	/dev/null

I'm not sure if you can put /dev/null directly in virtual.

Postfix logs ALOT of information to syslog. During our mail syslog would
take up to 70% CPU! This can be fixed by turning off synchronous writes
for maillog. I did this and also threw away some useless logs. The relevant
lines in syslog.conf:

# Log all the mail messages in one place.
# mail.*                                                      /var/log/maillog
mail.notice                                                   -/var/log/maillog

So everything above 'notice' goes to maillog asynchronously.

Next, /var/spool/postfix was placed on the 2nd drive, which was dedicated for
this spool directory. This filesystem should be mounted with options in fstab
defaults,noatime. There is no point keeping access time on every file and
looking through the Postfix source shows no usage of atime. You should also
mke2fs this filesystem with -m0 to save some space.

/var/spool/postfix should be symlinked to the other filesystem, or possibly
mounting directly. Some would argue the symlink adds overhead, and mounting
directly on /var/spool/postfix would be better.

Startup Postfix, then stop it. This is just to make sure all of the
directories are created successfully.

Next, I needed to modify /etc/postfix/postfix-script. Under the 'start)'
section, I commented out the following lines:

        # $config_directory/postfix-script check || {
        #       $FATAL Postfix integrity check failed!
        #       exit 1
        # }

This performs file checks. However, when there are 1 million spool files
lying around, the check can take hours. I didn't care if 1 of these messages
are corrupt. Next comment out the line:

		# $CHATTR $dir

Normally, Postfix will enable synchronous writes on the spool directories.
This is slow, and really unneeded in this scenario. Go into /var/spool/postfix
and do:

chattr -R -S *

Setup DNSCache
--------------

On the feeder machine, it is a good idea to install a DNS server. The best
one for this is djbdns. The other 2 machines should be setup to query the
1st machine for DNS (/etc/resolv.conf). More information on this is at:

http://cr.yp.to/djbdns.html

Having a dedicated DNS for these mass mails can speed them up very much.

What about ReiserFS?
--------------------

At some point I played around with putting the spool directories on ReiserFS.
Unfortunately, under heavy load I noticed many VM errors reported by the
kernel and a few panics. I had to move back to ext2. At this point, I don't
feel ReiserFS is ready for production usage. Your mileage may vary. I do
love ReiserFS though, and hope they keep up the good work. I run it at home
where there are less important files :).