Description ----------- This script provides a content filter program for the Stalker CommuniGate Pro mail server that seeks to limit the number of outgoing spam your local users can send, as was as limit the incoming spam from outsiders. In essence, it is a throttling script that starts rejecting mail once a certain threshold is reached. The current thresholds supported include: - number of separate messages sent within a certain time frame - sum total recipients calculated across all messages within a certain time frame - messages sent with the same subject that reach a certain message and/or recipient sum total limit Optionally, SpamAssasin support can be enabled to add headers for tagging spam or rejecting them immediately. The reasoning for this script is that it is difficult to catch spammers until after the fact they have already done lots of damage and spammed many users. It is simply an effort to stop them in their tracks in real-time so you can deal with them at a later time (like after 3am :). Most of the spam tools out there are for preventing spam from entering your server. The major drive for this filter is to prevent spam coming from YOUR users. We need to take responsiblity and implement tools to prevent our users from abusing our resources. This is one such tool. Design ------ The basic design is that certain headers of every outgoing and incoming message is stored in an database with a timestamp. For every message that is message that enters your mail system, calculations are performed based on the data in the database. When a user sends messages and reaches one of the defined limits, he is marked as an abuser for a number of seconds. At that point, all messages from the user are rejected until those seconds have elapsed. In effect, this throttles the user whenever he reaches a limit and penalizes him for a certain amount of time. When processing outgoing messages, the 'user' is distinguished by their SMTP AUTHed username. When processing incoming messages, the 'user' is distinguished by their IP address. During the time the user is 'punished', rejected messages are logged as well. By analyzing this data you can determine whether to take action against this account or not, and a script spam_report.pl is provided as an example for suspending local users. Ultimately, the user is stopped immediately from doing much damage. This will hopefully stop the damage most spammers will do until you can take action against their accounts. To give an example, in one production environment with 500,000 users, this filter caught on average about 2-3 local spammers per day, and prevented about 10,000 external email addresses from receiving the spam these users would have sent per day. This reduced the spam complaints for this environment to almost zero. This script starts N processes with one attached to the content filtering API of CommuniGate. Each request is sent to a non-busy child process to handle. Those child processes keep a database connection open, parse the messages they are given, and either reject or accept them depending on the thresholds they calculate from the database. In the database there are 5 main tables. See schema.sql for their fields, but here is a general description: abusers Stores the users marked as abusers exempt_users Stores the users that are exempt from any throttling incoming Stores some headers of all successful incoming messages from outsiders outgoing Stores some headers of all successful outgoing messages from local users rejected Stores some headers of all messages that are rejected The algorithm used for throttling is defined in the throttle() subroutine. It currently does the following: ------ If the user exists in exempt_users accept message fi If user is marked as an abuser and we are within his abuse time Log message to rejected table reject message fi If user has exceeded one of the rate limits Log message to rejected table Mark user as an abuser reject message fi (optional) If message is considered by SpamAssassin to be spam Add headers to message indicating spam OR reject message fi Accept message and log to outgoing table ------ This is a very general description of the algorithm. See below and the code for the details of the rate limits. If anyone has ideas to improve it, please let me know. In all cases where there are problems, such as a child process dying, database unavailable, or timeouts, the script tries to fail safe. In other words it will OK the message if it can't process it for some reason. This is safer than rejecting it or requesting CG to try again later in my opinion. Requirements ------------ You MUST be using SMTP AUTH for your local users. If you are not, then you shouldn't be hosting a mail server, period. This script determines whether a user is local by looking for an SMTP AUTH'ed line in the message. If you aren't using SMTP AUTH, or there is no SMTP AUTH line found, the message is considered an 'incoming' message. The script will process Webmail submitted messages as 'outgoing' message, since CommuniGate puts the authentication info in the message. Please make sure you understand the distinction between 'incoming' mail verses 'outgoing' if you plan to process both of these types of messages. See the section after this for a more detailed discussion. You must have a MySQL server setup with the schema defined in schema.sql which is included in this distribution. You can perhaps use other databases with minor modifications to the script, but I have not tested anything but MySQL. Also the script uses certain MySQL specific functions such as 'interval' which may not be available in other databases. You will need the Perl DBI and MySQL DBD modules installed. Performance ----------- I have tested this script in a production clustered environment hosting about 500,000 user accounts and multiple domains. I haven't noticed any performance issues with it, and in the stress tests I performed the script always failed safe. The test setup was 2 frontends, 2 backends, all on Solaris SPARC systems and a MySQL server running on an AIX system. This filter was run only on the frontends, and any mails generated on the backends were configured to go to the frontends. If you are using a cluster, please note the cluster settings below as they are very important. Incoming vs. Outgoing Mail -------------------------- Simply put, outgoing mail is any mail that is SMTP AUTHed by your system. Everything else is considered incoming mail. Here are some examples. Say you host a domain called 'mydomain.com' and there is an external domain not hosted by you called 'extdomain.com'. A SMTP AUTHed message from blah@mydomain.com (local user) to whatever@extdomain.com is considered 'outgoing'. A message from whatever@extdomain.com to blah@mydomain.com is considered 'incoming'. A SMTP AUTHed message from blah@mydomain.com (local user) to another@mydomain.com (another local user) is considered 'outgoing'. A message from blah@mydomain.com (local user, but NOT SMTP AUTHed) to another@mydomain.com (another local user) is considered 'incoming'. A message from blah@mydomain.com (your user, but NOT SMTP AUTHed) to whatever@extdomain.com is considered 'incoming'. This may not make sense, but one of the requirements of using this script is that your local users MUST be using SMTP AUTH for it to work correctly! System-generated messages, which do not contain 'Received:' lines in the headers unless you are using a cluster, are NOT processed by the script. They are simply OK'ed always. If using a cluster, and you have the cluster settings setup properly in the configuration of this script, it will behave the same way. In other words, it should work properly. When writing this script, I preferred not to supply the filter with a list of locally hosted domains. One reason for this is I wanted it to be as drop-in as possible, without any dependencies on maintaining a domain list, etc. I did not want to duplicate any info that CG already has. The script instead intelligently figures out what your local domains are because anything SMTP AUTHed MUST be your local domain. This way you don't have to maintain anything in the filter system when you add or change domains. Each mail, whether outgoing or incoming, is distinguished by some specific parameter to determine 'who' the user is to apply throttling on. This is represented by the 'auth' field in the DB tables. For outgoing mail, this is simply the SMTP AUTHed user, and throttling is straightforward based on that. For incoming mail, I have made the 'auth' field instead hold the IP address of where your system received the message from. Note this may NOT be the origin IP address of the sender if the mail has traversed multiple servers. It is IP address of the mail server or client right before your mail server received it. So the distinguishing parameter to determine 'who' the user is to apply throttling on for incoming mail is this IP address. I chose the IP address because everything else in the message can easily be forged by spammers. So in a nutshell, for outgoing mail you will be throttling based on SMTP AUTH regardless of what IP or IPs they connect from. For incoming mail, you will be throttling based on IP only. Setup ----- 1. Create a database in MySQL for this filter (you can call it 'throttle') mysqladmin -u root -p create throttle 2. Create a user for use with this database. mysql -u root -p mysql grant all on throttle.* to throttle@CG_IP identified by 'YOUR_PASSWD'; Where CG_IP = IP where you run CG (or localhost) YOUR_PASSWD = a password you want to use. 3. Create the required schema in your database from schema.sql. mysql -u throttle -p throttle < schema.sql 4. Edit the top of spam_filter.pl with basic configuration: The configuration is setup such that you can specify limits for outgoing and incoming mail separately. The format is similar to: $config{...} These represent global settings that are not specific to outgoing/incoming messages. $config{incoming/outgoing}{...} These represent settings specific to incoming or outgoing messages. By default, everything is set to relatively safe values and incoming mails is NOT processed. Depending on your environment, you may need to adjust these settings. Queue-specific Settings - These are all prefixed with either $config{incoming} or $config{outgoing} depending on what they apply to. THROTTLE_HIST_TIME = 60 This defines the time range used in the calculation of X messages per Z seconds. In other words, this is the Z value. 1 minute (60 seconds) is a good value. THROTTLE_MSGS = 10 This defines the number of messages a user can send within THROTTLE_HIST_TIME. In other words, this is the X value in the calculation above. We found that 10 messages per minute is a good indication of spamming. Set this to 0 to disable this type of throttling. THROTTLE_RCPTS = THROTTLE_MSGS * 5 This defines the number of recipients a user can send to per THROTTLE_HIST_TIME. If this is set to 10 for example, 2 messages sent by a user each CCed to 5 people within THROTTLE_HIST_TIME would reach the limit. THROTTLE_MSGS * 5 is safe value, perhaps even too high. Set this to 0 to disable this limit. SUBJ_LAST_MSG_COUNT = 5 SUBJ_LAST_MSG_RCPTS = 50 SUBJ_LAST_MSG_TIME = 24 * 60 * 60 These settings do subject throttling by working as follows: If, within SUBJ_LAST_MSG_TIME seconds, the last SUBJ_LAST_MSG_COUNT messages from a user were the same subject and the recipients would exceed SUBJ_LAST_MSG_RCPTS, this will be considered spamming. This is mainly to catch 'slow' spammers that wouldn't be caught otherwise. Set SUBJ_LAST_MSG_COUNT to 0 to disable this feature. SUBJ_LAST_MSG_STRICT_COUNT = 50 SUBJ_LAST_MSG_STRICT_TIME = 24 * 60 * 60 These settings do subject throttling by working as follows: If, within SUBJ_LAST_MSG_STRICT_TIME seconds, the last SUBJ_LAST_MSG_STRICT_COUNT messages from a user were the same subject, this will be considered spamming. Note this setting is separate from the first type of subject throttling above in that it does not require a recipient limit to be reached. Set SUBJ_LAST_MSG_STRICT_COUNT to 0 to disable this feature. ABUSE_TIME = 3600 This defines how long a user is penalized after he has been marked as spamming. During this time, any mail from the user will be bounced back with an undeliverable message. 1 hour (3600 seconds) is a good value. IGN_SYSGEN = 0 for outgoing, 1 for incoming Set this to 1 to ignore system-generated mail (such as from postmaster, mailer-daemon). The main reason for this is to not consider 'undeliverable' reports as spam. This should only be enabled for incoming mail, as outgoing will never need this. SA_ENABLE = 0 Set this to 1 to enable SpamAssassin support. Any mail that SpamAssassin regards as spam will either be accepted but SpamAssassin headers will be added to the message or completely rejected based on the SA_REJECT setting. There is currently no marking as abuser (punishment) for these messages. You must have the Mail::SpamAssassin PERL module to use this feature. Also note that CPU and memory usage of the filter will be quite large with this enabled. If you need to configure SpamAssassin (whitelists, etc), you will need to edit your system-wide SpamAssassin configuration. SA_REJECT = 0 Set this to 1 to reject SpamAssassin tagged messages instead of adding headers indicating it is spam. SA_NETCHECKS = 0 for outgoing, 1 for incoming Set this to 1 to make SpamAssassin do network (RBL) checks. It should really be set to 0 for outgoing mail. POLICY_DISCARD = 0 Set this to 1 to discard messages instead of rejecting them. This is useful to prevent sending bogus undeliverable messages. The sender will not know whether the message was rejected or not if this is set to 1. Global Settings - These settings are enclosed in $config{...} where ... is the global setting. NUM_PROCS = 10 This will define how many processes to keep running. In an environment with 500,000 users, processing about 7-10 mails per second, 10 was more than enough. THROTTLE_MSG = "SMTP throttle limit reached, possible spamming" This defines the message the user will get in his undeliverable message. PENDING_MAX = 60 If one of the child processes fails to process a message within this many seconds, that child process will be considered hung and restarted. The message that the child was processing will fail safe (it will be OK'ed). If using SpamAssassin, this should be at least 60 seconds because such scanning can take that long. CG_CLUSTER = 0 CLUSTER_BE_NET = '' If you are using a cluster, set CG_CLUSTER = 1 and CLUSTER_BE_NET should be a regexp that matches your cluster network. See the Cluster section below for more information. QUEUE_BYPASS_TIME = 1800 This defines how old a queue file must be before processing it. The main reason for this is to deal with CommuniGate restarts. When CG restarts, it re-queues everything. This can wrongly be assumed by the filter to be spamming. To avoid this, the script makes sure the queue file is NOT older than $QUEUE_BYPASS_TIME seconds, and if so it will just OK it without any processing. 30 minutes (1800 seconds) is a good value. PROCESS_INCOMING = 0 Set this to 1 to process incoming messages as well as outgoing. If this is set to 0, all of the $config{incoming} values will be ignored. SA_MAXBYTES = 256 * 1024 Only process messages less than this size in bytes with SpamAssassin. This is to avoid scanning large messages and using alot of memory. SA_PERSISTENT = 0 If this is set to 1, one SpamAssassin object will be created for each queue (incoming and outgoing), and these objects will be reused for each scan. If set to 0, a new object is created for every message being scanned. The default is 0 because I've seen memory growing too large when using persistent objects. This increases load, but in my tests it was not a big deal. DEBUG = 0 This defines the debug level. Set this to 10 or higher initially to make sure everything is working properly. Once verified, set to 0. Note that if you set this value >= 15, child processes will log to $LOGFILE.PID. BASE_DIR = "/var/CommuniGate" This defines the standard CommuniGate location. LOGFILE = "/tmp/filter.out" This defines where logs should be sent. DB_DRIVER = "mysql" DB_HOSTNAME = "localhost" DB_NAME = "throttle" DB_USER = "throttle" DB_PASSWD = "throttle" This defines the database connection setup. It should be self-explanitory. INSERT_DELAYED = 1 Set this to 1 to enable 'insert delayed' syntax in MySQL. This results in faster inserts, but only works with MyISAM tables. DUMPBAD = 0 If this is set to 1, any messages that have taken a long time to process will be dumped to /tmp. This is mainly for debugging purposes. 5. Test the script outside of CommuniGate. Try running the following command: ./spam_filter.pl and then type: 1 FILE blah You should get back immediately: 1 OK Press Control-D to exit the script. Look at LOGFILE for any problems. You should not see any database errors. You should only see a 'Permission denied' error because 'blah' cannot be found. 6. Test with CommuniGate I recommend you test this on a non-production system first. Keep the DEBUG level at 10 and make the following changes in CommuniGate: Settings->General->Helpers Use Filter: spam_filter (or other descriptive name) Program Path: Full path for the script Check the checkbox next to 'Use Filter'. Other settings can be left at their defaults, as the filter should handle problem cases on its own. Settings->Rules (if using a cluster, select Server-Wide, NOT Cluster-Wide) Create a new rule with Action 'ExternalFilter' and Parameters 'spam_filter' or whatever name you gave above. The Data/Operation/Parameter sections at the top can be left alone which will mean the filter will apply to all messages. Settings->Queue->Message Enqueuer Set processors to NUM_PROCS/2 (5 is good) Next try sending some mail while watching the filter LOGFILE. Make sure there aren't any database errors, etc. Also check the tables to make sure the messages are being inserted properly. Try 'spamming' to see if your messages get rejected after you peak the threshold. 7. It may be a good idea to leave DEBUG at 10 for awhile to watch for problems. Afterwards set it to 0 and keep a watch on the tables. Cluster ------- If you use CommuniGate in a cluster, you should set CG_CLUSTER and CLUSTER_BE_NET appropriately. This is because this script normally determines local user information from the first Received: line in the header. By searching for a CommuniGate generated string in the Received: line, the script is able to determine who are your local users (and domains). However, when using a cluster, webmail composed on a backend could be forwarded to the frontend where this script runs. In this case the account info will be in the 2nd 'Received:' line. If we just look at the first 2 of these always, a remote CG server sending to our system could be parsed incorrectly (their user being considered ours). The CLUSTER_BE_NET regexp is an effort to make this safer. If the current message's Received: line doesn't contain the account info, and this message was received from an IP that matches CLUSTER_BE_NET, it is safe to assume that the next Received: line may contain the account info. Then, and only then, is the next Received: line parsed. For example, if your cluster IPs are 1.2.3.0/24, you should use the following settings: CG_CLUSTER = 1 CLUSTER_BE_NET = '^1\.2\.3\.' Do NOT set CG_CLUSTER = 1 if you are not using a cluster. Exempt Users ------------ To exempt any user from throttling, insert the info into the exempt_users table. The auth field should be the SMTP AUTHed username for outgoing mail, or the IP address for incoming mail. Maintenance ----------- The tables should be archived and cleaned depending on how large they grow. A good method would be to archive all data every month. I've included a sample script archive_throttle.pl that shows how you can do this. This creates monthly tables and moves data older than 7 days to the corresponding monthly table. It should be run daily. Also included is a sample spam reporting script spam_report.pl. We use something similar to disable accounts automatically. Finding Local Users With Spam Dropboxes --------------------------------------- The script provides the ability to find local users that appear to be using their mailbox as a spam dropbox. In most cases, this happens when the spammer uses external mail servers to send spam but sets a From: address to a user in your domain. Even though the spam did not come from your server, you will most likely be involved in the complaint because this address involves your domain. We can make some 'guesses' as to when this occurs. When a spammer does this, what your mail server ends up getting is MANY undeliverable messages to the account the spammer is using as a dropbox. The reason for this is that most spammers are using a huge list of addresses, possible generated somehow, and many of these addresses are either incorrect or disabled. Those remote mail servers end up flooding your server with tons of undeliverable messages. In most cases the spammer is not even reading these messages, and just using up your resources. What can we do? Well by watching for these undeliverable messages, we can be relatively sure that if one account receives many of them, that account is most likely a spam dropbox. To find such accounts, you will need to use this script to process incoming mail as well as outgoing: $config{PROCESS_INCOMING} = 1; Next you will need to process and store incoming 'system generated' mail. This is mail that appears to be undeliverable errors or other daemon messages. $config{incoming}{IGN_SYSGEN} = 1; Finally you will need to store these 'system generated' messages: $config{STORE_SYSGEN} = 1; Running with this configuration you will have some data in the 'sysgen' table that you can use to find these spam dropboxes. Here is a very simple SQL statement that will show you the top 5 users receiving system generated mail and their counts: select hto,count(hto) a from sysgen group by hto order by a desc limit 5; Writing something to process this data and kill these users is left as an exercise. Bugs and Suggestions -------------------- The main problem I have found with this is if a user has alot of mail queued on his machine (say an offline user), and then connects and flushes all of his ougoing mail. In this case, it may appear the user is spamming. I have not found a good way to deal with this situation. I have never received complaints regarding this, so I feel it negligible. When processing incoming mail, if there is an external mail server that has a high amount of throughput of messages to your system, but many different remote users relay through that server, that IP address could be blocked when you don't want it to be. There is no real solution I can see for this other than adding the IP address to the exemption list, which means don't do any throttling from that IP. Or you could only apply the SUBJECT related limits for incoming mail and disable the others. If you find any problems or have suggestions, please let me know at valankar@bigfoot.com.