Fighting Spam/Viruses with Amavisd-New, Maia and P

Postfix is a mail transport agent written by security researcher Wietse Venema. Not surprisingly, Postfix is designed from the ground up to be a highly secure system. It consists of several components, each of which runs with least privilege and none of which trust data from the other without validating it themselves. Despite the extensive security emphasis in the system's architecture, Postfix is capable of very good performance in normal conditions; because of architectural decisions, it is also fault tolerant and capable of good performance under adverse conditions such as resource starvation.

Postfix supports three content inspection methods, ranging from light-weight one-line-at-a-time scanning before mail is queued, to heavy duty machinery that does sophisticated content analysis after mail is queued. Each approach serves a different purpose.

The method shown in this article inspects mail AFTER it is stored in the queue, and uses standard protocols such as SMTP. After-queue inspection allows you to use content filters of arbitrary complexity without causing timeouts while receiving mail, and without running out of memory resources under a peak load.

The SpamAssassin system is software for analyzing email messages, determining how likely they are to be spam, and reporting its conclusions. It is a rule-based system that compares different parts of email messages with a large set of rules. Each rule adds or removes points from a message's spam score. A message with a high enough score is reported to be spam.

Amavisd-New is a high-performance and reliable interface between mailer (MTA) and one or more content checkers: virus scanners, and SpamAssassin. It is written in Perl, assuring high reliability, portability and maintainability. It talks to MTA via (E)SMTP or LMTP, or by using helper programs. No timing gaps exist in the design, which could cause a mail loss.

It is normally positioned at or near a central mailer, not necessarily where user's mailboxes and final delivery takes place. When calling of Mail::SpamAssassin (SA) is enabled, it calls SA only once per message (regardless of the number of recipients), and tries very hard to correctly honour per-recipient preferences, such as pass/reject, and inserting spam-related mail header fields.

Integrating Amavisd-New with Postfix

This article does not explain how to install and setup Postfix / SpamAssassin, this is explained here:

Postfix -- the Sendmail Replacement

Fighting Spam with SpamAssassin and Postfix

If you want to set up a spam-checking gateway for all recipients, local or not, you need a way to perform spam-checking as mail is received, before final delivery. Postfix provides a general-purpose filtering directive called content_filter.

The content_filter directive specifies a mail transport that Postfix will invoke after receiving a message. The mail transport hands the message to a filtering program. The filter checks the message and then either refuses it (which will cause Postfix to generate a bounce message), discards it, or reinjects the modified message into Postfix for further delivery. Messages that pass the filter are reinjected so that Postfix can operate on them almost as if they were new messages; this allows Postfix to behave properly if the content filter rewrites message headers.

Installing Amavisd-New

Several content-filtering daemons that call SpamAssassin are available for Postfix. This article provides a complete sample installation of Amavisd-New, a particularly efficient filter that supports both spam-checking and virus-checking. Amavisd-New is written in Perl. Some of Amavisd-New's features include:

Messages can be rejected based on MIME type or extensions of attached filenames.

Messages can be checked with multiple virus scanners, and messages carrying viruses can be refused, discarded, or quarantined.

SpamAssassin can be invoked on a message, and spam can be refused, discarded, quarantined, or tagged.

Per-user configuration of Amavisd-New is possible through an SQL or LDAP database.

The rest of this article details the installation, configuration, and operation of Amavisd-New as an example of a full-scale, daemonized, content filter approach to using SpamAssassin with Postfix. Amavisd-New's other functions, such as virus-checking, are not covered; read the documentation to learn more about these other Amavisd-New features.

Download Amavisd-New from: http://www.ijs.si/software/amavisd/

Download the following modules from http://www.cpan.org/, if you search a single module, http://search.cpan.org/ can be very helpful.

Archive-Tar-1.23.tar.gz

Archive-Zip-1.14.tar.gz

BerkeleyDB-0.26.tar.gz

Compress-Zlib-1.33.tar.gz

Convert-TNEF-0.17.tar.gz

Convert-UUlib-1.03.tar.gz

DBD-mysql-2.9004.tar.gz

DBD-Oracle-1.16.tar.gz

DBI-1.46.tar.gz

Digest-MD5-2.33.tar.gz

IO-String-1.06.tar.gz

IO-stringy-2.109.tar.gz

IO-Zlib-1.04.tar.gz

libnet-1.19.tar.gz

MailTools-1.65.tar.gz

MIME-Base64-3.05.tar.gz

MIME-tools-5.415.tar.gz

Net-Server-0.87.tar.gz

Syslog-0.97.tar.gz

Time-HiRes-1.65.tar.gz

Unix-Syslog-0.100.tar.gz

Install the modules as follows:

tar xzvf <file-from-above>

cd <extracted module>

perl Makefile.PL

make

make test

make install

If you plan to do per-user configuration of Amavisd-New through SQL, you'll need appropriate Perl modules for database access (DBI and a DBD:: module for SQL).

Begin the install process by creating a new user account and group for running Amavisd-New; the usual name for both the user and group is «amavis». This user will own Amavisd-New's files, and the user (or group) must have access to SpamAssassin's configuration and database files as well.

Amavisd-New uses several important directories. It keeps two files in its home directory, one containing its current process ID, and the other used for locking. It uses a working directory for unpacking email messages and scanning them; by default, this is the home directory or the tmp subdirectory of the home directory. For optimal performance, this directory should be on a fast disk—even a RAM disk if your operating system supports it and you have enough memory to spare. Amavisd-New stores quarantined email messages in /var/virusmails by default, but you can select any directory for this purpose.

As root, copy the amavisd script to a suitable directory for executable daemons (e.g. /usr/local/bin), chown it to root, and use chmod to set its permissions to 0755 (readable and executable by all users, writable only by root).

Copy the amavisd.conf file to a suitable directory for configuration files (e.g /etc). By default, amavisd expects to find this file in /etc, and if you locate it anywhere else, you will have to add an extra command-line option (-c filename) when invoking amavisd to tell it the new location. The amavisd.conf file should also be owned by root and should have permissions 0644 (readable by all, writable only by root).

Configuring Amavisd-New

Amavisd-New is configured through the amavisd.conf file. amavisd.conf is parsed as a Perl script and can contain any legal Perl code. Because it is parsed as Perl, you must escape any at sign (@), question mark ($), or backslash (\) characters that appear in double-quoted strings by prepending a backslash.

Edit amavisd.conf to set the (many) available configuration options to control amavisd. The file is organized in logical sections; the most important options are in Section I, but you'll need to read through the entire file to customize the system completely.

Once you've configured the options in amavisd.conf, you're ready to test amavisd. During your first test, start amavisd with the debug argument. This causes amavisd to run in the foreground and produce debugging information that you can watch to be sure that it's working correctly.

Here is an example of amavisd.conf

# /usr/local/bin/amavisd -u amavis debug

starting. /usr/local/bin/amavisd at rabbit.akadia.com Amavisd-New-2.2.0 (20041102), Unicode aware, LANG=en_US

user=amavis, EUID: 502 (502); group=502, EGID: 502 502 (502 502)

Perl version 5.008

Net::Server: 2004/12/10-09:18:19 Amavis (type Net::Server::PreForkSimple)

Net::Server: Binding to TCP port 10024 on host 127.0.0.1

Net::Server: Group Not Defined. Defaulting to EGID '502 502'

Net::Server: User Not Defined. Defaulting to EUID '502'

Net::Server: Setting up serialization via flock

Module Amavis::Conf 2.033

Module Archive::Tar 1.23

Module Archive::Zip 1.14

Module BerkeleyDB 0.26

Module Compress::Zlib 1.33

Module Convert::TNEF 0.17

Module Convert::UUlib 1.03

Module DBI 1.46

Module DB_File 1.810

Module MIME::Entity 5.415

Module MIME::Parser 5.415

Module MIME::Tools 5.415

Module Mail::Header 1.65

Module Mail::Internet 1.65

Module Mail::SPF::Query 1.997

Module Mail::SpamAssassin 3.000001

Module Net::Cmd 2.26

Module Net::DNS 0.48

Module Net::SMTP 2.29

Module Net::Server 0.87

Module Time::HiRes 1.65

Module Unix::Syslog 0.100

Amavis::DB code loaded

Amavis::Cache code loaded

Lookup::SQL code NOT loaded

Lookup::LDAP code NOT loaded

AMCL-in protocol code NOT loaded

SMTP-in protocol code loaded

ANTI-VIRUS code NOT loaded

ANTI-SPAM code loaded

Unpackers code loaded

Found $file at /usr/bin/file

No $arc, not using it

Found $gzip at /usr/bin/gzip

Found $bzip2 at /usr/bin/bzip2

No $lzop, not using it

Found $lha at /usr/bin/lha

Found $unarj at /usr/bin/unarj

Found $uncompress at /usr/bin/uncompress

No $unfreeze, not using it

No $unrar, not using it

No $zoo, not using it

Found $cpio at /bin/cpio

Found $ar at /usr/bin/ar

Found $rpm2cpio at /usr/bin/rpm2cpio

No $cabextract, not using it

No $dspam, not using it

Creating db in /home/amavis/db/; BerkeleyDB 0.26, libdb 4.0

SpamControl: initializing Mail::SpamAssassin

debug: SpamAssassin version 3.0.1

debug: Score set 0 chosen.

debug: running in taint mode? yes

debug: Running in taint mode, removing unsafe env vars, and resetting PATH

debug: PATH included '/usr/local/sbin', keeping.

debug: PATH included '/usr/local/bin', keeping.

debug: PATH included '/usr/sbin', keeping.

debug: PATH included '/sbin', keeping.

debug: PATH included '/usr/bin', keeping.

debug: PATH included '/bin', keeping.

debug: Final PATH set to: /usr/local/sbin:/usr/local/bin:/usr/sbin:/sbin:/usr/bin:/bin

debug: ignore: test message to precompile patterns and load modules

debug: using "/etc/mail/spamassassin/init.pre" for site rules init.pre

debug: config: read file /etc/mail/spamassassin/init.pre

debug: using "/usr/share/spamassassin" for default rules dir

debug: config: read file /usr/share/spamassassin/10_misc.cf

debug: config: read file /usr/share/spamassassin/20_anti_ratware.cf

debug: config: read file /usr/share/spamassassin/20_body_tests.cf

debug: config: read file /usr/share/spamassassin/20_compensate.cf

debug: config: read file /usr/share/spamassassin/20_dnsbl_tests.cf

debug: config: read file /usr/share/spamassassin/20_drugs.cf

debug: config: read file /usr/share/spamassassin/20_fake_helo_tests.cf

debug: config: read file /usr/share/spamassassin/20_head_tests.cf

debug: config: read file /usr/share/spamassassin/20_html_tests.cf

debug: config: read file /usr/share/spamassassin/20_meta_tests.cf

debug: config: read file /usr/share/spamassassin/20_phrases.cf

debug: config: read file /usr/share/spamassassin/20_porn.cf

debug: config: read file /usr/share/spamassassin/20_ratware.cf

debug: config: read file /usr/share/spamassassin/20_uri_tests.cf

debug: config: read file /usr/share/spamassassin/23_bayes.cf

debug: config: read file /usr/share/spamassassin/25_body_tests_es.cf

debug: config: read file /usr/share/spamassassin/25_hashcash.cf

debug: config: read file /usr/share/spamassassin/25_spf.cf

debug: config: read file /usr/share/spamassassin/25_uribl.cf

debug: config: read file /usr/share/spamassassin/30_text_de.cf

debug: config: read file /usr/share/spamassassin/30_text_fr.cf

debug: config: read file /usr/share/spamassassin/30_text_nl.cf

debug: config: read file /usr/share/spamassassin/30_text_pl.cf

debug: config: read file /usr/share/spamassassin/50_scores.cf

debug: config: read file /usr/share/spamassassin/60_whitelist.cf

debug: using "/etc/mail/spamassassin" for site rules dir

debug: config: read file /etc/mail/spamassassin/local.cf

debug: using "/home/amavis/var/.spamassassin/user_prefs" for user prefs file

debug: plugin: loading Mail::SpamAssassin::Plugin::URIDNSBL from @INC

debug: plugin: registered Mail::SpamAssassin::Plugin::URIDNSBL=HASH(0x993df48)

debug: plugin: loading Mail::SpamAssassin::Plugin::Hashcash from @INC

debug: plugin: registered Mail::SpamAssassin::Plugin::Hashcash=HASH(0x9941c2c)

debug: plugin: loading Mail::SpamAssassin::Plugin::SPF from @INC

debug: plugin: registered Mail::SpamAssassin::Plugin::SPF=HASH(0x9952454)

debug: plugin: Mail::SpamAssassin::Plugin::URIDNSBL=HASH(0x993df48)

debug: bayes: 11157 tie-ing to DB file R/O /home/spamd/_toks

debug: bayes: 11157 tie-ing to DB file R/O /home/spamd/_seen

debug: bayes: found bayes db version 3

debug: bayes: Not available for scanning, only 198 spam(s) in Bayes DB < 200

debug: bayes: 11157 untie-ing

debug: bayes: 11157 untie-ing db_toks

debug: bayes: 11157 untie-ing db_seen

debug: Score set 1 chosen.

debug: ---- MIME PARSER START ----

debug: main message type: text/plain

debug: parsing normal part

debug: added part, type: text/plain

debug: ---- MIME PARSER END ----

....

debug: is spam? score=-1.053 required=3

debug: tests=ALL_TRUSTED,MISSING_DATE,MISSING_SUBJECT,NO_REAL_NAME

debug: subtests=__HAS_MSGID,__MSGID_OK_DIGITS,__MSGID_OK_HOST,__SANE_MSGID

SpamControl: done

Net::Server: Beginning prefork (5 processes)

Net::Server: Starting "5" children

Quarantine Handling

When amavisd detects a spam email, it logs a message to its log file by default. It can also quarantine the email and/or notify an administrator. It can then generate a bounce message to the sender. Finally, it can either accept and deliver the message, or discard the message. Many different configuration variables are involved in these decisions.

Enable a spam quarantine by setting the following variables:

$QUARANTINEDIR = '/var/virusmails'; # Quarantine Directory

$spam_quarantine_method = 'local:spam-%b-%i-%n'; # Filename in $QUARANTINEDIR

$spam_quarantine_to = 'spam-quarantine'; # Put Spam in Quarantine Directory

# $spam_quarantine_to = "postmaster\@$mydomain"; # Send Spam to Adminstrator

# $spam_quarantine_to = undef; # Do nothing with Spam

$final_spam_destiny = D_DISCARD;

$spam_admin = "martin.zahn\@$mydomain"; # Where to send Notification

The following symbolic constants can be used in $final_spam_destiny:

D_DISCARD: Mail will not be delivered to its recipients, sender will NOT be notified. Effectively we lose mail (but will be quarantined unless disabled). Losing mail is not decent for a mailer, but might be desired.

D_BOUNCE: Mail will not be delivered to its recipients, a non-delivery notification (bounce) will be sent to the sender by amavisd-new; Exception: bounce (DSN) will not be sent if a virus name matches $viruses_that_fake_sender_re, or to messages from mailing lists, or for spam level that exceeds the $sa_dsn_cutoff_level.

D_REJECT: mail will not be delivered to its recipients, sender should preferably get a reject, e.g. SMTP permanent reject response.

Globally Sender whitelists and blacklists

Hard sender whitelists and blacklists

amavisd can maintain whitelists and blacklists of message senders. It uses a message's envelope address (the one provided in the SMTP MAIL FROM command) as the sender address. Whitelisting ensures that amavisd will allow mail from a whitelisted sender to continue to its intended recipients; blacklisting ensures that amavisd will treat mail from a blacklisted sender as spam.

As with other amavisd address-matching features, you can specify addresses to globally whitelist:

by an array

Set the @whitelist_sender_maps array to a list of sender addresses to whitelist.

@whitelist_sender_maps = (['.example.org', '.example.net']);

@whitelist_sender_maps = ([qw(.example.org .example.net)]); # same thing

@whitelist_sender_maps = ( [".$mydomain"] ); # $mydomain and its subdomains

by keys of a hash

Create a hash lookup table constructed in-line, with keys lowercased

map { $whitelist_sender=1 } (qw(

cert-advisory-owner@cert.org

owner-alert@iss.net

slashdot@slashdot.org

bugtraq@securityfocus.com

));

by a set of regular expressions

Finally, you can specify senders to whitelist by providing a list of regular expressions that match the sender addresses to the new_RE function and assigning the result to $whitelist_sender_re. You may use domain names instead of sender addresses to whitelist all mail sent from a given domain.

@whitelist_sender_maps = ( new_RE(

qr'.*@akadia.com$'i,

qr'.*@swisscom\.com$'i,

qr'.*@sicap\.com$'i,

qr'.*@glue\.ch$'i,

qr'.*@\.*\.admin\.ch$'i

));

You can use a similar set of variables for globally blacklisting senders. The array is @blacklist_sender_maps.

The default amavisd.conf defines $blacklist_sender_maps as shown below. Many username patterns typical of spammers are blacklisted, such as investments; many addresses of well-known security and vendor mailing lists are whitelisted.

Default blacklist entries in amavisd.conf

$blacklist_sender_re = new_RE(

qr'^(investments|lose_weight_today|market.alert|money2you|MyGreenCard)@'i,

qr'^(new\.tld\.registry|opt-out|opt-in|optin|saveonlsmoking2002k)@'i,

qr'^(specialoffer|specialoffers|stockalert|stopsnoring|wantsome)@'i,

qr'^(workathome|yesitsfree|your_friend|greatoffers)@'i,

qr'^(inkjetplanet|marketopt|MakeMoney)\d*@'i,

);

Soft sender whitelists and blacklists

Instead of hard black- or whitelisting, a softer approach is to add score points (penalties) to the SpamAssassin score for mail from certain senders. Positive points lean towards blacklisting, negative towards whitelisting. This is much like adding SpamAssassin rules or using its white/blacklisting, except that here only envelope sender addresses are considered (not addresses in a mail header), and that score points can be assigned per-recipient (or globally), and the assigned penalties are customarily much lower than the default SpamAssassin white/blacklisting score.

@score_sender_maps = ({

'.' => [ # the '.' matches any recipient

new_RE (

[qr'.*@akadia.com$'i => -10.0],

[qr'.*@swisscom\.com$'i => -10.0],

[qr'.*@sicap\.com$'i => -10.0],

[qr'.*@glue\.ch$'i => -10.0],

[qr'.*@\.*\.admin\.ch$'i => -10.0]

)

]

});

Per-Recipient Sender Whitelists and Blacklists (Maia Mailgurad)

Maia Mailguard is a web-based interface and management system for the popular amavisd-new e-mail scanner and SpamAssassin. Written in Perl and PHP, Maia Mailguard gives end-users control over how their mail is processed by virus scanners and spam filters, while giving mail administrators the power to configure site-wide defaults and limits.

Maia Features

Users can manage their own content filtering settings

Users can maintain their own whitelists and blacklists

Lets users see a list of their quarantined spam, sorted by score

Lets users see a list of their quarantined viruses, sorted by date

Users can view text or HTML contents of quarantined mail in decoded or raw form in a "safe" mail viewer

Users can rescue quarantined items to have them redelivered, delete them, or confirm them as spam

Users can report "false negatives" (i.e. spam that gets past the filters)

Optional auto-whitelisting for senders of rescued quarantine items

Entity Relationship Diagram (ERM)

Click here for the Entity Relationship Diagram (ERM)

Users

Contains amavisd-new's per-address settings, and links e-mail addresses to Maia users.

+----------------+------------------+------+-----+---------+----------------+

+----------------+------------------+------+-----+---------+----------------+

| priority | int(11) | | | 7 | |

+----------------+------------------+------+-----+---------+----------------+

Priority

SQL lookups (e.g. for user+foo@example.com) are performed in order which is normally requested by:

$sql_select_policy =

'SELECT *,users.id FROM users,policy'

. ' WHERE (users.policy_id=policy.id) AND (users.email IN (%k))'

. ' ORDER BY users.priority DESC';

ORDER is used, if there is a chance that multiple records will match - the first match wins.

The following order (implemented by sorting on the 'priority' field in DESCending order, zero is low priority) is recommended, to follow the same specific-to-general principle as in other lookup tables:

- lookup for user+foo@example.com

- lookup for user@example.com # (only if $recipient_delimiter nonempty)

- lookup for user+foo # (only if domain part is local)

- lookup for user # (only local; only if $recipient_delimiter is nonempty)

- lookup for @example.com

- lookup for @. # (catchall)

POLICY

POLICY contains amavisd-new's policy settings, which Maia applies on a per-address basis (i.e. each e-mail address is assigned its own unique policy record. Each domain also has a set of policy defaults, and the system default policy is stored as the '@.' user's policy record.

+----------------------+------------------+------+-----+---------+----------------+

+----------------------+------------------+------+-----+---------+----------------+

| virus_lover | char(1) | YES | | Y | |

| spam_lover | char(1) | YES | | Y | |

| banned_files_lover | char(1) | YES | | Y | |

| bad_header_lover | char(1) | YES | | Y | |

| bypass_virus_checks | char(1) | YES | | Y | |

| bypass_spam_checks | char(1) | YES | | Y | |

| bypass_banned_checks | char(1) | YES | | Y | |

| bypass_header_checks | char(1) | YES | | Y | |

| discard_viruses | char(1) | YES | | N | |

| discard_spam | char(1) | YES | | N | |

| discard_banned_files | char(1) | YES | | N | |

| discard_bad_headers | char(1) | YES | | N | |

| spam_modifies_subj | char(1) | YES | | N | |

| spam_tag_level | float | YES | | 999 | |

| spam_tag2_level | float | YES | | 999 | |

| spam_kill_level | float | YES | | 999 | |

+----------------------+------------------+------+-----+---------+----------------+

MAILADDR

MAILADDR contains a list of sender e-mail addresses referenced by users' whitelists and blacklists.

+----------+------------------+------+-----+---------+----------------+

+----------+------------------+------+-----+---------+----------------+

| priority | int(11) | | | 7 | |

+----------+------------------+------+-----+---------+----------------+

Priority

The SQL select clause to check sender in per-recipient whitelist/blacklist. The first SELECT argument '?' will be users.id from recipient SQL lookup, the %k will be sender addresses (e.g. full address, domain only, catchall).

$sql_select_white_black_list =

'SELECT wblist.wb FROM wblist,mailaddr,users'

. ' WHERE (users.id=?)'

. ' AND (wblist.rid=users.maia_user_id)'

. ' AND (wblist.sid=mailaddr.id)'

. ' AND (mailaddr.email IN (%k))'

. ' ORDER BY mailaddr.priority DESC';

WBLIST

WBLIST contains the whitelist and blacklist records, on a per-user (not per-address) basis.

+-------+------------------+------+-----+---------+-------+

+-------+------------------+------+-----+---------+-------+

| rid | int(10) unsigned | | PRI | 0 | | # maia_users.id

| sid | int(10) unsigned | | PRI | 0 | | # mailaddr.id

| wb | char(1) | | | | |

+-------+------------------+------+-----+---------+-------+

MAIA_USERS

MAIA_USERS contains mail filter settings that apply to e-mail recipients who have registered with Maia. The user_level is stored as one of (U)ser, (A)dministrator, or (S)uper-Administrator.

+----------------------------+------------------+------+-----+------------+----------------+

+----------------------------+------------------+------+-----+------------+----------------+

+----------------------------+------------------+------+-----+------------+----------------+

MAIA_DOMAINS

MAIA_DOMAINS contains mail filter settings that apply to entire e-mail domains as defaults for users who are not registered with Maia.

+--------------------------+------------------+------+-----+------------+----------------+

+--------------------------+------------------+------+-----+------------+----------------+

+--------------------------+------------------+------+-----+------------+----------------+

MAIA_DOMAIN_ADMINS

MAIA_DOMAIN_ADMINS is a one-to-many mapping of domains to users with administrator privileges (domains can have multiple administrators).

+-----------+------------------+------+-----+---------+-------+

+-----------+------------------+------+-----+---------+-------+

+-----------+------------------+------+-----+---------+-------+

MAIA_STATS

MAIA_STATS is a table of per-user statistics that keeps track of the total number of mail items of each type, along with total sizes and superlatives (e.g. largest, smallest, oldest, newest, etc.).

+------------------------------+------------------+------+-----+---------+-------+

+------------------------------+------------------+------+-----+---------+-------+

| user_id | int(10) unsigned | | PRI | 0 | &