General-Purpose Mail Filter
Simple Mail Transfer Protocol (SMTP) which is the standard
for email transmissions across the Internet was designed in the
good old days when nobody could even think of the
possibility of e-mail being abused to send tons of unsolicited
messages of dubious contents. Therefore it lacks mechanisms that
could have prevented this abuse (spamming), or at least could
have made it difficult. Attempts to introduce such mechanisms (such
extension) are being made, but they are not in wide use yet and,
probably, their introduction will not be enough to stop the e-mail
abuse. Spamming is today’s grim reality and developers spend lots of
time and efforts designing new protection measures against it.
Mailfromd is one of such attempts.
The package is designed to work with any MTA supporting ‘Milter’ or ‘Pmilter’ protocol, such as ‘Sendmail’, ‘MeTA1’ or ‘Postfix’. It allows you to:
The idea of the utility appeared in 2005, and its first version
appeared soon afterward. Back then it was a simple implementation of
Sender Address Verification (see SAV) for ‘Sendmail’ (hence
its name –
mailfromd) with rudimentary tuning
After a short run on my mail servers, I discovered that the utility was not flexible enough. It took less than a month to implement a configuration file that allowed the user to control program and data flow during the ‘envfrom’ SMTP state. The new version, 1.0, appeared in June, 2005.
Next major release, 1.2 (1.1 contained mostly bugfixes), appeared two months later, and introduced mail sending rate control (see Rate Limit).
The program evolved during the next year, and the version 2.0 was
released in September, 2006. This version was a major change in the
main idea of the program. Configuration file become a flexible filter
script allowing the operator to control almost all SMTP states. The
program supplied in the script file was compiled into a pseudo-code at
startup, this code being subsequently evaluated each time the filter
was invoked. This caused a considerable speed-up in comparison with
the previous versions, where the run-time evaluator was traversing the
parse tree. This version also introduced (implicitly, at the time),
two separate data types for the entities declared in the script, which
also played its role in the speed improvement (in the previous
versions all data were considered strings). Lots of improvements were
made in the filter language (MFL, see MFL) itself, such
as user-defined functions, the
switch statement, the
statement for handling run-time errors, etc. The set of
built-in functions extended considerably. A testsuite (using
DejaGNU) was introduced in this version.
During this initial development period the limitations
libmilter implementation became obvious. Finally,
I felt they were stopping further development, and decided
mailfromd should use its own ‘Milter’
implementation. This new library,
libgacopyz was the main
new feature of the 3.0 release, which was released in November, 2006.
Another major feature was the --dump-macros option and
rc.mailfromd script, that were intended
to facilitate the configuration on ‘Sendmail’ side.
The development of 3.x (more properly, 3.1.x) series concentrated mainly on bug-fixes, while the main development was done on the next branch.
The version 4.0 appeared on May 12, 2007. A full
list of changes in this release is more than 500 lines long, so it is
impractical to list them here. In particular, this version introduced
lots of new features in MFL syntax and the
library of useful MFL functions. The runtime engine was
also improved, in particular, stack space become expandable which
eliminated many run-time errors. This version also provided a
foundation for MFL module system. The code generation
was re-implemented to facilitate introduction of object files in
future versions. Another new features in this release include
SPF support and
mtasim utility — an MTA
simulator designed for testing
(see mtasim). The test suite in this version was made portable by
rewriting it in Autotest.
Another big leap forward was the 5.0 release, which appeared on December 26, 2008. It largely enriched a set of available functions (61 new functions were introduced, which amounts to 41% of all the available functions in 5.0 release) and introduced several improvements in the MFL itself. Among others, function aliases and optional arguments in user-defined functions were introduced in this release. The new “run operation mode” allowed to execute arbitrary MFL functions from the command line. This release also raised the Mailutils version requirements to at least 2.0.
Version 6.0, which was released in on 12 December, 2009, introduced a full-fledged modular system, akin to that of Python, and quite a few improvements to the language. such as explicit type casts, concatenation operator, static variables, etc.
Starting from version 7.0, the focus of further development of
mailfromd has shifted. While previously it had been regarded
as a mail-filtering server, since then it was developed as a system for
extending MTA functionality in the broad sense, mail
filtering being only one of features it provides.
Version 7.0 makes the MFL syntax more consistent and the language itself more powerful. For example, it is no longer necessary to use prefixes before variables to dereference them. The new ‘try--catch’ construct allows for elegant handling of exceptions and errors. User-defined exceptions provide a way for programming complex loops and recursions with non-local exits.
This version introduces a concept of dedicated callout server.
mailfromd to defer verifications for a later
time if the remote server does not response within a reasonably short
period of time (see SMTP Timeouts).
Six years later the version 8.0 was released. This version was a major rewrite of the mailfromd codebase. It introduced a separate callout daemon that made it possible to separate the mailfromd server machine from machines performing callout checks. The MFL language was extended by a number of built-in functions.
Since version 8.3 (2017-11-02)
The version 8.7 released in July, 2020 introduced DKIM support.
Many people need to be thanked for their assistance in developing
mailfromd. After S. C. Johnson, I can say
that this program “owes much to a most stimulating collection of
users, who have goaded me beyond my inclination, and frequently beyond
my ability in their endless search for "one more feature". Their
irritating unwillingness to learn how to do things my way has usually
led to my doing things their way; most of the time, they have been right.”
A real test for a program like
mailfromd cannot be done
but in conditions of production environment. A decision to try it
in these conditions is by no means an easy one, it requires courage
and good faith in the intentions and abilities of the author. To
begin with, I would like to thank my contributors for these virtues.
Jan Rafaj has intrepidly been using
mailfromd since its
early releases and invested lots of efforts in improving the program
and its documentation. He is the author of many of the MFL
library functions, shipped with the package. Some of his ideas are
still waiting in my implementation queue, while new ones are
Peter Markeloff patiently tested every
release and helped discover and fix many bugs.
Zeus Panchenko contributed many ideas and gave lots of helpful
comments. He offered invaluable help in debugging and testing
mailfromd on FreeBSD platform.
Sergey Afonin proposed many improvements and new ideas. He also invested a lot of his time in finding bugs and testing bugfixes.
John McEleney and Ben McKeegan contributed the token bucket filter implementation (see TBF).
Con Tassios helped to find and fix various bugs and contributed the
new implementation of the
greylist function (see greylisting types).
The following people (in alphabetical order) provided bug reports and helpful comments for various versions of the program: Alan Dobkin, Brent Spencer, Jeff Ballard, Nacho González López, Phil Miller, Simon Christian, Thomas Lynch.
This document was generated on May 26, 2021 using makeinfo.Verbatim copying and distribution of this entire article is permitted in any medium, provided this notice is preserved.