Version 7.0

Item posted by Sergey Poznyakoff <gray> on Sat Aug 7 13:12:45 2010.

Mailfromd version 7.0 is available for download.

This version introduces a lot of improvements and new features over the previous version 6.0.1.

Incompatible changes

The following features, that had been marked as deprecated in 6.0, are now removed:

• old-style functional notation
• the use of functional operators
• implicit concatenations
• #pragma option
• #pragma database

Milter and Callout servers

This release introduces a notion of a `server', i.e. a special mailfromd entity responsible for handling a particular task. Two kinds of servers are supported: Milter servers, which handle the milter requests, and Callout servers, which run callout (or sender verification) SMTP sessions and update the cache database accordingly.

When a callout server is enabled, sender verification functions work the following way. First, the usual sender verification is performed with a set of so-called soft timeout values. If this verification yields a definite answer, that answer is stored in the cache database and returned to the calling procedure as usual. In that regard, this release works exactly as its predecessors did. If, however, the verification is aborted due to a timeout, the caller procedure is returned the e_temp_failure exception, and the session is scheduled for processing by a callout server. The latter processes the request using a set of hard timeouts, which are normally much longer than the soft ones (their default values are those required by RFC 2822; see below for a detailed description). This callout session runs independently of the milter session that initiated it and which, having initiated the callout, returns a temporary error to the sender, urging it to retry the connection later. In the meantime, the callout server has a chance to finish the requested sender verification and store its result in the cache database. When the sender retries the delivery, the milter server will obtain the already cached result from the database. If the callout server has not finished the request by the time the sender retries the connection, the latter is again returned a temporary error, and the process continues until the callout is finished.

Milter servers are declared using the following configuration statement:

server <type: milter | callout> {
  # Server ID.
  id <arg: string>;
  # Listen on this URL.
  listen <url: string>;
  # Maximum number of instances allowed for this server.
  max-instances <arg: number>;
  # Single-process mode.
  single-process <arg: boolean>;
  # Reuse existing socket (default).
  reuseaddr <arg: boolean>;
  <acl-statement>
}

If the type is callout, the server block statement may also contain the following sub-statement:

  default <arg: boolean>;

When arg is yes, this server is marked as the default callout server for all milter servers declared in the configuration.

Alternatively, you may use a remote callout server, i.e. a separate instance of mailfromd, dedicated for that purpose. In that case, the address of a callout server is declared using the following statement:

  callout-url <url: string>;

Timeout control for callout SMTP sessions

Callout SMTP sessions initiated by polling functions are controlled by two sets of timeouts: soft and hard. Soft timeouts are used by the mailfromd milter servers. Hard timeouts are used by callout servers. When a soft timeout is exceeded, the calling procedure is delivered the e_temp_failure exception and the session is scheduled for processing by a callout server. The latter re-runs the session using hard timeouts. If a hard timeout is exceeded, the address is marked as not_found and is stored in the cache database with that status.

Normally, soft timeouts are set to shorter values, suitable for the use in MFL scripts without disturbing the calling SMTP session. Hard timeouts are set to large values, as requested by RFC2822, which guarantee obtaining a definite answer (see below for the default values).

Individual timeouts may be set in the configuration file using the following statement:

smtp-timeout [soft | hard] {
  # Initial SMTP connection timeout.
  connection <time: string>;
  # Timeout for initial SMTP response.
  initial-response <time: string>;
  # Timeout for HELO resonse.
  helo <time: string>;
  # Timeout for MAIL response.
  mail <time: string>;
  # Timeout for RCPT response.
  rcpt <time: string>;
  # Timeout for RSET response.
  rset <time: string>;
  # Timeout for QUIT response.
  quit <time: string>;
};

The default timeout settings are:

 Timeout            Soft     Hard
 ---------------------------------
 connection         10s       5m 
 initial-response   30s       5m
 helo               I/O       5m
 mail               I/O      10m
 rcpt               I/O       5m
 rset               I/O       5m
 quit               I/O       2m

The entries marked with I/O, unless set explicitly by a corresponding smtp-timeout soft entry, are set to the value of the I/O timeout (see the io-timeout configuration statement and the --timeout command line option), which defaults to 3s.

Automatic export of the "i" macro

The "i" Sendmail macro is automatically requested for the lowest used stage handler. This ensures the log messages are marked with the corresponding queue ID.

Notice, however, that the MTA is free to honor or ignore the request. Generally speaking, this works well with Sendmail starting from 8.14.0 and MeTA1 starting from 1.0.PreAlpha29.0. The "i" macro is alost useless with Postfix 2.6 or higher, because it defines it only at the EOM stage.

Language Changes

The use of % in front of identifiers

It is no longer necessary to use % in front of an identifier. To
reference a variable, simply use its name, e.g.:

   set x var + z

However, old syntax is still supported, so the following statement will also work:

   set x %var + %z

It will, however, generate a warning message.

Of course, the use of % to reference variables within strings remains mandatory.

User-defined exceptions

In addition to the built-in exception codes, you may define your
own exceptions. For example:

  dclex myerror

This statement declares a new exception identifier myerror. This identifier may then be used in catch and throw statements, e.g.:

  throw myerror "My error encountered"

For a detailed discussion, see the manual, subsection 4.19.2
"User-defined Exceptions".

The try-catch construct

This version introduces a try-catch construct similar to that used in another programming languages. The construct is:

  try
  do
    STMTLIST-1
  done
  catch EX-LIST
  do
    STMTLIST-2
  done

where SMTLIST-1 and STMTLIST-2 are lists of MFL statements and EX-LIST is a list of exceptions. The control flow is as follows. First, the statements from STMTLIST-1 are executed. If the execution finishes successfully, control is passed to the first statement after the catch block. Otherwise, if an exception is signalled and this exception is listed in EX-LIST, the execution is passed to the STMTLIST-2.

The try-catch construct allows a better and more flexible error recovery.

For a detailed discussion, see the manual, subsection 4.19.3
"Exception Handling".

Handling of stderr in open("|...")

If the open function is used to start an external program (i.e. the argument to open() begins with a `|' or `|&'), the standard error of the program is closed prior to starting it. The following special constructs are provided for redirecting it to an output file or syslog:

- "|2>null: COMMAND"
Standard error is redirected to /dev/null.

- "|2>file:NAME COMMAND
Standard error is redirected to the file NAME. If the file exists, it will be truncated.

- "|2>>file:NAME COMMAND
Standard error is appended to the file NAME. If file does not exist, it will be created.

- "|2>syslog:FACILITY COMMAND"
- "|2>syslog:FACILITY.PRIORITY COMMAND"
Standard error is redirected to the given syslog facility and, if specified, priority. If the latter is omitted, LOG_ERR is assumed.

Message Modifications and Accept

Calling accept undoes any modifications to the message applied by, e.g., header_add and the like. This is due to requirements of the Milter protocol.

This behavior caused several false bug reports in the past. Starting with this version, calling accept after any modifications to the message results in the following warning message:

RUNTIME WARNING near /etc/mailfromd.mf:36: `accept' causes previous message modification commands to be ignored; call mmq_purge() prior to `accept', to suppress this warning

It is only a warning and the accept action itself is, of course, honored. If you see this diagnostics in your log, do the following:

*if the behavior was intended, call mmq_purge, as suggested (see
below for a description of this function);
*if it was not, read the manual, section 5.8, "Message Modification Queue", for information on how to handle it.

#pragma miltermacros

The new #pragma miltermacros declares macros that are referenced at a given stage. It is helpful when automatic macro negotiation is in use and mailfromd is unable to trace all macros referenced from each handler, i.e. when at least one of the handlers:

• calls functions that reference MTA macros;
• or refers to macros via macro_defined() and getmacro() functions.

See the manual, subsection 4.2.5, "Pragma miltermacros", for a
detailed description.

MFL Functions

progress

The progress function notifies the MTA that the filter is still processing the message and urges it to restart its timeouts. It is available only in the eom handler.

sleep

The sleep function takes an optional second argument. If given, it specifies the number of microseconds to wait. For example, to sleep for 1.5 seconds, use:

   sleep(1,500000)

dequote

string dequote(string email)

The function dequote removes angle brackets surrounding its argument and returns the resulting string. If there are no anglebrackets, or if they are unbalanced, it returns unchanged argument.

debug_spec

string debug_spec([string modnames[, number minlevel]])

Returns the current debugging level specification. Optional parameters supply conditions for abridging the amount of information returned.

The minlevel parameter instructs the function to return only those specifications that have the level part greater than or equal to the given value. The modnames parameter (a comma-separated list of module name parts) selects only those specification that match the supplied module names.

mmq_purge

The function mmq_purge purges internal message modification queue. This undoes the effect of the following functions, if they had been called previously: rcpt_add, rcpt_delete, header_add, header_insert, header_delete, header_replace, replbody, quarantine.

New built-in constant _git_

The __git__ built-in constant is defined for alpha versions only. Its value is the GIT tag of the recent commit corresponding to that version of the package.

smap

The smap utility has been removed. A new project was established
that provides and largely expands its functionality. See
http://smap.software.gnu.org.ua, for additional information,
including links to file downloads.

mtasim

New option --milter-timeout sets timeouts for Milter I/O operations.