SMAP
	A Socket Map Server	Sergey Poznyakoff

• Main
• Downloads
• Documentation

Smap Manual (split by chapter):

Section:

Chapter:

Doc:

?

6. Modules Shipped with Smap

Smap is shipped with a set of loadable modules, which are installed in its default module directory, ‘$prefix/lib/smap’. The modules are configurable on a per-module (see section Loadable Modules), and per-database (see section Databases) levels.

Smap version 1.1 contains the following modules: ‘echo’, ‘mailutils’ and ‘guile’. These are described in detail in the following sections.

6.1 Echo

The echo module is the simplest of all modules. It sends back a static reply string, no matter what the query was. This module is useful for default databases, which catch erroneous or not handled queries.

Loading

The module needs no additional arguments for initialization. Normal loading statement is:

module echo echo

Database

Database initialization function treats its arguments as a string to be sent in reply to all queries. An example database definition:

database default echo NOTFOUND [no such map]

Such a definition is normally used as a target of the ‘default’ dispatch rule:

dispatch default database default

6.2 Mailutils

This module uses GNU Mailutils (http://www.gnu.org/software/mailutils) and provides two main modes:

‘auth’

This mode uses GNU Mailutils authorization mechanism to obtain user data (similar to the system ‘getpwnam’ routine) and returns positive reply if the data were retrieved and negative reply otherwise. See section Example: Using smapd with MeTA1, for an example on how to use it as a local user and alias database.

‘mbq’

This mode allows to check whether the user's mailbox exceeded the allotted quota, and if not, whether it is able to accept a message of the given size without exceeding it. The mode name is an abbreviation of Mailbox Quota.

6.2.1 Variable Expansion

In the discussion below we often refer to meta-variable expansion in strings. This is a process, whereby any sequence ‘${variable}’ is replaced with the value of variable. The defined variables are:

db

The database name.

map

The map name.

key

The lookup key.

diag

If the key was not found or some error occurred, this variable expands to a short diagnostics string, suitable for return message. Otherwise, expands to empty string.

name

The ‘name’ field from the retrieved record. Empty string if the user not found.

passwd

The ‘passwd’ field from the retrieved record. Empty string if the user not found.

uid

The ‘uid’ field from the retrieved record. If the user was not found, expands to ‘-1’.

gid

The ‘gid’ field from the retrieved record. If the user was not found, expands to ‘-1’.

gecos

The ‘gecos’ field from the retrieved record. Empty string if the user not found.

dir

The ‘dir’ field from the retrieved record. Empty string if the user not found.

shell

The ‘shell’ field from the retrieved record. Empty string if the user not found.

mailbox

The ‘mailbox’ field from the retrieved record. Empty string if the user not found.

quota

The ‘quota’ field from the retrieved record. If the user was not found, expands to ‘NONE’.

mbsize

Mailbox size, in bytes. Defined only in ‘mbq’ mode.

msgsize

Expected message size, in bytes. Defined only in ‘mbq’ mode.

6.2.2 Mailutils Loading Sequence

module mailutils mailutils [args]

Arguments are:

‘config-verbose’

Verbosely trace the processing of the main Mailutils configuration files.

‘config-dump’

Dump the parse tree from the Mailutils configuration.

‘positive-reply=str’

Declare default positive reply string. This string is returned when the underlying database was able to found the requested key. Prior to returning, str is subject to meta-variable expansion, as described above.

Default positive reply string is ‘OK’.

‘negative-reply=str’

Declare default negative reply string. This string is returned when the underlying database failed to found the requested key. Prior to returning, str is subject to meta-variable expansion.

Default negative reply string is ‘NOTFOUND’.

‘onerror-reply=str’

Declare a reply to be returned on error. Prior to returning, str is subject to meta-variable expansion. Default string is ‘NOTFOUND’.

The module reads most of its configuration settings from the main Mailutils configuration file. See Mailutils Configuration File: (mailutils)configuration section `configuration' in GNU Mailutils Manual, for a description of GNU Mailutils configuration system. It looks for smap-specific settings in the section ‘program smap-mailutils’.

The module uses GNU Mailutils authorization databases to obtain the requested data. This concept is described in detail in Mailutils Configuration File: (mailutils)Auth Statement section `Auth Statement' in GNU Mailutils Manual.

6.2.3 Mailutils Databases

Mailutils databases are normally declared as follows:

database name mailutils mode=mode [args]

Here, name is the database name, mode is ‘auth’ if the database should work in auth mode, and ‘mbq’ if it should run in mbq mode. If the ‘mode’ argument is omitted, ‘auth’ is assumed. Optional args may be used to supply additional database configuration. These are:

‘positive-reply=str’

Declare positive reply string. This string is returned when the underlying database was able to found the requested key. Prior to returning, str is subject to meta-variable expansion, as described above.

Default positive reply string is ‘OK’, unless overridden by the module-level ‘positive-reply’ option (see section positive-reply.

‘negative-reply=str’

Declare negative reply string. This string is returned when the underlying database failed to found the requested key. Prior to returning, str is subject to meta-variable expansion.

Default negative reply string is ‘NOTFOUND’, unless overridden by the module-level ‘positive-reply’ option (see section negative-reply.

‘onerror-reply=str’

Declare a reply to be returned on error. Prior to returning, str is subject to meta-variable expansion. Default string is ‘NOTFOUND’, unless overridden by the module-level ‘positive-reply’ option (see section onerr-reply.

6.2.4 Mailutils Auth Mode

Mailutils module in ‘auth’ mode uses GNU Mailutils authorization mechanism to obtain user data. It returns ‘positive-reply’ if the data were retrieved and ‘negative-reply’ otherwise. This mode is often used for databases of local users and aliases. The key is normally a user name (either local part or fully qualified).

See section Example: Using smapd with MeTA1, for an example on how to use it.

6.2.5 Mailutils MBQ Mode

MBQ, or Mailbox Quota mode, uses key as the name of a local user. It obtains the user parameters via Mailutils authorization mechanism and then switches to this user privileges and opens his mailbox for a brief period of time. After opening it determines the mailbox size and closes it. The mode returns ‘positive-reply’ if the mailbox size is less than the quota, and ‘netagive-reply’ otherwise.

If the key value consists of two words, separated by whitespace, then the first word is used as a user name, and the second one as a size of a message which is about to be delivered to that user's mailbox (the size may be optionally prefixed by ‘SIZE=’). In this case, ‘positive-reply’ is returned if the actual mailbox size plus the message size is less than quota.

Two additional meta-variables may be used in reply templates to return quota-related information:

mbsize

Mailbox size, in bytes. Defined only in ‘mbq’ mode.

msgsize

Expected message size, in bytes. Defined only in ‘mbq’ mode.

The following example shows a definition of ‘mbq’ database which the author uses on his servers:

database mbq mailutils mode=mbq \
  positive-reply="OK [${diag}] ${mailbox} ${mbsize} ${quota}"\
  negative-reply="NOTFOUND [${diag}] ${mailbox} ${mbsize} ${quota}"\
  onerror-reply="NOTFOUND [${diag}]"

The ‘diag’ meta-variable contains a diagnostic string suitable for passing it back to the MTA. For example, in the case of ‘negative-reply’, ‘${diag}’ expands to:

mailbox quota exceeded for this recipient

if the mailbox has grown beyond the allowed quota, and

message would exceed maximum mailbox size for this recipient

if message of the given size cannot be delivered to mailbox without violating its quota.

Notice, that this mode requires superuser privileges.

6.3 Guile

Guile is an acronym for GNU's Ubiquitous Intelligent Language for Extensions. It provides a Scheme interpreter conforming to the R5RS language specification and a number of convenience functions. For information about the language, refer to (r5rs)Top section `Top' in Revised(5) Report on the Algorithmic Language Scheme. For a detailed description of Guile and its features, see (guile)Top section `Overview' in The Guile Reference Manual.

The guile module provides an interface to Guile which allows for writing Smap modules in Scheme. The module is loaded using the following configuration file statement:

module name guile [args]

Optional args are:

‘debug’

Enable Guile debugging and stack traces.

‘nodebug’

Disable Guile debugging and stack traces (default).

‘load-path=path’

Append directories from path to the list of directories which should be searched for Scheme modules and libraries. The path must be a list of directory names, separated by colons.

This option modifies the value of Guile's %load-path variable. See the section Configuration and Installation in the Guile Reference Manual.

‘init-script=script’

Specifies the name of a Scheme source file that must be loaded in order to initialize the module. The file is looked up using ‘%load-path’ variable.

‘init-args’

The init-args parameter supplies additional arguments to the module. They will be accessible to the ‘script’ via the command-line function.

‘init-fun’

This parameter specifies the name of a function that will be invoked to perform the initialization of the module and of particular databases. Default name is ‘init’. See section Guile Initialization, for a description of initialization sequence.

Guile databases are declared using the following syntax:

database dbname modname [args] [cmdline]

where: dbname gives the name for this database and modname is the name given to Guile module in module statement (see above).

Optional args override global settings given in the module statement. The following options are understood: init-script, init-args, and init-fun. Their meaning is the same as for module statement (see above), except that they affect only this particular database.

Any additional arguments, referenced as cmdline above, are be passed to the Guile open-db callback function (see open-db).

6.3.1 Virtual Functions

Any database handled by guile module is associated with a virtual function table. This table is an association list which supplies to the module the Scheme call-back functions implemented to perform particular tasks on that database. In this list, the car of each element contains the name of a function, and its cdr gives the corresponding function. The defined function names and their semantics are described in the following table:

init

Initialize the module.

done

Close the module, releasing any resources held by it.

open

Open the database.

close

Close the database.

query

Handle a socket map query

xform

Handle a transformation request (see section Transformations).

For example, the following is a valid virtual function table:

(list (cons "open" open-module)
      (cons "close" close-module)
      (cons "query" run-query))

Apart from per-database virtual tables, there is also a global virtual function table, which is used to supply the entries missing in the former. Both tables are created during the module initialization, as described in the next subsection.

Particular virtual functions are described in Guile API.

6.3.2 Guile Output Ports

Guile modules are executed in a specially prepared environment. Current error port is redirected so that everything written to it ends up in the smapd error stream. So, if smapd is writing its log to syslog, everything you write to ‘(current-error-port)’ will be written to syslog as well. The port is line-buffered. For example, the following code:

(with-output-to-port
  (current-error-port)
 (lambda ()
   (display "The diagnostics follows:")
   (newline)
   (display "Module opened")
   (newline)))

will result in two lines in your syslog file, which will look like

Jun 19 12:49:05 netbox smapd[7503]: The diagnostics follows
Jun 19 12:49:05 netbox smapd[7503]: Module opened

For any debugging output, use smap-debug-port. This port is configured so that everything written to it is explicitly marked as being debug output. If smapd logs to stderr, it will be prefixed with ‘DEBUG:’, and if it logs to syslog, the output will be logged with ‘LOG_DEBUG’ priority.

Finally, current output port is closed for any functions, excepting ‘query’ (see query-db). For ‘query’ function, it is redirected so that anything written to it is reformatted according to the socket map protocol (see section The Sockmap Protocol) and sent back as a reply to the client.

6.3.3 Guile Initialization

The module configuration statement causes loading and initialization of the guile module:

module modname guile [init-script=‘script’] \
                           [init-fun=function"]

Upon module initialization stage, the module attempts to load the file named ‘script’. The file is loaded using primitive-load-path call (see primitive-load-path: (guile)Loading section `Loading' in The Guile Reference Manual), i.e. it is searched in the Guile load path. The init-fun parameter supplies the name of the initialization function. This Scheme function returns virtual function tables for the module itself and for each database that uses this module. It must be declared as follows:

(define (function arg)
  …)

This function is called several times. First of all, it is called after script is loaded. This time it is given #f as its argument, and its return value is saved as a global function table. Then, it is called for each database statement that uses module modname (defined in the module statement above), e.g.:

database dbname modname …

This time, it is given dbname as its argument and its return is stored as the virtual function table for this particular database.

The following example function returns a complete virtual function table:

(define (my-smap-init arg)
  (list (cons "init" db-init)
        (cons "done" db-done)
        (cons "open" db-open)
        (cons "close" db-close)
        (cons "query" db-query)
        (cons "xform" db-xform)))

6.3.4 Guile API

This subsection describes callback functions that a Guile database module must provide. The description of each function begins with the function prototype and its entry in the virtual function table.

Guile Callback: open-db name . args

Virtual table: (cons "open" open-db)

Open the database. The argument name contains database name as given in dbname of the database declaration (see section Databases). Optional argument args is a list of command line parameters obtained from cmdline in database statement (see guile-cmdline). For example, if the configuration file contained:

database foo guile db=file 1 no

then the open-db callback will be called as:

(open-db "foo" '("db=file" "1" "no"))

The open-db callback returns a database handle, i.e. an opaque Scheme object which identifies this database, and keeps its internal state. This value, hereinafter named dbh, will be passed to another callback functions that need to access the database.

The unspecified return value indicates an error.

Guile Callback: close-db dbh

Virtual Table: (cons "close" close-db)

Close the database. This function is called during the cleanup procedure, before termination of smapd child process. The argument dbh is a database handle returned by open-db.

The return value from close-db is ignored. To communicate errors to the daemon, throw an exception.

Guile Callback: query-db dbh map key . rest

Virtual Table: (cons "close" close-db)
Perform the query. Arguments are:

dbh

A database handle returned by open-db.

map

The map name.

key

The lookup key

rest

If this query came over a UNIX socket, this argument is ‘()’. Otherwise, if the query came over an INET socket, rest is a list of two network socket addresses (see (guile)Network Socket Address section `Network Socket Address' in The Guile Reference Manual): first element is the address of the remote party (client), second element is the address of the server that is handling the query.

This function must write the reply, terminated with a newline, to the current output port, e.g.:

(define-public (smap-query handle map arg . rest) (display "NOTFOUND") (newline))

Guile Callback: xform-db dbh arg . rest

Virtual Table: (cons "xform" xform-db)
Transform the argument arg. Arguments dbh and rest have the same meaning as in query-db.

Returns transformed value or ‘#f’ if no transformation applies. This callback may be used to alter map or key values using ‘guile’ module (see section Transformations). The following example function removes optional domain part from its argument:

(define (smap-xform handle arg . rest) (let ((arg-parts (string-split arg #\@))) (if (null? (cdr arg-parts)) #f (car arg-parts))))

The following snippet from the ‘smapd.conf’ file shows how to apply it:

database localpart guile init-script=local.scm dispatch key like *@* transform key localpart

6.4 Mysql

The mysql module provides interface to MySQL database management system. It may be used to build smap databases over SQL ones.

The SQL database to use may be configured either globally, when loading the module, or locally, when defining a smap database. If a database definition lacks SQL configuration statements, then it attempts to use a globally defined connection.

Each database is configured with a SQL query template, and a set of smap reply templates to use. When dispatched a sockmap query, the database expands the SQL query template using the actual values of ‘${map}’ (the map name) and ‘${key}’ (the key value) and sends the expanded query to the | MySQL server. If the server responds with a non-empty set of tuples, the positive reply template is expanded and the result is used as a response. Otherwise, if the query produced an empty set, the smap database uses the negative reply template to create the response.

6.4.1 MySQL Configuration

The SQL database is configured using the following options:

‘config-file=file’

Set the name of the MySQL configuration file to read. By default ‘/etc/my.cnf’ is used.

‘config-group=name’

Set the name of the group in the MySQL configuration file, from where to read the configuration options.

The statements above allow to keep all security-sensitive information, such as MySQL username and password, in an external configuration file and thus to relax permission requirements for ‘smapd.conf’. For a detailed description of the format of such external configuration file (or option file in ‘MySQL’ parlance), see MySQL option files.

In case the use of option files is not feasible for some reason, you may specify MySQL connection and database parameters in ‘smapd.conf’ when loading the mysql module or defining a smap database. The following options are used to define MySQL connection parameters:

‘host=hostname’

Sets the hostname or IP address of the host running the MySQL server.

‘port=n’

Sets port number the MySQL server is listening on. Default is 3306.

‘socket=file’

Sets the socket name, if the server is listening on a UNIX socket.

‘ssl-ca=file’

Sets the pathname to the certificate authority file, if you wish to use a secure connection to the server via SSL.

Notice, that either ‘host’ and, optionally, ‘port’ or ‘socket’ must be used. Specifying both is senseless.

MySQL database and user credentials are defined using the following options:

‘database=name’

Sets the name of the MySQL database to use.

‘user=name’

Sets MySQL user name.

‘password=string’

Sets the password for accessing the MySQL database.

When using these options, it is reasonable to tighten the permissions on ‘smapd.conf’ so that no third person could see the MySQL password. The recommended permissions are ‘0600’.

6.4.2 MySQL Query and SMAP Replies

MySQL query is defined using the following option:

‘query=template’

Define MySQL query template.

The template may reference the following variables:

Table 6.1: MySQL query template variables

For example:

database alias mysql \
  query="SELECT alias FROM aliases WHERE email='$key'"

If the database definition lacks the query option, it will attempt to use one from the module statement. If the module statement lacked it as well, an error is reported.

Reply templates define the responses to be given. They are given by the following options:

‘positive-reply=template’

Defines a reply to be sent if the query returned a non-empty set of tuples. In addition to the variables described above (see mysql-query-vars), the template may also refer to the MySQL result columns, by using their names from the ‘SELECT’ part of the query. For example:

database alias mysql \ query="SELECT alias FROM aliases WHERE email='$key'" \ positive-reply="OK $alias"

The default positive-reply is ‘OK’.

‘negative-reply=template’

Defines a reply to be sent if the query returned an empty set of tuples. The template may refer to the variables described in mysql-query-vars.

Default value is ‘NOTFOUND’.

‘onerror-reply=template’

Defines a reply to be sent if an error occurred when executing the query. The template may refer to the variables described in mysql-query-vars.

Default value is ‘NOTFOUND’.

6.5 Postgres

The postgres module provides interface to PostgreSQL database management system. It may be used to build smap databases over SQL ones.

The module is in many regards similar to mysql module, described above. In particular, its overall functionality is exactly the same as described in Mysql, except, of course, that it uses PostgreSQL databases.

6.5.1 Postgres Configuration

A Postgres database is configured using a set of options understood by the Postgres PQconnectdb function. See http://www.postgresql.org/docs/8.4/static/libpq-connect.html, for a detailed description. The following is a short summary of the most useful options:

‘host=name’

Name of host to connect to. If this begins with a slash, it specifies Unix-domain communication rather than TCP/IP communication; the value is the name of the directory in which the socket file is stored.

‘hostaddr=ip’

Numeric IP address of host to connect to.

‘port=number’

Port number to connect to at the server host, or socket file name extension for Unix-domain connections.

‘dbname=name’

The database name.

‘user=name’

PostgreSQL user name to connect as. Defaults to be the same as the operating system name of the user running the smapd.

‘password=string’

Password to be used if the server demands password authentication.

‘connect_timeout=number’

Maximum wait for connection, in seconds. Zero or not specified means wait indefinitely.

‘options=string’

Any additional command-line options to send to the server at run-time. For example, setting this to ‘-c geqo=off’ sets the session's value of the ‘geqo’ parameter to ‘off’. For a detailed discussion of the available options, see Postgres documentation(7).

‘sslmode=mode’

This option determines whether or with what priority an SSL TCP/IP connection will be negotiated with the server. There are six modes: ‘disable’, ‘allow’, ‘prefer’, ‘require’, ‘verify-ca’ and ‘verify-full’(8).

‘sslcert=file’

This parameter specifies the file name of the client SSL certificate.

‘sslkey==file-or-engine-name’

This parameter specifies the location for the secret key used for the client certificate.

‘sslrootcert=file’

This parameter specifies the file name of the root SSL certificate.

‘sslcrl=name’

This parameter specifies the file name of the SSL certificate revocation list (CRL).

‘krbsrvname=name’

Kerberos service name to use when authenticating with Kerberos 5 or GSSAPI.

‘service=name’

Service name to use for additional parameters.

6.5.2 Postgres Query and SMAP Replies

Postgres SQL query and the smap replies are configured the same way as for mysql module (see section MySQL Query and SMAP Replies). The following is a short summary:

‘query=template’

Define the Postgres query template. The template may reference the following variables:

Variable	Meaning
map	Name of the map being queried
key	Lookup key

Table 6.2: Postgres query template variables

If the database definition lacks the query option, it will attempt to use one from the module statement. If the module statement lacked it as well, an error is reported.

‘positive-reply=template’

Defines a reply to be sent if the query returned a non-empty set of tuples. In addition to the variables described above (see postgres-query-vars), the template may also refer to the column names from the SQL result set.

The default positive-reply is ‘OK’.

‘negative-reply=template’

Defines a reply to be sent if the query returned an empty set of tuples. The template may refer to the variables described in postgres-query-vars.

Default value is ‘NOTFOUND’.

‘onerror-reply=template’

Defines a reply to be sent if an error occurred when executing the query. The template may refer to the variables described in postgres-query-vars.

Default value is ‘NOTFOUND’.

6.6 Sed

The ‘sed’ module applies sed-like s-expressions to strings to modify them. It is designed mainly for use in transformation rules (see section Transformations).

6.6.1 Loading sed module

module sed sed [args]

The following arguments may be given in the statement above:

‘icase’

Use case-insensitive expressions.

‘noicase’

Use case-sensitive expressions. This is the default.

‘extended’

Use extended regular expressions. This is the default.

‘noextended’

Use basic regular expressions.

Although sed module is designed for transformations in the first place, it may also be used as a conventional lookup database (see section Using Sed for Lookups). The following options modify its behavior in this mode:

‘positive-reply=str’

Use str for positive replies.

‘negative-reply=str’

Use str for negative replies.

‘onerror-reply=str’

Reply with str if an error occurred.

6.6.2 Defining Sed Databases

The definition of a sed databases requires a single argument: the s-expression to be applied. For example:

database dequote sed 's/<(.*)>/\1/g'

Be sure to properly quote the expression, especially if it contains backreferences. It is preferable to use single quotes, to avoid duplicating each backslash in the expression, as shown in the example below. If the expression itself contains single quote, you may either use double-quotes to quote the entire expression:

database foo sed "s/'utf8'(.*)/u8_\\1/"

or use escaped single quotes outside of quoted expression (a technique familiar for shell programmers):

database foo sed 's/'\''utf8'\''(.*)/u8_\1/'

All options valid for module initialization (see section Loading sed module) may also be used in database declarations. When used so, they take precedence over module initialization options. For example, the following database definition uses basic case-insensitive regular expressions:

database bar sed noextended noicase 's/test(\([^)]\))/\1/g'

6.6.3 S-expressions

The transformation expression is a sed-like replace expression of the form:

s/regexp/replace/[flags]

where regexp is a regular expression, replace is a replacement for each part of the input that matches regexp. Both regexp and replace are described in detail in The "s" Command: (sed)The "s" Command section `The `s' Command' in GNU sed.

As in sed, you can give several replace expressions, separated by a semicolon.

Supported flags are:

‘g’

Apply the replacement to all matches to the regexp, not just the first.

‘i’

Use case-insensitive matching

‘x’

regexp is an extended regular expression (see Extended regular expressions: (sed)Extended regexps section `Extended regular expressions' in GNU sed).

‘number’

Only replace the numberth match of the regexp.

Note: the POSIX standard does not specify what should happen when you mix the ‘g’ and number modifiers. The sed module follows the GNU sed implementation in this regard, so the interaction is defined to be: ignore matches before the numberth, and then match and replace all matches from the numberth on.

Any delimiter can be used in lieue of ‘/’, the only requirement being that it be used consistently throughout the expression. For example, the following two expressions are equivalent:

s/one/two/
s,one,two,

Changing delimiters is often useful when the regex contains slashes. For instance, it is more convenient to write s,/,-, than s/\//-/.

6.6.4 Using Sed for Lookups

The sed module is designed primarily for argument transformation. Nevertheless, it may also be used to define simple look-up databases. When used in a database clause of a dispatch rule, the module behaves as follows. The s-expression is applied to the key. If the result differs from the input key, the ‘positive-reply’ is returned. It the result is the same as the input key, ‘negative-reply’ is returned. If some error occurred, ‘onerror-reply’ is returned. The reply strings may be supplied as arguments to the database definition or to the module loading statement. The following variables are expanded within these strings:

‘map’

The map name.

‘key’

The key value.

‘xform’

Transformed key value. This variable is not defined for ‘onerror-reply’.

Default replies are:

?