automx2: Email client configuration made easy

This document explains how automx2 works, how Automated Mailbox Configuration works and what it takes to install and configure automx2. If you are already familiar with automated mailbox configuration methods you may want skip the following sections and jump right ahead to automx2 Installation and Configuring automx2.

1. How does automx2 work?

automx2 is a webservice. It sits behind a web server, e.g. NGINX, and waits for configuration requests. When a mail client requests configuration it contacts the web server. The web server then acts as a proxy and forwards all requests to the web service aka automx2.

2. How does Automated Mailbox Configuration work?

Modern mail clients can look for configuration data when a user begins to create a new account. They will either send the user’s mail address to a service and ask the service to reply with configuration that suits the user’s profile or they will query the DNS system for advice.

Using a specialized mail account configuration service allows for individualized setups. It also allows to enforce a specific policy, which for example configures the mail client to use a specific authentication mechanism. Quering the DNS for mail service locations allows for generic instructions, but it doesn’t give as much control over settings as a specialized service like automx2 will do.

As of today, there are four methods that help configuring a mail account. Three of them – autoconfig, autodiscover and mobileconfig – have been developed by vendors to cover their products' specific needs. The fourth is an RFC standard specifying the aformentioned more general DNS SRV Records method.

The vendor specific methods have in common that the mail client seeking configuration needs to send a request, which includes at least the user’s mail address, to a configuration service. The service will use the mail address to lookup configuration data and will return that data as response to the client. Format – XML response or file – and complexity differ depending on the method.

automx2 implements everything to configure a mailbox account. It does not implement functionality to e.g. also configure calendar or address book settings.

The following subsections will explain more detailed how the four methods work.

2.1. autoconfig

Autoconfig is a proprietary method developed by the Mozilla foundation. It was designed to configure a mail account within Thunderbird, and other email suites like Evolution and KMail have adopted the mechanism.

When a user begins to create a new mail account she is asked to enter her realname and mail address, e.g. alice@example.com. Thunderbird will then extract the domainpart (here: example.com) from the mail address and build a list of URIs to search for a configuration web service in the following order:

https://autoconfig.thunderbird.net/v1.1/$DOMAINPART (1)
https://autoconfig.example.com/mail/config-v1.1.xml?emailaddress=$MAILADDRESS (2)
https://$DOMAINPART/.well-known/autoconfig/mail/config-v1.1.xml
http://autoconfig.thunderbird.net/v1.1/$DOMAINPART
http://autoconfig.example.com/mail/config-v1.1.xml?emailaddress=$MAILADDRESS
http://$DOMAINPART/.well-known/autoconfig/mail/config-v1.1.xml

1	The `$DOMAINPART` variable represents the users mail addresses domainpart.
2	The `$MAILADDRESS` variable represents the users mail address.

A configuration service such as automx2 listening on one of the listed URIs will receive the request, process it and respond with a set of configuration instructions.

Thunderbird will use the instructions to automatically fill in the required fields in the account. The only remaining task for the user is to confirm the settings. After that she can immediately start to use her new mail account.

2.2. autodiscover

Autodiscover is a proprietary method developed by Microsoft. It was designed to configure a mail account within Outlook and has expanded to also configure Office 365.

When a user begins to create a new mail account she is asked to enter her realname and mail address, e.g. alice@example.com. Outlook will then extract the domainpart (here: example.com) from the mail address and build a list of URIs to search for a configuration web service in a specific order. If it can’t find a web service, it will search the DNS for a redirect:

https://$DOMAINPART/autodiscover/autodiscover.xml  (1)
https://autodiscover.$DOMAINPART/autodiscover/autodiscover.xml
http://autodiscover.$DOMAINPART/autodiscover/autodiscover.xml
dns: autodiscover.$DOMAINPART
dns: _autodiscover._tcp.$DOMAINPART

1	The `$DOMAINPART` variable represents the users mail addresses domainpart.

All HTTP(S) queries send a POST request and submit XML which contains information about the account that should be configured. The DNS queries search for a CNAME RR first, which is supposed to redirect the mail client to a resource outside of the mailbox owners domain, e.g. alice@example.com would be redirected to service.example-provider.com for configuration instructions. If the first DNS query fails the client may be redirected to a configuration service using a SRV RR like this:

_autodiscover._tcp.example.com.  0   443 service.example-provider.com.

The SRV RR used in the example above would send Alice’s client to service.example-provider.com and tell it to send the query to the configuration service on port 443.

2.3. mobileconfig

TODO - Requests and responses use proprietory content types and the PLIST data format.

2.4. DNS SRV Records

_imap._tcp.example.com          SRV  10  20  143  mail.example.com.
_imaps._tcp.example.com         SRV  0   1   993  .
_pop3._tcp.example.com          SRV  0   1   110  .
_pop3s._tcp.example.com         SRV  0   1   995  .
_smtp._tcp.example.com.         SRV  0   1   25   .
_submission._tcp.example.com.   SRV  10  20  587  mail.example.com.

3. automx2 Installation

automx2 requires Python 3.7 or higher, ideally in the form of a virtual Python environment, to run. Check the python3 version like this:

$ python3 --version
Python 3.8.0

If you see version 3.6 or lower, you’ll need to either change the active Python version for the shell session or edit setupvenv.sh after downloading the script.

Don’t run as root

If you use a port number greater than 1024 (we suggest 4243), the application does not require super user privileges when running. It also does not need to be installed as root. It is recommended that you create a user account specifically for automx2, but other unprivileged users will do as well.

Prepare the virtual environment for the automx2 web service, adjusting the installation path to your taste (automx2 itself does not care).

mkdir -p /srv/web/automx2
cd /srv/web/automx2

Download the script that will download and setup your automx2 service:

wget -O setupvenv.sh 'https://gitlab.com/automx/automx2/raw/master/contrib/setupvenv.sh?inline=false'
chmod u+x setupvenv.sh

Execute the setup script. It will create a Python virtual environment called venv in the current directory:

./setupvenv.sh

Activate the virtual environment and install the latest automx2 release from PyPI. Make sure to pick the correct activation for your shell from the venv/bin directory. This is an example for BASH:

. venv/bin/activate
pip install automx2

Updating automx2

Change to the directory where automx2 has been installed previously. Activate the virtual environment as usual and use pip’s -U option to update automx2:

cd /srv/web/automx2
. venv/bin/activate
pip install -U automx2

The next section explains how to configure automx2.

4. Configuring automx2

automx2 uses a file to read runtime instructions from and a database to lookup mail account configuration data.

4.1. Runtime configuration

The configuration file defines automx2 runtime behaviour and it specifies the backend automx2 should read mailbox account configuration data from.

Running without runtime config

If you launch automx2 without a configuration file, it will use internal defaults. These are suitable for testing only. Launched without a config it will use an in-memory SQLite database and all data will be lost once the application terminates.

When started automx2 looks for runtime configuration instructions at these locations:

AUTOMX2_CONF  (1)
~/.automx2.conf  (2)
/etc/automx2/automx2.conf
/etc/automx2.conf

1	If this environment variable exists, it will be used. The value must point to a location where automx2 can read configuration from. It will proceed if it can’t find the file at the moment.
2	If automx2 finds `.automx2.conf` in the `$HOME` of the user that runs automx2, it will be used.

To specify parameters and options automx2 uses an INI-style configuration syntax. The example configuration that ships with automx2 looks like this:

[automx2]
# A typical production setup would use loglevel = WARNING
loglevel = DEBUG
# Echo SQL commands into log? Used for debugging.
db_echo = yes
# In-memory SQLite database
db_uri = sqlite:///:memory:

# SQLite database in a UNIX-like file system
#db_uri = sqlite:////var/lib/automx2/db.sqlite

# MySQL database on a remote server. This example does not use an encrypted
# connection and is therefore *not* recommended for production use.
#db_uri = mysql://username:password@server.example.com/db

# Number of proxy servers between automx2 and the client (default: 0).
# If your logs only show 127.0.0.1 or ::1 as the source IP for incoming
# connections, proxy_count probably needs to be changed.
#proxy_count = 1

Place the content of the example configuration into one of the configuration locations automx2 looks for and adapt it to your needs. Then configure the database backend with data that suits your setup.

4.2. Testing standalone automx2

If you want to verify a vanilla install of automx2 works you can populate it with the (internal) test data. Start it as described in section Running automx2 and send the following request to populate your database with the internal test data:

curl http://127.0.0.1:4243/initdb/  (1)

1	This example assumes you are running automx2 on localhost listening on port `4243`, which is the suggested default port.

Once you have populated the database with sample data you can test if automx2 works. Use curl to send an account configuration request for user@example.com:

curl 'http://127.0.0.1:4243/mail/config-v1.1.xml?emailaddress=user@example.com'

Make sure to use quotes as shown, or your shell might attempt some pattern matching (FISH definitely does).

4.3. Database configuration

automx2 uses the SQLAlchemy toolkit to access databases. This allows a variety of databases, aka dialects, to be used, simply by defining the appropriate connection URL.

While you probably already have SQLite support available on your local machine, you may need to install additional Python packages for PostgreSQL, MySQL, etc. Detailed instructions to support a particular database dialect are out of scope for this document, but there are numerous guides available.

This section demonstrates what you need to do to in order to use a SQLite database as backend for automx2.

Editing account configuration data

At the moment you will need to add database entries manually. We plan to change this in an upcoming version.

You may use a helper script to populate your database with account configuration data. Change the script’s user-configurable section to reflect your domain and server names, then run the script to generate the necessary SQL statements:

#!/usr/bin/env bash
# vim:tabstop=4:noexpandtab
#
# Generates SQL statements to add a provider, servers, and domain.
# Adapt the configurable section below to your needs.

set -e

# User configurable section -- START
PROVIDER_NAME='Me, myself and I'
PROVIDER_SHORTNAME='Me'
PROVIDER_ID=123
DOMAIN='example.com'
IMAP_SERVER="imap.${DOMAIN}"
SMTP_SERVER="smtp.${DOMAIN}"
# User configurable section -- END

s1id=$((PROVIDER_ID+1))
s2id=$((PROVIDER_ID+2))
domid=$((PROVIDER_ID+3))

cat <<EOT
INSERT INTO provider VALUES(${PROVIDER_ID}, '${PROVIDER_NAME}', '${PROVIDER_SHORTNAME}');
INSERT INTO server VALUES(${s1id}, 993, 'imap', '${IMAP_SERVER}', 'SSL', '%EMAILLOCALPART%', 'plain');
INSERT INTO server VALUES(${s2id}, 587, 'smtp', '${SMTP_SERVER}', 'STARTTLS', '%EMAILLOCALPART%', 'plain');
INSERT INTO domain VALUES(${domid}, ${PROVIDER_ID}, '${DOMAIN}');
INSERT INTO server_domain VALUES(${s1id}, ${domid});
INSERT INTO server_domain VALUES(${s2id}, ${domid});
EOT

You can download the script from contrib/sqlite-generate.sh, adapt the "User configurable section" and pipe the scripts output into the sqlite3 command like this:

Placeholders

See Mozilla’s Thunderbird:Autoconfiguration:ConfigFileFormat for a documentation of available placeholders like %EMAILLOCALPART%.

contrib/sqlite-generate.sh | sqlite3 /var/lib/automx2/db.sqlite

Once you have populated the database automx2 is ready to run.

4.4. Running automx2

Change as the user that should run automx2 into the virtual environment directory and start the contrib/flask.sh script:

cd /srv/web/automx2
contrib/flask.sh run

This will start the flask service listening on IP address 127.0.0.1 and port 5000. These are the standard values for Flask and might collide with other software you have installed. We therefore recommend that you use the parameters --host and --port to override the defaults:

flask.sh run --host=192.0.2.1 --port=1234

Handling terminal output

The flask.sh script will deliberately keep automx2 running in the foreground, and log data will be displayed in the terminal. If you press Ctrl-C or close the shell session, the application will terminate. To run automx2 in the background, you can use a window manager like GNU Screen

Now that automx2 is up and running, you need to configure the web server proxy that will receive requests from the outside and forwards them to automx2.

4.5. Configuring a web server

While it is technically possible to run automx2 without a web server sitting in front of it, we don’t recommend doing that in a production environment. A web server can provide features automx2 was designed not to have. Features such as transport layer encryption aka HTTPS or, for example, the capability to rate-limit clients are handled very well by full-fledged web servers working as reverse proxies. It would be a waste to re-implement all this in a web service.

This section will explain how to configure a web server as a reverse proxy in front of automx2. Before you set up the proxy you need to tell automx2 it operates behind one. Add the proxy_count parameter to your automx2 configuration file or uncomment the parameter if it is already there:

[automx2]
# A typical production setup would use loglevel = WARNING
loglevel = WARNING

# Echo SQL commands into log? Used for debugging.
db_echo = no

# SQLite database in a UNIX-like file system
db_uri = sqlite:////var/lib/automx2/db.sqlite

# Number of proxy servers between automx2 and the client (default: 0).
# If your logs only show 127.0.0.1 or ::1 as the source IP for incoming
# connections, proxy_count probably needs to be changed.
proxy_count = 1  (1)

1	Set the number to reflect the number of proxies chained in front of automx2, i.e. the number of "proxy hops" a client’s request must pass before it reaches automx2.

4.5.1. NGINX

The following example defines a HTTP server, which will listen on port 80 for requests to either autoconfig.example.com or autodiscover.example.com. All requests will be forwarded (proxied) to automx2, which listens on 127.0.0.1 on port 4243 in this example. Requests to /initdb are restricted to clients from 127.0.0.1 only. The proxy_set_header directives will cause NGINX to pass relevant data about incoming requests' origins.

# NGINX example configuration snippet to forward incoming requests to automx2.
# vim:ts=4:et:ft=nginx

http {
    server {
        listen *:80;
        listen [::]:80;
        server_name autoconfig.example.com autodiscover.example.com;
        location /initdb {
            # Database may only be initialised from localhost
            allow 127.0.0.1;
            deny all;
        }
        location / {
            # Forward all traffic to local automx2 service
            proxy_pass http://127.0.0.1:4243/;
            proxy_set_header Host $host;
            # Set config parameter proxy_count=1 to have automx2 process these headers
            proxy_set_header X-Forwarded-Proto http;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Real-IP $remote_addr;
        }
    }
}

4.5.2. Apache

The following example defines a HTTP server, which will listen on port 80 for requests to either autoconfig.example.com or autodiscover.example.com. All requests will be forwarded (proxied) to automx2, which listens on 127.0.0.1 on port 4243 in this example. Requests to /initdb are restricted to clients from 127.0.0.1 only. The ProxyPreserveHost directives will cause apache to pass relevant data about incoming requests' origins.

# apache2.4 example configuration snippet to forward incoming requests to automx2.
# vim:ts=4:et:ft=apache

<VirtualHost *:80>
    ServerName autoconfig.automx.org
    ServerAlias autodiscover.automx.org
    ProxyPreserveHost On
    ProxyPass "/"  "http://127.0.0.1:4243/"
    ProxyPassReverse "/"  "http://127.0.0.1:4243/"
    <Location /initdb>
        Order Deny,Allow
        Deny from all
        Allow from 127.0.0.1
    </Location>
</VirtualHost>

5. About automx2

The project resides as automx/automx2 on GitLab. Please use the projects issue tracker if you find bugs. To discuss usage and other topics join us on the automx-users mailing list.

automx2 was written by Ralph Seichter for sys4 AG. It has 100% test coverage and is mirrored to rseichter/automx2 on GitHub.