Local IMAP Server as Mail Archive

I am a big fan of data privacy in the sense, that I want to be the only one hosting my data. The internet should only be used as a transmit zone, not as data storage. For that reason, I was looking for a method how to bring my mails to my NAS but without hosting a fully blown mail server and without the need of exposing it to the internet.

The mail provider I use does a fairly good job and I want to use his mail services (e.g., send & receive mails, filtering spam, …) but without the need to have all my mails stored on his data centres. In the past I used MS Outlook for this and downloaded my mails to my local computer with POP3. But that bears a few problems (or at least annoyances):

• The .PST files from Outlook grow rather big over time and I had to repair them every now and then.
• To get those files backed up, I either need to have them on the NAS (SMB Share – since those are backed up nightly), but then I can access my mails only in my LAN, since Outlook can’t handle those big PST files over slow VPN connections
• Having the files locally on my computer makes it usable even over slow VPN connections, but I need to back it up manually
• I can only use my computer and Outlook to access my mails

The basic idea I wanted to realise looked somewhat like this:

pasted-from-clipboard.png

Searching the Internet, I found out that somebody had the same idea years ago. He described it here: https://web.archive.org/web/20200807233801/https://joel.porquet.org/wiki/hacking/getmail_dovecot/ .

He implemented it using Getmail and Dovecot. There are even some container images implementing this idea. The most current one (as of this writing) is probably this one https://github.com/tctlrd/getmail-container .

I liked the idea of a containerized solution but I decided against it for the following reasons:

• None of the images implemented it the way I wanted to use it.
• Most of the images were rather old and I’ve not seen any of it getting regular updates. The risk of security vulnerabilities gets too high, when you don’t update the software.
• I found an official container image, that gets updated frequently only for dovecot but not for Getmail6.
• I want to use the usernames and passwords of the users of my NAS also for the IMAP server. That would need a bit of a hack to get this into a container which I simply wasn’t in the mood to do.

My solution covers the following:

• Retrieving mails from the mail provider using IMAP IDLE, so I don’t have to poll every x minutes, but it rather works like a push connection until it times out.
• Every NAS user can define its own connections / mail accounts.
• Get mails from multiple mail providers / mail accounts per user.
• User accounts and passwords are taken from the Debian System using PAM.
• Mails are NOT stored in the regular $HOME directories of the system users.
• Config files for Getmail6 could be stored in the regular $HOME directories of the system users or elsewhere.
• Mail storage is on a different place (HDDs) than index files (SSD). Reason for this is speed and backup.
• Mails in dovecot are stored in Maildir++ structure (one file per mail; Subfolder possible).
• Full Text Search on server side included.
• Virtual folders on server side included.
• SSL for the local IMAP server on the NAS can be implemented.
• Full backup and restore of the mail data.

My main pieces of SW are Dovecot CE (IMAP server; see: https://www.dovecot.org/), Getmail6 (Python3 enabled port of the original Getmail; see: https://getmail6.org/) and Thunderbird (free e-mail client; also covers calendar and contacts; see: https://www.thunderbird.net/), but you could probably use any other e-Mail client capable of IMAP. This somehow excludes Outlook, since it is known since years for its weaknesses regarding IMAP.

I will first explain how to install the IMAP server (Dovecot) and make sure, we can connect to it using Thunderbird. When the IMAP server is up and running and especially the authentication topic is proven to be working, I am going to describe how to setup Getmail6 to retrieve mails.

I urge you to try every of those steps upfront in a dry-run; i.e., installing and configuring Dovecot, installing and configuring Getmail6 to retrieve mails from your mail provider (create yourself an additional account); create the special Outlook-Profile needed to transfer all the existing Mails from PST to the IMAP-Server, etc.

Prerequisites

Installation

Install the SW packages for Dovecot and Getmail using the Web-UI of OVM. The names of the packages are: dovecot-imap, dovecot-fts-xapian, getmail6.

Don’t use backports for Dovecot! I started with the backported version and got problems later with full text search (missing dependencies). This was resolved, after downgrading to the regular version from the stable branch. Since Dovecot even in this quite minimalistic installation involves several modules from the repository, I clearly recommend to stick with the stable version.

I use the following directory structure.

This is on a slow(er) disk (HDD):

/srv/additional-services/mail
├── dovecot
│   └── user
│       └── Maildir
├── getmailrc
│   └── user
│       └── state

This is the data, that goes to the long-term backup.

This is on a fast(er) disk (SSD):

/srv/fast-data/mail
├── indexes
│   └── user

In case of a disaster, the index files can (and should) easily be recreated. They need not be part of the backup.

Fallback

Latest now make a copy of /etc/docevcot.

Logging

To enable logging and define the path where to place the logfile, edit /etc/dovecot/10-logging.conf

log_path = /srv/logs/mail/dovecot.log

Authentication

DE-activate SSL and allow plaintext passwords (for the moment). Authentication has enough pitfalls on its own. I don’t want the additional complexity of encryption right now.

Edit /etc/dovecot/10-auth.conf

disable_plaintext_auth = no

Edit /etc/dovecot/10-ssl.conf

ssl = no

Mail Directory

By default Dovecot uses the $HOME directories of the system users, but that’s not what I want. Even if I use the system users for authentication, I’d rather follow the concept of the virtual users and want all the data belonging to this service to be below one central directory. Well, maybe except the index files. Those are located on a faster SSD instead of the slower HDD.

Using the system users for authentication is mainly, because I want to have the same password for all the services on my NAS and if one user changes its password, it should be changed for all the services without the need to manually keep several service specific files in sync.

One more word about directories. The directories containing the user specific data must also have the owner set to that user and this user must also have access rights for that directory. If that’s not the case, the mail service will throw errors at you, you might not at first associate with access rights and the ownership of those directories.

To set the path where Dovecot stores its internal state as well as mails and index files, edit /etc/dovecot/ 10-mail.conf:

mail_home=/srv/additional-services/mail/dovecot/%u
mail_location = maildir:/srv/additional-services/mail/dovecot/%u/Maildir:INDEX=/srv/fast-data/mail/indexes/%u

Note that %u will be substituted by the user name.

Mails will be retrieved with getmail and “delivered” (i.e. stored in the dovecot specific location below mail_location) by dovecot-lda. By default Dovecot will not automatically create any missing mailboxes or subfolders, when storing mails with dovecot-lda. The next line changes this.

Edit /etc/dovecot/15-lda.conf

lda_mailbox_autocreate = yes

First Login

Now let’s test the IMAP server. First check if the configuration contains any (syntactical) errors.

dovecot -n

This outputs the configuration, that differs from the default values. If there were any (syntactical) errors, it would tell us. Now let’s start the service:

systemctl start dovecot

To manually login issue the following command

telnet localhost imap

If everything works well, you should see a line similar to this:

OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ AUTH=PLAIN] Dovecot (Debian) ready.

Now we are talking directly to the open port of the freshly installed dovecot server. To test if the authentication works as expected enter the following command (of course replacing “user” and “password” with one of the regular users of your system).

a login user password
OK [CAPABILITY … ] Logged in

If you see this, then Dovecot is working as expected. Now logout

a logout

Here’s a list of some commands to play with:

A LOGIN <username> <pass>                  # Log in
A LIST "" *                                # List all mail folders
A LIST INBOX *                             # List subfolders of INBOX
A SELECT INBOX                             # Set folder INBOX for the following commands
A FETCH 2 all                              # Get message 2
A FETCH 2 body[text]                       # get body of message 2
A LOGOUT                                   # Ausloggen

Now stop Dovecot again.

systemctl stop dovecot

SSL

If you plan using SSL to secure your communication with the IMAP server, now is the time to do so. To be honest, I currently don’t care too much about it, since in my setting it will NEVER be exposed to the internet and only be accessed locally in my LAN. If I need remote access, I will use my VPN. But since I have my own little CA due to other reasons, I’m going to implement it anyhow.

Edit again /etc/dovecot/10-ssl.conf

ssl = yes
ssl_cert = </etc/dovecot/private/Nas_IMAP.public.crt
ssl_key = </etc/dovecot/private/Nas_IMAP.private.pem

Also edit /etc/dovecot/10-auth.conf

disable_plaintext_auth = yes

Please note:

When you enable SSL and use the self-signed certificate from OMV, Thunderbird might (silently) refuse to connect. Therefore, if you do so, first test if Dovecot is working correctly:

openssl s_client -connect localhost:993

should give you some meaningful output, the last line looking similar to this:

* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ AUTH=PLAIN] Dovecot (Debian) ready.

…you are ready to log in as above. If logging in and logging out works fine, Dovecot is working as expected with SSL.

Mailboxes

This is optional, but I find it quite useful. It defines a few “special” mailboxes (e.g. Trash, Spam, Draft, …), automatically creates them for every new user and lets the mail client automatically subscribe to them.

Edit /etc/dovecot/15-mailboxes.conf

namespace inbox {
  # These mailboxes are widely used and could perhaps be created automatically:
  mailbox Drafts {
    special_use = \Drafts
    auto = subscribe
  }
  mailbox Junk {
    special_use = \Junk
    auto = subscribe
  }
  mailbox Trash {
    special_use = \Trash
    auto = subscribe
  }

Don’t be mistaken about this “special_use” flag. It doesn’t add any extra functionality on its own. It only adds some meta information to that folder, so that the e-mail client knows, this is e.g. the folder to put a copy of sent mails, or where to move deleted mails. For most of those “special_use” flags, it is up to the client to implement the functionality. This is different though for the “special_use” flags “\All” and “\Flagged”. They require extra functionality on the server side. We will look into this later. For the moment “\All” and “\Flagged” remain DE-activated.

If your mail provider has another name for e.g. “Junk”, maybe “Spam” or “Spam suspected”, the entry looks like this:

  mailbox “Spam suspected” {
    special_use = \Junk
    auto = subscribe
  }

Dovecot as a Service

To configure Dovecot as a service and having it started at boot time, issue this command:

systemctl enable dovecot@service

I had the problem, that dovecot could not (yet) find the logfile after a reboot and therefore refused to start. I had to add an “override” to the default system definition. If this also happens to you, issue the following command:

systemctl edit dovecot@service

In the editor enter:

[Unit]
After=local-fs.target network-online.target multi-user.target

Thunderbird

I won’t go into detail how to install Thunderbird. But once it was installed, I’ve setup an account to connect to the IMAP server (Dovecot) and my mail provider. This is somewhat tricky and you MUST NOT use the assistant. So, here’s what I did:

As soon as you enter something like “a@b” as e-mail address, you can switch to “manual setup” (there should be a link in blue left to the buttons where you can click on). Then immediately switch to “Extended Setup” (I’m just guessing the names, since I don’t have an English system). For those already familiar with TBird: this “Extended Setup” is what you see, when you switch to the accounts in a running TBird.

In the Extended Setup first configure the connection to the SMTP server of your mail provider as you would do anyhow (SMTP server, port, encryption). Your mail provider will have instructions available for it.

Then in “Server Settings” enter the connection data for the IMAP server (name or IP of the NAS, port, encryption, user name, password). There is a tab called “Copies” (or similar): Make sure, that for every target the IMAP server is selected and not “Local Folder”.

If this works out fine, you can try to connect with Thunderbird to the IMAP server. If you experience any connection problems with Thunderbird (using self-signed certificates) the problem is most likely with Thunderbird.

You have several options then:

• The obvious one: Don’t use SSL. If used purely internal in a LAN it’s not needed
• Try reconnecting several times until Thunderbird asks you if it should accept the self-signed certificate
• Create your own small Root-CA with the tool XCA
  (As I said, that’s what I have in place, but I’m not going into those details here)

If the connection to the IMAP server is in place, but Thunderbird doesn’t show any folder (or some folders are missing) you may need to explicitly subscribe it. (Right-click on your mailbox and select "Subscribe," then check the box of the desired folder and confirm with OK.)

Restart Thunderbird. When prompted, enter the password for the SMTP and / or the IMAP server. You might want to hide “local Folders”, since they are unused (3 dots on the top (just above the box with the mail accounts) to the right).

Now TBird is connected to our IMAP server, shows the mails stored there and can send mails using the SMTP relay of the mail provider.

Get the Mails

Now it’s time to fetch the mails from the mail provider and do deliver them to the IMAP server. Gertmail6 should be installed by now.

For every (external) mail account there needs to be a config file. By default getmail6 searches for $HOME/.getmail/getmailrc. I’m not going to go down that path. For one, I probably need more than one file (for more than one external mail account). And second, I simply don’t want to put it in the home directories of the system users. Instead, I want all the data of this mail installation in one place below one sub-directory. I suggest to create a directory for getmail and a sub-directory per user to store all the config files (one per external mail account). The naming scheme for the config files should be “user.accout-at-mailprovider.org”.

This is what such a config file looks like for GMX.

# getmailrc email configuration

[retriever]
type = SimpleIMAPSSLRetriever
server = imap.gmx.net
port = 993
username = user@gmx.net
password = PASSWORD
mailboxes = ALL

[destination]
# for a first test run use this
type = Maildir
path = /srv/additional-services/mail/dovecot/user/Maildir
user = user

[options]
read_all = no
# for a test run uncomment the next line and comment the line after
#delete = no
delete_after = 30
delivered_to = false
received = true
# set to 1 for a test run
verbose = 0
message_log = /srv/additional-services/mail/getmail/user/getmail.log
message_log_verbose = false

That would be the main account, since it delivers all mails in several subfolders, just as it is set up for your mail provider. Let’s look at the file a bit more in depth.

Section [retriever] contains all the data to access the server located at the mail provider. Since this contains a password, you should make sure, the access rights are set appropriately. If you don’t want to place the password in clear text in that file, there are also other ways. See the official Getmail6 documentation for it.

Section [destination] defines how the mails shall be handled. For a first test run, you should point it to a path (as you can see above). This gives you the possibility to easily check if it works as expected, since you can easily inspect the downloaded text files.

Section [options] defines e.g., how long mails shall remain on the server at the mail provider (delete, delete_after). For test runs I propose to use delete = no. For the final configuration delete_after = 30 (or whichever value suites you). Only use one of them, not both.

verbose, message_log and message_log_verbose define the amount of messages printed to the console or logfile and where to put the logfile. For test runs I propose verbose = 1.

To start getmail enter the following command:

getmail ––rcfile=/srv/additional-services/mail/getmail/user/user.account-at-gmx.net
         ––getmaildir=/srv/additional-services/mail/getmail/user/state

(This is one line. I've only placed it on two lines for readability)

--rcfile points to the config file; --getmaildir points to a directory, where getmail places some files, in which it tracks which mails have already been retrieved. I use a subdirectory below the config file for the files with tracking information.

If that works as expected, replace the lines below [destination]

type = Maildir
path = /srv/additional-services/mail/dovecot/user/Maildir
user = user

with the following lines

type = MDA_external
path = /usr/lib/dovecot/dovecot-lda
arguments = ("-e", "-m", "%(mailbox)", "-d", "user")
allow_root_commands = yes

This uses the delivery agent from dovecot (you could also use deliver, it’s the same program). Doing so, you make sure, that all index files etc. for Dovecot get updated accordingly.

For testing and installation purposes you might maybe run this as root user. In this case -d is needed to tell Dovecot for which user the mails are intended. Later this will be run with user privileges. Then -d must be omitted. Dovecot gets the user from the user ID of the calling process.

-e lets the program print messages to stderr in case of errors and exiting.

-m instructs it in which mailbox (i.e. subfolder) to deliver the retrieved mail, i.e. each folder from your provider is also created here.

Delete alle the mails you already received, including the state files from Getmail, then run again the command

getmail ––rcfile=/srv/additional-services/mail/getmail/user/user.account-at-gmx.net
        ––getmaildir=/srv/additional-services/mail/getmail/user/state

and you have successfully retrieved your first set of mails from your mail provider and handed them over to Dovecot.

Start TBird and see if all messages arrived successfully. You should also check, if all subfolders are displayed, since TBird won’t automatically display subfolders that were created by getmail. You must instruct TBird to do so. (open context menu on the IMAP account on the NAS and select “Subscription”.)

For a second mail provider, the line

arguments = ("-e", "-m", "%(mailbox)", "-d", "user")

would most likely rather look like this:

arguments = ("-e", "-m", "other Mailprovider", "-d", "user")

Thus, placing all the mails from that other provider in a single subfolder. Of course, you could do this also for your main mail provider.

One last little trick is to continuously run getmail, so that my own IMAP-Server (on the NAS) is always up to date. To achieve this, I simply add “--idle INBOX” to the list of command line params. This makes use of a feature called “IMAP idle”. Every message that is delivered to the INBOX of my mail provider gets now immediately pushed to my own IMAP server in the NAS. Depending on the provider there might be timeouts configured. For GMX this is roughly 3h, then the connection gets closed and getmail needs to reconnect. Skymail.de either did not have any timeout or it is more than 10h.

Getmail as a Service

To have getmail to continuously reconnect to my mail provider, I installed it as a service.

For the daily / continuous usage, getmail should be run with user privileges. I don’t want a SW with external connections to be run as root. To create a service, that will run with user privileges and that can be used with multiple external mail accounts per user, I wrapped it in the following service definition:

First of all, in file user.accout-at-mailprovider.org remove "-d", "user" from the line with the arguments.

arguments = ("-e", "-m", "%(mailbox)")

Then create

/etc/systemd/user/getmail-adr@.service

## Run getmail --idle to retrieve mails from a specific mail account.
## Must be called as the user of that mail account NOT as root / system.
##

[Unit]
Description=Getmail --idle for a mail account. MUST be run as user.
Wants=network-online.target
After=network-online.target

[Service]
Type=exec
ExecStart=/usr/bin/getmail --rcfile=/srv/additional-services/mail/getmail/%u/%i
                --getmaildir=/srv/additional-services/mail/getmail/%u/state/ --idle INBOX
Restart=always
RestartSec=30

[Install]
#WantedBy=multi-user.target
WantedBy=default.target

Run the following command to start the service and see if it works as expected. Doing this with an empty mailbox helps, but is not necessary.

systemctl --user start getmail-adr@user.account-at-gmx.net.service

Check the log files to see if it worked or why not.

Short explanation:

%i is substituted by what comes after the @ and before .service. In this case that’s the name of the config file.

%u is the name of the user running the above command.

If it works as expected, stop it. Then issue the following:

systemctl --user enable getmail-adr@user.account-at-gmx.net.service
systemctl --user start  getmail-adr@user.account-at-gmx.net.service
systemctl --user status getmail-adr@user.account-at-gmx.net.service

You should get something like this:

$ systemctl --user status getmail-adr@user.account-at-gmx.net.service
● getmail-adr@user.account-at-gmx.net.service - Getmail --idle for a mail account. MUST be run as user.
     Loaded: loaded (/etc/xdg/systemd/user/getmail-adr@.service; enabled; preset: enabled)
     Active: active (running) since Wed 2025-03-12 21:29:07 CET; 2h 39min ago
   Main PID: 2148526 (getmail)
      Tasks: 1 (limit: 23758)
     Memory: 6.1M
        CPU: 1.250s
     CGroup: /user.slice/user-1111.slice/user@1111.service/app.slice/app-getmail\x2dadr.slice/getmail-adr@user.account-at-gmx.net.service
             └─2148526 /usr/bin/python3 /usr/bin/getmail --rcfile=/srv/additional-services/mail/getmail/user/user.account-at-gmx.net --getmaildir=/srv/additional-services/mail/getmail/user/state/ --idle INBOX

Mar 12 21:29:07 data-nas systemd[1059]: Starting getmail-adr@user.account-at-gmx.net.service - Getmail --idle for a mail account. MUST be run as user....
Mar 12 21:29:07 data-nas systemd[1059]: Started getmail-adr@user.account-at-gmx.net.service - Getmail --idle for a mail account. MUST be run as user..

Take a special look at the time running (here 2h 39min). After some time, this should clearly be more than just a few seconds. If it doesn’t that means, that there is something wrong with the command. The logfile should provide insights as to what might be the reason.

This way every user can start his own getmail services and he can even fetch mails from multiple mail providers. But so far, this service would start when the user logs in and it would get stopped, when the user logs out. For an interactive machine that would be all that’s needed. But the users don’t log in interactively, they just connect to the file server or the IMAP server. So, the user service needs to get started, when the NAS boots.

To achieve this, issue the following command:

loginctl enable-linger

Note:

When I first tried to run systemctl with user privileges on OMV, I got an error that /usr/bin/pkttyagent was missing. I had to install polkitd using apt install to get rid of that error.

Server side functionality

Full Text Search (FTS)

I wanted to have full text search on the server side. Luckily the Debian team has already prepared the corresponding plugin (dovecot-fts-xapian). I also installed the plugin using the Web-UI. Then added the following to my config files:

### Full Text Search (FTS)

# from 10-mail.conf
mail_plugins = $mail_plugins fts fts_xapian

# from 90-plugin.conf
plugin {
    fts = xapian
    #fts_xapian = verbose=0
    fts_xapian = partial=2 full=20 attachments=0 verbose=0

    fts_autoindex = yes
    fts_enforced = yes

    fts_autoindex_exclude = \Trash
}

service indexer-worker {
    # Increase vsz_limit to 2GB or above.
    # Or 0 if you have rather large memory usable on your server, which is preferred for performance)
    vsz_limit = 0G
    # This one must be 0
    process_limit = 0
}

(Re-)start dovecot or reload configuration.

systemctl reload-or-restart dovecot

Initialize the full text search:

doveadm index -A -q \*

This command must be run on a regular basis (e.g. daily or weekly via cron) to optimize the search index:

doveadm fts optimize -A

Please note, that the quick search bars in TBird only search locally. If you want to search on the server, use “Edit / Search / Search Messages…”. This will open a dialog box where you can select “search on server”.

You can use the Thunderbird addon “Expression Search - NG”, which enables you to use the quick search to search on server side, but only in the folder you are currently looking at. To use this add-on to search through all mails on server side, a virtual folder containing all the mails is needed.

NB.: There is a more than 10 years old feature request for server side search in TBird. Doesn’t look like it will be implemented soon…

Virtual folders

As seen above, there is some need for virtual folders on server side. To do so, Dovecot must know about the existence of those folders (i.e. their path in the mailbox structure) and it must know what should be displayed inside those folders. These are two different config items, that must play together. I wanted two such folders, called “All” and “Flagged”. The first shall contain all mails within my mailbox (except for the ones in Spam or that are deleted). The last shall contain all mails with a flag.

In Dovecot virtual folders MUST be configured inside a separate namespace. That’s because they are “stored” differently than normal mails. You may think of it, like using a different driver for accessing it.

First of all, virtual folders are activated by including the plugin “virtual” in the line mail_plugins = .

mail_plugins = $mail_plugins virtual

namespace virtual {
  location = virtual:/etc/dovecot/virtual
                    :INDEX=/srv/fast-data/mail/indexes/%u/virtual
                    :CONTROL=/srv/fast-data/mail/dovecot/%u/virtual
  type = private
  prefix = virtual/
  separator = /
  hidden = yes
  list = no
}

The namespace itself could be named anything, there is no need to name it “virtual”. What is shown behind prefix =  will be shown in the mail client. I suggest to use the same prefix as the name for the namespace (just for reasons of consistency and readability). The most important part is the line location. Please note, that in the config file this is one line, I only placed it on several lines for better readability. Let’s look at the line in detail:

The keyword “virtual” is what I called “driver to access the information”. One namespace can (and must) only have one such “driver” associated. That’s the reason why we need a second namespace for the virtual folders. The first path entry (here: /etc/dovecot/virtual) is the location where Dovecot will look for the names of the virtual folders and what should be display inside those virtual folders. The path behind INDEX= tells dovecot where to store the index files for the virtual folders. If possible, this should be on a fast disk (SSD). The path behind CONTROL= contains state information of those virtual folders.

As for the other keywords: type = private tells dovecot, that this contains private content (as opposed to shared or public content). separator = / tells dovecot which separator to use. It is important to use the same separator in all namespaces. If not, the mail client can get confused. I suggest to also set the separator for the inbox namespace, just to be on the safe side. (I had some weird, but minor annoyances, that I could solve that way). hidden = yes and list = no prevent, that the virtual folders appear together with real mail folders when the client issues some of the IMAP commands.

Next step is to tell Dovecot which virtual folders and what content to display. To get two virtual folders called “ALL” and “Flagged”, create exactly those two folders below /etc/dovecot/virtual. Inside those two folders a file with the name dovecot-virtual is needed. The content of that file defines what is displayed in the virtual folder.

Content of file /etc/dovecot/virtual/All/dovecot-virtual.

*
-Trash
-Trash/*
-Spam
-Spam/*
-Junk
-Junk/*
-virtual/Flagged
-virtual/Flagged/*
  all

This will display all mails but the ones in the folders (and their sub-folders) after “-“. Note the line containing “virtual/Flagged”. This prevents the content from the other virtual folder to be displayed here. That would lead to have the flagged mails to appear twice.

Content of file /etc/dovecot/virtual/Flagged/dovecot-virtual.

*
-Trash
-Trash/*
-Spam
-Spam/*
-Junk
-Junk/*
-virtual/All
-virtual/All/*
  flagged

This will display all flagged mails but the ones in the folders (and their sub-folders) after “-“. Note the line containing “virtual/All”. This prevents the content from the other virtual folder to be displayed here. That would lead to have all mails to appear twice.

Now that the virtual folders for all or flagged mails are defined on server side, we can as well tell the client about it.

Edit /etc/dovecot/15-mailboxes.conf

namespace inbox {

[…]

  # If you have a virtual "All messages" mailbox:
  mailbox virtual/All {
    special_use = \All
    comment = All my messages
    auto = subscribe
  }

  # If you have a virtual "Flagged" mailbox:
  mailbox virtual/Flagged {
    special_use = \Flagged
    comment = All my flagged messages
    auto = subscribe
  }

[…]

}

The line mailbox virtual/All establishes the link to the previously defined virtual folders, where the part virtual/ must be the same as the one after prefix = in the namespace definition. “All” is the name of the directory below /etc/dovecot/virtual, where the corresponding file dovecot-virtual resides.

Again, adding the special_use tag to a mail folder, doesn’t add (or change) the functionality of that folder by itself. The virtual folders will appear in Thunderbird even without this (At least they did in my installation. But there are users reporting, that they first had to add this, before the virtual folders appeared for them). But if the client knows about their special use, maybe a different icon is displayed or its name is localized to your language.

Small drawback:

The virtual folders are shown below a folder called “virtual”. I wanted them to appear on top level, but it seems, this is (currently?) not possible, due to the two different namespaces that are needed. I spent quite some time, trying to change this, but wasn’t able to find anything. So, in case somebody knows how this can be achieved, please leave a message.

Transfer existing mails

Last step I did, was to transfer my existing mails from the Outlook PST files to the IMAP server. In any case, make copies of all your PST files, just in case something goes wrong.

First, I changed my existing POP3 configuration in Outlook. In the original configuration I left the mails for up to 30 days on the server before they got deleted. I changed this to 1 or 2 days, because I wanted to minimize the number of mails I would get twice later.

Start Outlook and keep it running for a few minutes, just to give it time to delete the mails on the server. This will delete on the server of your provider all mails older than 1 or 2 days. Additionally, I made sure, that indeed all mails were stored in local PST files.

Next step is to transfer all the existing / old mails from the PST files to our IMAP server.

This next step was surprisingly tricky. Even though Thunderbird offers an import of Outlook mails, it appeared to be either buggy or at least incomplete. And even if it would work, all the mails would only be transferred to the “Local Folder” and not to the connected IMAP server. So, I can’t recommend using it.

I decided to manually transfer them to the IMAP server using Outlook. To achieve this, I created a new Outlook Profile with the sole purpose to import all my existing mails from the PST files to the IMAP server. At this profile I attached all my PST files and I connected it to the IMAP server on the NAS.

To my surprise, I realized that Outlook absolutely needs a SMTP server to be configured in the profile. Without it, it won’t even create or accept the profile! Even worse: I had to first set up a complete (dummy) mail account at (some arbitrary) mail provider. At least I didn’t found a way to do it differently.

The following steps were needed:

• Create an SMTP & IMAP connection to (some arbitrary) mail provider
  (I created a temporary mail account for that purpose with some other mail provider, because I didn’t want to have any connection to my real mail provider during the import)
• When that Outlook profile is created and works, it’s possible to slowly modify it, using the Control Panel from Windows and selecting the Icon “Mail (Microsoft Outlook)”. Avoid starting Outlook itself.
• Open the profile.
• Disable automatic test.
• Click on “more settings”.
• If you wish to, give it a meaningful name.
• On tab “Outbound mail server”: select that the SMTP server needs a different username and password (and provide them).
• On tab “Extended” set port and SSL encryption according to the configuration of Dovecot.
• Click OK.
• Set the server’s name or IP address of the local IMAP server on the NAS.
• Set the username and the password for the IMAP server on the NAS.

That should do the trick. Starting Outlook and selecting this profile, I was able to read all my PST-Files and transfer their content to the IMAP Server on the NAS.

Backup

I’m using BTRFS as the underlying file system and create nightly snapshots of the sub-volume containing the mail data. Of course, this snapshot is subsequently sent to another NAS for backup purposes. This is handled by btrbk.

In case of a disaster, you can simply delete the index data (of both real and virtual mail folders) as well as the control data for the virtual folders. Then restore the mail data for the affected user. So far, I have only tested backup (and restore) of the mail data for one user, but that worked well. I have not yet found an easy way to restore single messages, e.g., in case someone accidentally deletes an important mail or a folder (and is not able to retrieve it from trash). There are some tools, that claim to be able to read the Maildir++ file structure Dovecot uses. They could come in handy here, but I’ve not tested them.

There is a command doveadm backup. I can’t recommend using it. At least not locally on the same machine. This command is intended to replicate the data from one IMAP server to another. Doing so it updates also index files, control files, etc. Thus, it may appear to be a good idea to use this to have a consistent copy of the mail data suitable for a backup.

If you use this command to replicate the mail data on the local machine it WILL update the local index files. It will do so repeatedly, but since these are the index files of the running Dovecot installation the result is somewhat unpredictable – or at least undesired. It’s probably fine to use it, if you do have another machine, for the replicated data. Then it will be the index files on that machine, that get updated, not the local ones.

Best Practice

It is seen as best practice to have all those changes in a single file at /etc/dovecot/local.conf instead of modifying all the default configuration files.

Dashboard

To add Dovecot to the dashboard of OMV, add the following file:

/usr/share/openmediavault/engined/module/dovecot.inc

<?php

/*
 * After copying the file to /usr/share/openmediavault/engined/module run
 *   > monit restart omv-engined
 *   > omv-mkworkbench all
 * on the shell
 */

class OMVModuleDovecot extends \OMV\Engine\Module\ServiceAbstract
    implements \OMV\Engine\Module\IServiceStatus
{
    public function getName()
    {
        return "Dovecot";
    }

    public function getStatus()
    {
        $systemCtl = new \OMV\System\SystemCtl("dovecot");
        return array(
            "name" => $this->getName(),
            "title" => gettext("Dovecot"),
            "enabled" => $systemCtl->isEnabled(),
            "running" => $systemCtl->isActive()
        );
    }
}

To get an entry for Getmail, add the following file:

/usr/share/openmediavault/engined/module/getmail.inc

<?php

/*
 * After copying the file to /usr/share/openmediavault/engined/module run
 *   > monit restart omv-engined
 *   > omv-mkworkbench all
 * on the shell
 */


class OMVModuleGetmail extends \OMV\Engine\Module\ServiceAbstract
    implements \OMV\Engine\Module\IServiceStatus
{
    public function getName()
    {
        return "getmail6";
    }

    public function getStatus()
    {
        $cmd = new \OMV\System\Process("systemctl  --user --machine user@ is-active getmail-adr@user.account-at-gmx.net.service");
        $cmd->setQuiet(true);
        $cmd->execute($output, $exitStatus);
        return array(
            "name" => $this->getName(),
            "title" => gettext("Getmail"),
            "enabled" => true,
            "running" => (0 == $exitStatus) ? true : false
        );
    }
}

Then run with root privileges:

monit restart omv-engined
omv-mkworkbench all

It’s green, when the Dovecot is running, red when it’s stopped or not running.

pasted-from-clipboard.png

pasted-from-clipboard.png

When it’s disabled, it doesn’t show:

pasted-from-clipboard.png

Open Topics

What is still open is adding a Gmail account using OAuth for authentication. Getmail claims to be able to do that, and there are a few How-To’s explaining it. See:

getmail-5.6 XOAUTH2 configuration for gmail

https://web.archive.org/web/20240421182205/https://www3.isi.edu/~johnh/OTHER/LINUX/OAUTH2/index.html

But even though I do have a Gmail account, this is currently not my main priority. I will add it here, if / when I get this working.

Links

Dovecot | The Secure IMAP server

https://web.archive.org/web/20200807233801/https://joel.porquet.org/wiki/hacking/getmail_dovecot/

GitHub - tctlrd/getmail-container: dovecot and getmail6 docker image for private local imap storage when using a remote smtp server

dovecot and getmail6 docker image for private local imap storage when using a remote smtp server - tctlrd/getmail-container

github.com

Dovecot-IMAP-Server installieren (Debian) – MattWiki

https://www.baeldung.com/linux/systemd-create-user-services

Dovecot Full-Text Search - creative chaos

GitHub - grosjo/fts-xapian: Dovecot FTS plugin based on Xapian

Dovecot FTS plugin based on Xapian. Contribute to grosjo/fts-xapian development by creating an account on GitHub.

github.com

getmail-5.6 XOAUTH2 configuration for gmail

https://web.archive.org/web/20240421182205/https://www3.isi.edu/~johnh/OTHER/LINUX/OAUTH2/index.html

reserved