Monday, 18 August 2014

Setting up Dovecot and Roundcube on Ubuntu

One of the default applications I always install on Ubuntu is Postfix as an MTA. I find it very reliable, and along with Procmail is a good method for triggering jobs using inbound emails. However, I've never installed an end-to-end email solution - one where I could compose, view, and respond to emails from off the server. Recent changes to my current email provider has made me look at this option, and I now have a working solution. As with most all OSS there's if anything too much information out on the internet, and it's hard to find the wood.

Anyway, here is what I've done. Bear in mind that this is only for a 'personal' mail server ... not a corporate setup of 100s of users.

Dovecot

I installed Dovecot from the repositories - dovecot, dovecot-imap, and dovecot-pop3d.

Post-installation I amended my system users' mailbox format from mbox to maildir. This is quite straightforward as there was no mail I wanted to keep, so just building the ~/Maildir folders and adding them into /etc/skels was enough.

The configuration of Dovecot hardly needed any updates:

- changed 10-mail.conf to:

mail_location = maildir:~/Maildir

- changed 10-ssl.conf to:

ssl = yes

and I also created a separate log-file for dovecot in /var/log/dovecot.log with logrotation in /etc/logrotate.d

These changes meant I would be connecting to Dovecot via SSL and it would understand/expect Maildir format mail on the server.

Procmail

I already had Procmail running spamassassin, and some extra rules to intercept certain messages. Rather than replacing this with Sieve I kept these and just used the Dovecat LDA to deliver inbound mail to Maildir at the end of the existing rules.

Changes were limited to just:


MAILDIR=$HOME/Maildir/

DELIVER="/usr/lib/dovecot/deliver"


and -

:0 w
| $DELIVER

as the last lines in .procmailrc (I use user-level procmailrc files)

Postfix

As Postfix was already delivering to Procmail, I still used it's virtual_hosts and virtual.db to point external email addresses to internal system users. No virtual users (within Dovecot) were used.

Delivering mail now delivered email to the correct users (as before) but I now had external access via POP3S and IMAPS routes.

Mail

The 'mail' command in the default install for Ubuntu doesn't understand Maildir format, so I installed heirloom-mailx to replace it. You also need to update a few pam modules to let some services know to go to Maildir and not mbox. I changed ssh -

session    optional     pam_mail.so dir=~/Maildir standard # [1]

and su -

session    optional   pam_mail.so dir=~/Maildir nopen

So the notices you receive at logon regarding mail look in the correct place, and when switching user you can view the new users mail correctly.

Logging

Postfix was (still) logging to /var/log/mail.log and Dovecot to /var/log/dovecot.log. IMAP/POP3 authentication was in /var/log/auth.log.

Roundcube

For a web front-end I decided to install Roundcube. The version in the repositories is quite old (v0.7) and much newer versions are available to download from myroundcube. I followed the installation procedure 

(it's a little odd, but basically:

download gzip, de-compress and place somewhere your web server can access it;
create a mysql (for me anyway) database, and user, initialise with the SQL/initial.sql script
then point browser to the installer folder e.g. mymail/installer and follow instructions
)

This then enables you to logon to your email via this web interface using your system user/password.

Security

A couple of things worried me about this setup. One was that I don't use very strong passwords for my system users (as I use a PKI for security and disallow password logon), and as you access over IMAP/POP3 from the localhost I didn't want to leave just any user exposed. There are a myriad of solutions to this, but I went for this.

Fail2ban
I left access as localhost, rather than the server's IP address (I might want to genuinely access via a real external server at some point) so the auth.log and dovecot.log are a bit useless as they'll just report localhost as the remote IP address. Regardless I still set up fail2ban for dovecot checking and banning. The fail2ban I have doesn't have IPv6 capability so I updated dovecot.conf to "listen = *" so it would only listen on IPv4 addresses, and I'd avoid spurious warnings from fail2ban whenever a logon failed.

I added a fail2ban jail for roundcube itself as it logs logon failures. The Roundcube log by default is in /var/lib/roundcube/logs/errors so I created a filter in /etc/fail2ban/filter.d as roundcube.conf -


#

failregex = .*(IMAP|POP3) Error: Login failed for .* from \..*


# Option:  ignoreregex
# Notes.:  regex to ignore. If this regex matches, the line is ignored.
# Values:  TEXT
#
ignoreregex = 

- as the error message looked like:

[17-Aug-2014 18:30:40 +0100]: IMAP Error: Login failed for xxxx from aa.bb.cc.dd AUTHENTICAT... etc.

Then just included this new jail in jail.local:


[roundcube]



enabled = true
port    = http,https
filter  = roundcube
logpath = /var/lib/roundcube/logs/errors
maxretry = 3

So this means I'll block genuine failures at the web end, even though the IMAP/POP3 routes are still exposed.

In addition, rather than let any system user logon via Dovecot I decided to create new system users just for email with strong passwords and then explicitly give Dovecot access to these users. So I amended the pam for Dovecot to add this line -

auth required pam_listfile.so item=user sense=allow file=/etc/dovecot/users onerr=fail

and then in /etc/dovecot/users have the list of users who I want to logon in this way. This means I'll not have anyone brute-force using Dovecot to hack the root password (if there was one) or any other users than those I explicitly allow.

Roundcube Changes

There are a few quirks with Roundcube which I didn't much like so made a few simple changes.

Paragraph Spacing
There are options in Roundcube to determine what sort of HTML 'brackets' a new paragraph. Some of these replace the <p> with <br /> but as in the Roundcube FAQ I agree that a genuine 'return' should create a new paragraph. This results in 1em gaps between paragraphs which is different to how most email clients show paragraph breaks.

To amend this update the 'skin' being used to add a p{margin:0} - I found this needed changing both in the editor-content.css (for writing mail) and in the styles.css (for previewing/showing email).

Default Fonts
Although you can set the default font in user settings, this sets the font displayed in the editor, not the font selected in the editor (which is TinyMCE). There's a good solution in this Stack Exchange post and the code just goes in js/editor.js -

(within definition of conf, line 30-ish, add)
setup : function(ed) {
        ed.onInit.add(function(ed) {
            ed.execCommand("fontName", false, "Arial");
            ed.execCommand("fontSize", false, "2");
        });
    }
(I think there ought to be a way of extended the conf via the setting rcmail_editor_settings but haven't looked into this yet)

Email Replies
I don't like the indentation mechanism for replies. This leads to many indents and very narrow text on large conversations. I prefer to just give the original text and add to the top (like Outlook or Yahoo Mail) so I've changed the include of program/steps/mail/compose.inc:

(in rcmail_create_reply_body ... this probably doesn't show well, but you catch the drift!) -

        // // build reply (quote content)
        // $prefix = '<p>' . rcube::Q($prefix) . "</p>\n";
        // $prefix .= '<blockquote>';

        // if (intval($RCMAIL->config->get('reply_mode')) > 0) { // top-posting
        //     $prefix = '<br>' . $prefix;
        //     $suffix = '</blockquote>';
        // }
        // else {
        //     $suffix = '</blockquote><p></p>';
        // }

        $suffix = '';
        $prefix = sprintf(
            "<br /><p><hr/></p>" .
            "<b>%s:</b> %s<br />" .
            "<b>%s:</b> %s<br />" .
            "<b>%s:</b> %s<br />" .
            "<b>%s:</b> %s<br />" .
            "<br />",
            $RCMAIL->gettext('from'), rcube::Q($MESSAGE->get_header('from'), 'replace'),
            $RCMAIL->gettext('to'), rcube::Q($MESSAGE->get_header('to'), 'replace'),
            $RCMAIL->gettext('sent'), rcube::Q(format_date($MESSAGE->headers->date, $RCMAIL->config->get('date_long'))),
            $RCMAIL->gettext('subject'), rcube::Q($MESSAGE->subject));

        if ($cc = $MESSAGE->headers->get('cc'))
            $prefix .= sprintf("<tr><th align=\"right\" nowrap=\"nowrap\" valign=\"baseline\">%s: </th><td>%s</td></tr>",
                $RCMAIL->gettext('cc'), rcube::Q($cc, 'replace'));

        if (($replyto = $MESSAGE->headers->get('reply-to')) && $replyto != $MESSAGE->get_header('from'))
            $prefix .= sprintf("<tr><th align=\"right\" nowrap=\"nowrap\" valign=\"baseline\">%s: </th><td>%s</td></tr>",
                $RCMAIL->gettext('replyto'), rcube::Q($replyto, 'replace'));

This creates replies that look like:

Here is a response to that message

From: Me
To: "you@yourplace.co.uk"
Sent: 18/08/2014 11:01
Subject: Testing to me
Will this work?


And that's just about it!

UPDATE 2025
After updating to Roundcube 1.6.10, changes to TinyMCE CSS aren't automatically picked up. To handle this, amend the 'cache_suffix' value in program/js/editor.js (and editor.min.js, probably) - it just needs to be a different value to what was there before. Then, any changes to the TinyMCE CSS files (under program/js/tinymce/skins/...) will be picked up when the page refreshes.


ANOTHER UPDATE
To clarify the above, use browser debugger to determine which skin TinyMCE is using. In my case it was program/js/tinymce/skins/ui/oxide (the oxide one). In there, there are about 6 files, but the content.min.css is the one to edit - add the p{margin:0} as another line (it can be .mce-content-body prefixed if you want but the whole editor is in an iframe). Then edit program/js/editor.min.js cache_suffix value (as above). Re-load and it should work. The cache_suffix change might no longer be needed, as seems to append file modified timestamp to GET requests, but...

Saturday, 19 July 2014

Ubuntu - some log files not rotating?

I find that quite of Ubuntu images aren't correctly set-up to rotate logs, as this is something not every user of a VM is interested in.

Two areas that typically have issues are:

Mail messages logged twice

This is where you'll find all mail server messages ending up in both /var/log/mail.log and then in one of mail.info/warn/err.  This is controlled within /etc/syslog.conf which maps system logger messages to their appropriate log file. 

Part of the config gives:

mail.* -/var/log/mail.log
user.* -/var/log/user.log

#
# Logging for the mail system.  Split it up so that
# it is easy to write scripts to parse these files.
#
mail.info -/var/log/mail.info
mail.warning -/var/log/mail.warn
mail.err         -/var/log/mail.err

which is the root of the problem - mail.* messages go once to mail.log and then split into three separate files. Just comment out the lower 3 lines to sort this out ... unless you actually want to split mail server messages into 3 log files.

Log Files Not being rotated

This occasionally happens. You'll see /var/log/syslog being rotated daily, and mysql (for instance) but not mail.log or kern.log or messages. This is due to a conflict between log rotation mechanisms employed by Ubuntu. 

From 12.04 (about) all log rotation moved into the domain of 'logrotate' utility (config in /etc/logrotate.conf and /etc/logrotate.d) but earlier versions split the work between this and the system logger itself. The system logger uses sysklogd CRON entries to perform log file rotation, which uses the utility /usr/sbin/syslogd-listfiles to obtain the list of files to rotate. This queries /etc/syslog.conf to determine whether a log file should be rotated daily or weekly.

If syslog is being rotated daily but no others, first check the output of "/usr/sbin/syslogd-listfiles --weekly" - this will list the files to rotate weekly. If this list is correct (i.e. it lists the other log files) then the problem might simply be that there's no sysklogd file in /etc/cron.weekly.  To fix this, just copy the file from cron.daily and amend the line after "cd /var/log" to :

logs=$(syslogd-listfiles --weekly)

The files should now start to be rotated weekly.

If there are no sysklogd entries in CRON then your system is using logrotate only, and the issue is not this one(!) 

Sunday, 12 May 2013

Installing Git on Ubuntu

I'm in the middle of replacing Subversion with Git - more details on Git to come shortly, but here's a quick overview on installation.

The details on the Git website/book itself do a full installation. As I only need it on a server, then rather than the full UI additions, a simple installation from source is lighter ...

From the Git source repositories, grab the latest version -

https://code.google.com/p/git-core/downloads/list

and untar this using tar -xvf "downloaded file"

To install within a typical /usr/bin location, first make then make install as per the INSTALLATION file. So, that's just 'make', no 'make all', 'make doc'; just 'make' with an appropriate prefix.
Then run a make install (as root).

This will give you git without any documentation. To get the documentation, rather than build it, which is a heavy process (installing asciidoc is large!) grab the manpages tarball from the same location above, and untar it to the man pages area:

tar -xf "downloaded manpages file" -C /usr/share/man --no-same-owner

(more useful info here).

Tuesday, 19 March 2013

Replacing Nautilus with Nemo

As posted previously I've replaced Unity (meh) with Cinnamon on my Ubuntu laptop.

Cinnamon comes with a fork of the Nautilus file manager, called Nemo, which is a slight improvement I find. However, although various posts go through how to install Nemo, I found that it and Nautilus were competing to draw the desktop - such that on every other re-boot I was running one or the other.

So, as well as ensuring that any folder launch commands and file associations use Nemo over Nautilus, I also had to prevent Nautilus from auto-starting:

In Ubuntu the X-Server desktop menu set-up is held at /etc/xdg. Within this, folder autostart determines what starts automatically with the desktop. Remove the nautilus-autostart.desktop file from there (and make sure the Nemo install has added one). This should mean Nemo is preferred everywhere.

More excellent details here.

Friday, 1 March 2013

Setting up vsftp for Virtual Users (Ubuntu)


vsftpd setup for virtual users is fairly straght-forward but most guides don't go through the 'why', so here I'm attempting to explain this in brief.


Install vsftpd

- I'll assume you've already done this(!)


Update Configuration

The configuration file is typically in /etc/vsftpd.conf.

Make the following changes:


virtual_use_local_privs=YES
guest_enable=YES
guest_username=xxxxx
user_sub_token=$USER
local_root=/home/ftpusers/$USER
chroot_local_user=YES
hide_ids=YES
user_config_dir=/etc/vsftpd/vsftpd-user-conf


where,

guest_enable=YES

sets 'guest' logon allowed. With Virtual Users, you'll actually be logging on as this guest logon, so this is the id that needs the appropriate access to any folders you point the virtual user at.

guest_username=xxxxx

the local user to actually run under - should be a 'real' user, not a system one. Something like 'vftp' might be appropriate. Create them as 'normal' and either root the virtual user's in /home/vftp; or create a new folder structure and chown it to 'vftp'.

user_sub_token=$USER

how to identify the remote/virtual user. In this case using their 'user' name (what they logon with).

local_root=/home/vftp/$USER
which local root/home directory they will be sent to. So for a user 'test' they will start in '/home/vftp/test' in this example.

chroot_local_user=YES
'YES' means they will be confined to that local_root directory.

user_config_dir=/etc/vsftpd/vsftpd-user-conf 
A directory to use for 'per user' overrides. This is optional, but if you want one of your virtual users to operate under a different local user, or go to a different home directory you can create a file in this folder to do this.

Example 

You have virtual users 'test' and 'admin'. Test should have the basic access of '/home/vftp/test' and will create files as the (local) user 'xxxxx'. The 'admin' user you want to be able to access and update any of the virtual user's data within '/home/vftps'. 
Therefore create a file called 'admin' in /etc/vsftpd/vsftpd-user-conf with:

local_root=/home/vftp

which overrides the local_root setting and puts the user into the server at the higher-up node. 
This is where you could restrict access to 'read-only' if required at user-level (using write_enable=NO).

Amend PAM

The typical pam used (although it's set in the .conf file) is /etc/pam.d/vsftpd.

Update this to use just password authentication:

auth required pam_pwdfile.so pwdfile /etc/ftphtpasswd
account required pam_permit.so


where /etc/ftphtpasswd is the user/password file to hold your virtual users.
Everything else in the PAM can go - that's for enabling only locally defined (real) users to logon via ftp.
(there's a chance that the local PAM does not include pam_pwdfile ability - it's missing from /lib/security - in which case it can be installed from package libpam-pwdfile)

Create Users

Build the file mentioned above - /etc/ftphtpasswd using the htpasswd utility.


Verify

Logon with a suitable FTP client to check that each user is confined to the appropriate location, with the correct access rights.

(this info gathered in part from this post)


EDIT
*****
Ubuntu 12.04 (or maybe vsftpd) at some point decided not to like the crypt passwords htpasswd creates, so following guide here, use:

  sudo htpasswd -c -p -b ftphtpasswd user $(openssl passwd -1 -noverify password)