Using Apache as a Reverse Proxy with Tailscale backend server

Apache as a Reverse Proxy

I started to use Tailscale to access my computers externally after moving to an ISP who use CGNAT. This enabled backups to and from my NAS from external servers I rented. I decided against opening up any of the machine within the Tailnet to full worldwide access, as that seemed too "open".

This works well, but I still had one website that I hosted on a RaspberryPi at home which I needed web access to. I originally just added my phone to the Tailnet, but there were problems with re-issuing LetsEncrypt certificates, and I wanted to retain my own domain name for the website, rather than use the .ts.net ones.

I had an external server on my Tailnet already running Apache, and decided the simple solution was to use that existing service to sit in front of the RaspberryPi and forward HTTP requests to it. After all, I only was using this website; it wasn't used by anyone else. This proved a little trickier than hoped, but eventually I got there, and these are the steps I went through...

Add VirtualHost to Apache

On the external server I added an website definition to Apache (i.e. in /etc/apache2/sites-available) which just consisted of the VirtualHost definition for the website domain:

<VirtualHost *:80>

    ServerName mydomain.com

    ProxyPass         "/" "http://tailnetname.xxxx-yyyy.ts.net/"

    ProxyPassReverse  "/" "https://mydomain.com/"

</VirtualHost>

where "mydomain.com" is the domain name you are exposing to the outside world, and "tailnetname.xxxx-yyyy" is the Tailnet name of the server actually hosting the website (in my case the RaspberryPi running locally).

Enable that site (a2ensite) and restart Apache.

Note that I'm using Apache to handle the SSL connection, and talking to the backend server over HTTP. This is just as secure, as all tailnet traffic is encrypted, and the tailnet name (.ts.net) is not exposed to the web (it has a CGNAT IP address, as do all machines in a tailnet). You could use HTTPS to get to the backend, but that seems a pointless overhead to me.

Issues I had later on were due to originally setting the ProxyPassReverse to the tailnet name, rather than mydomain.com - you want Apache to add headers to retain the external name of the website, not the tailnet name.

Adding LetsEncrypt Certificate

With the definition in Apache, use certbot to create your SSL certificate. certbot was already installed, so it was just a case of running "sudo cetbot certonly --apache" and letting certbot offer me the website to add the certificate to. I don't recall if this worked without any changes to the backend server, but I think it did. I manually updated the site definition in Apache, but you can let certbot do that by removing the 'certonly' parameter. Either way, you end up with the site definition amended with the following lines:

<VirtualHost *:80>

    ServerName mydomain.com

    RewriteEngine on

    RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]

</VirtualHost>

<VirtualHost *:443>

SSLCertificateFile /etc/letsencrypt/live/mydomain.com/fullchain.pem

SSLCertificateKeyFile /etc/letsencrypt/live/mydomain.com/privkey.pem

Include /etc/letsencrypt/options-ssl-apache.conf

- which are the standard lines to redirect any HTTP traffic to HTTPS, and point the site at the SSL certificates just created.

Changing the backend server

At this point, requests to https://mydomain.com should go via the external server, over your tailnet to the correct backend server (providing you have set the tailnet access rules appropriately - I have fairly open rules so nothing needed changing).

Although the connection was working, I kept getting 404 errors in the browser. This was caused by a number of issues, which needed fixing.

Firstly, the website definition was still setup to handle SSL traffic from the outside world, (as above) so the HTTP requests were being redirected to HTTPS.

The mod_rewrite rules and VirtualHost for port 443 needed removing.

Secondly, the VirtualHost definition still included mydomain.com as the ServerName. This meant Apache (on the RaspberryPi) was confused by the response pointing at mydomain.com, as it could be served locally, I think. In short I needed to just have the ServerName set as "tailnetname.xxxx-yyyy.ts.net" and remove any references to mydomain.com.

The final issue was I had the flask-talisman module installed in the web application - this does its own redirect of HTTP to HTTPS requests, and this was the final cause of the 404 responses (as there was no local handler for port 443 on the website). Maybe I could have avoided two of these issues by sticking with SSL on the backend, but there we are. I initially fixed this by just removing talisman, but eventually just changed the options, as I still wanted it to add the CSP headers, and so on.

Additional Changes

As the backend is now always receiving requests via the proxy, logging needs to handle the different HTTP headers, to record the actual external details, rather than those of the proxy.

onclick link not working? Check your input field names!

Had two almost identical web-pages, both using 'buttons' with onclick events linking to another page:

button onclick=location.href="...somewhere..."

After exhaustive fiddling, adding/removing javascript, checking the form method, playing around with submit types, it turned out that one of the input fields on the form was:

name='location'

so the onclick was just updating this input field, and not linking to the new page.

See my submit !=== submit page(!)

Upgrading/Replacing HDD with SSD

With the falling price of SSDs, it's a good time to dump the Hard Drive, and go for one.

Typically SSDs cost more per GB, and have less capacity - 1TB drives are appearing, but they are expensive. I decided to swap out my 500GB HDD for a 480GB SSD. 

It's quite straight-forward, and I adopted an approach of not updating any partition sizes on the old HDD, so I always had a working fall-back.

I split my drive into three areas, one for /boot, another for the root directory (/) and the final one for /home, where all user data resides. You might just have a single partition, making the whole process simpler.

Partition New Drive

First, attach the new drive to your existing computer, and use something like GPartEd to create a Partition Table, and add some partitions to the drive. It makes sense to keep this roughly in line with your old HDD, so in my case, I created an msdos Partition Table, and built 4 partitions:

• sda1 (/boot) for the boot partition of 500MB (ample for Ubuntu)
• sda5 as a swap partition (same size as your RAM)
• sda6 (/) for the root directory of 26GB (again, ample for Ubuntu)
• sda7 (/home) ... whatever is left

Mark sda1 as 'boot' and this is all there is to it.

The crucial point, is to make sure the boot and root partitions (sda1 and sda6 in my case) are at least as large as their existing counterparts on your HDD. You're going to clone the old partitions into them, and you can't clone into a smaller partition (obviously).

Copy Basic Partitions

Re-boot from a LiveCD (most have GPartEd on them) and with both old/new drives unmounted, copy across the partitions, using Copy/Paste in GPartEd. This will build the contents of sda1 and sda6 in my case.

You now have a working drive.

Copy User Data

My old /home folder was about 435GB; the new one 410GB, so I just used rsync to copy across the used data areas:

      rsync -av old/mount/point new/mount/point/       (watch trailing slashes!)

This obviously takes some time. Go to bed and let it whirl away.

You have now copied all data to the new drive.

Update fstab

The drive mount information held in /etc/fstab uses UUIDs (typically) or labels to identify the drives to mount at boot time, and where to mount them. It's likely that (at least) the (smaller) /home partition will have a different UUID, so use 'sudo blkid' to obtain a list of all the UUIDs and ensure that /etc/fstab (on the new SSD ... you'll need to mount the new drive, navigate to its /etc/fstab - for me on sda6 - and update the file there).

Switch to new Drive and boot

Remove the old HDD and replace with the SSD. Boot up.

(Repair Grub)

This might well fail. I was stuck on the prompt as copying the boot partition won't update Grub for your new SSD, however much you might hope it does.

From the grub prompt, check you have the drive partitions accessible by entering 'ls'. There should be your drive, and its partitions shown (as 'hd0,1', etc.) 

You need to first set the root, based on where your /boot partition is. For me, it's sda1:

grub> set root=(hd0,1)

Then point to a Ubuntu image ...

grub> linux /vmlinuz[-xxxxxx] root=/dev/sda6

(after "/vmlinuz" press TAB to see the image options you have, and choose the newest one which will fill in the [-xxxxxxx] part)

For 'root=' point to the root partition (not the boot partition) - sda6 for me.

Do similar TAB completion on the initrd setting:

grub> initrd /initrd[-xxxxxxx]

picking the initrd.img to match the version from above.

Then boot:

grub> boot

All being well, you should now boot to Ubuntu.

Make fix permanent

To re-build the Grub menu options, from command prompt, in a working, booted system:

sudo update-grub

(this will build the Grub menu options as normal).

sudo grub install /dev/sda

(place the generated options into the /boot partition)

Hopefully, you're now done.

Ubuntu 14.10 Upgrade

I'm now trying to keep fairly up-to-date on Ubuntu versions, and the upgrades are becoming easier to do - very little needs 'fixing' post-upgrade, even if you're not using a vanilla install (I don't user Unity but Cairo-dock, and prefer Nemo over Nautilus). I'm also having issue both recovering from Suspend (using Nouveau) and stability problems on some pages with Chromium.

Here are the set of changes required post upgrade to 14.10 from 14.04:

Zeitgeist
Zeitgeist is an irritating Unity hangover which builds a large sqlite database of files/searches/etc. on your local computer. Even with all options disabled it runs in the background, so I always kill it off by deleting it's autostart file from /etc/xdg.

Nemo
The Nemo package in the official repo's no longer handles the desktop (the effect of this is that the desktop does redraw the screen correctly ... the 'old' image is left behind when you move windows around) However there is a PPA which is maintained by the webupd8team and this fixes the issue. Add, or re-enable, their PPA, then uninstall Nemo (it will have been replaced with the official one and won't play happily with the PPA version) then reinstall it from the PPA.

... and that's about it!  

MySQL Loading Files

The method of loading records one at a time (via SQL Insert) is too slow when dealing with a large number of records, even if you choose to disable indexes before the load starts. Loading via a 'LOAD DATA' command is much faster, but there are a number of hurdles which you might face.

Firstly, there are two variations of this command:

• LOAD DATA INFILE, and 
• LOAD DATA LOCAL INFILE

- the difference between the two is where the file to be imported resides. In this case, 'LOCAL' means on the local machine performing the SQL command (in which case the MySQL client reads, the file, and it is then transferred to the MySQL server and loaded) and ommiting 'LOCAL' means the file is already on the server hosting the MySQL server.

Therefore, in either case the appropriate user (either the local, client user) or the remote, server user, must be able to read the file in question. 

A number of issues can arise with access being required at all folder levels up to the location of the file. See various ServerFault discussions on the matter. Best to place the upload file in a general temporary area which is "world-readable".

With LOCAL there are also additional security concerns, covered here.

These concerns mean that most distributions of MySQL do not, out of the box, permit use of the 'LOCAL' parameter, and you receive the "not supported in this version" error. This isn't strictly true - to make it work you need to start both Server and Client with a parameter to enable the use of local files. For the Server, it's 

• local-infile=1 in /etc/mysql/my.cnf

For the client, it's necessary to set the local_infile option in the database connect command (which varies by client) e.g. local_infile: true in database.yml for a Rails application.

However, it's better to avoid these issues, and just upload from a file on the MySQL server (if you have access). Again, the file itself must be readable, by the (typically) 'mysql' user ... on *nix, /tmp would be one such location.

One further problem you might encounter is that you get a 'file not found' error still ... one that doesn't imply a permissions error (e.g. not a ErrFile 13) but simply that the file doesn't exist even though it does, and is world-readable. If this occurs then check either auth.log or syslog to see if AppArmor is the problem:

Jan 26 12:43:45 localhost kernel: [13726.977235] type=1400 audit(1422276225.103:76): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/tmp/upload_file" pid=20692 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=114 ouid=1000

Here, you can see that before MySQL is passed the file, AppArmor has denied it access - effectively saying to MySQL that the file doesn't exist. To resolve this issue, update the MySQL permissions in AppArmor to include read-access to the location you wish to upload from, by editing /etc/apparmor.d/usr.bin.mysqld (or the local/ version):

  /tmp/* r,                        (...... your upload location)

  /run/mysqld/mysqld.pid rw,

  /run/mysqld/mysqld.sock w,

If you change the MySQL or AppArmor configuration, then restart the service.

Additionally, the connecting MySQL user will need the FILE privilege setting.

Update
Latest MySQL (5.7+) now lets you declare the location you'd like to use for loading your data from in secure_file_priv. Use SHOW VARIABLES LIKE "secure_file_priv" to see the location, and put your files there (setting the value if need be - can't be dynamically changed, so add to mysql.conf.d/mysqld.cnf ... or wherever). 

Apache Server Side Includes (SSI)

I have an ancient website on a shared hosting server ... one of those cheap deals that offers you a public_html sub-folder on a VM. Recently I needed to make some changes so copied the site across to a newer server. Everything was easy to set up, set for the life of me I couldn't get Server-side Includes (SSI) to work. This is pretty old technology, and I could have moved them into a different framework but I didn't have the time.

The steps that eventually sorted it were:

• enable mod-include
• update Options to include "+Include"
• if you're using .html rather than .shtml then also set "XBitHack On"

I wasn't getting any errors with this yet still the includes weren't working. I knew the file attributes must be right as they were copied from the working site. As this was a newer version of Apache I also added the 'Require all granted' in case that was the issue, but in the end it was this server fault entry finally sorted the problem -

if you are going to serve SSI via HTML you need to tell Apache that's your intention:

 AddHandler server-parsed .html
 AddType text/html .html
 AddOutputFilter INCLUDES .html

Setting up Dovecot and Roundcube on Ubuntu

One of the default applications I always install on Ubuntu is Postfix as an MTA. I find it very reliable, and along with Procmail is a good method for triggering jobs using inbound emails. However, I've never installed an end-to-end email solution - one where I could compose, view, and respond to emails from off the server. Recent changes to my current email provider has made me look at this option, and I now have a working solution. As with most all OSS there's if anything too much information out on the internet, and it's hard to find the wood.

Anyway, here is what I've done. Bear in mind that this is only for a 'personal' mail server ... not a corporate setup of 100s of users.

Dovecot

I installed Dovecot from the repositories - dovecot, dovecot-imap, and dovecot-pop3d.

Post-installation I amended my system users' mailbox format from mbox to maildir. This is quite straightforward as there was no mail I wanted to keep, so just building the ~/Maildir folders and adding them into /etc/skels was enough.

The configuration of Dovecot hardly needed any updates:

- changed 10-mail.conf to:

mail_location = maildir:~/Maildir

- changed 10-ssl.conf to:

ssl = yes

and I also created a separate log-file for dovecot in /var/log/dovecot.log with logrotation in /etc/logrotate.d

These changes meant I would be connecting to Dovecot via SSL and it would understand/expect Maildir format mail on the server.

Procmail

I already had Procmail running spamassassin, and some extra rules to intercept certain messages. Rather than replacing this with Sieve I kept these and just used the Dovecat LDA to deliver inbound mail to Maildir at the end of the existing rules.

Changes were limited to just:

MAILDIR=$HOME/Maildir/

DELIVER="/usr/lib/dovecot/deliver"

and -

:0 w

| $DELIVER

as the last lines in .procmailrc (I use user-level procmailrc files)

Postfix

As Postfix was already delivering to Procmail, I still used it's virtual_hosts and virtual.db to point external email addresses to internal system users. No virtual users (within Dovecot) were used.

Delivering mail now delivered email to the correct users (as before) but I now had external access via POP3S and IMAPS routes.

Mail

The 'mail' command in the default install for Ubuntu doesn't understand Maildir format, so I installed heirloom-mailx to replace it. You also need to update a few pam modules to let some services know to go to Maildir and not mbox. I changed ssh -

session    optional     pam_mail.so dir=~/Maildir standard # [1]

and su -

session    optional   pam_mail.so dir=~/Maildir nopen

So the notices you receive at logon regarding mail look in the correct place, and when switching user you can view the new users mail correctly.

Logging

Postfix was (still) logging to /var/log/mail.log and Dovecot to /var/log/dovecot.log. IMAP/POP3 authentication was in /var/log/auth.log.

Roundcube

For a web front-end I decided to install Roundcube. The version in the repositories is quite old (v0.7) and much newer versions are available to download from myroundcube. I followed the installation procedure 

(it's a little odd, but basically:

download gzip, de-compress and place somewhere your web server can access it;

create a mysql (for me anyway) database, and user, initialise with the SQL/initial.sql script

then point browser to the installer folder e.g. mymail/installer and follow instructions

)

This then enables you to logon to your email via this web interface using your system user/password.

Security

A couple of things worried me about this setup. One was that I don't use very strong passwords for my system users (as I use a PKI for security and disallow password logon), and as you access over IMAP/POP3 from the localhost I didn't want to leave just any user exposed. There are a myriad of solutions to this, but I went for this.

Fail2ban

I left access as localhost, rather than the server's IP address (I might want to genuinely access via a real external server at some point) so the auth.log and dovecot.log are a bit useless as they'll just report localhost as the remote IP address. Regardless I still set up fail2ban for dovecot checking and banning. The fail2ban I have doesn't have IPv6 capability so I updated dovecot.conf to "listen = *" so it would only listen on IPv4 addresses, and I'd avoid spurious warnings from fail2ban whenever a logon failed.

I added a fail2ban jail for roundcube itself as it logs logon failures. The Roundcube log by default is in /var/lib/roundcube/logs/errors so I created a filter in /etc/fail2ban/filter.d as roundcube.conf -

#

failregex = .*(IMAP|POP3) Error: Login failed for .* from \..*

# Option:  ignoreregex

# Notes.:  regex to ignore. If this regex matches, the line is ignored.

# Values:  TEXT

#

ignoreregex = 

- as the error message looked like:

[17-Aug-2014 18:30:40 +0100]: IMAP Error: Login failed for xxxx from aa.bb.cc.dd AUTHENTICAT... etc.

Then just included this new jail in jail.local:

[roundcube]

enabled = true

port    = http,https

filter  = roundcube

logpath = /var/lib/roundcube/logs/errors

maxretry = 3

So this means I'll block genuine failures at the web end, even though the IMAP/POP3 routes are still exposed.

In addition, rather than let any system user logon via Dovecot I decided to create new system users just for email with strong passwords and then explicitly give Dovecot access to these users. So I amended the pam for Dovecot to add this line -

auth required pam_listfile.so item=user sense=allow file=/etc/dovecot/users onerr=fail

and then in /etc/dovecot/users have the list of users who I want to logon in this way. This means I'll not have anyone brute-force using Dovecot to hack the root password (if there was one) or any other users than those I explicitly allow.

Roundcube Changes

There are a few quirks with Roundcube which I didn't much like so made a few simple changes.

Paragraph Spacing

There are options in Roundcube to determine what sort of HTML 'brackets' a new paragraph. Some of these replace the <p> with <br /> but as in the Roundcube FAQ I agree that a genuine 'return' should create a new paragraph. This results in 1em gaps between paragraphs which is different to how most email clients show paragraph breaks.

To amend this update the 'skin' being used to add a p{margin:0} - I found this needed changing both in the editor-content.css (for writing mail) and in the styles.css (for previewing/showing email).

Default Fonts

Although you can set the default font in user settings, this sets the font displayed in the editor, not the font selected in the editor (which is TinyMCE). There's a good solution in this Stack Exchange post and the code just goes in js/editor.js -

(within definition of conf, line 30-ish, add)

setup : function(ed) {
        ed.onInit.add(function(ed) {
            ed.execCommand("fontName", false, "Arial");
            ed.execCommand("fontSize", false, "2");
        });
    }

(I think there ought to be a way of extended the conf via the setting rcmail_editor_settings but haven't looked into this yet)

Email Replies

I don't like the indentation mechanism for replies. This leads to many indents and very narrow text on large conversations. I prefer to just give the original text and add to the top (like Outlook or Yahoo Mail) so I've changed the include of program/steps/mail/compose.inc:

(in rcmail_create_reply_body ... this probably doesn't show well, but you catch the drift!) -

        // // build reply (quote content)
        // $prefix = '<p>' . rcube::Q($prefix) . "</p>\n";
        // $prefix .= '<blockquote>';

        // if (intval($RCMAIL->config->get('reply_mode')) > 0) { // top-posting
        //     $prefix = '<br>' . $prefix;
        //     $suffix = '</blockquote>';
        // }
        // else {
        //     $suffix = '</blockquote><p></p>';
        // }

        $suffix = '';
        $prefix = sprintf(
            "<br /><p><hr/></p>" .
            "<b>%s:</b> %s<br />" .
            "<b>%s:</b> %s<br />" .
            "<b>%s:</b> %s<br />" .
            "<b>%s:</b> %s<br />" .
            "<br />",
            $RCMAIL->gettext('from'), rcube::Q($MESSAGE->get_header('from'), 'replace'),
            $RCMAIL->gettext('to'), rcube::Q($MESSAGE->get_header('to'), 'replace'),
            $RCMAIL->gettext('sent'), rcube::Q(format_date($MESSAGE->headers->date, $RCMAIL->config->get('date_long'))),
            $RCMAIL->gettext('subject'), rcube::Q($MESSAGE->subject));

        if ($cc = $MESSAGE->headers->get('cc'))
            $prefix .= sprintf("<tr><th align=\"right\" nowrap=\"nowrap\" valign=\"baseline\">%s: </th><td>%s</td></tr>",
                $RCMAIL->gettext('cc'), rcube::Q($cc, 'replace'));

        if (($replyto = $MESSAGE->headers->get('reply-to')) && $replyto != $MESSAGE->get_header('from'))
            $prefix .= sprintf("<tr><th align=\"right\" nowrap=\"nowrap\" valign=\"baseline\">%s: </th><td>%s</td></tr>",
                $RCMAIL->gettext('replyto'), rcube::Q($replyto, 'replace'));

This creates replies that look like:

Here is a response to that message

---

From: Me
To: "you@yourplace.co.uk"
Sent: 18/08/2014 11:01
Subject: Testing to me

Will this work?

And that's just about it!

UPDATE 2025

After updating to Roundcube 1.6.10, changes to TinyMCE CSS aren't automatically picked up. To handle this, amend the 'cache_suffix' value in program/js/editor.js (and editor.min.js, probably) - it just needs to be a different value to what was there before. Then, any changes to the TinyMCE CSS files (under program/js/tinymce/skins/...) will be picked up when the page refreshes.

ANOTHER UPDATE

To clarify the above, use browser debugger to determine which skin TinyMCE is using. In my case it was program/js/tinymce/skins/ui/oxide (the oxide one). In there, there are about 6 files, but the content.min.css is the one to edit - add the p{margin:0} as another line (it can be .mce-content-body prefixed if you want but the whole editor is in an iframe). Then edit program/js/editor.min.js cache_suffix value (as above). Re-load and it should work. The cache_suffix change might no longer be needed, as seems to append file modified timestamp to GET requests, but...

Ubuntu - some log files not rotating?

I find that quite of Ubuntu images aren't correctly set-up to rotate logs, as this is something not every user of a VM is interested in.

Two areas that typically have issues are:

Mail messages logged twice

This is where you'll find all mail server messages ending up in both /var/log/mail.log and then in one of mail.info/warn/err.  This is controlled within /etc/syslog.conf which maps system logger messages to their appropriate log file. 

Part of the config gives:

mail.* -/var/log/mail.log
user.* -/var/log/user.log

#
# Logging for the mail system.  Split it up so that
# it is easy to write scripts to parse these files.
#
mail.info -/var/log/mail.info
mail.warning -/var/log/mail.warn
mail.err         -/var/log/mail.err

which is the root of the problem - mail.* messages go once to mail.log and then split into three separate files. Just comment out the lower 3 lines to sort this out ... unless you actually want to split mail server messages into 3 log files.

Log Files Not being rotated

This occasionally happens. You'll see /var/log/syslog being rotated daily, and mysql (for instance) but not mail.log or kern.log or messages. This is due to a conflict between log rotation mechanisms employed by Ubuntu. 

From 12.04 (about) all log rotation moved into the domain of 'logrotate' utility (config in /etc/logrotate.conf and /etc/logrotate.d) but earlier versions split the work between this and the system logger itself. The system logger uses sysklogd CRON entries to perform log file rotation, which uses the utility /usr/sbin/syslogd-listfiles to obtain the list of files to rotate. This queries /etc/syslog.conf to determine whether a log file should be rotated daily or weekly.

If syslog is being rotated daily but no others, first check the output of "/usr/sbin/syslogd-listfiles --weekly" - this will list the files to rotate weekly. If this list is correct (i.e. it lists the other log files) then the problem might simply be that there's no sysklogd file in /etc/cron.weekly.  To fix this, just copy the file from cron.daily and amend the line after "cd /var/log" to :

logs=$(syslogd-listfiles --weekly)

The files should now start to be rotated weekly.

If there are no sysklogd entries in CRON then your system is using logrotate only, and the issue is not this one(!) 

Installing Git on Ubuntu

I'm in the middle of replacing Subversion with Git - more details on Git to come shortly, but here's a quick overview on installation.

The details on the Git website/book itself do a full installation. As I only need it on a server, then rather than the full UI additions, a simple installation from source is lighter ...

From the Git source repositories, grab the latest version -

https://code.google.com/p/git-core/downloads/list

and untar this using tar -xvf "downloaded file"

To install within a typical /usr/bin location, first make then make install as per the INSTALLATION file. So, that's just 'make', no 'make all', 'make doc'; just 'make' with an appropriate prefix.
Then run a make install (as root).

This will give you git without any documentation. To get the documentation, rather than build it, which is a heavy process (installing asciidoc is large!) grab the manpages tarball from the same location above, and untar it to the man pages area:

tar -xf "downloaded manpages file" -C /usr/share/man --no-same-owner

(more useful info here).

Replacing Nautilus with Nemo

As posted previously I've replaced Unity (meh) with Cinnamon on my Ubuntu laptop.

Cinnamon comes with a fork of the Nautilus file manager, called Nemo, which is a slight improvement I find. However, although various posts go through how to install Nemo, I found that it and Nautilus were competing to draw the desktop - such that on every other re-boot I was running one or the other.

So, as well as ensuring that any folder launch commands and file associations use Nemo over Nautilus, I also had to prevent Nautilus from auto-starting:

In Ubuntu the X-Server desktop menu set-up is held at /etc/xdg. Within this, folder autostart determines what starts automatically with the desktop. Remove the nautilus-autostart.desktop file from there (and make sure the Nemo install has added one). This should mean Nemo is preferred everywhere.

More excellent details here.

Setting up vsftp for Virtual Users (Ubuntu)

vsftpd setup for virtual users is fairly straght-forward but most guides don't go through the 'why', so here I'm attempting to explain this in brief.

Install vsftpd

- I'll assume you've already done this(!)

Update Configuration

The configuration file is typically in /etc/vsftpd.conf.

Make the following changes:

virtual_use_local_privs=YES
guest_enable=YES
guest_username=xxxxx
user_sub_token=$USER
local_root=/home/ftpusers/$USER
chroot_local_user=YES
hide_ids=YES
user_config_dir=/etc/vsftpd/vsftpd-user-conf

where,

guest_enable=YES
sets 'guest' logon allowed. With Virtual Users, you'll actually be logging on as this guest logon, so this is the id that needs the appropriate access to any folders you point the virtual user at.

guest_username=xxxxx
the local user to actually run under - should be a 'real' user, not a system one. Something like 'vftp' might be appropriate. Create them as 'normal' and either root the virtual user's in /home/vftp; or create a new folder structure and chown it to 'vftp'.

user_sub_token=$USER
how to identify the remote/virtual user. In this case using their 'user' name (what they logon with).

local_root=/home/vftp/$USER
which local root/home directory they will be sent to. So for a user 'test' they will start in '/home/vftp/test' in this example.

chroot_local_user=YES
'YES' means they will be confined to that local_root directory.

user_config_dir=/etc/vsftpd/vsftpd-user-conf 
A directory to use for 'per user' overrides. This is optional, but if you want one of your virtual users to operate under a different local user, or go to a different home directory you can create a file in this folder to do this.

Example 
You have virtual users 'test' and 'admin'. Test should have the basic access of '/home/vftp/test' and will create files as the (local) user 'xxxxx'. The 'admin' user you want to be able to access and update any of the virtual user's data within '/home/vftps'. 
Therefore create a file called 'admin' in /etc/vsftpd/vsftpd-user-conf with:

local_root=/home/vftp

which overrides the local_root setting and puts the user into the server at the higher-up node. 
This is where you could restrict access to 'read-only' if required at user-level (using write_enable=NO).

Amend PAM

The typical pam used (although it's set in the .conf file) is /etc/pam.d/vsftpd.

Update this to use just password authentication:

auth required pam_pwdfile.so pwdfile /etc/ftphtpasswd
account required pam_permit.so

where /etc/ftphtpasswd is the user/password file to hold your virtual users.
Everything else in the PAM can go - that's for enabling only locally defined (real) users to logon via ftp.
(there's a chance that the local PAM does not include pam_pwdfile ability - it's missing from /lib/security - in which case it can be installed from package libpam-pwdfile)

Create Users

Build the file mentioned above - /etc/ftphtpasswd using the htpasswd utility.

Verify

Logon with a suitable FTP client to check that each user is confined to the appropriate location, with the correct access rights.

(this info gathered in part from this post)


EDIT
*****
Ubuntu 12.04 (or maybe vsftpd) at some point decided not to like the crypt passwords htpasswd creates, so following guide here, use:

  sudo htpasswd -c -p -b ftphtpasswd user $(openssl passwd -1 -noverify password)

Installing R Language support in Ruby

The R Language is a similar (yet open source) version of Metalab's MATLAB. R is available on most platforms, and there is a binding Gem for Ruby called RSRuby.

The installation is easy enough once you have R correctly installed.

Here are the steps on Ruby 1.8.7, with Gems >1.8 (probably the same on all version).

If you just try the basic Gem installation you get:

sudo gem install rsruby

Fetching: rsruby-0.5.1.1.gem (100%)

Building native extensions.  This could take a while...

ERROR:  Error installing rsruby:

ERROR: Failed to build gem native extension.

        /usr/bin/ruby1.8 extconf.rb

checking for main() in -lR... no

ERROR: Cannot find the R library, aborting.

*** extconf.rb failed ***

Could not create Makefile due to some reason, probably lack of

necessary libraries and/or headers.  Check the mkmf.log file for more

details.  You may need configuration options.

Provided configuration options:

--with-opt-dir

--without-opt-dir

--with-opt-include

--without-opt-include=${opt-dir}/include

--with-opt-lib

--without-opt-lib=${opt-dir}/lib

--with-make-prog

--without-make-prog

--srcdir=.

--curdir

--ruby=/usr/bin/ruby1.8

--with-R-dir

--without-R-dir

--with-R-include

--without-R-include=${R-dir}/include

--with-R-lib

--without-R-lib=${R-dir}/lib

--with-Rlib

--without-Rlib

So you need to add some appropriate switches to enable the builder to find the R libraries.

Assuming you have R already installed, they are likely (under Ubuntu) to be at:

/usr/lib/R                       = binaries

/usr/share/R/include       = build libraries

First create an environment variable for the R binaries:

~$ export R_HOME=/usr/lib/R

then add this to your load path:

~$ export LD_LIBRARY_PATH=\$LD_LIBRARY_PATH:R_HOME/bin

Now, retry the installation:

~$ sudo gem install rsruby -- --with-R-dir=/usr/lib/R --with-R-include=/usr/share/R/include

Building native extensions.  This could take a while...

Successfully installed rsruby-0.5.1.1

1 gem installed

Installing ri documentation for rsruby-0.5.1.1...

Installing RDoc documentation for rsruby-0.5.1.1...

Test out the rsruby gem:

~$ irb

irb(main):001:0> require 'rubygems'

=> true

irb(main):002:0> require 'rsruby'

=> true

irb(main):003:0> r = RSRuby.instance

=> #-2147483648, "parse"=>#, "help"=>#, "NaN"=>NaN, "FALSE"=>false, "TRUE"=>true, "eval"=>#, "F"=>false, "T"=>true, "helpfun"=>#, "get"=>#}, @proc_table={}, @class_table={}, @default_mode=-1>

irb(main):004:0> r.help

=> "/usr/lib/R/library/base/help/NULL"

irb(main):005:0> exit

Just remember to set R_HOME permanently for your user in wherever you choose to add environment variables.

Cinnamon on Ubuntu

Since switching to Ubuntu 12.04 I've moved to using Cinnamon as my DE. I did give Unity a serious go for a couple of weeks, but after one evening trying to use it to support a software release - switching between 3 browser windows and 2 SSH sessions, it became obvious it wasn't suitable. For many reasons, documented all over the web.

So I installed Cinnamon and despite a few teething problems (I think mainly caused by my nVidia card) it works beautifully. Nice weather, battery, and monitor applets; a simple, clean look, and easily configurable. As nice as Gnome used to be.

One issue I had recently though is that Wine applications place their menu items within the 'Other' area of the Menu (see this post), and when attempting to tidy this up, Cinnamon locked up and refused to restart. Even re-booting didn't solve the problem. I was kind of stuck as I couldn't see where the configuration was, or how it had failed. A remove/purge and re-install didn't fix anything (as configuration is held at user-level this is 'safe' to do. In that, you don't lose any settings/applets).

The solution arrived via this post (searching for the syslog failure message - Cinnamon seemed stuck at launch) - however as I'd never edited my Menu prior to this incident I had no 'backup' of the cinnamon-applications.menu file. However, reading through the first post, above, the installation version seems to be at /etc/xdg/menus/cinnamon-applications.menu so copying that to ~/.config/menus and restarting sorted things out.

Incidentally, following this stack-exchange entry I found that deleting entries from ~/.local/share/applications/wine/Programs and again in the ~/.config/menus/applications-merged area achieved much the same thing as editing the menu in the DE, with less chance of issues(!)

Definitive Guide to Active and Passive FTP

...yes, another one :)

To start, let's get one thing clear. With TCP traffic there are always TWO ports involved, yours and mine; the client's and the server's.

Imagine I'm the client. And I'm calling a website (invoking a web server). I'll send in an HTTP request. With nothing 'special' on the url, I'll be calling the server's port 80. From one of 'my' ports. My port will be an Ephemeral one (literally).

So:
         Me                               You
         '50001'           ==>          80

I initiate the request on an ephemeral port, targeting a specific server port. You respond to me on that open (ephemeral) port.

Active FTP
Now, let's consider FTP Active mode.

This works in the same sort of way. But FTP uses two ports. One for the Commands (logging on, request a file, and so on) and another to ship data on. However, first the Command line is established.

Like HTTP, FTP will generally use a fixed port, 21, for the initial Command channel.


So:
         Me                               You
         '50001'           ==>          21

I initiate the request on an ephemeral port, targeting a specific server port. You respond to me on that open (ephemeral) port.

So far, so like HTTP.

Now, I ask for a file from You - a 'get' request. In Active mode I start this off by sending you a PORT command. This tells you which port to use my side for the data transfer. I'm being 'Active' and telling you which port to open my side.

         Me                               You                  Content
         '50001'           ==>          21                    PORT a,b,c,d,p1,p2


(the port command is my IP address a.b.c.d and the port to use which is found as (256 x p1) + p2)

At this point, I've told You (the client has told the server) to open a port on the client, to use as the Data channel. So the Server then opens that port on the Client, and sends the data down.


         Me                               You
         56001           <==          '32001'

The difference here is, although I've told you which port my side to use, it's You who initiates the transfer; you open the port on my side. Of course, like any 'client' (You are acting like a client now), You use an ephemeral port your side, to open that specific (although it'll be one of my ephemeral ones) port my side.

It's 'Active' as I'm the one controlling Port usage. The ephemeral bit is always the same - the client picks an outgoing port from the ephemeral pool, but always target a specific port on the server you call. Initially, and always, for the Command channel it's the standard you've agreed (typically, 21), but the Data channel will be an ephemeral one on the Client side, even though it's being specifically requested by the FTP server (on the advice of the active FTP client).

And this is the "issue" with Active FTP...

Consider HTTP traffic ... HTTP client (web browser) targets server port 80 of web server, using an ephemeral client port; server responds to request;
Consider SMTP traffic ... SMTP client (email agent) targets server port 587 of mail server, using an ephemeral client port; server responds to request;

If you're a firewall watching this network traffic, the client is always targeting known, specific remote targets; it's always initiating the request; it's always getting the response back on the open channel that it started.

But for FTP, although the Command channel follows that model, the Data channel doesn't. The firewall sees all these (seemingly) unsolicited requests coming in on non-standard ports, to non-server computers. It's the opposite direction to 'normal' traffic; and firewalls don't like that. This is why Active FTP can seem to work for a while, then stop when your firewall gets interesting.

Passive FTP
This options attempts to make FTP data transfers work in the same way as normal TCP traffic. The client always the client; the server always the server.

The Command channel works in the same way, but when a file is requested by the client, the response from the Server this time is You telling me which port to use on the Server.


So:
         Me                               You                          Content
         '50001'           ==>          21                            get file


The response to this request is You saying which port to use to fetch the file on. The port on the Server.


Thus:
         Me                               You
         '50002'           ==>          36001

i.e. the response gives Me the number of one of your ephemeral ports, which I then target specifically (with one of my ephemeral ports of course). In the response to this command, I get the file I want. I'm being 'passive'; You are telling Me which port on Your side to use. The direction of data flowing now matches the same profile as all other TCP standards, and no firewalls get anxious about things (well, the server needs to set aside a range of its ephemeral ports to pick from, and these are usually quite high, well away from any 'interesting' ones).


And there you have it :)  Use Passive FTP. You are being passive, ports are selected by the server, but it means that data flows in a standard manner.

ADDENDUM
With a Linux machine you can use the netstat command to see the current state of your TCP connections. This makes clear both the difference between My and Your port numbers, and shows which connections are currently in use:

netstat -an

tcp        0      0 192.168.0.7:45619       176.255.246.57:80       TIME_WAIT  
tcp        0      0 192.168.0.7:33474       212.73.212.117:80       ESTABLISHED
tcp        0      0 192.168.0.7:36383       176.255.246.56:80       TIME_WAIT  
tcp        1      0 192.168.0.7:50603       77.67.21.25:80          CLOSE_WAIT 
tcp        0      0 192.168.0.7:53067       176.255.246.72:80       TIME_WAIT  
tcp        0      0 192.168.0.7:58593       74.125.230.207:443      ESTABLISHED
tcp        0      0 192.168.0.7:59389       176.255.246.40:80       TIME_WAIT  
tcp        0      0 192.168.0.7:47264       199.59.150.41:443       ESTABLISHED
tcp        0      0 192.168.0.7:57111       74.125.132.125:5222     ESTABLISHED
tcp        0      0 192.168.0.7:47645       63.135.90.55:80         ESTABLISHED
tcp        0      0 192.168.0.7:59765       66.196.66.212:80        ESTABLISHED
tcp        1      0 192.168.0.7:54573       90.223.233.145:80       CLOSE_WAIT 
tcp        0      0 192.168.0.7:38578       64.13.137.78:80         TIME_WAIT  
tcp        0      0 192.168.0.7:59410       173.194.41.135:443      ESTABLISHED

The left-hand figures are My ports ... all in the ephemeral range as I've initiated the connection; Your ports are all standard numbers for what they are (80 being HTTP and 443 for HTTPS).