Definitive Guide to Active and Passive FTP

...yes, another one :)

To start, let's get one thing clear. With TCP traffic there are always TWO ports involved, yours and mine; the client's and the server's.

Imagine I'm the client. And I'm calling a website (invoking a web server). I'll send in an HTTP request. With nothing 'special' on the url, I'll be calling the server's port 80. From one of 'my' ports. My port will be an Ephemeral one (literally).

So:
         Me                               You
         '50001'           ==>          80

I initiate the request on an ephemeral port, targeting a specific server port. You respond to me on that open (ephemeral) port.

Active FTP
Now, let's consider FTP Active mode.

This works in the same sort of way. But FTP uses two ports. One for the Commands (logging on, request a file, and so on) and another to ship data on. However, first the Command line is established.

Like HTTP, FTP will generally use a fixed port, 21, for the initial Command channel.


So:
         Me                               You
         '50001'           ==>          21

I initiate the request on an ephemeral port, targeting a specific server port. You respond to me on that open (ephemeral) port.

So far, so like HTTP.

Now, I ask for a file from You - a 'get' request. In Active mode I start this off by sending you a PORT command. This tells you which port to use my side for the data transfer. I'm being 'Active' and telling you which port to open my side.

         Me                               You                  Content
         '50001'           ==>          21                    PORT a,b,c,d,p1,p2


(the port command is my IP address a.b.c.d and the port to use which is found as (256 x p1) + p2)

At this point, I've told You (the client has told the server) to open a port on the client, to use as the Data channel. So the Server then opens that port on the Client, and sends the data down.


         Me                               You
         56001           <==          '32001'

The difference here is, although I've told you which port my side to use, it's You who initiates the transfer; you open the port on my side. Of course, like any 'client' (You are acting like a client now), You use an ephemeral port your side, to open that specific (although it'll be one of my ephemeral ones) port my side.

It's 'Active' as I'm the one controlling Port usage. The ephemeral bit is always the same - the client picks an outgoing port from the ephemeral pool, but always target a specific port on the server you call. Initially, and always, for the Command channel it's the standard you've agreed (typically, 21), but the Data channel will be an ephemeral one on the Client side, even though it's being specifically requested by the FTP server (on the advice of the active FTP client).

And this is the "issue" with Active FTP...

Consider HTTP traffic ... HTTP client (web browser) targets server port 80 of web server, using an ephemeral client port; server responds to request;
Consider SMTP traffic ... SMTP client (email agent) targets server port 587 of mail server, using an ephemeral client port; server responds to request;

If you're a firewall watching this network traffic, the client is always targeting known, specific remote targets; it's always initiating the request; it's always getting the response back on the open channel that it started.

But for FTP, although the Command channel follows that model, the Data channel doesn't. The firewall sees all these (seemingly) unsolicited requests coming in on non-standard ports, to non-server computers. It's the opposite direction to 'normal' traffic; and firewalls don't like that. This is why Active FTP can seem to work for a while, then stop when your firewall gets interesting.

Passive FTP
This options attempts to make FTP data transfers work in the same way as normal TCP traffic. The client always the client; the server always the server.

The Command channel works in the same way, but when a file is requested by the client, the response from the Server this time is You telling me which port to use on the Server.


So:
         Me                               You                          Content
         '50001'           ==>          21                            get file


The response to this request is You saying which port to use to fetch the file on. The port on the Server.


Thus:
         Me                               You
         '50002'           ==>          36001

i.e. the response gives Me the number of one of your ephemeral ports, which I then target specifically (with one of my ephemeral ports of course). In the response to this command, I get the file I want. I'm being 'passive'; You are telling Me which port on Your side to use. The direction of data flowing now matches the same profile as all other TCP standards, and no firewalls get anxious about things (well, the server needs to set aside a range of its ephemeral ports to pick from, and these are usually quite high, well away from any 'interesting' ones).


And there you have it :)  Use Passive FTP. You are being passive, ports are selected by the server, but it means that data flows in a standard manner.

ADDENDUM
With a Linux machine you can use the netstat command to see the current state of your TCP connections. This makes clear both the difference between My and Your port numbers, and shows which connections are currently in use:

netstat -an

tcp        0      0 192.168.0.7:45619       176.255.246.57:80       TIME_WAIT  
tcp        0      0 192.168.0.7:33474       212.73.212.117:80       ESTABLISHED
tcp        0      0 192.168.0.7:36383       176.255.246.56:80       TIME_WAIT  
tcp        1      0 192.168.0.7:50603       77.67.21.25:80          CLOSE_WAIT 
tcp        0      0 192.168.0.7:53067       176.255.246.72:80       TIME_WAIT  
tcp        0      0 192.168.0.7:58593       74.125.230.207:443      ESTABLISHED
tcp        0      0 192.168.0.7:59389       176.255.246.40:80       TIME_WAIT  
tcp        0      0 192.168.0.7:47264       199.59.150.41:443       ESTABLISHED
tcp        0      0 192.168.0.7:57111       74.125.132.125:5222     ESTABLISHED
tcp        0      0 192.168.0.7:47645       63.135.90.55:80         ESTABLISHED
tcp        0      0 192.168.0.7:59765       66.196.66.212:80        ESTABLISHED
tcp        1      0 192.168.0.7:54573       90.223.233.145:80       CLOSE_WAIT 
tcp        0      0 192.168.0.7:38578       64.13.137.78:80         TIME_WAIT  
tcp        0      0 192.168.0.7:59410       173.194.41.135:443      ESTABLISHED

The left-hand figures are My ports ... all in the ephemeral range as I've initiated the connection; Your ports are all standard numbers for what they are (80 being HTTP and 443 for HTTPS).

Firefox "Caret Browsing"

A very short post ...

For a while my Firefox has been showing a blinking caret on all pages, even on pop-ups. Turns out I must have inadvertently hit 'f7' at some point which turns on a traditional text editor style caret for text selection (i.e one you can see).

There's nothing obvious in the options to disable this. But pressing 'f7' again sorts it out.   I wouldn't have thought to search for 'caret' in the options(!)

Crontab Shells

Sometimes you'll be able to execute a Linux shell command fine from the command prompt, but fail when you use the same command from Cron. You might even have remembered to address the script/whatever correctly, and run Cron from the appropriate user, yet it might complain about the syntax of the actual command.

I've found that this is generally down to different shell interpreters being used. Although the documentation strongly implies that when a SHELL directive is not specified in a user-level Crontab, it will use the one specified in /etc/passwd, this doesn't seem to be the case.

I have a user with /bin/bash set-up, but it's clear in the output mailed by Cron that the SHELL is /bin/sh. 

So you might have to explicitly set a SHELL= environment in your crontab.

(I don't think it helps when Ubuntu distro's jump between bash and dash, but that's probably another tale...)

Addendum
Another important note for CRONTAB entries. The CRON program treats '%' as a line terminator. If your commands are failing, and the email you get seems to show only part of the command line, you probably have a '%' in the command. You need to either escape the % with a preceding '\' (e.g. "date +\%e") or pipe the command through sed.

Monit for Monitoring - check your default page!

I have a few servers that I look after at work, and having spent too long staring at them day after day, night after night, just to make sure they are working, I decided I needed some better alerting and monitoring.

Being a Ruby fan I looked closely at both 'god' and 'blue-pill', but at the end of the day decided against both. Partly due to concerns raised about memory-usage with a Ruby-based package (although most of these fears seem to have been allayed in recent versions), but also because you really don't want a big footprint on a monitoring product. You should not need to monitor a monitor.

So at the end of a little research I installed 'Monit' on the servers. This has a nice little web interface (secured) that will enable you both to monitor, and restart monitored services. Ideal if you can't SSH onto a server and just need to get things back up and running. The monitoring file syntax is simple and there are examples for the common sorts of services you might want to monitor (MySQL, Postfix, Apache, ...) You can be simply alerted on certain conditions (storage, CPU, #children) or have corrective action taken (restarting a service; stopping it; so on).

I've found at lest one thing to be wary of though. You can monitor a "web process" (e.g. Apache) by checking once per 'monitoring cycle' that your specific port responds to a specific protocol. This can go as far as checking the response from the protocol - e.g. test that page 'home.html' returns some regular expression. Quite nice, but rarely needed - Apache is rock solid in my experience. Simply knowing the process is active is good enough. However, if you *do* choose to run a default  'protocol HTTP' test, then this will 'GET /' from your site. I was looking at my MySQL logs and noticed a connection and a couple of SELECTs every 5 minutes. Baffling, until I noticed they coincided with the Monit probes as seen in the Apache logs. The website in question sits at the base of the domain, and when sending the home page also checks to see if a cookie is present in the database, and if not builds a default response, from default settings held in the database - hence the calls to the database. I'm not happy about polling my database server every 5 minutes (see one of my previous posts!) so I've now reverted to a simple port 80 TCP probe - that just checks the port is open rather than sends in an HTTP request. I feel this is good enough ... as I said, if Apache is there at all, then it's working.

So you might have a large home page with lots of dynamic content. Be aware that that page will be fetched by Monit whenever it checks your web server. Probably not an issue for most, but weigh the cost against the value.

When submit !=== submit

One of those little 'gotchas' in HTML/Javascript ... if you're happily attempting to submit a form using the 'document.form.submit()' tied to an event (such as an 'onChange' on a select box) then you must make sure that 'submit' is actually the function/method 'submit' and not an element on your page.

By that, I mean, you might have a "normal" submit button that you've called 'submit'. And that will "not be a function" so it won't go submitting the form if you invoke it.

Nice post on it: http://thedesignspace.net/MT2archives/000292.html

In the past I've always "blanket-covered" my submit button by using id='submit', name='submit', and type='submit'. Not so in future.