Sunday, 24 February 2013

Installing R Language support in Ruby

The R Language is a similar (yet open source) version of Metalab's MATLAB. R is available on most platforms, and there is a binding Gem for Ruby called RSRuby.

The installation is easy enough once you have R correctly installed.

Here are the steps on Ruby 1.8.7, with Gems >1.8 (probably the same on all version).

If you just try the basic Gem installation you get:

sudo gem install rsruby
Fetching: rsruby-0.5.1.1.gem (100%)
Building native extensions.  This could take a while...
ERROR:  Error installing rsruby:
ERROR: Failed to build gem native extension.

        /usr/bin/ruby1.8 extconf.rb
checking for main() in -lR... no

ERROR: Cannot find the R library, aborting.

*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers.  Check the mkmf.log file for more
details.  You may need configuration options.

Provided configuration options:
--with-opt-dir
--without-opt-dir
--with-opt-include
--without-opt-include=${opt-dir}/include
--with-opt-lib
--without-opt-lib=${opt-dir}/lib
--with-make-prog
--without-make-prog
--srcdir=.
--curdir
--ruby=/usr/bin/ruby1.8
--with-R-dir
--without-R-dir
--with-R-include
--without-R-include=${R-dir}/include
--with-R-lib
--without-R-lib=${R-dir}/lib
--with-Rlib
--without-Rlib

So you need to add some appropriate switches to enable the builder to find the R libraries.

Assuming you have R already installed, they are likely (under Ubuntu) to be at:

/usr/lib/R                       = binaries
/usr/share/R/include       = build libraries

First create an environment variable for the R binaries:

~$ export R_HOME=/usr/lib/R

then add this to your load path:

~$ export LD_LIBRARY_PATH=\$LD_LIBRARY_PATH:R_HOME/bin

Now, retry the installation:

~$ sudo gem install rsruby -- --with-R-dir=/usr/lib/R --with-R-include=/usr/share/R/include
Building native extensions.  This could take a while...
Successfully installed rsruby-0.5.1.1
1 gem installed
Installing ri documentation for rsruby-0.5.1.1...
Installing RDoc documentation for rsruby-0.5.1.1...

Test out the rsruby gem:

~$ irb
irb(main):001:0> require 'rubygems'
=> true
irb(main):002:0> require 'rsruby'
=> true
irb(main):003:0> r = RSRuby.instance
=> #-2147483648, "parse"=>#, "help"=>#, "NaN"=>NaN, "FALSE"=>false, "TRUE"=>true, "eval"=>#, "F"=>false, "T"=>true, "helpfun"=>#, "get"=>#}, @proc_table={}, @class_table={}, @default_mode=-1>
irb(main):004:0> r.help
=> "/usr/lib/R/library/base/help/NULL"
irb(main):005:0> exit

Just remember to set R_HOME permanently for your user in wherever you choose to add environment variables.

Saturday, 19 January 2013

Cinnamon on Ubuntu


Since switching to Ubuntu 12.04 I've moved to using Cinnamon as my DE. I did give Unity a serious go for a couple of weeks, but after one evening trying to use it to support a software release - switching between 3 browser windows and 2 SSH sessions, it became obvious it wasn't suitable. For many reasons, documented all over the web.

So I installed Cinnamon and despite a few teething problems (I think mainly caused by my nVidia card) it works beautifully. Nice weather, battery, and monitor applets; a simple, clean look, and easily configurable. As nice as Gnome used to be.

One issue I had recently though is that Wine applications place their menu items within the 'Other' area of the Menu (see this post), and when attempting to tidy this up, Cinnamon locked up and refused to restart. Even re-booting didn't solve the problem. I was kind of stuck as I couldn't see where the configuration was, or how it had failed. A remove/purge and re-install didn't fix anything (as configuration is held at user-level this is 'safe' to do. In that, you don't lose any settings/applets).

The solution arrived via this post (searching for the syslog failure message - Cinnamon seemed stuck at launch) - however as I'd never edited my Menu prior to this incident I had no 'backup' of the cinnamon-applications.menu file. However, reading through the first post, above, the installation version seems to be at /etc/xdg/menus/cinnamon-applications.menu so copying that to ~/.config/menus and restarting sorted things out.

Incidentally, following this stack-exchange entry I found that deleting entries from ~/.local/share/applications/wine/Programs and again in the ~/.config/menus/applications-merged area achieved much the same thing as editing the menu in the DE, with less chance of issues(!)

Thursday, 20 September 2012

Definitive Guide to Active and Passive FTP

...yes, another one :)

To start, let's get one thing clear. With TCP traffic there are always TWO ports involved, yours and mine; the client's and the server's.

Imagine I'm the client. And I'm calling a website (invoking a web server). I'll send in an HTTP request. With nothing 'special' on the url, I'll be calling the server's port 80. From one of 'my' ports. My port will be an Ephemeral one (literally).

So:
         Me                               You
         '50001'           ==>          80

I initiate the request on an ephemeral port, targeting a specific server port. You respond to me on that open (ephemeral) port.

Active FTP
Now, let's consider FTP Active mode.

This works in the same sort of way. But FTP uses two ports. One for the Commands (logging on, request a file, and so on) and another to ship data on. However, first the Command line is established.

Like HTTP, FTP will generally use a fixed port, 21, for the initial Command channel.


So:
         Me                               You
         '50001'           ==>          21

I initiate the request on an ephemeral port, targeting a specific server port. You respond to me on that open (ephemeral) port.

So far, so like HTTP.

Now, I ask for a file from You - a 'get' request. In Active mode I start this off by sending you a PORT command. This tells you which port to use my side for the data transfer. I'm being 'Active' and telling you which port to open my side.

         Me                               You                  Content
         '50001'           ==>          21                    PORT a,b,c,d,p1,p2


(the port command is my IP address a.b.c.d and the port to use which is found as (256 x p1) + p2)

At this point, I've told You (the client has told the server) to open a port on the client, to use as the Data channel. So the Server then opens that port on the Client, and sends the data down.


         Me                               You
         56001           <==          '32001'

The difference here is, although I've told you which port my side to use, it's You who initiates the transfer; you open the port on my side. Of course, like any 'client' (You are acting like a client now), You use an ephemeral port your side, to open that specific (although it'll be one of my ephemeral ones) port my side.

It's 'Active' as I'm the one controlling Port usage. The ephemeral bit is always the same - the client picks an outgoing port from the ephemeral pool, but always target a specific port on the server you call. Initially, and always, for the Command channel it's the standard you've agreed (typically, 21), but the Data channel will be an ephemeral one on the Client side, even though it's being specifically requested by the FTP server (on the advice of the active FTP client).

And this is the "issue" with Active FTP...

Consider HTTP traffic ... HTTP client (web browser) targets server port 80 of web server, using an ephemeral client port; server responds to request;
Consider SMTP traffic ... SMTP client (email agent) targets server port 587 of mail server, using an ephemeral client port; server responds to request;

If you're a firewall watching this network traffic, the client is always targeting known, specific remote targets; it's always initiating the request; it's always getting the response back on the open channel that it started.

But for FTP, although the Command channel follows that model, the Data channel doesn't. The firewall sees all these (seemingly) unsolicited requests coming in on non-standard ports, to non-server computers. It's the opposite direction to 'normal' traffic; and firewalls don't like that. This is why Active FTP can seem to work for a while, then stop when your firewall gets interesting.

Passive FTP
This options attempts to make FTP data transfers work in the same way as normal TCP traffic. The client always the client; the server always the server.

The Command channel works in the same way, but when a file is requested by the client, the response from the Server this time is You telling me which port to use on the Server.


So:
         Me                               You                          Content
         '50001'           ==>          21                            get file


The response to this request is You saying which port to use to fetch the file on. The port on the Server.


Thus:
         Me                               You
         '50002'           ==>          36001

i.e. the response gives Me the number of one of your ephemeral ports, which I then target specifically (with one of my ephemeral ports of course). In the response to this command, I get the file I want. I'm being 'passive'; You are telling Me which port on Your side to use. The direction of data flowing now matches the same profile as all other TCP standards, and no firewalls get anxious about things (well, the server needs to set aside a range of its ephemeral ports to pick from, and these are usually quite high, well away from any 'interesting' ones).


And there you have it :)  Use Passive FTP. You are being passive, ports are selected by the server, but it means that data flows in a standard manner.

ADDENDUM
With a Linux machine you can use the netstat command to see the current state of your TCP connections. This makes clear both the difference between My and Your port numbers, and shows which connections are currently in use:

netstat -an

tcp        0      0 192.168.0.7:45619       176.255.246.57:80       TIME_WAIT  
tcp        0      0 192.168.0.7:33474       212.73.212.117:80       ESTABLISHED
tcp        0      0 192.168.0.7:36383       176.255.246.56:80       TIME_WAIT  
tcp        1      0 192.168.0.7:50603       77.67.21.25:80          CLOSE_WAIT 
tcp        0      0 192.168.0.7:53067       176.255.246.72:80       TIME_WAIT  
tcp        0      0 192.168.0.7:58593       74.125.230.207:443      ESTABLISHED
tcp        0      0 192.168.0.7:59389       176.255.246.40:80       TIME_WAIT  
tcp        0      0 192.168.0.7:47264       199.59.150.41:443       ESTABLISHED
tcp        0      0 192.168.0.7:57111       74.125.132.125:5222     ESTABLISHED
tcp        0      0 192.168.0.7:47645       63.135.90.55:80         ESTABLISHED
tcp        0      0 192.168.0.7:59765       66.196.66.212:80        ESTABLISHED
tcp        1      0 192.168.0.7:54573       90.223.233.145:80       CLOSE_WAIT 
tcp        0      0 192.168.0.7:38578       64.13.137.78:80         TIME_WAIT  
tcp        0      0 192.168.0.7:59410       173.194.41.135:443      ESTABLISHED


The left-hand figures are My ports ... all in the ephemeral range as I've initiated the connection; Your ports are all standard numbers for what they are (80 being HTTP and 443 for HTTPS).

Saturday, 27 August 2011

Firefox "Caret Browsing"

A very short post ...

For a while my Firefox has been showing a blinking caret on all pages, even on pop-ups. Turns out I must have inadvertently hit 'f7' at some point which turns on a traditional text editor style caret for text selection (i.e one you can see).

There's nothing obvious in the options to disable this. But pressing 'f7' again sorts it out.   I wouldn't have thought to search for 'caret' in the options(!)


Thursday, 21 July 2011

Crontab Shells

Sometimes you'll be able to execute a Linux shell command fine from the command prompt, but fail when you use the same command from Cron. You might even have remembered to address the script/whatever correctly, and run Cron from the appropriate user, yet it might complain about the syntax of the actual command.

I've found that this is generally down to different shell interpreters being used. Although the documentation strongly implies that when a SHELL directive is not specified in a user-level Crontab, it will use the one specified in /etc/passwd, this doesn't seem to be the case.

I have a user with /bin/bash set-up, but it's clear in the output mailed by Cron that the SHELL is /bin/sh. 

So you might have to explicitly set a SHELL= environment in your crontab.

(I don't think it helps when Ubuntu distro's jump between bash and dash, but that's probably another tale...)

Addendum
Another important note for CRONTAB entries. The CRON program treats '%' as a line terminator. If your commands are failing, and the email you get seems to show only part of the command line, you probably have a '%' in the command. You need to either escape the % with a preceding '\' (e.g. "date +\%e") or pipe the command through sed.