Remote tail

You might have wondered why I felt the urge to specify “local” in the title of last post. Well, fast forward a few days since then, for a similar set of tests I also needed to check a log file on a remote Linux machine – that is, I needed some kind of remote tail.

ssh tail -f

We already know select will be part of our tool set. On top of that, we’ll need to forward the command across the network – and a nice way of doing that is over SSH. In Python, this task is relatively simple if you choose to use paramiko, a third party library implementing the SSHv2 protocol.

A few caveats here, as well. The following snippet is a raw prototype to demonstrate the functionality. It fit my bills, but YMMV. Of course many aspects can be improved, starting from instance with isBeginningOfMessage, which is much better placed in a derived class, so that different BOM patterns can be handled. Closing the SSH channel cleanly is also something you might want to polish before using this class.

import paramiko
import re
import select
import Queue


class SSHTail(object):
    """Tail a remote file and store new messages in a queue
    """
    READ_ONLY = select.POLLIN | select.POLLPRI | select.POLLHUP | select.POLLERR
    TIMEOUT = 1000  # milliseconds
    BUF_SIZE = 1024
    NEWLINE_CHARS = {'\n', '\r'}

    def __init__(self, host, path):
        self.host = host
        self.path = path
        self.poller = select.poll()
        self.messageQueue = Queue.deque()

    def start(self):
        """Start the tail command and return the queue used to
        store read messages
        """
        client = paramiko.SSHClient()
        client.load_system_host_keys()
        client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
        client.connect(self.host)
        self.client = client

        transport = self.client.get_transport()
        transport.set_keepalive(1)
        self.transport = transport

        channel = self.transport.open_session()
        channel.exec_command("tail -F %s" % self.path)
        self.channel = channel

        self.poller.register(self.channel, self.READ_ONLY)

        return self.messageQueue

    @staticmethod
    def isBeginningOfMessage(line):
        """Return True if the line starts with the hardcoded Beginning of
        Message pattern
        """
        BOMPattern = ''
        return re.match(BOMPattern, line)

    def loop(self):
        """Whilst the SSH tunnel is active, keep polling for new
        content and call parseBuffer() to parse it into messages
        """
        while self.transport.is_active():
            events = self.poller.poll(self.TIMEOUT)
            for fd, flag in events:
                if flag & (select.POLLIN | select.POLLPRI):
                    buf = self.channel.recv(self.BUF_SIZE)
                    self.parseBuffer(buf)

    def parseBuffer(self, buf):
        """Given a buffer buf, split it into messages and glue it together to
        previous messages, if buf is not the beginning of a message.
        
        Note: assumes each message is on its on line.
        """
        if buf:
            messages = buf.splitlines()

            oldest = messages[0]
            if not self.isBeginningOfMessage(oldest):
                try:
                    messages[0] = self.messageQueue.popleft() + oldest
                except IndexError:
                    pass

            for message in messages:
                self.messageQueue.appendleft(message)

Foreground and background (boy you turn me)

I’m sure Diana Ross would have named so her song, had she been a huge *nix fun.

What are we talking about? Job control. It’s simple if you already know it, but today I learnt some people didn’t know about putting shell jobs in background or restoring them in foreground.

The basics

When you run a command in a shell, it will take over the shell input until it’s done. And that’s called foreground.

Append an & at the end of the command line and the job will run in background, leaving the shell input free for other tasks. The job number will appear between square brackets.

To bring a foreground job in background, suspend it (Ctrl-Z) and issue bg.

To bring it back on foreground, type fg.

To see the list of jobs running in background, use the builtin jobs.

More fun

To reference a job in particular, you can use %n where n is the job number. So you could do, fg %3 or kill %7.

Alternatively, you can use %name where name is the command name used (for instance if you do man bash &, then you can refer to the job as %man).

? can be used to partially match strings: in the previous example, fg %?n or fg %?m would have referenced man bash.

jobs‘ output also tells you something more than just the list of jobs. A + is appended to the current job (last job stopped) and a - is appended to the previous job (i.e. that run before the current).
Guess what, you can use both signs to retrieve those jobs: enter %+ and %-.

For reasons I can’t explain, also % and %% refer to the current job, so watchout for typos.

Finally, % can also be used to run commands. In other words: %2 & is equivalent to bg %2 and %3 is equivalent to fg %3.

Change directory like a pro – 5 minutes intro

pushd and popd: it might take a bit to get used to them, but it’s worth it.

The basics

pushd some/path will simply take you to that directory (just as cd some/path does), with the added bonus that the old directory is kept in a FIFO queue.
popd will take you to the last path that was pushed to the queue.

The list of paths populating the queue will be output as soon as you hit either command.
Bonus: use dirs to list the entries in the queue without changing directory.

Sample session:

you@box:~> pwd
/home/you
you@box:~> pushd ..
/home ~
you@box:/home> dirs
/home ~
you@box:/home> pushd ~/bin/
~/bin /home ~
you@box:~/bin> popd
/home ~
you@box:/home>

Additional tips and tricks

  • If you push the same path multiple times… the same path will appear multiple times.
  • Don’t want to see the output? Alias the commands and redirect the output to /dev/null
  • Want to save a few keystrokes? Alias the commands to something like p and o
  • Want to impress your friends? Move around with p [+-]n where n is the number of entries to skip in the queue. Example:
    you@box:~> dirs
    ~ ~/bin ~/bin /home
    you@box:~> p +3
    /home ~ ~/bin ~/bin
    you@box:/home>

    (explanation: with the first command you can see that a) you find yourself in ~, b) the second and third entry is ~/bin, and c) /home is in fourth position. Count 3 from where you are, and you know you’ll land on /home).

Here a nice page that gives you more details on the subject. (I stole the aliases from there :)