Calling external commands in awk

Today I had a file in which some lines were displaying a base 10 number. I wanted to translate this number in base36, possibly keeping the format as it was in the original file.
The entry was something along the lines of:


I decided to resort to awk and calling an external command to do the conversion.

Turns out, there are a couple of ways you can tell awk to fire off an external command. One is using system. The other is by simply stating the command in double quotes. There’s a nice forum discussion around this topic, and for once it’s not on StackOverflow.

Here’s the first draft of the solution.

awk '{
  if (/thisField=.([0-9]+)./) {
    match($1, "([0-9]+)", m);
    "python -c \"import numpy as np; print np.base_repr(" m[0] ", 36)\" " | getline converted;
    sub("([0-9]+)", converted);
  } else {
}' orig.dat > converted.dat

The trick is to pipe the result of the command to getline so that it can be saved into a variable, that is later used.
As a side topic, notice that this rough solution is extremely slow: a numpy import will happen for every line that matches. Other solutions for this particular problem exist, such as using bc after setting a proper obase or awk only.


Tricks for dealing with panes in tmux

Straight to the point: here’s a list of config lines/commands that I hope will ease your life as they have done with mine!

Write on all open panes at the same time

CTRL+B : to enter the tmux command line and setw synchronize-panes on.
Of course, set it “off” to disable sync’d writing.

Change panes layout

CTRL+Space to cycle through all layouts
CTRL+B META+15 to select one specific preset layout (even-horizontal, even-vertical, main-horizontal, main-vertical, tiled).
On my machine CTRL+B ESC 15 seems to do the trick instead.

Move panes around

CTRL+B CTRL+o to rotate them.
CTRL+B CTRL+{ and CTRL+B CTRL+} to move the active pane left/up or right/down.

Remote tail

You might have wondered why I felt the urge to specify “local” in the title of last post. Well, fast forward a few days since then, for a similar set of tests I also needed to check a log file on a remote Linux machine – that is, I needed some kind of remote tail.

ssh tail -f

We already know select will be part of our tool set. On top of that, we’ll need to forward the command across the network – and a nice way of doing that is over SSH. In Python, this task is relatively simple if you choose to use paramiko, a third party library implementing the SSHv2 protocol.

A few caveats here, as well. The following snippet is a raw prototype to demonstrate the functionality. It fit my bills, but YMMV. Of course many aspects can be improved, starting from instance with isBeginningOfMessage, which is much better placed in a derived class, so that different BOM patterns can be handled. Closing the SSH channel cleanly is also something you might want to polish before using this class.

import paramiko
import re
import select
import Queue

class SSHTail(object):
    """Tail a remote file and store new messages in a queue
    READ_ONLY = select.POLLIN | select.POLLPRI | select.POLLHUP | select.POLLERR
    TIMEOUT = 1000  # milliseconds
    BUF_SIZE = 1024
    NEWLINE_CHARS = {'\n', '\r'}

    def __init__(self, host, path): = host
        self.path = path
        self.poller = select.poll()
        self.messageQueue = Queue.deque()

    def start(self):
        """Start the tail command and return the queue used to
        store read messages
        client = paramiko.SSHClient()
        self.client = client

        transport = self.client.get_transport()
        self.transport = transport

        channel = self.transport.open_session()
        channel.exec_command("tail -F %s" % self.path) = channel

        self.poller.register(, self.READ_ONLY)

        return self.messageQueue

    def isBeginningOfMessage(line):
        """Return True if the line starts with the hardcoded Beginning of
        Message pattern
        BOMPattern = ''
        return re.match(BOMPattern, line)

    def loop(self):
        """Whilst the SSH tunnel is active, keep polling for new
        content and call parseBuffer() to parse it into messages
        while self.transport.is_active():
            events = self.poller.poll(self.TIMEOUT)
            for fd, flag in events:
                if flag & (select.POLLIN | select.POLLPRI):
                    buf =

    def parseBuffer(self, buf):
        """Given a buffer buf, split it into messages and glue it together to
        previous messages, if buf is not the beginning of a message.
        Note: assumes each message is on its on line.
        if buf:
            messages = buf.splitlines()

            oldest = messages[0]
            if not self.isBeginningOfMessage(oldest):
                    messages[0] = self.messageQueue.popleft() + oldest
                except IndexError:

            for message in messages:

Foreground and background (boy you turn me)

I’m sure Diana Ross would have named so her song, had she been a huge *nix fun.

What are we talking about? Job control. It’s simple if you already know it, but today I learnt some people didn’t know about putting shell jobs in background or restoring them in foreground.

The basics

When you run a command in a shell, it will take over the shell input until it’s done. And that’s called foreground.

Append an & at the end of the command line and the job will run in background, leaving the shell input free for other tasks. The job number will appear between square brackets.

To bring a foreground job in background, suspend it (Ctrl-Z) and issue bg.

To bring it back on foreground, type fg.

To see the list of jobs running in background, use the builtin jobs.

More fun

To reference a job in particular, you can use %n where n is the job number. So you could do, fg %3 or kill %7.

Alternatively, you can use %name where name is the command name used (for instance if you do man bash &, then you can refer to the job as %man).

? can be used to partially match strings: in the previous example, fg %?n or fg %?m would have referenced man bash.

jobs‘ output also tells you something more than just the list of jobs. A + is appended to the current job (last job stopped) and a - is appended to the previous job (i.e. that run before the current).
Guess what, you can use both signs to retrieve those jobs: enter %+ and %-.

For reasons I can’t explain, also % and %% refer to the current job, so watchout for typos.

Finally, % can also be used to run commands. In other words: %2 & is equivalent to bg %2 and %3 is equivalent to fg %3.

Change directory like a pro – 5 minutes intro

pushd and popd: it might take a bit to get used to them, but it’s worth it.

The basics

pushd some/path will simply take you to that directory (just as cd some/path does), with the added bonus that the old directory is kept in a FIFO queue.
popd will take you to the last path that was pushed to the queue.

The list of paths populating the queue will be output as soon as you hit either command.
Bonus: use dirs to list the entries in the queue without changing directory.

Sample session:

you@box:~> pwd
you@box:~> pushd ..
/home ~
you@box:/home> dirs
/home ~
you@box:/home> pushd ~/bin/
~/bin /home ~
you@box:~/bin> popd
/home ~

Additional tips and tricks

  • If you push the same path multiple times… the same path will appear multiple times.
  • Don’t want to see the output? Alias the commands and redirect the output to /dev/null
  • Want to save a few keystrokes? Alias the commands to something like p and o
  • Want to impress your friends? Move around with p [+-]n where n is the number of entries to skip in the queue. Example:
    you@box:~> dirs
    ~ ~/bin ~/bin /home
    you@box:~> p +3
    /home ~ ~/bin ~/bin

    (explanation: with the first command you can see that a) you find yourself in ~, b) the second and third entry is ~/bin, and c) /home is in fourth position. Count 3 from where you are, and you know you’ll land on /home).

Here a nice page that gives you more details on the subject. (I stole the aliases from there :)