Calling external commands in awk

Today I had a file in which some lines were displaying a base 10 number. I wanted to translate this number in base36, possibly keeping the format as it was in the original file.
The entry was something along the lines of:


I decided to resort to awk and calling an external command to do the conversion.

Turns out, there are a couple of ways you can tell awk to fire off an external command. One is using system. The other is by simply stating the command in double quotes. There’s a nice forum discussion around this topic, and for once it’s not on StackOverflow.

Here’s the first draft of the solution.

awk '{
  if (/thisField=.([0-9]+)./) {
    match($1, "([0-9]+)", m);
    "python -c \"import numpy as np; print np.base_repr(" m[0] ", 36)\" " | getline converted;
    sub("([0-9]+)", converted);
  } else {
}' orig.dat > converted.dat

The trick is to pipe the result of the command to getline so that it can be saved into a variable, that is later used.
As a side topic, notice that this rough solution is extremely slow: a numpy import will happen for every line that matches. Other solutions for this particular problem exist, such as using bc after setting a proper obase or awk only.


sort -k

TIL, when you call sort -k3, you’re not just sorting by the third field, but by whatever the value between the third field up to the end of the line is.
Not only that, in the case of ties, by default it will use also the first field.

Consider this example.

$ cat data
theta AAA 2
gamma AAA 2
alpha BBB 2
alpha AAA 3

Sorting with -k2 gives:

$ sort data -k2 --debug
sort: using simple byte comparison
gamma AAA 2
theta AAA 2
alpha AAA 3
alpha BBB 2 

Notice I’ve also added --debug, to show which parts are used in the comparisons.
So, first comes “AAA 2”, then “AAA 3”.
Also, for the two lines that have “AAA 2”, the first field is used, so “gamma” comes before “theta”.

Forget about the ties for now.
To consider field 2 only, rather than field 2 and all following fields, you need to specify a stop. This is done by adding “,2” to the -k switch. More in general, -km,n means “sort by field m up to n, boundaries included”.

$ sort data -k2,2 --debug
sort: using simple byte comparison
alpha AAA 3
gamma AAA 2
theta AAA 2
alpha BBB 2 

As you can see, field 2 only is taken into account at first.
“AAA 3” comes before “AAA 2” because, being a tie, the first field is used as a second comparison.

Taking this a step further, to actually only consider field 2 and resort to the original order in case of ties, that is, to have a stable sort, you need to pass the -s switch.

$ sort data -k2,2 -s --debug
sort: using simple byte comparison
theta AAA 2
gamma AAA 2
alpha AAA 3
alpha BBB 2 

This look similar to the first snippet, but actually the first two lines in the output are swapped. Here they appear in the original order.

Remote tail

You might have wondered why I felt the urge to specify “local” in the title of last post. Well, fast forward a few days since then, for a similar set of tests I also needed to check a log file on a remote Linux machine – that is, I needed some kind of remote tail.

ssh tail -f

We already know select will be part of our tool set. On top of that, we’ll need to forward the command across the network – and a nice way of doing that is over SSH. In Python, this task is relatively simple if you choose to use paramiko, a third party library implementing the SSHv2 protocol.

A few caveats here, as well. The following snippet is a raw prototype to demonstrate the functionality. It fit my bills, but YMMV. Of course many aspects can be improved, starting from instance with isBeginningOfMessage, which is much better placed in a derived class, so that different BOM patterns can be handled. Closing the SSH channel cleanly is also something you might want to polish before using this class.

import paramiko
import re
import select
import Queue

class SSHTail(object):
    """Tail a remote file and store new messages in a queue
    READ_ONLY = select.POLLIN | select.POLLPRI | select.POLLHUP | select.POLLERR
    TIMEOUT = 1000  # milliseconds
    BUF_SIZE = 1024
    NEWLINE_CHARS = {'\n', '\r'}

    def __init__(self, host, path): = host
        self.path = path
        self.poller = select.poll()
        self.messageQueue = Queue.deque()

    def start(self):
        """Start the tail command and return the queue used to
        store read messages
        client = paramiko.SSHClient()
        self.client = client

        transport = self.client.get_transport()
        self.transport = transport

        channel = self.transport.open_session()
        channel.exec_command("tail -F %s" % self.path) = channel

        self.poller.register(, self.READ_ONLY)

        return self.messageQueue

    def isBeginningOfMessage(line):
        """Return True if the line starts with the hardcoded Beginning of
        Message pattern
        BOMPattern = ''
        return re.match(BOMPattern, line)

    def loop(self):
        """Whilst the SSH tunnel is active, keep polling for new
        content and call parseBuffer() to parse it into messages
        while self.transport.is_active():
            events = self.poller.poll(self.TIMEOUT)
            for fd, flag in events:
                if flag & (select.POLLIN | select.POLLPRI):
                    buf =

    def parseBuffer(self, buf):
        """Given a buffer buf, split it into messages and glue it together to
        previous messages, if buf is not the beginning of a message.
        Note: assumes each message is on its on line.
        if buf:
            messages = buf.splitlines()

            oldest = messages[0]
            if not self.isBeginningOfMessage(oldest):
                    messages[0] = self.messageQueue.popleft() + oldest
                except IndexError:

            for message in messages:

Local tail

tail -f

A while ago I needed, for one of my tests, to monitor a log file on a Linux system and store any new lines, so that I could access the added content at the end of the test. In a sense, I needed a kind of buffered tail -f on a local file.

A quick search led me to the select module.
Without further ado, here’s the code to watch one or more files, and to store anything added to those files in a message queue.

It’s a quick and dirty version which can be improved in many ways. For starters, the keys to access the message queues are the sockets themselves, pretty useless in general, but good enough in my case. Second, notice the file is never closed explicitly: definitely not ideal.

import Queue
import select

class Watcher(object):
    TIMEOUT = 1000
    READ_ONLY = select.POLLIN | select.POLLPRI | select.POLLHUP | select.POLLERR

    def __init__(self):
        """Initialize the Watcher"""
        self.poller = select.poll()

        self.fd_to_socket = {}
        self.message_queues = {}

    def addFile(self, path):
        """Add a file to monitor.
        :path: absolute path of the file, including the filename
        f = open(path)
        self.poller.register(f, self.READ_ONLY)
        self.fd_to_socket[f.fileno()] = f 
        self.message_queues[f] = Queue.deque()

    def start(self):
        """Start polling files"""
        while True:
            events = self.poller.poll(self.TIMEOUT)
            for fd, flag in events:
                s = self.fd_to_socket[fd]
                if flag & (select.POLLIN | select.POLLPRI):
                    lines = s.readlines()
                    if lines:

Fear and loathing when compiling the kernel

Make menuconfig.

Carefully sift through the myriad of kernel options and select only the ones you need, dreaming of how fast your machine will be with the kernel you’re crafting.

Wait for your family to be asleep and spend the rest of the night on the interwebs to find out whether CONFIG_PPS_CLIENT_GPIO should be compiled as a module or built-in, if you should enable CONFIG_CAN as an excuse to buy that dongle you’ve been longing for a while and questioning every “if unsure, say N” – because you might well be unsure, but you certainly don’t want to miss out on that cool feature.

Be pleased with yourself when you have what you think is the perfect kernel for your box.

Save the config, but don’t use the default .config name: give it a different name for no particular reason, like kconfig, and exit.

make menuconfig again, load .config while you want to perform your last few tweaks. Be surprised when you find out CONFIG_HW_RANDOM_AMD, CONFIG_HW_RANDOM_INTEL and CONFIG_HW_RANDOM_VIA are all enabled, though you were sure all architecture-dependent options were already taken care of. Disable the two that don’t apply it and save as .config. Save it also as kconfig, just in case. Realise you’ve just overwritten the result of a dozen nights of lost sleep.

Count to ten, trying not to curse. Curse yourself when you reach 23. Exhale, slowly. Close the lid and get some sleep.

GOTO step 1 the day after.

Be even more pleased with yourself when you have another new shiny .config – also saved with different names to 8 other places, including a USB stick that you buried in the socks drawer and the SD card of your camera. Love the moment when you see you could shave off another handful of options and allow yourself to grin all day long when you go and come back to work. Remember you haven’t shaved since the first make menuconfig when a kid offers you a place on the bus, despite the fact you’re just 32.

make-kpkg your kernel and dpkg -i it. Smile and reboot.

Expect to select your kernel in GRUB and be a bit disappointed that you entirely missed the boot menu. Realise you’re still smiling anyway.

At the splash screen, wonder for a split second why neither keyboard nor touchpad work. Realise you can’t log in into the box and stop smiling suddenly.

Detach yourself and say to no one in particular “alright, I’ll just ssh in from the other box.” Try to remember whether sshd has started.

Start to realise a few things.

Realise in horror it’s not even installed.

Realise that you can’t reboot using the the older kernel, since the boot menu is hidden (you didn’t miss it: it really wasn’t there).

Realise you’re stuck. You’ve sort of locked yourself out your laptop. Realise this is the series of dumbest rookie errors you’ve done in ages.

Realise that, if there were an award for the stupidest thing, you’d win it hands down.

Give up on ssh and think about logging in using the on-screen keyboard. You still need a way to input characters.

Pray that you’ve enabled the support for a USB mouse. It’s kind of basic, but after what happened you’re doubting yourself about every single thing.

Locate in your head where you last saw a mouse in the house and run like there’s no tomorrow praying it’s still there. Find it and almost shed a tear of joy when you carry your mouse back to the laptop with both hands, like you just found the Graal.

Select each character of your 20-char password and be glad you’re alone and you won’t have to change your password(s) after these shenanigans.

Try to fire up the on-screen keyboard now that you’re logged in. Be lost wondering why this doesn’t work. Breathe slowly and resort to the char map. Admittedly not the best, decidedly better than nothing. Take three minutes to write sudo su and the password, char by char. As you can only find letters but no numbers or punctuation charaters in this stupid char map, open another shell and paste ll. Now you can copy paste numbers and some punctuation.

Can’t find the dash but you can write Devanagari in your shell

Open another shell and copy paste h i s t o r y. OK, now you’ve even got a selection of the most common commands. Realize you’re on the road to victory. Or at least, pretend so.

Now copy paste apt-get install sshd. Error. apt-cache search for it. Obviously it’s not sshd. It’s openssh-server. Install it. Error. Can’t find the repo. Move your head towards the top right corner and despair when you understand that wireless support is broken.

You will not be able to ssh in here. Ever.

Panic a bit and try a dpkg -P linux-image-4.1.6. Read a kind notice warning you you’re about to do a(nother) stupid(TM) thing. Wonder how you can select either ‘OK’ or ‘Cancel’ in this dialog, since you can’t paste control characters. Maybe this is a good thing after all.

Decide to edit the GRUB menu. Last time you checked, you needed to edit menu.lst. Learn that things have moved on and you now have to operate on /etc/defaults/grub. Open nautilus as root, thank the gods when you see that gedit is installed and regret all the times you said “I don’t need anything else outside vi“.

See that GRUB_HIDDEN_TIMEOUT_QUIET is set to true and GRUB_TIMEOUT is set to 0. Wonder why people think it’s always best when you hide things from sight. Change to true, 10. update-grub and reboot.

See the GRUB menu in all its uselessness, displaying a countdown from 10 with no other option.

You’re still on that crippled 4.1.6. Repeat all the weird stuff you just did, but at 10x speed. You’re now a semi-pro of on-screen keyboards, char maps and char copy/pasting.

Open /boot/grub/grub.cfg. Pause a second to thank for gedit again. See that the most-recent non-custom kernel is the third entry in the menu. So, without thinking twice set GRUB_DEFAULT from 0 to 2 in /etc/default/grub. Update-grub and reboot.

Accept that you messed up big time, as the system now enters the BIOS without even showing the useless countdown. You’re now on the path to desperation. Change boot settings and get yourself into a busybox environment. Feel like you’ve just seen the Matrix. At least now the keyboard works. <TAB> <TAB>. There are maybe 30 commands. vi is not one of them.

Think about mounting your partition, you’ll think how to modify grub.cfg once you get there.
Try to remember the layout of your HD. Screw it, just do mount -t ext4 /dev/sdaN /tmp/hd for N in 1 .. 6. Cry of joy at N=4. Try /tmp/hd/usr/bin/vi.tiny /tmp/hd/boot/grub/grub.cfg and be relieved watching vi compose the text of the file. Manually change every pointer to vmlinuz-4.1.6 to the last working kernel. Save. Reboot.

Back to normality.

Grep for ‘KEYBOARD‘ and ‘MOUSE‘ just for fun you kconfig (and all other 8 copies) and let your jaw drop when you see that neither CONFIG_KEYBOARD_ATKBD nor CONFIG_MOUSE_PS2 were selected.

GOTO beginning, again. Make menuconfig and go for another spin.

But not tonight.

Misconfigured Python pretty printers in GDB

I noticed that running an executable in gdb displayed an error – and only the first time you run the program. The error reads:

(gdb) r
Starting program: /home/chris/workinprogress/cpp/book/a.out 
Traceback (most recent call last):
  File "/usr/share/gdb/auto-load/usr/lib/x86_64-linux-gnu/", line 63, in 
    from libstdcxx.v6.printers import register_libstdcxx_printers
ImportError: No module named 'libstdcxx'
[Inferior 1 (process 7215) exited normally]

The problem is, the libstdcxx directory is not in the path.
The very simple fix is to add the directory to the Python path in gdbinit.

$ cat ~/.gdbinit
import sys
sys.path.insert(0, '/usr/share/gcc-4.8/python')
from libstdcxx.v6.printers import register_libstdcxx_printers
register_libstdcxx_printers (None)

The journey from a crash logged in /var/log/messages to the offending line in the code

Recently I had to investigate a process crash. Usually, I grab the generated coredump and feed it to gdb. Usually, this works. Unless someone has already deleted the core file by mistake.

Fear not, there’s a solution. And here I am to document it before I forget it and spend another half an hour on Google.
Look in /var/log/messages the line that references the crash:

> sudo cat /var/log/messages | grep binary_that_crashed
Feb 30 17:20:22 zeus kernel: [9232272.493642] binary_that_crashed[12868] trap divide error ip:7f11de8d4095 sp:7f11daf7d990 error:0 in[7f11de876000+88000]

Then work out the location in the code by subtracting the address where the lib was loaded from the address of the instruction pointer:

> python -c "print '0x%x' % (0x7f11de8d4095-0x7f11de876000)"

Finally, get the offending line with either addr2line:

> addr2line 0x5e095 -fe /usr/lib/

where the mangled name of the method, _ZN3XYZ14ThisMethodName4plusEv can be demangled using c++filt:

> c++filt _ZN3XYZ14ThisMethodName4plusEv

or objdump:

> objdump -DCgl | grep -B1 5e095
   5e095:  48 f7 f1             div    %rcx

And while we’re more or less on topic, here’s a not-too-random selection of pointers to the stack (ha!):

Foreground and background (boy you turn me)

I’m sure Diana Ross would have named so her song, had she been a huge *nix fun.

What are we talking about? Job control. It’s simple if you already know it, but today I learnt some people didn’t know about putting shell jobs in background or restoring them in foreground.

The basics

When you run a command in a shell, it will take over the shell input until it’s done. And that’s called foreground.

Append an & at the end of the command line and the job will run in background, leaving the shell input free for other tasks. The job number will appear between square brackets.

To bring a foreground job in background, suspend it (Ctrl-Z) and issue bg.

To bring it back on foreground, type fg.

To see the list of jobs running in background, use the builtin jobs.

More fun

To reference a job in particular, you can use %n where n is the job number. So you could do, fg %3 or kill %7.

Alternatively, you can use %name where name is the command name used (for instance if you do man bash &, then you can refer to the job as %man).

? can be used to partially match strings: in the previous example, fg %?n or fg %?m would have referenced man bash.

jobs‘ output also tells you something more than just the list of jobs. A + is appended to the current job (last job stopped) and a - is appended to the previous job (i.e. that run before the current).
Guess what, you can use both signs to retrieve those jobs: enter %+ and %-.

For reasons I can’t explain, also % and %% refer to the current job, so watchout for typos.

Finally, % can also be used to run commands. In other words: %2 & is equivalent to bg %2 and %3 is equivalent to fg %3.

Cannot run a program as root because you “Cannot open display”

Say you’re trying a program as root. You sudo, you type the program name and then… bang.

(wireshark:2901): Gtk-WARNING **: cannot open display: localhost:30.0

Here’s the fix: remember the “localhost:30.0”, sudo into a root shell, run xauth to create/merge root’s Xauthority file and use the same display:

sudo -i
xauth merge ~your_username/.Xauthority
export DISPLAY=localhost:30.0

If you don’t get the display name (in the error message) when first running the program, you can retrieve it by echoing $DISPLAY.