Foreground and background (boy you turn me)

I’m sure Diana Ross would have named so her song, had she been a huge *nix fun.

What are we talking about? Job control. It’s simple if you already know it, but today I learnt some people didn’t know about putting shell jobs in background or restoring them in foreground.

The basics

When you run a command in a shell, it will take over the shell input until it’s done. And that’s called foreground.

Append an & at the end of the command line and the job will run in background, leaving the shell input free for other tasks. The job number will appear between square brackets.

To bring a foreground job in background, suspend it (Ctrl-Z) and issue bg.

To bring it back on foreground, type fg.

To see the list of jobs running in background, use the builtin jobs.

More fun

To reference a job in particular, you can use %n where n is the job number. So you could do, fg %3 or kill %7.

Alternatively, you can use %name where name is the command name used (for instance if you do man bash &, then you can refer to the job as %man).

? can be used to partially match strings: in the previous example, fg %?n or fg %?m would have referenced man bash.

jobs‘ output also tells you something more than just the list of jobs. A + is appended to the current job (last job stopped) and a - is appended to the previous job (i.e. that run before the current).
Guess what, you can use both signs to retrieve those jobs: enter %+ and %-.

For reasons I can’t explain, also % and %% refer to the current job, so watchout for typos.

Finally, % can also be used to run commands. In other words: %2 & is equivalent to bg %2 and %3 is equivalent to fg %3.

Cannot run a program as root because you “Cannot open display”

Say you’re trying a program as root. You sudo, you type the program name and then… bang.

(wireshark:2901): Gtk-WARNING **: cannot open display: localhost:30.0

Here’s the fix: remember the “localhost:30.0”, sudo into a root shell, run xauth to create/merge root’s Xauthority file and use the same display:

sudo -i
xauth merge ~your_username/.Xauthority
export DISPLAY=localhost:30.0

If you don’t get the display name (in the error message) when first running the program, you can retrieve it by echoing $DISPLAY.

Pdb cheat sheet

I often need a table with all the commands for the Python debugger (Pdb) but so far I couldn’t find a comprehensive one. So I wrote one. Enjoy!

h(elp) or ? print available commands
h(elp) command print help about command
q(uit) quit debugger
! stmt or exec stmt execute the one-line statement stmt
p(rint) expr print the value of expression
pp expr pretty-print the value of expression
whatis obj print the type of obj
bt or w(here) print stack trace from oldest frame to newest
l(ist) list 11 lines of source code around the current line
l m, n list from line m to line n
args print the arguments of the current function
alias [name [command [parameter parameter ...] ]] create an alias called name that executes command
unalias name remove alias
<ENTER> repeat the last command entered
n(ext) execute the current statement (step over)
s(tep) step into function
j(ump) n set line n the next line to be executed
r(eturn) continue execution until the current function returns
c(ontinue) continue execution until a breakpoints is encountered
unt(il) continue execution until reaching the line greater than the current one or the current frame returns
run or restart (re-)run the program
u(p) move one level up in the stack trace
d(own) move one level down in the stack trace
b(reak) show breakpoints
tb(reak) [filename:]n set a temporary breakpoint at line n of file filename
b [filename:]n set a breakpoint at line n of file filename
b fun set a breakpoint at function fun
disable bn1 [bn2, ...] disable breakpoint(s) bn1, bn2, …
enable bn1 [bn2, ...] enable breakpoint(s) bn1, bn2, …
clear [bn] clear breakpoint number bn; if no number is specified, clear them all
commands [bn] specify commands to run at breakpoint bn
condition bn expr expression expr must evaluate to true in order for breakpoint bn to be hit. If no expression is specified, the breakpoint is made unconditional

String-building performance – a small experiment

I wanted to create a collection of all ASCII letters (both upper- and lowercase) and single digits in Python.

The first (IMO most Pythonic) way that I came up with uses the standard string concatenation operator:

import string
a = string.ascii_letters + string.digits

Then I wondered, would it be faster to use the usual methods to format strings?
Say, a = "%s%s" % (string.ascii_letters, string.digits) or a = "{0}{1}".format(string.ascii_letters, string.digits)?

Only timeit.timeit can tell.

>>> timeit("a = string.ascii_letters + string.digits", setup="import string", number=10000000)
1.4662139415740967
>>> timeit("a = \"%s%s\" % (string.ascii_letters, string.digits)", setup="import string", number=10000000)
2.8996360301971436
>>> timeit("a = \"{0}{1}\".format(string.ascii_letters, string.digits)", setup="import string", number=10000000)
12.925632953643799

Interesting, innit. Now, a word of caution: these results are limited to this particular example. Results may vary (and probably do) if the “domain of the experiment” changes – that is, if you use different strings in terms of number, length, characters, etc.

PEP 257 and your docstrings

PEP 257 is the PEP devoted to Docstrings.

I’ll be brief here, just listing main do’s and dont’s from the PEP – go to the source if you’re hungry for more information.

Basics

What happens if you don’t comply with PEP 257?

  • You’ll be contributing to increasing the entropy, thereby accelerating the end of the universe.
  • Your co-workers might also be slightly upset with you.
  • And if you use Docutils or similar tools, results could be disappointing.

General guidelines

  • Use triple double quotes (even if the string fits on one line).
  • No blank lines before or after the docstring (except for classes, see below).
  • A docstring is a phrase ending in a period written as a command (return the input rounded) not as a description (this function returns...).
  • Don’t put the signature of a function: this can be retrieved (starting with Py3.3) via the __annotations__ attribute.
  • One-line docstrings:
    • The closing quotes are on the same line as the opening quotes.
  • Multi-line docstrings
    • First line should be a summary. Second line is a blank line. Following lines detail the behaviour of the object.
    • All lines to be indented as the opening quotes (first character on any line is at the same level of the first ").

Object-specific guidelines

  • Script-specific:
    • the docstring should be re-used (re-usable, really) as the “usage” message.
  • Module specific:
    • List any object exported by the module (classes, exceptions, functions, etc.), with a one-line summary of each. Further detail is given in each of those objects’ docstring.
  • Package-specific:
    • the docstring of the package’s __init__.py module should list the modules and subpackages exported by the package.
  • Function/method-specific:
    • summarise its behavior and document its arguments (both mandatory and optional), return values, side effects, exceptions raised, and restrictions on when it can be called.
  • Class-specific:
    • Summarise its behavior, list the public methods and instance variables.
    • Insert a blank line after the docstring, before all methods.
    • The class constructor should be documented in the docstring for its __init__ method.
  • Subclass-specific:
    • Use the verb “override” to indicate that a subclass method replaces a superclass method and does not call the superclass method.
    • Use the verb “extend” to indicate that a subclass method calls the superclass method.

Change directory like a pro – 5 minutes intro

pushd and popd: it might take a bit to get used to them, but it’s worth it.

The basics

pushd some/path will simply take you to that directory (just as cd some/path does), with the added bonus that the old directory is kept in a FIFO queue.
popd will take you to the last path that was pushed to the queue.

The list of paths populating the queue will be output as soon as you hit either command.
Bonus: use dirs to list the entries in the queue without changing directory.

Sample session:

you@box:~> pwd
/home/you
you@box:~> pushd ..
/home ~
you@box:/home> dirs
/home ~
you@box:/home> pushd ~/bin/
~/bin /home ~
you@box:~/bin> popd
/home ~
you@box:/home>

Additional tips and tricks

  • If you push the same path multiple times… the same path will appear multiple times.
  • Don’t want to see the output? Alias the commands and redirect the output to /dev/null
  • Want to save a few keystrokes? Alias the commands to something like p and o
  • Want to impress your friends? Move around with p [+-]n where n is the number of entries to skip in the queue. Example:
    you@box:~> dirs
    ~ ~/bin ~/bin /home
    you@box:~> p +3
    /home ~ ~/bin ~/bin
    you@box:/home>

    (explanation: with the first command you can see that a) you find yourself in ~, b) the second and third entry is ~/bin, and c) /home is in fourth position. Count 3 from where you are, and you know you’ll land on /home).

Here a nice page that gives you more details on the subject. (I stole the aliases from there :)

Nosetests test generators – make your bool values more explicit

I had to write a test generator in Python + nostests to drive the same test in two different conditions.

The way I initially wrote the generator initially resembled something like this:

def testChildIsPopulatedCorrectly(self):
    for case in [True, False]:
        yield self.checkChildIsPopulatedCorrectly, case

Although concise, the test as it’s written is not really easy to follow: there is no way to know what the test does without delving into the checkChildIsPopulatedCorrectly method and see which paths case triggers.

A second problem happens when the test runs. The output will be:

testChildIsPopulatedCorrectly(True,) ... ok
testChildIsPopulatedCorrectly(False,) ... ok

Again, it is not very clear what True and False mean in that context.

A solution is to create two simple classes, within the generator[1]:

def testChildIsPopulatedCorrectly(self):
    class WaitForChildren(object):
        def __bool__(self): return True  # [2]
        def __repr__(self): return 'Wait until all the children in the tree are populated'
    class DontWaitForChildren(object):
        def __bool__(self): return False
        def __repr__(self): return 'Do not wait until all the children in the tree are populated'
    for case in [WaitForChildren(), DontWaitForChildren()]:
        yield self.checkChildIsPopulatedCorrectly, case

The next time you run the tests, you’ll get:

testChildIsPopulatedCorrectly(Wait until all the children in the tree are populated,) ... ok
testChildIsPopulatedCorrectly(Do not wait until all the children in the tree are populated,) ... ok

Now both the test generator and the output are clearer.

[1] This is a proof of concept. The complete solution I implemented involves the definition of a more generic class in a different module.

[2] just a heads-up that you should include __nonzero__ = __bool__ if you are using Python 2.x.

Raising (and catching!) exceptions in separate threads

When coding some system tests, I needed to create a Watchdog class: a context manager that upon entering the context starts a threading.Timer and upon exiting cancels it. If the timer goes off before the context is exited, an exception is thrown.

When writing unit tests for the class, I realised that the Watchdog would raise the exception in its own thread, with the end result that I could not catch it in the main thread.
The test output was something like this:

test_watchdog.TestWatchdog.test_raiseException ... Exception in thread Thread-1 [...]

You can fix that by means of a function in the Python API, PyThreadState_SetAsyncExc (ctypes module), which raises an asynchronous exception in a separate thread.

I later discovered that there is a module that does pretty much the same (with some bonus features such as the definition of a ‘timeoutable’ decorator): it is called stopit. Go get it if you’re into that sort of things!

Switch the case of words matching a pattern in Vim

Suppose you want to switch lowercase all instances of, say, “IMPORT” in a file.

Of course you can search for the pattern (/IMPORT), then hit gue and then continue by repeating n and . until all occurrences have been changed.

But there’s a smarter way to do that, that leverages :%s, that is :%s/\(IMPORT\)/\L\1/g.
This calls the \L function and applies it to the first matched group (\1).

There’s also a \U function, which, as you may have guessed already, can be used to switch to uppercase.

Some food for thought on the matter can be found on the Vim Wikia and the official Vim doc.