From Vim to Spacemacs

I’m very fond of Vim. It’s lean, it’s everywhere (if not, there’s vi anyway), it does the job. I also love Neovim and I’ve set it up with some plugins to ease Python development, such as vim-fugitive, a Git wrapper, and jedi-vim, autocompletion and static analysis for Python.
Undeniably, there are quite a few resources on the interwebs on how to create your “perfect” IDE out of Vim.

All nice and lovely, and it gets the job down but… sometimes some plugins let me down. They get confused, cannot find where a variable has been defined, autocompletion stops working properly and undefined vars are just not signalled. At this point I’m forced to close all my tabs and start again.

Even though it’s bearable, I got lured by some colleagues of mine to the dark side of Spacemacs, and I have decided to give that a try. Here are the bits I tweaked, to make Spacemacs feel more comfortable (to me).

Disclaimer

The following applies to Spacemacs on Linux.
I’m still learning my ropes on Spacemacs, so it might well be that some tweaks can be done in a better way or are not effective in all edge cases. Any feedback is, of course, most welcome!

Mouse, be gone

I only use my editors in the terminal, and I tend to use the mouse the same way I use it in other terminals: selecting text copies it, the middle button of the mouse pastes it.
Spacemacs, by default, allows you to jump through the code under the mouse pointer and highlights the pointed line. This is a no-no for me.
To disable the mouse, I resorted by adding under dotspacemacs/user-config the following line:

(xterm-mouse-mode -1)

To completely disable the mouse, this StackExchange answer hints at downloading the disable-mouse package and then using:

(global-disable-mouse-mode)

Make * and # search as in Vim

Apparently, I’m not the only one who noticed that * and # in Spacemacs do not do the RightThing(TM) out of the box: an Emacs StackExchange user asked this very question.
The answer is nice and effective: extend dotspacemacs/user-config with:

(with-eval-after-load 'auto-highlight-symbol
  (define-key evil-motion-state-map (kbd "*") 'evil-search-word-forward)
  (define-key evil-motion-state-map (kbd "#") 'evil-search-word-backward))

There’s still an unsolved nagging, though. Searches done through * and # do not populate the search history.

Make C-a and C-x increment and decrement numbers

For this one, I had to make a compromise. C-x is used as prefix for a gazillion other commands. I didn’t feel comfortable changing the prefix of all those commands: after all, incrementing and decrementing numbers is something I do regularly, yet not very often.

I decided I could settle for C-x a to increment and C-x x to decrement. Again, StackExchange comes to help: there a user suggests that the package “evil-numbers” does just that.

This time, dotspacemacs/user-config should be extended with these two lines:

(global-set-key (kbd "C-x a") 'evil-numbers/inc-at-pt)
(global-set-key (kbd "C-x x") 'evil-numbers/dec-at-pt)

Trim trailing spaces and newlines when saving a file

This is to comply with PEP8, mainly – though I admit I love not having trailing spaces at the end of the lines or unnecessary new lines at the end of a file.

Initially I found a solution that works fine, but is specific to plain emacs.

In Spacemacs, it turns out, there’s an easier baked-in way to do just this. Search under dotspacemacs/init the dotspace-whitespace-cleanup entry and set it to 'trailing.

Search history

If you search some term with / and then try again later by hitting first / and then the up arrow, you’ll be disappointed to see that Spacemacs comes out of the search mode.
Fear not, for there is a solution. Extend dotspacemacs/user-init with:

(setq-default evil-search-module 'evil-search)

Display the absolute path of a file (somewhere)

I’d love to see the full path of the file I am editing.
Since C-g is not an option, given that is a pretty fundamental command in Emacs, I’d be OK settling for SPC f y as the bug report suggests.

However, to always display the path of the file being edited, you can also extend your dotspacemacs/user-config function with:

(spaceline-define-segment buffer-id
  (if (buffer-file-name)
      (abbreviate-file-name (buffer-file-name))
      (powerline-buffer-id)))

Line numbers

This one is another “baked-in” value. In your dotspacemacs/init just set totspacemacs-line-numbers to something like t or prog-mode.

However, in the terminal, I saw that the line numbers are awfully close to the text. Others have noticed it and the fix is to add to your dotspacemacs/user-config the following lines:

(unless (display-graphic-p)
  (setq linum-format (concat linum-format " ")))

Lint as you write

In Vim I used to use python-mode.
Though sometimes it got confused and stopped working, it would catch most of the obvious issues, such as undefined names, unused variables, etc.

In Spacemacs there’s a similar checker, which is not enabled by default. For starters you have to uncomment syntax-checking under dotspacemacs-configuration-layers.
Once done, you can toggle the syntax checker using SPC t s.
To configure the checker (flake8, by default, if I’m not mistaken), create a ~/.config/flake8 file as described here.
You can list errors in a buffer by hitting SPC e L while there are a few other key combos worth learning.

I’ll update the post or write a follow-up on how to use pylint instead.

Misc

Opening files at a certain line

With Vim, I’d do vim file +8 or vim +8 file. Emacs only understand the latter: emacs +8 file.
The reason I mention this, is because I’ve set up a function that opens a file as specified by grep -rn.
To give an example, suppose that grep -rn ' x ' returns this:

somedir/fileA.txt:2873:  foo x bar
someotherdir/pyfile.py:28:from foo import x as y

I’d do v someotherdir/pyfile.py:28:from and be sure I’d open the file at that line.
So, the v() function would look like:

v() {
    FILE=$(echo "$1" | cut -d: -f1,2 --output-delimiter=" +")
    vim $FILE
}

Now, I have an e() function that differs only by the position of the arguments:

e() {
    FILE=$(echo "$1" | cut -d: -f1,2 --output-delimiter=" +" | awk '{print $2, $1}')
    emacs $FILE
}

Admittedly, it could all be done in awk, I believe, but spawning one more process is not critical in my situation.

Conclusions (for now)

There are still a few things I’m not very fond of, and I’ll try to address, namely:

  • There are no tabs in Spacemacs. I simply can’t live without tabs. It’ll be a hard compromise, this one, I’m afraid.
  • * and # to populate the search history – and persist across sessions! That one seems a bit fundamental to me.
  • Python motions given by python-mode. I noticed [M and ]M in Vim have been translated to [[ and ]] here. Not sure how to get to classes, yet. Probably minor anyway.
  • Learn how to create commands myself. Or re-learn a bunch of new commands. Again, not a biggie, I guess.
  • Start-up times. Spacemacs offers you lots, but it’s a behemoth when starting up. This can be sped up I believe, by running an emacs daemon.

Overall, the first impressions are good, but there’s still some road to do before I can say I’m alright in Spacemacs.

Till next time!

Advertisements

My minimalistic .pdbrc

My .pdbrc file is heavily based on Ned Batchelder’s .pdbrc, with only a few additions.
Here are those additions:

# pretty print the content of an object
alias i pp dir(%1)
# pretty print OrderedDicts and any other mapping structures that can be unpacked
alias ppd pp dict(**%1)

Publishing nosetests results in Jenkins CI

While trying to publish test results in Jenkins CI via the xUnit plugin, I’ve set up the Post-build Actions section in Jenkins according to what the nosetests documentation suggests, that is, to “tick the box named “Publish JUnit test result report”” and referenced the correct file.

However, I was repeatedly stumbling upon an error:

WARNING: The file '/var/lib/jenkins/workspace/Project/nosetests.xml' is an invalid file.
WARNING: At line 1 of file:/var/lib/jenkins/workspace/Project/nosetests.xml:cvc-complex-type.3.2.2: Attribute 'skip' is not allowed to appear in element 'testsuite'.
ERROR: The result file '/var/lib/jenkins/workspace/Project/nosetests.xml' for the metric 'JUnit' is not valid. The result file has been skipped.

Turns out, my nosetests.xml file does indeed contain the skip attribute. And that’s also in line with the example in the official documentation.
Much to my surprise, though, running a few web searches made me realize there aren’t many other users of Jenkins + xUnit plugin + nosetests who have the same problem.

To fix this, it looked like I had to write my one XLS file. All you need to do is create two templates, one matching the “skip” attribute, and the other matching everything else.

The “skip”-matching template will simply get rid of the attribute altogether. The other will pass everything else as it is.

By taking cue from this StackOverflow answer, the result is as follows:

<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    <xsl:output method="xml" version="1.0" encoding="UTF-8" omit-xml-declaration="no" indent="yes"/>
    <xsl:strip-space elements="*"/>

    <xsl:template match="@skip" />
    <xsl:template match="@*|node()">
        
            <xsl:apply-templates select="@*|node()"/>
        
    


Save it as nosetests.xsl. Now, armed with your shiny XSL file, put it in the $JENKINS_HOME/userContent directory, say in userContent/xunit_xsl/nosetests.xsl.

In the job config page, under the Post-Build Actions section, change the type from JUnit to Custom, and provide a reference to the XSD file you’ve created, in the form of $JENKINS_HOME/userContent/xunit_xsl/nosetests.xsl.

Save, run, and happily enjoy your test results!

Rentalcars.com dark patterns

Wikipedia defines dark pattern “a user interface that has been carefully crafted to trick users into doing things”.

Then, “carefully crafting fake notifications about customers renting cars” clearly falls into the same category.

This is what rentalcars.com does. Once you select a city and some dates, you’ll get the list of cars available, and something else: a set of notifications that spawn with a random interval from one another implying other customers are renting cars in the same city you’re looking for a car to rent.

Does this put some pressure on you, hastening you into buying a car before all disappear? Yes.
Should you trust it? Actually, no.

If you inspect the source code, this is what you’ll see (click to view it full size):
rentalcarsdarkpattern
Have a look yourself and you’ll find that as the page loads, a function will set to display, in the top right corner, some “pre-cooked” notification, lasting for 4.8 seconds, popping at random intervals from one another.

Reload the same page a few times and you’ll see those 4 notifications pop up time and again.

Sure it can’t be that legal, can it?

Navigate the stack backward

Took me a bit to figure it out correctly, so here it is: a recursive generator to navigate the stack backward (upward, if you want)

def getUpperFrames(frame):
    yield frame
    if not hasattr(frame, 'f_back'):
        return
    for frame in getUpperFrames(frame.f_back):
        yield frame

And if you’re on Python 3 – which you should – then you get the bonus of using yield from instead of using for ... yield

Create a valid ISIN

For a suite of tests I recently wrote, I had to create valid ISINs. The official page doesn’t give away much on how the final checksum digit should be computed and the examples on Wikipedia are, in my opinion, not particularly clear.

While I was trying to understand more, I stumbled upon a related unanswered question on stack overflow, which was basically after what I was trying to do, so I took the time to answer it.

Here I report a slightly modified version of the code snippet linked above, to create the checksum digit for the first 11 characters of an ISIN.

import string

def digit_sum(n):
    return (n // 10) + (n % 10)

alphabet = {letter: value for (value, letter) in
            enumerate(''.join(str(n) for n in range(10)) + string.uppercase)}

def isinChecksumDigit(isin):
    isin_to_digits = ''.join(str(d) for d in (alphabet[v] for v in isin))
    isin_sum = 0
    for (i, c) in enumerate(reversed(isin_to_digits), 1):
        if i % 2 == 1:
            isin_sum += digit_sum(2*int(c))
        else:
            isin_sum += int(c)

    checksum_digit = abs(- isin_sum % 10)
    return checksum_digit

Assuming countries is a list of valid, i.e. ISO-6166 compliant, country codes, you call the isinChecksumDigit as follows:

In [1]: isin = '{:2s}{:09d}'.format(random.choice(countries), random.randint(1, 10E8))

In [2]: isin
Out[2]: 'KR681111517'

In [3]: validIsin = isin + str(isinChecksumDigit(isin))

In [4]: validIsin
Out[4]: 'KR6811115171

Calling external commands in awk

Today I had a file in which some lines were displaying a base 10 number. I wanted to translate this number in base36, possibly keeping the format as it was in the original file.
The entry was something along the lines of:

    thisField='49173058277224'

I decided to resort to awk and calling an external command to do the conversion.

Turns out, there are a couple of ways you can tell awk to fire off an external command. One is using system. The other is by simply stating the command in double quotes. There’s a nice forum discussion around this topic, and for once it’s not on StackOverflow.

Here’s the first draft of the solution.

awk '{
  if (/thisField=.([0-9]+)./) {
    match($1, "([0-9]+)", m);
    "python -c \"import numpy as np; print np.base_repr(" m[0] ", 36)\" " | getline converted;
    sub("([0-9]+)", converted);
    print;
  } else {
    print;
  }
}' orig.dat > converted.dat

The trick is to pipe the result of the command to getline so that it can be saved into a variable, that is later used.
As a side topic, notice that this rough solution is extremely slow: a numpy import will happen for every line that matches. Other solutions for this particular problem exist, such as using bc after setting a proper obase or awk only.

Git: apply a lost stash

Today’s scenario: stash save something in order to get the latest source code, stash pop to re-apply the last changes, do something silly and lose all in-flight changes.
Fear not, there’s a way to recover the “lost” stash, assuming you still have the output of the stash pop.
In fact, after the pop operation there should be a line that goes like this:

Dropped refs/stash@{0} (f2acd7f56e93236bbb813a9dab3bba18e124da04)

Armed with the SHA code, just cherry-pick it:

git cherry-pick f2acd7f56e93236bbb813a9dab3bba18e124da04 -m 1

.
Watch out: this will commit your change as it is. You may want to git reset --soft HEAD^1 and git commit --amend accordingly!

Monitoring WiFi packets

It’s been a long time since I last enjoyed sniffing WiFi packets… the kernel version was 2.4.x to 2.6.x, the first Centrinos made their appearance, OpenWrt was the next big thing, and we all tried to put our hands on Prism II PCMCIA PC Cards. WEP was around, along with the first tools to crack it. It was good fun.

Now, fast forward quite a few years, I can’t believe I still have to debug my own local network by means of dumpcap to monitor AP associations, etc.

So that next time I don’t have to sift through the official Wireshark help page (which, by the way, is actually very thorough), here are some quick instructions to sniff packets on a WLAN.

  1. Leave your wifi device on and connected to the access point (no need to do the ifconfig up/down and iwconfig dance anymore!)
  2. If you don’t have it already, install aircrack-ng
  3. Run airmon-ng: airmon-ng start wlan1
  4. If it says monitor mode is enabled on some device (might be a brand new one, such as mon0) you’re good to go. Sniff some packets from Wireshark (tick the “monitor mode” preference) or using dumpcap: dumpcap -i mon0 -I

Better yet, you can fake an AP and have the device you’re debugging connected.

Unpickling GitPython datetimes

Pickles

I’ve been playing around with GitPython recently, in an effort to analyse the relation between commits and software quality.

One by-product of this analysis was a Pandas Series of the number of commits on a given day. Since this turned out to be a time-consuming operation (as I needed to repoint head back in time for each day I was interested in), I opted to pickle the Series. Imagine the horror when, the day after I had run the script, I discovered that unpickling the data raised an exception.

In [4]: commits = pd.read_pickle('commits.pkl')
...
TypeError: __init__() takes at least 2 arguments (1 given)

That error comes from pickle_compat.py, part of the Pandas library.
However, no mention is made of which class actually raised it.

Entering %debug and going up and down the stack didn’t reveal much either, so I decided to go closer to the actual unpickling operation, using cPickle.

In [10]: commits = cPickle.load(open('commits.pkl'))
...
TypeError: ('__init__() takes at least 2 arguments (1 given)', <class 'git.objects.util.tzoffset'>, ())

Still an error, but a more meaningful one. Let’s see what a brief inspection of tzoffset shows.

In [11]: import git.objects.util

In [12]: git.objects.util.tzoffset?
Init signature: git.objects.util.tzoffset(self, secs_west_of_utc, name=None)
Docstring:
File: /opt/bats/lib/python2.7/site-packages/git/objects/util.py
Type: type

So __init__ expects a secs_west_of_utc positional argument (no default).

To still be able to unpickle your data without the need for running the script again, you just need to mock that class with a slightly modified one. Thank partial applications for that.

In [20]: git.objects.util.tzoffset = partial(git.objects.util.tzoffset, secs_west_of_utc=0)
In [21]: commits = pickle.load(open('commits.pkl'))
In [22]:

Job done – thank you functools!