Org-Babel Ipython and Elpy Conflict

Short-take

If you use elpy and ob-ipython together and use the jupyter option when configuring the elpy interpreter settings it will break the execution of ob-ipython code blocks, so use the ipython version instead. In other words use this in your init.el file:

(setq python-shell-interpreter "ipython"
      python-shell-interpreter-args "-i --simple-prompt")

Background

I've become quite reliant on ob-ipython for emacs (it lets you use ipython with org-babel to create literate programming documents). I used to use pweave, and if I weren't an emacs user I probably would still use it, and of course there's jupyter notebooks, and the attendant emacs-ipython-notebook that I've used as well (and sometimes still use - as when I need to hand in a Coursera assignment, for instance), but org-mode and ipython seems to hit the sweet spot for me (at least this week).

So I was more than a little disturbed when I tried to execute a code block on my new laptop and found that nothing would run. This is a description of what happened and the fix, in case I forget the next time.

The Problem

I tried executing some python code in an ipython code block but I kept getting an error. This is a toy example of what an ob-ipython block looks like:

#+BEGIN_SRC ipython :session test :results output
print("tester")
#+END_SRC

To execute it you'd put your cursor somewhere in the block and enter Control-c Control-c.

This is the error I was getting:

There was a fatal error trying to process the request. See *ob-ipython-debug*

And in the *ob-ipython-debug* buffer would be this:

Error executing Jupyter command '/home/dogen/.emacs.d/elpa/ob-ipython-20180113.929/client.py': [Errno 2] No such file or directory

I tried googling for the error, which brought up some Jupyter questions on Stack Overflow that didn't seem relevant. There were also some issues on the ob-ipython github site, but none of them looked exactly the same. They seemed to suggest that the wrong python interpreter was being used, but when I brought up the interpreter in emacs (C-c C-v C-z) it showed the correct interpreter version and I could import the jupyter module… but since that was what people said was the problem and I was using a virtualenv I decided to try setting up pyenv, which apparently lets you switch between python versions fairly easily.

So, I jumped on my desktop to test it out (I ran into the org-mode problem on my laptop, but I was home so I figured I'd switch), but first I brought up emacs and tried executing the same code-block that failed on my laptop, and of course it ran perfectly. So then I went into a long death-spiral of trying to download the ob-ipython git repository and going back in the git-history to see if I could find the place where it might work (my desktop version of ob-ipython was from October of 2017, my laptop's version is from January 2018), and of course none of them worked. Then I tried copying my init.el file from my desktop to my laptop, and suddenly things worked - so it was a configuration problem, but what was it?

The Fix

If you go to the elpy documentation (elpy is a mode for emacs that makes editing python much easier), there is a section on setting up which interpreter to use - and if you're wondering why I'm suddenly talking about elpy instead of ob-ipython, well, it's because it was the problem (or at least using them together was the problem).

The documentation gives you three options for setting the interpreter - python, ipython, or jupyter. This is what my laptop had:

(setq python-shell-interpreter "jupyter"
      python-shell-interpreter-args "console --simple-prompt")

This is what my desktop had:

(setq python-shell-interpreter "ipython"
      python-shell-interpreter-args "-i --simple-prompt")

I don't know why I configured them differently, but I tried using the ipython setting instead of the jupyter setting and all of a sudden - it worked. Maybe I've now broken something else, but it's fixed for now.

#+BEGIN_SRC ipython :session test :results output
print("tester")
#+END_SRC

#+RESULTS:
: tester

Summary

This documents a problem that I ran into wherein the elpy settings I used broke the execution of code blocks by ob-ipython . Just remember to use ipython instead of jupyter and it should be okay. Another day of my life lost to troubleshooting (well, not a whole day, but too much).

Flask-Bibliography

Web Sockets

Flask-SocketIO

This combines flask with socket.io (so there's an extra thing to learn, maybe). The author made a blog post about it that gives a little more background around it and what advantages he thinks it has over flask-sockets. He also gave a talk at PyCon Ireland 2017 that is on YouTube.

Flask Sockets

This is a project from Kenneth Reitz, the guy who made requests. On the one hand, it looks much simpler, on the other it doesn't look like it has as many options. Maybe a good starter that would later need to be swapped out for Socket IO if it got complex enough.

Flask-uWSGI-Websocket

This looks like it's halfway between flask-sockets and flask-socketIO.

Nikola Inter-Site Links

Background

While creating a bibliography using nikola I found about nikola path handlers. It allows you to make reference to other posts without needing the full address (e.g. http://necromuralist.github.io/pages/nikola-inter-site-links).

Instead you can use the special link:// protocol and nikola will resolve it for you. It uses the form:

link://<kind>/<name>/

There is (of course) another form that lets you doc it using the :doc: restructured text directive. This form is more succinct, and is what I'll probably stick with in the future. I originally went with the link version because I thought it would work with other markup languages, but it didn't (or at least not with org-mode).

Post-Paths

In this case I'm trying to link to another post so the link kind is post_path, which will resolve to either posts or pages. Because it can be either one, you need to extend the path to this form:

link://post_path/posts/<post-slug>/

Note

You can get the slug in the meta-data in the post itself.

Warning

When I forgot the last forward-slash (after the slug) nikola couldn't resolve it and returned a 404 error.

Example

Here's how to link to another post that I titled Pruning Large Files From Git with BFG, with a slug of pruning-large-files-from-git-with-bfg.

`link <link://post_path/posts/pruning-large-files-from-git-with-bfg/>`__

This creates this link.

Doc Role

This is a restructured-text role that nikola supports to make it easier to install links. The previous example using this style would be done like this:

:doc:`link <pruning-large-files-from-git-with-bfg>`

Which creates this link.

Conclusion

The doc method seems much easier for posts, but the links method covers many other cases and might be useful otherwise. It should be noted that both methods work for pages as well (stories), although the link version will need the path changed. In my case, all the pages are under the stories folder so the link would be change to something like:

`link <link://post_path/stories/story-slug/>`__

Removing Large Files From git Using BFG and a Local Repository

1 Background

This is about removinge large files that have been committed to a git repository. Github has a 100 MB file limit, and I'm using Century Link's ADSL (Fiber to the Node) so my upload speeds are much slower than my download speeds, so I'd like to keep large files out of the repository. The problem when working with data is that you can often end up with very large files that accidentally get committed, then need to be removed. Here's two cases I've run into:

  • I accidentally committed some large files to a git repository and couldn't push my changes up to github, having exceeded their file-size limit
  • I committed a folder that had some large files that were under the limit, but were just taking too long to upload.

I stumbled upon the BFG Repo-Cleaner while looking for a solution and kept these notes so I don't forget what I did.

The instructions on the website are pretty clear, but it assumes that the large files are on a remote repository (I talk about that case here) but in my case my changes were local so I needed to change the first step from cloning the repository to cleaning the git repository (I'll call the repository big_files). In this case I decided to remove anything bigger than 50 MegaBytes.

2 The Commands

For reference, these are all the commands. I'll go over them in the sections that follow. The java -jar line pointing to the BFG jar file will have to match your download file name and location.

cd big_files
git gc
cd ../
java -jar bfg-1.12.7.jar --strip-blobs-bigger-than 50M big_files
cd big_files
git reflog expire --expire=now --all
git gc --prune=now --aggressive

I'll break this down into three parts:

  1. Clean the repository
  2. Run BFG to remove the file from the git history
  3. Clean the repository again

2.1 Clean The repository

The git gc (garbage collection) command will clean up unnecessary files and perform some optimizations that will let us run BFG on it.

cd big_files
git gc
cd ../

2.2 Run BFG

java -jar bfg-1.12.7.jar --strip-blobs-bigger-than 50M big_files
  • You have to run in the folder just above the git repository
  • Change it to have the correct path to the jar (e.g. java -jar /home/username/bin/bfg-1.12.7.jar).
  • Then tell it to remove files bigger than 50 Megabytes (or whatever size limit you want) in the big_files folder

2.3 Clean Up the Git Repository Again

BFG will tell you this part in it's output, although I'm not using bash so it worked better when I ran the commands separately, instead of all on one line using && between the commands (fish doesn't support it). These are git commands so you have to change back into the repository folder.

cd big_files
git reflog expire --expire=now --all
git gc --prune=now --aggressive

Pruning Large Files From Git with BFG

These are my notes on removing large files from a git repository with the BFG Repo-Cleaner. It assumes that you have already pushed it to a remote repository. If you've committed the changes but haven't pushed them, see this post.

1 Background

I accidentally committed an 89 MB file to git and pushed it upstream to github. This is within the allowed file-size limit, but when I looked in the file it was filled with the same error message over and over again, so it wasn't useful to keep, anyway. I was just working with small log-files, so any large files indicated an error, anyway, so I decided to clean anything over 50 MB from the repository using BFG. The instructions on the home page mostly work, but didn't exactly work so I'm making some notes here for the next time.

2 The Process

Assuming you've downloaded the BFG jar file, this is what you need to do.

git clone --mirror <remote git address>

The --mirror flag creates a bare repository so it will look a little odd (the top level has the contents of what are normally in the .git folder). When I first tried this I thought I could clone my local copy but when I ran BFG on the clone it said that it couldn't find any large files. As noted in this bug-report the large files would be in the packfile if you clone it from a remote repository, but not necessarily in the local repository, so I had to clone it from github.

Next run the BFG.

java -jar BFG-1.12.15.jar --strip-blobs-bigger-than 50M <path to cloned mirror>

Now you have to change into the cloned repository and execute some commands to update it.

git reflog expire --expire-now --all
git gc --prune=now --aggressive

Now push it back to github.

git push

At this point the original local copy of the repository will still have the large file(s) in the history so if you just do a git pull it will think you are ahead of the remote, so you have to remove your original local repository and re-clone the remote.

rm -r <original local repository>
git clone <remote repository>

One thing that tripped me up a little was that I had removed the master branch but the BFG re-added it, so it originally looked like I had lost some changes. Once I changed back into my working branch everything was as I had expected.

Restructured Text Anonymous Links

Background

I'm creating an annotated bibliography using nikola, and added links to the PDFs online so I could download them again if needed. I called them all PDF Link but when I built the web-page I found that there were some restructured text errors in it that looked like this.

backlink_rst_error.png

This stack overflow answer pointed me to anonymous links (which I had never heard of) to fix the problem.

The Experience of Updating Ubuntu 17.04 To 17.10

I recently switched to Ubuntu 17.10 (from 17.04) and found it to be surprisingly difficult. This is meant to document what I noticed so that if I decide to upgrade another machine I'll have something to jog my memory about what it was like.

Complete Re-Install Required

I upgraded four machines (I don't have all their details at hand - maybe I'll update this later). In each case they were running Ubuntu 17.04 so I used do-release-upgrade initially. I didn't upgrade them all on the same day, but did upgrade all of them within the past week.

  • HP Z600
  • eMachine
  • Zareason
  • Toshiba portege (laptop)

The first sign that something bad was happening was that the lock-screen appeared to break after the update so I couldn't log back in. The field where you would normally enter the password was gone so there was no way to authenticate myself. In retrospect, maybe since I did this four times I should have tried a headless install with GNU screen to at least see what was going on, but hindsight's always fifty-fifty, isn't it? My tendency to repeat my mistakes is why I'm writing this down, after all.

On trying to reboot I found that none of the machines would fully start. All of them got stuck on a black screen with a single white underscore in the top left corner. Unlike the other machines, the Toshiba let me enter the disk-decryption password, but then it ended up at the same black screen as the rest. Once it reached this state there was no way to get to the consoles either (e.g. using ctrl-alt-F1). So, in every case I had to boot from a thumb drive and do a fresh install.

Backwards Steps

Alternate Keyboard Layouts

It used to be that the installer would ask you to set up a disk-encryption password before asking you for your keyboard layout, making it harder for people like me that use alternate layouts (Programmer Dvorak in my case). They eventually fixed that, but now it's back again, forcing you to enter the password to decrypt your drive using qwerty. The initial login to decrypt the drive after you boot the computer also has no way to switch the keyboard layout, so once again you have to use qwerty. On top of that, the visual keyboard is missing keys (well, it's missing the underscore, any way) which I used to use when I found myself struggling too much with qwerty, so instead I have to use the physical keyboard, hunting and pecking and trying to forget where I think the keys should be. Then, once you get past that first screen, it switches back to dvorak (although the visual keyboard doesn't, for some reason) so you have to know that the user login is using a different keyboard layout from the disk decryption prompt - even though there's no way to actually check what the layout is without just trying the variations until you find the one that works.

Video Drivers

On top of the keyboard problem, the default Nouveau video drivers seem to have re-introduced an old bug that causes the GUI to freeze up periodically (on my ZaReason and HP computers, at least. I haven't seen it on the other two, but I don't use them as much, anyway). Switching to the Nvidia driver fixes the freezing problem, but it then makes the initial disk-decryption login non-functional - it seems to freeze everything so you can't do anything but restart the computer and use the Grub menu to drop into recovery mode and enter the password at the command line before resuming the startup. As a 'fix' I followed the advice from this Stack Overflow answer and disabled the initial splash screen altogether by editing the /etc/default/grub file to have this line:

GRUB_CMDLINE_LINUX_DEFAULT="text"

Then update grub (it doesn't load this file on booting up, you have to explicitly update it).

sudo update-grub

This makes the initial boot-screen text-only, like an old-school terminal. Not fancy, but at least I don't have to remember to drop into recovery-mode every time I start the computer.

Apt-Fast Bug

This is a bug in apt-fast, not an ubuntu bug, but since apt-fast is one of the first things I usually install, I figure I should document the fix. Open up /var/lib/dpkg/info/apt-fast.config (as root), jump to line 212 and change this line:

if ($downloadcmd =~ /\${_MAXNUM}/){

to this:

if ($downloadcmd =~ /\$\{_MAXNUM}/){

In case you can't see it I added a backslash (\) between the $ and {_MAXNUM.

A Little Less Whining

Although the disk-decryption login is a major negative, I must note that I kind of like the new aesthetic, I was just really, really, surprised at how defective this process seemed to be, given how far along Ubuntu has come. There's still some unexpected behaviors (switching workspaces seems to drag applications with them, even though I didn't explicitly ask to, for instance, and byobu seems to ignore my alternate escape key sometimes), but I like it overall. Maybe tomorrow I'll feel different.

Shell Output With Org-Babel Ipython

One of the nice things about ipython is the ability to run shell commands with !. When you use it in an ob-ipython block, though, it will dump the output into a popup rather than in the same emacs org-file. One alternative is to use a shell block instead. I wrote earlier that this requires you to put an echo command in the block to prevent the text going to a pop-up.

#+BEGIN_SRC sh
head Future_Connections.csv
echo
#+END_SRC

#+RESULTS:
|            | Future Connection |
| (6, 840)   |               0.0 |
| (4, 197)   |               0.0 |
| (620, 979) |               0.0 |
| (519, 872) |               0.0 |
| (382, 423) |               0.0 |
| (97, 226)  |               1.0 |
| (349, 905) |               0.0 |
| (429, 860) |               0.0 |
| (309, 989) |               0.0 |

It turns out that there's a similar solution to the ob-ipython popup, only since it's in python you have to use a print this time.

#+BEGIN_SRC ipython :session futures :results output
!head Future_Connections.csv
print()
#+END_SRC

#+RESULTS:
#+begin_example
,Future Connection
"(6, 840)",0.0
"(4, 197)",0.0
"(620, 979)",0.0
"(519, 872)",0.0
"(382, 423)",0.0
"(97, 226)",1.0
"(349, 905)",0.0
"(429, 860)",0.0
"(309, 989)",0.0

#+end_example

I actually prefer the shell version, and since this is org-babel it makes more sense to use it, but I forgot about it just now and it took me a little while to figure out how to get the ob-ipython block to work so *note to future self: This is how to make it work, but use a shell block instead.

How To Change The Nikola Bootswatch Theme

I had my theme for this site set to use Spacelab, which I liked, but the gradients were starting to annoy me so I decided to switch to Journal, which matches my aesthetic tastes a little better. This took more work than I thought it would (mostly because I don't mess with the nikola configuration very often). So these are my notes for the next time.

Overview

  1. Install the bootswatch 3 nikola theme (if you haven't already) (nikola install_theme bootstrap3).
  2. Install the journal bootswatch theme (nikola bootswatch_theme -s journal).
  3. Get the base template (nikola theme -c base.tmpl).
  4. In base.tmpl, change navbar-inverse to navbar-default.
  5. Change the anchor color in bootstrap.css to something less creamsicle (and do whatever other CSS adjustments you need to do).
  6. If you need to, minify the CSS (cleancss -o bootstrap.min.css bootstrap.css).
  7. Build the posts (nikola build).
  8. Admire the fruits of your (well, other people's) labor.

Installing the theme

The Bootswatch 3 Theme

I already had the bootswatch 3 theme installed, since I was using Spacelab, but if you don't have it installed this is how to set the base theme:

nikola install_theme bootstrap3

The Journal Theme

Since I want to change the theme to Journal this is the next step (you can also specify a parent theme, but the default is bootstrap3, which is what I wanted anyway):

nikola bootswatch_theme -s journal

This will set up the default bootstrap theme, but you need to re-build the HTML to have it show up on the development-server's web-page (something that I didn't realize initially, causing me to think something was broken).

nikola build

Updating the Template

On re-loading the page I found that it mostly worked, but for some reason it looked more orange than I expected. It turns out that the nikola developer decided that navbar-inverse should be the default style for the navbar (instead of navbar-default) and the section headers are hyperlinks, which in the Journal theme are orange (by orange I really mean something closer to a creamsicle). Following the instructions in this bug report (which says that it isn't a bug, it's a feature) I copied the base template from the nikola installation (in site-packages) into the site using a built-in nikola command.

nikola theme -c base.tmpl

This command creates a file (themes/custom/templates/base.tmpl) which you can edit to override the built-in base template. I opened it up and changed the navbar-inverse CSS class to navbar-default.

<nav class="navbar navbar-default navbar-static-top">

The I re-built the nikola posts, and reloaded the page and there it was, the white navbar I wanted.

Updating the CSS

Fixing the Headline Color

"Fixing" the headlines was more straight-forward. When you install the journal theme it creates a file at themes/custom/assets/css/bootstrap.css (along with a mini-fied version) which you can edit. I decided to leave the anchor-hover value the same but changed the plain anchor color to #990033 so that it's more of a blood red than a creamsicle.

Changing the Font

Since I was editing the CSS anyway I made some other changes. First, I changed the text-font to Goudy Bookletter. It's a little old-fashioned looking, but I've always liked Goudy's fonts. To do this I first added a link to the font:

@import url("https://fonts.googleapis.com/css?family=Goudy+Bookletter+1911:400,700");

Then I updated the font-family.

body {
  font-family: "Goudy Bookletter 1911", "EB Garamond",  Georgia, "Times New Roman", Times, serif;
  font-size: 3em;
  line-height: 1.42857143;
  color: #777777;
  background-color: #ffffff;
}

Fixing the headline

For some reason the title of the site has a CSS rule (.navbar-brand #blog-title) that sets the top padding to 15 px even though the other elements have a padding of 19.5 px so I overrode it to make it align with the other elementes.

.navbar-header .navbar-brand #blog-title {
    padding-top: 19.5px;
}

Minifying

When I wrote this originally, editing the bootstrap.css file worked, but on one of my other sites the CSS file that is being created uses the bootstrap.mini.css file instead. While this seems like it would be preferable, anyway, the change came about unexpectedly - I still don't know why Nikola is doing it on that site and not this one (I guess maybe I'll have to read the manual one of these days). So for that case, after editing the bootstrap.css file you have to minify it (or just make a copy called bootstrap.min.css). To minify it I use Clean CSS CLI.

npm install clean-css-cli -g
cd themes/custom_theme/assets/css/
cleancss -o bootstrap.min.css bootstrap.css

Where the themes folder is in the root of my nikola repository, next to the conf.py file.

Deploy

This isn't really to do with theme-changing, but for completeness, I'll mention it anyway.

nikola github_deploy

Summary

Changing the bootswatch theme isn't as hard as it seemed, in retrospect. You just have to know the basic commands (nikola install_theme bootstrap3, nikola bootswatch_theme -s journal), know that the inverse navbar is the default (and how to change it - nikola theme -c base.tmpl), know how to update the CSS to whatever you want (by editing bootstrap.css and minifying it if necessary), and understand that you have to build the site to see the changes on the development server (nikola build).

Org To Jupyter Notebook

I've grown to really like org-mode, particularly with ob-ipython, but there are times when a real Jupyter notebook is better - if you want to render it on github or use the %matplotlib notebook feature, for instance. I found a blog post the Kitchin Research Group site announcing the creation of an exporter and this is some documentation about how to set it up.

Download

When I originally set this up it was only available from a link on the blog-post, but it has since been added to a much larger github repository called scimax. I'm assuming this is where the latest version will be kept. Unfortunately there's no license so I don't think you can fork the repository, even more importantly, I can't get it to work, so skip it for now and grab the old one.

cd ~/.emacs.d
mkdir lisp
wget http://kitchingroup.cheme.cmu.edu/media/ox-ipynb.el

Install

Make sure that the lisp folder is in the emacs path by putting this line in init.el.

(add-to-list 'load-path "~/.emacs.d/lisp")

Also make sure that it's loaded by putting this in init.el.

(require 'ox-ipynb)

Use

In your org-mode file use the export-ipython function to export it to an ipynb file.

M-x export-ipynb-buffer

Note that this will only work if you have ob-ipython installed and at least one ipython code block in the org-file.

#+BEGIN_SRC ipython :session
print("a")
#+END_SRC