PuDB Remote

In the Beginning

This is a post on using PuDB via telnet.

Short Version

In case I forgot what to do and just want to read this to remember.

  1. In the screen where you are going to run telnet, run tput cols and tput lines to find out the number of columns and lines that the screen is using.
  2. In the code where you want the break, instead of the usual import pudb; pudb.set_trace() use:
from pudb.remote import set_trace
set_trace(term_size=(columns, lines))

When the code hits the breakpoint it will start up a telnet service and you can log into it from the other screen:

telnet 127.0.0.1 6899

Okay, So Now Why Would You Do This?

I was trying to build one of my sites that uses nikola and this strange error came out saying that one of the shortcodes was getting the "site" argument multiple times. I had no idea what was going on and nikola doesn't pre-define the arguments to the shortcode-plugins (it parses the text and then passes the arguments along using the *args **kwargs convention) which makes it flexible, but figuring out a problem with the arguments is pretty tough, so I decided to turn to my old standby PuDB, which I've been using for so many years now, and is my favorite of the python debuggers I've tried.

I did my usual thing and found the line in the nikola code that was raising the error and inserted a breakpoint further up in the code.

if name == "lancelot":
    import pudb
    pudb.set_trace()

I figured the problem was with my shortcode, named lancelot, and the conditional allowed me to skip all the other shortcodes being used. But then, when I ran the build (nikola build -v2)… disaster.

nil

For some reason it wouldn't use the entire screen, making it impossible (or at least really hard) to read some of the variables that I wanted to check - and even worse, when I tried to open up the ipython terminal in it I got an error message.

nil

The error-message is one of those error-after-the-error messages that you sometimes run into. It was trying to notify me of an error by popping up some kind of dialog but then the dialog wouldn't open so it told me about the dialog error and didn't get around to telling me what the original error was. In any case, something was broken, so I had to resort to desperate measures - I went to read the documentation.

Surprisingly, there actually is some (there wasn't really much when I first started using it).

Dead Ends

The first couple of things I tried seemed promising, but they didn't work.

Another Screen

There wasn't anything about re-sizing the window, but it did mention that you can have the output come out in a different terminal from the one where you run the code, so I gave it a go. I made a different screen and got its file-path using the tty command (/dev/pts/6 in this case). Then I set an environment variable to hold the file-path in the screen where I was running the code (set -x PUDB_TTY /dev/pts/6). This caused PuDB to pipe the output to the next screen - and it did work to send the first PuDB output to the other screen, and it did use the whole window, but then it quit instead of letting me use the debugger. Not quite what I wanted, so I unset the environment variable and moved on.

term_size

The documentation also showed how to use PuDB as a remote debugger, and in the example they passed in the argument term_size to set_trace so I thought, since the functions have the same name, that the set_trace I was using would take the same arguments. So I tried it.

import pudb
pudb.set_trace(term_size=(236, 61))

Using the values that I had gotten from tput using tput cols and tput lines. But that just raised an ArgumentError. The functions have the same name, but not the same arguments.

Telnet

So then I decided to try their remote version. It doesn't really make sense to me that it would work better than the regular version, but I didn't see any other choice. So instead of the usual code to insert a breakpoint I used:

from pudb.remote import set_trace
set_trace(term_size=(236, 61))

When I ran the build it stopped and told me to telnet into the localhost address at port 6899.

nil

So I changed into the other screen and ran telnet.

telnet 127.0.0.1 6899

And what do you know.

nil

This turns out to not be a complete fix. Hitting ! to get to the ipython terminal froze PuDB, but this was enough for me to inspect the variables and realize that I just needed to move one of the parameters in the definition of my shortcode method and it worked.

But It's Not Fixed?

Well, if this were a more intense debug I really would want the ipython~/~ptipython terminal, but since this is the first time I've tried to run PuDB under KUbuntu's Konsole instead of the GNOME terminal I'm hoping that just switching back to the other terminal will be enough - I'll have to test that once I'm more motivated.

Emacs Scrollbar Artifact on Kubuntu

What's this then?

I switch back and forth between Kubuntu and Ubuntu (Ubuntu seems to work better, but I like the aesthetics of Kubuntu) and one of the problems I had was that when I launched emacs in Kubuntu it had a permanent scrollbar in the center of the window that blocked out whatever text was there.

nil

It's more of an annoyance than anything else but since it doesn't happen on Ubuntu I figured I'd try and fix it. It took me a couple of different searches to find the answer so I thought I'd document it in case I need to remember this later.

The Cause

This is the desktop that's causing the problem:

nil

It turns out that it's because my monitors are of different resolutions and in order to be able to read anything on the higher-resolution monitor I had to set the display scale to 200%, but this causes a problem with the scaling of the widgets (at least that's what it said on the reddit post where I found the solution).

The Fix

The fix for me was to edit the ~/.local/share/applications/emacs.desktop file so that the EXEC line read:

Exec= /usr/bin/env GDK_SCALE= emacs

Once this was in place the artifact went away.

nil

The Source

I linked to it above, but this is the reddit post where I found the fix:

Building fastai's Documentation

What is this about?

I've decided to try and build as much of the documentation that I use all the time on my local system, not just so that I'll have it if my internet connection goes down but also so that I won't be distracted by what's happening on the web. This is about building fastai's documentation, which was a little trickier than I thought it would be so I decided it would be worth it to make a note for the future.

You can skip to the In A Nutshell section of the post to get a summary of the steps without all the exposition that the middle section has.

What happened?

The Repository

The first thing I did was clone the fastai git repository from github. If you inspect it there's a folder called docs_src which seemed to logically mean that that's where the source files for the documentation are but when you go in there you won't find an index.html file, which seemed peculiar. There's a Makefile at the root of the repository so I inspected it and found that there's a rule:

docs: $(SRC)
        rsync -a docs_src/ docs
        nbdev_build_docs

So I tried a naive make docs but of course it failed because there's nothing called nbdev_build_docs, so I searched online and found out that nbdev is a fastai project to make jupyter notebooks into a Literate Programming system and that nbdev_build_docs is one of their command-line commands, so I installed it through pip:

pip install nbdev

And re-ran the make command, which did nothing because the rsync command had already created the docs folder and for some reason this made the nbdev_build_docs command not work. So I removed the docs folder and re-ran it, which produced a big dump of errors because in converting the notebooks nbdev was importing a bunch of python code that wasn't installed. Interestingly, at this point the docs folder actually has enough to run the site, despite all the error-messages, but if you just try to load the files into a browser you can see that it's kind of broken, so then I went looking for what was going on.

Jekyll and Hide

For some reason I couldn't find anything in the documentation on building it, but searching for "fastai build documentation" brought an outdated page that tells you how to build the documentation but was written for the prior version of fastai (v1) so much of it doesn't make sense for v2 (e.g. it refers to a non-existent tools folder), which I didn't figure out at first because the sites for v1 and v2 don't really identify their version, except in the URL for the old site.

All You Need

Reading that documentation it turns out that they're using Jekyll, so if you install it you just need to run Jekyll in the docs folder.

cd docs
bundle exec jekyll serve

And the site is ready to read at http://localhost:4000 and at this point you're good to go - but, of course, I didn't realize that and tried to fix the error messages first, which is what the rest of this post is about.

Fixing the Imports

There's three things you need to do to fix the imports:

Installing fastai

The old documentation recommended installing it in development mode. I don't know if that's strictly necessary, but it fixed a lot of things so it seems like a good idea.

In the root of the fastai repository run pip.

pip install -e ".[dev]"

This installs a lot of stuff so you might want to go get a cup of coffee (or maybe a cocktail) at this point while it does its thing. The settings.ini file lists the dev_requirements and the regular requirements if you want to see what needs to be installed in either case.

Installing Flask Compress

This is pretty straight-forward, just use pip.

pip install flask-compress

Install Azureml-core

  • The Problem

    This wasn't quite so straight-forward, which is why I put it in a separate section. If you try to install it in Ubuntu 21.04 (or 20.04, etc.) you will get a big blob of error messages ending in this.

    ERROR: Command errored out with exit status 1: /home/hades/.virtualenvs/fastai-clean/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-srnkqokl/ruamel-yaml_803314568
    e8f4fa49015a45528d277b2/setup.py'"'"'; __file__='"'"'/tmp/pip-install-srnkqokl/ruamel-yaml_803314568e8f4fa49015a45528d277b2/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(_
    _file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip
    -record-yfvqflby/install-record.txt --single-version-externally-managed --compile --install-headers /home/hades/.virtualenvs/fastai-clean/include/site/python3.9/ruamel.yaml Check the logs for full command output
    

    Which isn't really all that helpful. Scrolling up, it looks like the problem was with something called ruamel.yaml, so investigating this seemed like a place to start, but, of course, the error messages are completely inscrutable now that I haven't programmed in C for so many years so I decided to search the web instead of trying to debug it directly, figuring that someone else must have had this problem.

    This lead to a long search through various posts, but what it turned out to be was that both ruamel.yaml and azureml-core don't support python 3.9 yet (there are some bug reports on GitHub for it already) so you can't install it with the version that currently ships with Ubuntu (3.9.5) or anything above python 3.8.

  • The Fix

    The fix I decided to use was to install pyenv using their installer. Once you run the installer and follow the rest of their installation instructiors it's fairly straightforward to set up so I won't go into it.

    I decided to use python 3.8.10 so to install it you do this:

    pyenv install 3.8.10
    

    The only thing that didn't work for me was their pyenv which function which is supposed to show you the location of the python installation. The command might work but I couldn't figure out the arguments to use (updating the example they gave didn't work for me). It turned out the python binary was at:

    ~/.pyenv/versions/3.8.10/bin/python
    

    pyenv has it's own system for creating a virtual environment, but since I'm already using virtualfish and didn't want to try and troubleshoot yet another method I created a virtual environment the way I usually do it.

    ~/.pyenv/versions/3.8.10/bin/python -m venv fastai-doc
    

    At this point I activated the new virtual environment and had to re-do previous installation steps (for fastai and flask_compress) as well as the azure-ml installation.

    pip install -e ".[dev]"
    pip install flask-compress azureml-core
    

    The installation of fastai installs nbdev as one of the requirements so that didn't have to be re-done. And now I built the documentation and ran the jekyll server. Easy-peasy.

    make docs
    cd docs
    bundle exec jekyll serve
    

In A Nutshell

The Minimum to Get the Documentation

  • Clone the fastai git repository from github
  • Install jekyll and nbdev
  • Change into the root of the fastai repository you cloned
  • Run make docs and ignore the error-messages
  • Change into the docs folder that was created and run the jekyll server (bundle exec jekyll serve)

To Fix All the Errors

This isn't really necessary to get the documentation, but I think it's better, since you don't have to ignore all the error messages.

  • Clone the fastai git repository from github
  • Install jekyll
  • Get python 3.8 working (I used pyenv)
  • Use pip to install fastai in development mode
  • Use pip to install flask_compress and azureml-core
  • Change into the root of the fastai repository you cloned
  • Run make docs
  • Change into the docs folder that was created and run the jekyll server (bundle exec jekyll serve)

Coding Strip

Abstract

Programming is difficult for some people to learn because the concepts are abstract and while there have been efforts to make them more concrete, efforts that focused either on telling stories (e.g. picture books and adventure comics) or transforming the code into something more tangible (e.g. manipulating physical blocks or using graphic representations to create programs) but these prior works didn't make the step from the concrete systems they created to actual code so this team created and tested a system (Coding Strips) to develop comics that could be directly tied to real code.

Cite

  • Suh S, Lee M, Xia G. Coding strip: A pedagogical tool for teaching and learning programming concepts through comics. In2020 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) 2020 Aug 10 (pp. 1-10). IEEE.

Coding Comics: Recursion

What Is This?

This is a re-working of the Coding Strip Recursion example. Not because I can do it better, but because I've never done one before so stealing their idea seems like an easier way to start. In the original they had a comic showing a character who wants to buy a ticket but there's a long line so she asks the person in front of her how many people are in front of him, and he asks the person in front of him, and so on. They then followed up the comics with some code that translated the comic to a concrete function.

The Comic

(Coming Soon)

In English-Ish

Forward

To find the length of the line, each person asks the person in front how many people are in front of them.

Base Case

When the person at the front of the line is reached, he reports that there's no one in front of him (zero).

Backwards

Once the front of the line is reached, each person then relays back how many people the person in front reports and adds one to include the person who reported the count until the back of the line is reached.

The Code

Here's some code to illustrate the idea of asking the person in front of you how many people are in front of them and having that question propagate forward and then have the answer propagate back.

A Person

I originally thought of using a list, but then you'd have to criple the length method… so I'm making a linked list of a sorts, where each person knows the person in front of them.

class Person:
    """A person in line
    """
    person_in_front = None

The Recursion

def hey_fella_how_many_people_are_in_front_of_me(fella: Person):
    """Finds out how many people are in front of current person

    Args:
     fella: the current person being asked

    Returns:
     Number of people in front (including last person)
    """
    COUNT_THIS_FELLA = 1    
    if fella.person_in_front is None:
        return COUNT_THIS_FELLA
    return (hey_fella_how_many_people_are_in_front_of_me(fella.person_in_front)
            + COUNT_THIS_FELLA)

Check If It Works

Now I'll create a line of unknown length so we can check it.

from string import ascii_letters
import random

waiting = random.randrange(1, 1000)

def line_of_people(people: int) -> Person:
    """Builds the lengthless line

    Args:
     people: how many people to queue up

    Returns:
     line of people
    """
    line = this_person = Person()
    for person in range(1, waiting):
        this_person.person_in_front = Person()
        this_person = this_person.person_in_front
    return line

in_line = line_of_people(waiting)

So at this point we have a line of people of unknown length. Each person only knows the existence of the person in front of them so there's no way to get the length of the line directly, but we can use the recursive function to find out how many people there are.

reported = hey_fella_how_many_people_are_in_front_of_me(in_line)
print(f"Expected: {waiting}, Actual: {reported}")

assert waiting == reported
Expected: 539, Actual: 539

Seems to be working.