These are tutorials I found on github.

• TensorFlow Examples
• HVASS Labs Tutorials
• Morvan Zhou
• alrojo
• pkmital

I was looking for a way to update python dependencies that I'd installed with pip when I stumbled upon pip-tools. I'm not particularly good about keeping everything in sync and up-to-date so I'm hoping that this will make it easier to do and thus more likely that I'll do it. It's been a little while since I first used it and I had to look it up, so these are my notes to my future self.

pip-tools installs a command named pip-compile which will use either the requirements you put in your setup.py file or a special file named requirements.in (if you call it this you don't have to pass in the filename, otherwise you have to tell it where to look). Unless there's only a few requirements I prefer to use a separate file, rather than setup.py, since it makes it clearer and more likely that I'll keep it up to date. The requirements.in file is a list of your dependencies but unlike the requirements.txt file, it doesn't have version numbers, the version numbers are added when you call the pip-compile command.

So where does the requirements.in file come from? You have to make it. But if you're editing things by hand, doesn't this kind of make it less likely you'll maintain it? Yes, which is where pipdeptree comes in. pipdeptree will list all the python dependencies you installed as well as everything those dependencies pulled in as their dependencies. It's usefull to take a look at how a dependency you didn't directly install got into your virtual environment. You can install it from pypi.

pip install pipdeptree

Here's its help output.

pipdeptree -h

usage: pipdeptree [-h] [-v] [-f] [-a] [-l] [-u] [-w [{silence,suppress,fail}]]
                  [-r] [-p PACKAGES] [-j] [--json-tree]
                  [--graph-output OUTPUT_FORMAT]

Dependency tree of the installed python packages

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit
  -f, --freeze          Print names so as to write freeze files
  -a, --all             list all deps at top level
  -l, --local-only      If in a virtualenv that has global access do not show
                        globally installed packages
  -u, --user-only       Only show installations in the user site dir
  -w [{silence,suppress,fail}], --warn [{silence,suppress,fail}]
                        Warning control. "suppress" will show warnings but
                        return 0 whether or not they are present. "silence"
                        will not show warnings at all and always return 0.
                        "fail" will show warnings and return 1 if any are
                        present. The default is "suppress".
  -r, --reverse         Shows the dependency tree in the reverse fashion ie.
                        the sub-dependencies are listed with the list of
                        packages that need them under them.
  -p PACKAGES, --packages PACKAGES
                        Comma separated list of select packages to show in the
                        output. If set, --all will be ignored.
  -j, --json            Display dependency tree as json. This will yield "raw"
                        output that may be used by external tools. This option
                        overrides all other options.
  --json-tree           Display dependency tree as json which is nested the
                        same way as the plain text output printed by default.
                        This option overrides all other options (except
                        --json).
  --graph-output OUTPUT_FORMAT
                        Print a dependency graph in the specified output
                        format. Available are all formats supported by
                        GraphViz, e.g.: dot, jpeg, pdf, png, svg

If you look at the options you can see that there's a --freeze option, that's what we'll be using. Let's look at some of what that looks like.

pipdeptree --freeze | head

So it looks like the output of pip freeze except it puts the packages you installed flush-left and then uses indentation to indicate what that package installed. In the example above, I installed Nikola, then Nikola installed doit, and doit installed cloudpickle and pyinotify. I kind of remember needing to install pyinotify myself, but maybe pydeptree caught that it was a dependency that doit is using. Anyway.

For our requirements.in file we only want the names, and although there might be a reason to keep the entire tree, I think it makes it easier to understand what I'm using if the file only holds the dependencies at the top-level (the ones that I'm using directly, rather than being a dependency of a dependency). So, we'll use a little grep. First, since I'm a python-programmer I'm going to give it the -P flag to use perl escape codes. Next, we want to only match the lines that have alpha-numeric characters as the first character in the line.

First, let's see how many total dependencies there are.

pipdeptree --freeze | wc -l

: 160

So there are 160 dependencies total. How many did I install?

pipdeptree --freeze | grep --perl-regexp "^\w+" | wc -l

Out of the 160 only 11 were directly installed by me.

So we're done, right? Not yet, we need to get rid of the == and version numbers. I hadn't known that grep had this feature, since I normally use python instead of grep, but grep has an --only-matching option that will discard the parts of the line that don't match.

pipdeptree --freeze | grep --only-matching --perl-regexp "^\w+"

If you look at the first entry you'll notice it says ghp, but the actual name of the package is ghp-import, but the hyphen isn't part of the alpha-numeric set, so we'll have to add it.

pipdeptree --freeze | grep -oP "^[\w\-]+"

This looks like what we want, but there's a couple of things that we should take care of that would happen if this were for an installed package.

• there's a bug in ubuntu that causes pip to include pkg-resources, which isn't something you can install.
• it will add an extra entry for your python egg, something like this:

-e git+git@github.com:russell-n/iperflexer.git@65f4d3ca72670591f584efa6fa9bfd64c18a925b#egg=iperflexer

So we should filter those out.

pipdeptree --freeze | grep --only-matching --perl-regexp "^[\w\-]+" | grep --invert-match "\-e\|pkg"

ghp-import2
graphviz
Nikola
notebook
pip-tools
pipdeptree
virtualfish
watchdog
webassets
wheel
ws4py

There are probaby other exceptions that have to be added for other installations, but this looks like enough for us. Now we can redirect this to a requirements.in file and we're ready for pip-tools.

pipdeptree --freeze | grep --only-matching --perl-regexp "^[\w\-]+" | grep --invert-match "\-e\|pkg" > requirements.in

pip-compile will read in the requirements.in file and add the version numbers and can create a requirements.txt file. It will automatically look for the requirements.in file or you can explicitly pass in the filename.

pip-compile | head

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --output-file requirements.txt requirements.in
#
argh==0.26.2              # via watchdog
backcall==0.1.0           # via ipython
bleach==2.1.3             # via nbconvert
blinker==1.4              # via nikola

You'll notice it adds in the dependencies of the dependencies and shows what requries them.

If we stopped at this point we'd have:

• a way to check who installed what using pipdeptree (as well as a way to plot the dependencies as a graph)
• a way to separate out our dependencies into a separate file (requirements.in) to make it easier to read
• a way to create our requirements.txt file using our requirements.in file

I think that's kind of nice already, especially if you end up with a lot of dependencies. Try working with sphinx and scikit-learn and you'll see things start to explode. But of course, there's always more.

So there you have it, how to keep dependencies synced. The README for pip-tools is much briefer, but I thought I'd add a little more detail to the part of it that I plan to use the most.

I'm trying to set-up a wireless packet monitor (it's something I've long thought might be an interesting source of data, and now I need it for work too). My thought was that I'd set up a raspberry pi to experiment with - I don't think it is powerful enough to work, but it should work just to mess with code, and a distributed system might get some interesting results, but anyway - but when I tried to put my raspberry pi's wireless interface into monitor mode I got an error.

iwconfig wlan0 mode monitor
Error for wireless request "Set Mode" (8B06) :
    SET failed on device wlan0 ; Operation not supported.

Looking around on the web I found this reddit post as well as some Stack Overflow posts that said that monitor mode isn't supported on the Raspberry Pi. There is a project called nexmon that apparently lets up add a firmware patch to enable it, which I'll probably try later, but before I tried that I remembered that I have a Realtek 8812AU USB WiFi adapter that I bought a while ago for an old desktop I had that I wasn't using so I decided to try it.

The solution was in this StackOverflow post - the make program is being pointed to a folder named arm7l (thats 'arm seven ell') but it should actually be pointed to one named arm. The simple solution is to create an alias with the correct name.

sudo ln -s /usr/src/linux-headers-4.4.38-v7+/arch/arm/ /usr/src/linux-headers-4.4.38-v7+/arch/armv7l

This turns out to fix the build problem and after a reboot the network interface showed up.

The Raspberry Pi 3 doesn't support monitor mode for its wireless interface out of the box, and while there is a firmware patch to enable it, I chose to use a Realtek RTL 8812AU USB WiFi adapter instead. You need a little bit of extra work to get it going, but it does seem to work. One thing I noticed is that iwconfig will put it in monitor mode but airmon-ng doesn't (I haven't figured out why yet). It doesn't report an error, it just doesn't seem to work. Also, iw always reports the interface as managed, even when it isn't… maybe I'll try the firmware patch after all.

• [R1] Rehman S, Turletti T, Dabbous W. A Roadmap for Benchmarking in Wireless Networks. 2011 Aug 9; (ResearchGate Link)

• [R3] Schulman A, Levin D, Spring N. On the fidelity of 802.11 packet traces. InInternational Conference on Passive and Active Network Measurement 2008 Apr 29 (pp. 132-141). Springer, Berlin, Heidelberg. (PDF Link)
• [R4] Kashyap A, Paul U, Das SR. Deconstructing interference relations in WiFi networks. InSensor Mesh and Ad Hoc Communications and Networks (SECON), 2010 7th Annual IEEE Communications Society Conference on 2010 Jun 21 (pp. 1-9). IEEE. (PDF Link)
• [R5] Dujovne D, Turletti T, Filali F. A taxonomy of IEEE 802.11 wireless parameters and open source measurement tools. IEEE Communications Surveys & Tutorials. 2010 Apr 1;12(2):249-62. (PDF Link)
• [R6] Serrano P, Zink M, Kurose J. Assessing the fidelity of COTS 802.11 sniffers. InINFOCOM 2009, IEEE 2009 Apr 19 (pp. 1089-1097). IEEE. (PDF Link)

• [R1] Veselý V. Extended Comparison Study on Merging PCAP Files. ElectroScope. 2012 Oct 1;2012:1–6.

  This paper talks about the three main packet-capture file-formats (LibPCAP, PCAPng, and NetMon) and some of the programs used to merge the different formats.

• [R2] Dulaunoy A. Packet Capture, Filtering and Analysis - Today’s Challenges with 20 Years Old Issues. :38.

  This is a presentation for a security conference. It has a list of common packet capture tools as well as ways to attack some of them while they are capturing packets.

• [R<number>] - Research Paper

This is documented on the Flask site, but I was trying to help someone debug some old server code that I'd written and couldn't remember how to debug it, so I'm documenting it here as I go through remembering it again so I'll have a single reference to use the next time. Some of the settings look different from what I remember using so I think that Flask has changed a little over time, but since I didn't document it the first time I don't have a record to compare against (well, I probably have some notes in a notebook but it's hard to refer to that when sitting at someone else's desk).

The Flask Quickstart tells you what to do, but for some reason when we googled it, the instructions were different, I think it might have lead us to an older form of the documentation. This is the current version (May 20, 2018.)

• It has more filtering capabilities and can filter while capturing packets, but it doesn't have the analytical tools that something like wireshark has¹.

• (Wireshark Revealed⁵) The tcpdump website
• (Wireshark Revealed⁵) A friendly tutorial
• (Wireshark Revealed⁵) The official man page

This is an illustration of how to use bokeh with org-mode in nikola. There is a more extensive and flexible explanation of how to do this in this post on cherian.net but I made these notes to understand how it works and to have a simpler example to refer to.

I was interested in doing this because I'm trying to re-work some of what I did for the Coursera Data Science With Python specialization by changing the data-sets and building them as blog posts. I might convert the posts to restructured text at some point, but while I'm working with them I'm using org-mode. Also, while most of the time I use matplotlib for plotting since this is going to be a blog-first approach I decided to go with bokeh. I had previously written about how to get bokeh into Nikola using restructured text, but as an intermediate step I want to do the work in org-mode and still be able to see the output as I'm working.

The magic mix for this seems to be to use:

• Nikola to build the HTML posts
• org-mode, an emacs mode to format the posts
• the orgmode-plugin for nikola
• ob-ipython to get jupyter/ipython in the org-mode posts
• bokeh to make the plots

Finally I'll save the javascript and HTML files needed and then output the blob needed to embed the plot into this post. The autoload_static function takes the bokeh plot object (fig), the bokeh javascript that I loaded earlier (bokeh), and the name of the javascript file that you want it to creat (test.js) and returns the javascript to save (javascript) and the HTML fragment that will include the javascript (source). Note that because of the way nikola structures things I have to create a folder named files/posts/bokeh-org-mode and save the files there. Since nikola will automatically look in this folder the name you pass into autoload_static should just be the filename without the path, but when you save the javascript file you will save it there so you need to add the relative path. If my explanation seems a little convoluted, just look at the code below, it's fairly simple.

First I'll create a variable to hold the path to the folder to save the files in. All files for nikola posts go into sub-folders of files/posts/ and since the source file for this post is called bokeh-org-mode.org, the files to include in it go into the folder files/posts/bokeh-org-mode (files/posts/ plus the slug for the post).

FOLDER_PATH = "../files/posts/bokeh-org-mode/"

To get a bokeh figure into an org-mode document in nikola:

1. Create the bokeh plot.
2. Create a folder in the files/posts/ folder that matches the slug for the post.
3. Use autoload_static to convert the bokeh object to javascript and create the HTML tag to embed it.
4. Save the javascript in the files/posts/<slug>/ folder that you created
5. Print the HTML fragment in an org-mode #+BEGIN_EXPORT html block.