Packet Capturing Bibliography

Packet Capturing

  • [R3] Schulman A, Levin D, Spring N. On the fidelity of 802.11 packet traces. InInternational Conference on Passive and Active Network Measurement 2008 Apr 29 (pp. 132-141). Springer, Berlin, Heidelberg. (PDF Link)
  • [R4] Kashyap A, Paul U, Das SR. Deconstructing interference relations in WiFi networks. InSensor Mesh and Ad Hoc Communications and Networks (SECON), 2010 7th Annual IEEE Communications Society Conference on 2010 Jun 21 (pp. 1-9). IEEE. (PDF Link)
  • [R5] Dujovne D, Turletti T, Filali F. A taxonomy of IEEE 802.11 wireless parameters and open source measurement tools. IEEE Communications Surveys & Tutorials. 2010 Apr 1;12(2):249-62. (PDF Link)
  • [R6] Serrano P, Zink M, Kurose J. Assessing the fidelity of COTS 802.11 sniffers. InINFOCOM 2009, IEEE 2009 Apr 19 (pp. 1089-1097). IEEE. (PDF Link)

Merging PCAP Files

Key

  • [R<number>] - Research Paper

Networking Tools

Packet Capturing

Gulp

Gulp purports to be better at capturing packets than tcpdump (although they can work together).

There is more than one version out there:

- This one says it applied a patch to it five years ago.

  • This one says it is the original but hasn't been updated in six years.
  • This blog post has updated versions of it including one in 2017 that says it has a major bug fix (but I don't know if it's a gulp bug or not)

tcpdump

More easily obtainable and better documentation available (although still not enough).

ssldump

Captures packets and decodes SSL/TLS packets.

Packet Examining

Compressed PCAP Packet Indexing Prograpm (cppip)

This adds indexing to bgzip compressed LibPCAP files which then lets you extract them while the original files are still compressed.

tcpslice

This lets you extract part of or combine files created by tcpdump when using file rotation.

ngrep

Describes itself as like GNU grep but for packets.

pylibpcap, pypcap

Python code to work with libpcap.

WireShark

These are installed when you install wireshark.

tshark/wireshark

Packet capturing and examining (better documented than most of the other programs)

reordercap

Reorders the packets by timestamp.

capinfos

This prints summary information about packe files (works with gzipped files).

mergecap

Merges multiple packet files together. Mergecap will try to keep timestamps in order when merging, but it assumes each individual file to merge is already in order.

Packet Flows

tcpick

Track, reassemble, reorder TCP streams.

tcptrace

Gives connection information taken from a capture file.

tcpflow

Separates out TCP flows into separate files.

Capture Summarizing

ipsumdump

Summarizes packet information in ASCII format

tcpdstat

Gives summary statistics for a pcap file

Network Monitoring

ntop

Like top but for the network.

Miscellaneous

tcpreplay

A suite of programs to edit and replay pcap files.

netcat

Read from and write to TCP/UDP network connections.

socat

Route data between byte streams.

The Flask Debug Server

Background

This is documented on the Flask site, but I was trying to help someone debug some old server code that I'd written and couldn't remember how to debug it, so I'm documenting it here as I go through remembering it again so I'll have a single reference to use the next time. Some of the settings look different from what I remember using so I think that Flask has changed a little over time, but since I didn't document it the first time I don't have a record to compare against (well, I probably have some notes in a notebook but it's hard to refer to that when sitting at someone else's desk).

Setup

The Flask Quickstart tells you what to do, but for some reason when we googled it, the instructions were different, I think it might have lead us to an older form of the documentation. This is the current version (May 20, 2018.)

The Environment Variables

The Flask App

First you have to tell flask which file contains your flask app by setting the FLASK_APP environment variable. In my case I'm using connexion, an oddly-named adapter that adds support for swagger/OpenApi to Flask. So the file that has the app has this line in it.

application = connexion.FlaskApp(__name__)

In this case that's a file named api.py which we'll say is in the server folder (it isn't, but that's okay) so we need to set our environment accordingly. I use the fish shell so the syntax is slightly different from the Quick Start example. Also - and this caused me a lot of trouble - when I didn't pass in the name of my FlaskApp instance I got this error:

Error: Failed to find application in module "server.api".  Are you sure it contains a Flask application?  Maybe you wrapped it in a WSGI middleware or you are using a factory function.

So I had to specifically tell flask the name of my app by appending it to the end of the setting (perhaps it is looking for app specifically, but I called mine application).

set -x FLASK_APP server.api:application

Development Mode

If you want the server to automatically reload when it detects changes then you should to set the FLASK_ENV environment variable to development. This is similar to using FLASK_DEBUG but I think it adds the reloading. Anyway, it does more than just set debug mode.

set -x FLASK_ENV development

Run It

The Development server

This is the output of the help string for the development server, note that it uses -h for host so you need to pass in --help to see this output or you will get an error.

flask run --help

Public Server

The default server runs on localhost, but since I'm hosting the code on a raspberry pi sitting on the network somewhere but debugging it remotely, I need to run it on a public address.

flask run --host=0.0.0.0

Make it repeatable

Monitor Mode With airmon-ng

Introduction

I'm looking into setting up a wireless (WiFi and bluetooth) monitoring station to collect data that correlates with how my network is performing and what the state of the network is and I thought that, as a first step, I'd get some packet capturing logs going. I'm primarily a python programmer who's kept my toe in the Linux command-line world but it's been a little while since I really dove into the wireless networking world. I had some vague notion about doing it with iwconfig or iw, but then I found airmon-ng and realized that it was what I was really looking for. Why is it better? Well, to be honest, I'm not informed enough to say that it's better, but when I tried to use iw it failed without really telling me why, while airmon-ng not only didn't fail, but it told me that there were other processes already using my wireless interface which is likely why iw failed and it told me how to fix it. On the one hand, since it's hiding so much from you airmon-ng lets you be a little ignorant and still do stuff, on the other - what's wrong with that?

Setting Up

I'm using Ubuntu 18.04 (Bionic Beaver) - which seems to have both fixed and broken a surprising amount of stuff (nice that you let me log in with Dvorak now, but maybe you should let me know the keyboard layout has changed ahead of time) - so these instructions are based on that. First, airmon-ng is part of the aircrack-ng package so you need to install it to get what we want.

sudo apt install aircrack-ng

Once you do this you'll see that airmon-ng is installed.

which airmon-ng

Interestingly, if you check it out, you'll see that all it is is a bash script.

file `which airmon-ng`

The file is kind of long.

wc -l `which airmon-ng`

So I won't list it here - you can check it out if you're interested. It's actually very informative if you want to learn how to do this kind of stuff, but for this case, we just need to know it works.

Monitor Mode

Starting Up Monitor Mode

Finding your interface

In the good old days you could be pretty sure that your wireless interface was wlan0 (assuming you only had one) but then ubuntu/freedesktop went and changed things so now you should probably check what your interface name is using iw.

iw dev

So it looks like we have a wireless interface named wlp2s0 that we want to change from managed to monitor mode.

Okay, now monitor it

The syntax to start monitor mode is airmon-ng start <interface>.

sudo airmon-ng start wlp2s0
Found 5 processes that could cause trouble.
If airodump-ng, aireplay-ng or airtun-ng stops working after
a short period of time, you may want to run 'airmon-ng check kill'

  PID Name
 1505 wpa_supplicant
 1524 NetworkManager
 1541 avahi-daemon
 1748 avahi-daemon
 2298 dhclient

PHY     Interface       Driver          Chipset

phy0    wlp2s0          iwlwifi         Intel Corporation Wireless 7260 (rev 73)

                (mac80211 monitor mode vif enabled for [phy0]wlp2s0 on [phy0]wlp2s0mon)
                (mac80211 station mode vif disabled for [phy0]wlp2s0)
#+END_SRC

The first thing you should notice is that there are five potentially interfering processes. This is probably what interferes with the =iw= method, but we'll leave it alone and see if it works. Why don't we check on the interface.

#+BEGIN_SRC bash :results raw
iw dev
#+END_SRC

#+RESULTS:
phy#0
	Interface wlp2s0mon
		ifindex 5
		wdev 0x3
		addr 7c:5c:f8:f7:f5:c6
		type monitor
		channel 10 (2457 MHz), width: 20 MHz (no HT), center1: 2457 MHz
		txpower 0.00 dBm

So you can see that running =airmon-ng start= killed our original =wlp2s0= interface and replaced it with =wlp2s0mo= which is in monitor mode on channel 10. Unforturnately I wanted channel 6 but forgot to specify it. Let's try that again.

The first thing we have to do is to turn off monitor mode.

#+BEGIN_EXAMPLE
sudo airmon-ng stop wlp2s0mon

Note that we are stopping the new monitor-mode interface, not our original wireless interface. Now we can start the monitor-mode interface set to channel 6. The syntax is airmon-ng start <interface> <channel>.

sudo airmon-ng start wlp2s0 6
#+END_SRC

There's some output from the command, but we want to know what =iw= thinks is going on.

#+BEGIN_SRC bash :results raw
iw dev
#+END_SRC

#+RESULTS:
phy#0
	Interface wlp2s0mon
		ifindex 7
		wdev 0x6
		addr 7c:5c:f8:f7:f5:c6
		type monitor
		channel 6 (2437 MHz), width: 20 MHz (no HT), center1: 2437 MHz
		txpower 0.00 dBm

So now we have an interface (=wlp2s0mon=) on channel 6 in monitor mode. We can make sure that it's working using [[https://tcpdump.org][tcpdump]].

#+BEGIN_EXAMPLE
sudo tcpdump -i wlp2s0mon -n

Note that we need to use the new interface name. Also, if it wasn't obvious up to now, putting the interface into monitor mode will break any networking capabilities for that interface on your computer (so if it was your internet connection, don't expect to access the web when it's in monitor mode).

Cleaning Up

We already got a preview of turning off monitor mode earlier. The syntax is airmon-ng stop <interface>.

sudo airmon-ng stop wlp2s0mon

This will bring back the original wireless interface, but it won't (likely) re-establish your connection to your wireless access point. To get back onto the network you will probably need to open network manager and go through the setup process again.

Summary

These were my notes on setting up monitor mode using airmon-ng. The main point I wanted to get across is how easy it is to do using airmon-ng as opposed to the other methods. I didn't actually show how much harder it is to use iwconfig, but if you have tried you might know what it entails. In any case, hopefully these notes will help me in the future as I keep watching the packets.

TCP Dump Notes

These are notes I made while surfing the web looking into TCP Dump. You will most likely need to use sudo to run most of the commands, but I'm leaving it off to make it shorter.

About TCP Dump

  • It has more filtering capabilities and can filter while capturing packets, but it doesn't have the analytical tools that something like wireshark has1.

Some Examples

Listing interfaces

You can ask tcpdump which interfaces it is able to listen to2.

tcpdump -D

Capture packets on an interface

To capture packets on an interface you pass its name to the -i flag2 (here the interface I'll use is eno1).

tcpdump -i eno1

Save the packet capture to a file

The default behavior is for tcpdump to send the output to standard output, to have it save the packets to a files use the -w flag2 (you can call it anything, I'll call it dump.pcap).

tcpdump -i eno1 -w dump.pcap

Increase the verbosity of the capture

To increase the amount if information that's captured, pass multiple v arguments2 (in this case I'll use -vvv).

tcpdump -i eno1 -vvv -w dump.pcap

Filtering

By IP address

You can get all the packets being sent or received by a host using the host argument3.

tcpdump host 192.168.1.12

By Sender IP Address

You can filter out all the packets except those that are being sent by a host using the src host argument2.

tcpdump -i eno1 src host 192.168.1.12

You can leave off the host argument and just use src3

By Target IP Address

To filter out all the packets except those that are going to a specific target use the dst host argument2.

tcpdump -i eno1 dst host 192.168.1.1

Sender and Target IP Addresses

You can combine parameters using the logical operators and, or, and not3.

tcpdump 'src 192.168.1.1 and dst 192.168.1.12'

The single quotes are optional and are just used to group the arguments together.

By Subnet

You can grab all the packets on a network or subnet using the net argument and CIDR notation3. This example grabs all the packets on the 192.168.1.* subnet.

tcpdump net 192.168.1.0/24

By port and/or protocol

If you want to only catch activity on a certain port and by a certain protocol then you use the port argument and the name of the protocol (e.g. udp)3. This would catch all the tcp traffic over SSH.

tcpdump tcp port 22

You can use tcp, udp, or icmp for the protocols and add multiple ports using a comma4.

tcpdump tcp port 22,80

Turn off hostname and port translation

The default behavior for tcpdump is to translate the hostnames and ports to something human-readable if possible. To turn this off you pass in the -n argument3. Since this stops having to look things up it will reduce the amount of overhead needed by tcpdump.

tcpdump -n -i eno1 port 22

Links

Sources

Footnotes:

1

Diogenes, Y. & Ozkaya, E. (2018). Cybersecurity, Attack and Defense Strategies : infrastructure security with Red Team and Blue Team tactics. Birmingham, UK: Packt Publishing.]

2

Johansen, G. (2017). Digital forensics and incident response : an intelligent way to respond to attacks. Birmingham, UK: Packt Publishing.

3

Beltrame, J. (2017). Penetration testing bootcamp : quickly get up and running with pentesting techniques. Birmingham, UK: Packt Publishing.

4

McPhee. & Beltrame, J. (2016). Penetration testing with Raspberry Pi : learn the art of building a low-cost, portable hacking arsenal using Raspberry Pi 3 and Kali Linux 2. Birmingham, UK: Packt Publishing.

5

Baxter, J., Orzach, Y. & Mishra, C. (2017). Wireshark revealed : essential skills for IT professionals : get up and running with Wireshark to analyze your network effectively. Birmingham, UK: Packt Publishing.

MNIST Digits With Keras

Imports

These are the parts that will make up the model.

The Sequential Model

The Keras Sequential Model is a stack of layers that will make up the neural network.

from keras.models import Sequential

The Dense Layers

The Keras Dense layer is a densely-connected layer within our neural network.

from keras.layers.core import Activation

Activation

The Activation represents the activation function for each layer (e.g. relu).

from keras.layers.core import Activation

Adam

To tune the model to the data we'll use the Adam optimizer

from keras.optimizers import Adam

Categorical Converter

Finally, since our problem is a classification problem (identify which of 10 digits an image represents) I'll import the Keras to_categorical function to enable classification of our data.

from keras.utils import np_utils

The MNIST dataset is made up of human-classified hand-written digits. Keras includes it as part of their installation so we can load it directly from keras.

from keras.datasets import mnist

We're going to use numpy to reshape the data.

import numpy

To make our output the same every time, I'll set the random seed to April 28, 2018 as a string of digits.

numpy.random.seed(4282018)

bokeh org-mode

Introduction

This is an illustration of how to use bokeh with org-mode in nikola. There is a more extensive and flexible explanation of how to do this in this post on cherian.net but I made these notes to understand how it works and to have a simpler example to refer to.

I was interested in doing this because I'm trying to re-work some of what I did for the Coursera Data Science With Python specialization by changing the data-sets and building them as blog posts. I might convert the posts to restructured text at some point, but while I'm working with them I'm using org-mode. Also, while most of the time I use matplotlib for plotting since this is going to be a blog-first approach I decided to go with bokeh. I had previously written about how to get bokeh into Nikola using restructured text, but as an intermediate step I want to do the work in org-mode and still be able to see the output as I'm working.

The magic mix for this seems to be to use:

Creating the Bokeh Plot

Imports

These are the dependencies. It's really all bokeh, numpy is just there to generate the data-values.

# from pypi
from bokeh.models import HoverTool
from bokeh.plotting import figure, ColumnDataSource
from bokeh.embed import autoload_static, file_html
import bokeh.resources
import numpy

I probably should save bokeh to this repository to keep the post from breaking in the future, but I'm lazy so I'm just going to import it from a CDN.

bokeh = bokeh.resources.CDN

The Data

To get a simple example going I'm just going to use some random outputs generated by numpy.

X = numpy.arange(10)
Y = numpy.random.randint(0, 10, 10)

In order to create a data-structure that bokeh can use (similar to a pandas dataframe) you need to use a ColumnDataSource.

source = ColumnDataSource(data=dict(
    x=X,
    y=Y,
    desc=["a", "b", "c", "d", "e", "f", "g", "h", "i", "j"],
))

The keys in the data-dict are essentially the column headers.

Key Description
x the x-axis values
y the y-axis values
desc The labels for the tooltip

Now to get some tool-tips to pop up when you hover over the plot, I'll create a HoverTool.

hover = HoverTool(tooltips=[
    ('index', '$index'),
    ('(x, y)', '($x, $y)'),
    ('desc', '@desc'),
])

The tooltips list maps the labels that will show up in the tooltip (the first argument to each tuple) to variables in the ColumnDataSource (if preceded by an @) or generated value (if preceded by a $) The index value is the index in the array where the data point sits (so for the first point it will be 0, the second will be 1, etc.). The (x, y) values are the coordinate locations of your pointer when you hover over the data points, and the desc will be replaced by the label I set in the ColumnDataSource for a particular data-point.

The Plot

Now I'll create the actual plot (figure).

fig = figure(title="Random Example", x_axis_label="x", y_axis_label="y",
	     tools=[hover])
fig.line('x', 'y', source=source)
fig.circle('x', 'y', size=10, source=source)

Getting the Bokeh Plot Into The Post

Finally I'll save the javascript and HTML files needed and then output the blob needed to embed the plot into this post. The autoload_static function takes the bokeh plot object (fig), the bokeh javascript that I loaded earlier (bokeh), and the name of the javascript file that you want it to creat (test.js) and returns the javascript to save (javascript) and the HTML fragment that will include the javascript (source). Note that because of the way nikola structures things I have to create a folder named files/posts/bokeh-org-mode and save the files there. Since nikola will automatically look in this folder the name you pass into autoload_static should just be the filename without the path, but when you save the javascript file you will save it there so you need to add the relative path. If my explanation seems a little convoluted, just look at the code below, it's fairly simple.

First I'll create a variable to hold the path to the folder to save the files in. All files for nikola posts go into sub-folders of files/posts/ and since the source file for this post is called bokeh-org-mode.org, the files to include in it go into the folder files/posts/bokeh-org-mode (files/posts/ plus the slug for the post).

FOLDER_PATH = "../files/posts/bokeh-org-mode/"

The Javascript

Now, I'll create the javascript source for the plot.

FILE_NAME = "test.js"
javascript, source = autoload_static(fig, bokeh, FILE_NAME)

with open(FOLDER_PATH + FILE_NAME, "w") as writer:
    writer.write(javascript)

The javascript variable holds the actual javascript source code (which then gets saved) while the source variable holds the string with the HTML to embed the javascript into this post (which I show at the end of this post).

Embedding the Plot

Finally, we need to print out the string that is stored in the source variable which then tells org-mode to embed the files into this post. I'll output the full org-block so you can see the header arguments.

#+BEGIN_SRC python :session bokeh :results output raw :exports results
print('''#+BEGIN_EXPORT html
{}
#+END_EXPORT'''.format(source))
#+END_SRC

And there you have it. I don't have a lot to say about it, other than that if you hover over the data with your cursor and then look up above at the ColumnDataSource above, you'll see that the variables match the inputs

Summary

To get a bokeh figure into an org-mode document in nikola:

  1. Create the bokeh plot.
  2. Create a folder in the files/posts/ folder that matches the slug for the post.
  3. Use autoload_static to convert the bokeh object to javascript and create the HTML tag to embed it.
  4. Save the javascript in the files/posts/<slug>/ folder that you created
  5. Print the HTML fragment in an org-mode #+BEGIN_EXPORT html block.

Data Visualization Bibliography

MW.1. Playing With The. (2018) Drawing animated GIFs with matplotlib - Eli Bendersky's website. Retrieved February 27, 2018, from https://eli.thegreenplace.net/2016/drawing-animated-gifs-with-matplotlib/

This shows how to save the animation as a GIF using ImageMagick.

MW.2. Repeatedly Calling A. (2018) animation module — Matplotlib 2.1.2 documentation. Retrieved February 27, 2018, from https://matplotlib.org/api/animation_api.html

This is the matplotlib documentation for creating animations.