This is a post to generate and document the pytest.ini configuration for testing.

There are some things that need to be installed to get the code to run (like numpy and nltk) but for the testing this is what's needed.

expects
faker
pytest
pytest-bdd
pytest-mock
pytest-xdist

This list is in a text-file (tests/testing-requriements) so they can be installed by changing into the tests folder and running pip.

pip install --requirement testing-requirements

Pytest can read in an INI formatted file. It has a bunch of configuration options but I haven't really explored them much. I mostly use it to set the command line arguments that I would otherwise pass in.

• Faker 18.11.2 documentation [Internet]. [cited 2023 Jul 2]. Available from: https://faker.readthedocs.io/en/master/index.html
• INI file. In: Wikipedia [Internet]. 2023 [cited 2023 Jul 2]. Available from: https://en.wikipedia.org/w/index.php?title=INI_file&oldid=1153411356
• Pytest-BDD 6.1.1 documentation [Internet]. [cited 2023 Jul 2]. Available from: https://pytest-bdd.readthedocs.io/en/latest/#bdd-library-for-the-pytest-runner
• pip documentation v23.1.2 [Internet]. [cited 2023 Jul 2]. Available from: https://pip.pypa.io/en/stable/
• pudb 2022.1.3 documentation [Internet]. [cited 2023 Jul 2]. Available from: https://documen.tician.de/pudb/
• pytest-mock documentation [Internet]. [cited 2023 Jul 2]. Available from: https://pytest-mock.readthedocs.io/en/latest/
• pytest-xdist — pytest-xdist documentation [Internet]. [cited 2023 Jul 2]. Available from: https://pytest-xdist.readthedocs.io/en/stable/

This is a post for some notes on the ddg_images function from the duckduckgo-search library (link to GitHub) which downloads images using duckduckgo (and thus bing in a way).

Let's start by looking at the arguments that the function takes.

from duckduckgo_search import ddg_images

print(ddg_images.__doc__)

DuckDuckGo images search. Query params: https://duckduckgo.com/params

    Args:
        keywords (str): keywords for query.
        region (str, optional): wt-wt, us-en, uk-en, ru-ru, etc. Defaults to "wt-wt".
        safesearch (str, optional): On, Moderate, Off. Defaults to "Moderate".
        time (Optional[str], optional): Day, Week, Month, Year. Defaults to None.
        size (Optional[str], optional): Small, Medium, Large, Wallpaper. Defaults to None.
        color (Optional[str], optional): color, Monochrome, Red, Orange, Yellow, Green, Blue,
            Purple, Pink, Brown, Black, Gray, Teal, White. Defaults to None.
        type_image (Optional[str], optional): photo, clipart, gif, transparent, line.
            Defaults to None.
        layout (Optional[str], optional): Square, Tall, Wide. Defaults to None.
        license_image (Optional[str], optional): any (All Creative Commons), Public (PublicDomain),
            Share (Free to Share and Use), ShareCommercially (Free to Share and Use Commercially),
            Modify (Free to Modify, Share, and Use), ModifyCommercially (Free to Modify, Share, and
            Use Commercially). Defaults to None.
        max_results (int, optional): maximum number of results, max=1000. Defaults to 100.
        output (Optional[str], optional): csv, json, print. Defaults to None.
        download (bool, optional): if True, download and save images to 'keywords' folder.
            Defaults to False.

    Returns:
        Optional[List[dict]]: DuckDuckGo text search results.

Hopefully the arguments are pretty straight forward. I couldn't find an official help page for image searches but they seem pretty straight-forward.

Let's do a search for images of lop-eared rabbits and then take a look at what ddg_images returns.

output = ddg_images("rabbit lop", type_image="photo",
                    license_image="public", 
                    max_results=1)

pprint(output[0])

{'height': 848,
 'image': 'https://cdn.pixabay.com/photo/2015/12/22/20/27/animals-1104748_1280.jpg',
 'source': 'Bing',
 'thumbnail': 'https://tse1.mm.bing.net/th?id=OIP.lEqFD_LPmRGbc1GyIUoNygHaE6&pid=Api',
 'title': 'Flemish Lop Rabbit Very · Free photo on Pixabay',
 'url': 'https://pixabay.com/en/flemish-lop-rabbit-rabbit-1104748/',
 'width': 1280}

I thought that the output argument would change the format of the returned values but it instead seems to direct the values to a file (counting stdout as a file) and then return the same function output as before. Here's what "print" does'.

output = ddg_images("rabbit lop", type_image="photo",
                    license_image="public",
                    output="print",
                    max_results=1)

print(output[0])

1. {
    "title": "Flemish Lop Rabbit Very · Free photo on Pixabay",
    "image": "https://cdn.pixabay.com/photo/2015/12/22/20/27/animals-1104748_1280.jpg",
    "thumbnail": "https://tse1.mm.bing.net/th?id=OIP.lEqFD_LPmRGbc1GyIUoNygHaE6&pid=Api",
    "url": "https://pixabay.com/en/flemish-lop-rabbit-rabbit-1104748/",
    "height": 848,
    "width": 1280,
    "source": "Bing"
}
{'title': 'Flemish Lop Rabbit Very · Free photo on Pixabay', 'image': 'https://cdn.pixabay.com/photo/2015/12/22/20/27/animals-1104748_1280.jpg', 'thumbnail': 'https://tse1.mm.bing.net/th?id=OIP.lEqFD_LPmRGbc1GyIUoNygHaE6&pid=Api', 'url': 'https://pixabay.com/en/flemish-lop-rabbit-rabbit-1104748/', 'height': 848, 'width': 1280, 'source': 'Bing'}

I was hoping that I'd be able to link to some official documentation to explain what's going on, but from what I can tell duckduckgo doesn't have an advertised API for its image search, and duckduckgo-search isn't too heavily documented, but if you look at the duckduckgo-search code it appears to be using a special token (vqd) that you can use to make queries to a special endpoint (https://www.duckduckgo.com/i.js) that you can use to get the search results from, which I couldn't find documented in the duckduckgo-search repository but is mentioned in this StackOverflow answer. I don't know how they figured it out, but using the vqd parameter and o=json makes it work pretty much like an API, although the code is also handing pagination so it seems almost like a hybrid web-scraping and API request.

• duckduckgo-search on pypi
• duckduckgo-search on GitHub

In this notebook we'll go over the fastai course lesson 3 - "Which image models are best?". We'll use the benchmarking data from timm, a collection of pyTorch IMage Models to compare how different computer vision models performed using time-per-image and accuracy as our metrics.

We'll be using data that's part of the git repository for timm . Once you clone the repository the first file within it that we want will be results/results-imagenet.csv. This is the result of using the Imagenet Validation set to validate the models.

RESULTS = Path("~/projects/third-party/"
               "pytorch-image-models/results").expanduser()
DATA = RESULTS/"results-imagenet.csv"
validation = pandas.read_csv(DATA)

print(TABLE(validation.iloc[0].to_frame()))

This table shows the first row of the results-imagenet CSV. Each row represents a computer vision model and some information about how it performed during validation. The documentation says that top1 and top5 are "top-1/top-5 differences from clean validation." Which means… what? Looking at the validate.py file it appears that top1 and top5 are measures of accuracy. Looking in the utils.metrics.py module the function accuracy has a docstring that says: Computes the accuracy over the k top predictions for the specified values of k. The top1 and top5 are AverageMeter objects that keep a running average of their accuracies.

This seems straightforward enough, but if you look at that first row the top1 is smaller than the top5 and has a larger error…

Guessing by the name, the model in our row is an instance of "BEIT: BERT Pre-Training of Image Transformers (https://arxiv.org/abs/2106.08254)" found in timm's beit.py module.

print(validation.shape)

(668, 9)

The model column is the string you use when creating a model and also refers to a function in one of the pytorch-image-models/timm/models modules. If you want to see how the model in our example row is defined, look in the timm/models/beit.py module for a function named "beit_large_patch16_512". You should find something like this.

@register_model
def beit_large_patch16_512(pretrained=False, **kwargs):
    model_kwargs = dict(
        img_size=512, patch_size=16, embed_dim=1024, depth=24, num_heads=16, mlp_ratio=4, qkv_bias=True,
        use_abs_pos_emb=False, use_rel_pos_bias=True, init_values=1e-5, **kwargs)
    model = _create_beit('beit_large_patch16_512', pretrained=pretrained, **model_kwargs)
    return model

So we can now see that besides being a BEIT model the name tells us that it used an image size of 512 and a patch size of 16. Further up the file is this configuration:

'beit_large_patch16_512': _cfg(
    url='https://conversationhub.blob.core.windows.net/beit-share-public/beit/beit_large_patch16_512_pt22k_ft22kto1k.pth',
        input_size=(3, 512, 512), crop_pct=1.0,

Which tells you where the pretrained weights came from.

We're going to merge our "validation" data with two "benchmark" files (also in the "results" folder) doing some cryptic filtering and data wrangling. It's not obvious what everything is doing so let's use it first and maybe figure out most of it later. The main things to note is that we're adding a family column made by taking the first token from the model name (e.g. the model beit_large_patch16_512 gets the family beit), we're adding a secs column by inverting the samples-per-second column, and filtering the models down to a subset that are useful to look at.

BENCHMARK_FILE = ("benchmark-{infer_or_train}"
                  "-amp-nhwc-pt111-cu113-rtx3090.csv")
SAMPLE_RATE = "{infer_or_train}_samples_per_sec"
FAMILY_REGEX = r'^([a-z]+?(?:v2)?)(?:\d|_|$)'
FAMILY_FILTER = r'^re[sg]netd?|beit|convnext|levit|efficient|vit|vgg'

def get_data(infer_or_train: str,
             validation: pandas.DataFrame=validation) -> pandas.DataFrame:
    """Load a benchmark dataframe

    Args:
     infer_or_train: part of filename with label (infer or train)
     validation: DataFrame created from validation results file

    Returns:
     benchmark data merged with validation
    """

    frame = pandas.read_csv(
        RESULTS/BENCHMARK_FILE.format(
            infer_or_train=infer_or_train)).merge(
        validation, on='model')
    frame['secs'] = 1. / frame[SAMPLE_RATE.format(infer_or_train=infer_or_train)]
    frame['family'] = frame.model.str.extract(FAMILY_REGEX)
    frame = frame[~frame.model.str.endswith('gn')]
    IN_FILTERER = frame.model.str.contains('in22'), "family"
    frame.loc[IN_FILTERER] = frame.loc[IN_FILTERER] + '_in22'

    RESNET_FILTERER = frame.model.str.contains('resnet.*d'),'family'
    frame.loc[RESNET_FILTERER] = frame.loc[RESNET_FILTERER] + 'd'
    return frame[frame.family.str.contains(FAMILY_FILTER)]

The first benchmarking data we're going to add is the inference data. Unfortunately I haven't been able to find out what this means, exactly - was this a test of categorizing a test set? It only adds the average sample time to what we're going to plot, which perhaps isn't as interesting as the accuracy anyway.

inference = get_data('infer')
print(TABLE(inference.iloc[0].to_frame()))

Let's look at a row of what was added to our original validation data.

added = inference[list(set(inference.columns) - set(validation.columns))].iloc[0]
print(TABLE(added.to_frame()))

If you look back at get_data you'll see that we added the sec column which is defined as \(\frac{1}{\textit{samples per second}}\). So it's the averaged(?) seconds per sample. I think.

Let's see how evenly distributed the families are.

counts = inference.family.value_counts().to_frame().reset_index().rename(
    columns = {"index": "Family", "family": "Count"})

chart = altair.Chart(counts).mark_bar().encode(
    x="Count", y=altair.Y("Family", sort="-x"), tooltip=["Count"],
).properties(
    width=PLOT_WIDTH,
    height=PLOT_HEIGHT,
    title="Inference Family Counts"
)

save_it(chart, "inference-family-counts")

There doesn't seem to be an even representation of model families. Let's look at the accuracy vs the speed for the models.

plot_it(inference, title="Inference", 
        filename="inference-benchmark",
        infer_or_train="infer")

While we still don't have an explanation of exactly what we're looking at, in the broadest it's a plot of the time it takes for a model to process an image (in seconds on a logarithmic scale) versus the accuracy when categorizing the Imagenet dataset.

• The color matches the family in the legend.
• The size is proportional to the number of seconds it took.
• Clicking on a family in the legend will highlight it and suppress the other families.
• Hovering over a circle gives the exact information for that point.

I believe that the accuracy is the best performance for a model, so even though a family might have multiple points in the plot, each model will only have one point to represent its best accuracy and the time it took.

• PyTorch Image Models: Documentation for the timm pre-built computer vision models for pytorch.
• Pytorch Image Models on github: Repository for timm.
• timm on paperswithcode.com: Table of timm models showing what dataset was used for training and links to publications about each model, and links to a detail page for each model.
• README for the timm results folder on GitHub.

# python
from pathlib import Path

#fastai
from fastai.vision.all import (
    error_rate,
    get_image_files,
    ImageDataLoaders,
    load_learner,
    Resize,
    resnet34,
    untar_data,
    URLs,
    vision_learner,
)

path = untar_data(URLs.PETS)/'images'

def its_a_cat(filename: str) -> bool:
    """Checks if the filename looks like a cat

    Args:
     filename: name of the image file

    Returns:
     True if the first letter of the filename is upper-cased
    """
    return "cat" if filename[0].isupper() else "dog"

loader = ImageDataLoaders.from_name_func(
    path,
    get_image_files(path),
    valid_pct=0.2,
    seed=42,
    label_func=its_a_cat,
    item_tfms=Resize(224)
)

learner = vision_learner(
    loader, resnet34, metrics=error_rate)

model = learner.to_fp16()
with model.no_bar():
    model.fine_tune(1)

[0, 0.12685242295265198, 0.019196458160877228, 0.00405954010784626, '00:19']
[0, 0.06199510768055916, 0.016171308234333992, 0.0067658997140824795, '00:25']

You can either save the underlying pytorch model or the fastai Learner. We want the simpler way so we'll save the fastai Learner.

MODEL_PATH = '/tmp/model.pkl'
model.export(MODEL_PATH)

Weirdly, the original fastai jupyter notebook doesn't tell you how to load the model once you've saved it, but I'm assuming that this is the way to do it.

relearner = load_learner(fname=MODEL_PATH)

def check_model(image_path: str) -> None:
    image_path = Path(image_path).expanduser()

    with relearner.no_bar():
        category, location, probablilities = relearner.predict(image_path)
    print(f"I think this is a {category}.")
    print(f"The probability that it's a {category} is"
          f" {probablilities[location.item()].item():.2f}")
    return

check_model("~/test-cat.jpg")

I think this is a cat.
The probability that it's a cat is 1.00

check_model("~/test-dog.jpg")

I think this is a dog.
The probability that it's a dog is 1.00

• Cat Scratching (Wikimedia Commons)
• Golden Retriever

This is a run-through of some of the ideas from Lesson 0 of the FastAI Practical Deep Learning for Coders course (sort of, there's a 2022 version that I'm using which doesn't seem to exactly match the lectures on the website). In it we search for photos using a search engine and build a neural-network to classify the images that belong to one of the two classes of photos that we use. This is an image classification example, like the Cats vs Dogs post but it has the added feature of demonstrating how to build your own dataset using a search-engine. I'll be using Tanuki (the Japanese Raccoon Dog) and Raccoon images as the categories to classify.

We're going to create an alias for the ddg_images function to make the search and then return only the URLs of the images (or their thumbnails) that DuckDuckGo finds.

def get_image_urls(keywords: str,
                   max_images: int=200,
                   license_image: str="any",
                   key="image") -> list:
    """Search duckduckgo images

    Args:
     keywords: A string with keywords to give to duckduckgo
     max_images: the upper limit for how many images to return

    Returns:
     a list-like object with the URLs of the images found
    """
    return [output.get(key) for output in 
            ddg_images(keywords,
                       type_image="photo",
                       license_image=license_image,
                       max_results=max_images)]

loaders = DataBlock(
    blocks=(ImageBlock, CategoryBlock), 
    get_items=get_image_files, 
    splitter=RandomSplitter(valid_pct=0.2, seed=42),
    get_y=parent_label,
    item_tfms=[Resize(192, method='squish')]
).dataloaders(DATA_PATH)

And now we train the categorizer.

with warnings.catch_warnings() as catcher:
    warnings.simplefilter("ignore")
    learner = vision_learner(loaders, resnet18, metrics=error_rate)

with learner.no_bar() as nobar, Timer() as timmy:
    learner.fine_tune(3)

Started: 2022-12-08 18:28:55.158226
[0, 0.29742079973220825, 0.0608036108314991, 0.020555555820465088, '00:12']
[0, 0.05481018126010895, 0.0367087759077549, 0.009444444440305233, '00:16']
[1, 0.029521549120545387, 0.0178204495459795, 0.004999999888241291, '00:17']
[2, 0.012186083942651749, 0.013458597473800182, 0.006666666828095913, '00:17']
Ended: 2022-12-08 18:29:59.272703
Elapsed: 0:01:04.114477

I put the supression of the warnings in because somebody (I assume FastAI) is calling pytorch with deprecated arguments.

The final loss for the model during training was pretty low (less than 1%) but it wasn't able to identify our one tanuki test image. On the one hand, less than 1% loss isn't 0 loss, so I might just have chosen one example that is particularly hard. It might also be important that tanuki and raccoons do look quite a bit alike, so this is a harder problem than, say, cats versus dogs. Also, our method for gathering images isn't checking that the images are unique (although the URLs are, they might be redundant postings), and tanuki might be obscure enough that there aren't a huge variety of images out there, making it harder for the model to train to identify them.

• FastAI QuickStart: This is where I got the beginnings of the stuff here but it stops before showing you how to use the model you build.
• FastAI Tabular Training page: This is where I got most of this stuffy.
• StackOverflow answer on how to select pandas columns by data-type.

The top post for the quickstart posts is this one and the previous post was on image segmentation.

I think as long as you pass in adjectives that it encountered in the reviews it works well enough. In a way it's more interesting to see how terms match up relative to each other (e.g. terrible isn't as bad as hated and okay is worse than wasn't very good) as it gives you a sense of how the reviewers use descriptive words in their reviews.

This is a look at the part of the fastai Quick Start that demonstrates image segmentation (Wikipedia Article). The goal here is to break images up into sub-parts.

The top post for the quickstart posts is this one and the previous post was the cat image identifier.

The top post for the quickstart posts is this one and the next post will be on sentiment analysis.

This is a run-through of the fastai Computer Vision Quickstart that shows how to build an image classification model from a public dataset hosted on fastai's site. It is similar to the post on classifying rabbits and pigs except in the other post we create our own dataset by searching duckduckgo for images.

# python standard library
from pathlib import Path

As noted on Stack Overflow, FastAI does a lot of monkey patching, so if you just import something from where it's defined (to make it clearer where things are coming from) it might not have the methods or attributes you expect. In this case, for instance, the vision_learner function is defined in fastai.vision.learner but if you try and import it from there the object you get back won't have the to_fp16 method that we're going to use so you have to import it from fastai.vision.all instead. Since there's no good way to avoid using all I'll import objects from there but I'll try and also point to the original modules where things are defined to make it easier to look things up.

from fastai.vision.all import (
    ImageDataLoaders,
    PILImage,
    Resize, 
    URLs,
    error_rate,
    get_image_files,
    resnet34,
    untar_data,
    vision_learner,
)

This downloads the Oxford-IIIT Pet Dataset. Despite the name, there are only cats and dogs in the dataset (37 breeds across the species).

By default this will download the data to ~/.fastai/data but both untar_data and URLs (note the s at the end is lowercase) take an argument c_key that allows changing this but I don't know what the difference is between using one or the other.

path = untar_data(URLs.PETS)/"images"
print(path)

/home/athena/.fastai/data/oxford-iiit-pet/images

The names of the files give the breed of the pet (either cat or dog) with dog names all in lower case (e.g. "yorkshire_terrire_9.jpg") and cats with the first initials capitalized (e.g. "Abyssinian_100.jpg"). So our function to categorize the training data will check if the first letter is a capital letter and label it True if it is, False if it isn't, using the following function.

def its_a_cat(filename: str) -> bool:
    """Decide if file is a picture of a cat

    Args:
     filename: name of file where first letter is capitalized if it's a cat

    Returns:
     True if first letter is capitalized (so it's a picture of a cat)
    """
    return filename[0].isupper()

This next bit creates a batch data loader for us.

loader = ImageDataLoaders.from_name_func(
    path,
    get_image_files(path),
    valid_pct=0.2,
    seed=42,
    label_func=its_a_cat,
    item_tfms=Resize(224)
)

Now we create the model that learns to detect cats.

learner = vision_learner(
    loader, resnet34, metrics=error_rate)

cat_model = learner.to_fp16()

Pretty much all of this is inexplicable if you haven't used some kind of neural network library before, but that last call (``to_fp16``) seems especially mysterious. This first part is just about making sure things work, though, so I'll wait until I get to the more detailed explanations to figure it out, although their article "Mixed Precision Training Explained" explains it pretty well.

We're using a pre-trained model so we just have to do some transfer learning - freezing the weights of most of the layers and training the last layer to make a cat or not a cat classification.

For some reason fastai assumes that you'll only run it in a jupyter notebook and dumps out a progress bar with no simple way to disable it permanently. As a workaround I'll use the context-manager no_bar to turn off the progress bar temporarily.

with cat_model.no_bar():
    cat_model.fine_tune(1)

[0, 0.17085878551006317, 0.019044965505599976, 0.005412719678133726, '00:20']
[0, 0.05584857985377312, 0.01942548155784607, 0.0067658997140824795, '00:25']

Fastai really seems to want to force you to use their system the way they do - the output from fine_tune is printed to standard out and not returned as some kind of object so I can't re-format it to make it nicer looking here (using org-mode), but for reference, the columns for the two rows of output are:

• epoch
• train_loss
• valid_loss
• error_rate
• time

Given these labels, the output of the last block shows that the error rate for the second epoch was 0.005, and it took about twenty and twenty-five seconds per epoch.

We're going to apply our model to some images of cats and a dog to see what it tells us about the image. Since it's the same process for each image I'll create a function check_image to handle it.

def check_image(path: str) -> None:
    """Loads the image and checks if it's a cat

    Args:
     path: string with path to the image
    """
    POSITIVE, NEGATIVE = " think", " don't think"

    image = PILImage.create(Path(path).expanduser())

    with cat_model.no_bar():
        ees_cat, _, probablilities = cat_model.predict(image)
    print(f"I{POSITIVE if ees_cat=='True' else NEGATIVE} this is a cat.")
    print(f"The probability that it's a cat is {probablilities[1].item():.2f}")
    return

So there you go, not really exciting, which I suppose is sort of the point of fastai - it should be simple, almost boring, to do image classification. This is just a rehash of what they did, of course, a better check would be to try something different, but since this is the first take it'll have to do for now.

The top post for the quickstart posts is this one and the next post will be on Image Segmentation.